CW

Snowflake Data Mesh Architecture

Free Lesson

Advertisement

Snowflake Data Mesh Architecture

Data Mesh is a decentralized data architecture that organizes data around business domains, treating data as a product. Snowflake's architecture naturally supports Data Mesh patterns.

Architecture Overview

<svg width="800" height="500" viewBox="0 0 800 500" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <linearGradient id="domainGrad" x1="0%" y1="0%" x2="0%" y2="100%">
      <stop offset="0%" style="stop-color:#FF6B6B;stop-opacity:1" />
      <stop offset="100%" style="stop-color:#FF8E8E;stop-opacity:1" />
    </linearGradient>
    <linearGradient id="platformGrad" x1="0%" y1="0%" x2="0%" y2="100%">
      <stop offset="0%" style="stop-color:#4ECDC4;stop-opacity:1" />
      <stop offset="100%" style="stop-color:#7EDDD6;stop-opacity:1" />
    </linearGradient>
  </defs>

  <text x="400" y="30" text-anchor="middle" font-size="18" font-weight="bold" fill="#333">Snowflake Data Mesh Architecture</text>
  <rect x="30" y="60" width="180" height="200" rx="10" fill="url(#domainGrad)" opacity="0.9"/>
  <text x="120" y="85" text-anchor="middle" font-size="12" fill="white" font-weight="bold">Sales Domain</text>
  <rect x="45" y="95" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="120" y="117" text-anchor="middle" font-size="10" fill="#333">Data Product: Revenue</text>
  <rect x="45" y="140" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="120" y="162" text-anchor="middle" font-size="10" fill="#333">Data Product: Pipeline</text>
  <rect x="45" y="185" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="120" y="207" text-anchor="middle" font-size="10" fill="#333">Data Product: Customers</text>
  <rect x="45" y="230" width="150" height="25" rx="5" fill="white" opacity="0.3"/>
  <text x="120" y="247" text-anchor="middle" font-size="9" fill="white">SLAs: 99.9% uptime</text>
  <rect x="230" y="60" width="180" height="200" rx="10" fill="#9B59B6" opacity="0.9"/>
  <text x="320" y="85" text-anchor="middle" font-size="12" fill="white" font-weight="bold">Marketing Domain</text>
  <rect x="245" y="95" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="320" y="117" text-anchor="middle" font-size="10" fill="#333">Data Product: Campaigns</text>
  <rect x="245" y="140" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="320" y="162" text-anchor="middle" font-size="10" fill="#333">Data Product: Attribution</text>
  <rect x="245" y="185" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="320" y="207" text-anchor="middle" font-size="10" fill="#333">Data Product: Segments</text>
  <rect x="245" y="230" width="150" height="25" rx="5" fill="white" opacity="0.3"/>
  <text x="320" y="247" text-anchor="middle" font-size="9" fill="white">SLAs: 99.5% uptime</text>
  <rect x="430" y="60" width="180" height="200" rx="10" fill="#3498DB" opacity="0.9"/>
  <text x="520" y="85" text-anchor="middle" font-size="12" fill="white" font-weight="bold">Operations Domain</text>
  <rect x="445" y="95" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="520" y="117" text-anchor="middle" font-size="10" fill="#333">Data Product: Inventory</text>
  <rect x="445" y="140" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="520" y="162" text-anchor="middle" font-size="10" fill="#333">Data Product: Logistics</text>
  <rect x="445" y="185" width="150" height="35" rx="5" fill="white" opacity="0.9"/>
  <text x="520" y="207" text-anchor="middle" font-size="10" fill="#333">Data Product: Quality</text>
  <rect x="445" y="230" width="150" height="25" rx="5" fill="white" opacity="0.3"/>
  <text x="520" y="247" text-anchor="middle" font-size="9" fill="white">SLAs: 99.99% uptime</text>
  <rect x="640" y="60" width="140" height="200" rx="10" fill="url(#platformGrad)" opacity="0.9"/>
  <text x="710" y="85" text-anchor="middle" font-size="12" fill="white" font-weight="bold">Platform</text>
  <text x="710" y="110" text-anchor="middle" font-size="10" fill="white">Discovery</text>
  <text x="710" y="125" text-anchor="middle" font-size="10" fill="white">Catalog</text>
  <text x="710" y="140" text-anchor="middle" font-size="10" fill="white">Access</text>
  <text x="710" y="155" text-anchor="middle" font-size="10" fill="white">Lineage</text>
  <text x="710" y="170" text-anchor="middle" font-size="10" fill="white">Quality</text>
  <text x="710" y="185" text-anchor="middle" font-size="10" fill="white">Monitoring</text>
  <text x="710" y="210" text-anchor="middle" font-size="10" fill="white">Self-service</text>
  <text x="710" y="225" text-anchor="middle" font-size="10" fill="white">APIs</text>
  <text x="710" y="240" text-anchor="middle" font-size="10" fill="white">Governance</text>
  <path d="M210 160 L230 160" stroke="#333" stroke-width="1" stroke-dasharray="5,5"/>
  <path d="M410 160 L430 160" stroke="#333" stroke-width="1" stroke-dasharray="5,5"/>
  <rect x="30" y="290" width="740" height="190" rx="10" fill="#F39C12" opacity="0.9"/>
  <text x="400" y="315" text-anchor="middle" font-size="14" fill="white" font-weight="bold">Data Product Specification</text>

  <rect x="50" y="330" width="170" height="130" rx="8" fill="white"/>
  <text x="135" y="350" text-anchor="middle" font-size="11" fill="#333" font-weight="bold">Identity</text>
  <text x="135" y="370" text-anchor="middle" font-size="9" fill="#666">Domain ownership</text>
  <text x="135" y="385" text-anchor="middle" font-size="9" fill="#666">Versioning</text>
  <text x="135" y="400" text-anchor="middle" font-size="9" fill="#666">SLA definitions</text>
  <text x="135" y="415" text-anchor="middle" font-size="9" fill="#666">Quality guarantees</text>

  <rect x="240" y="330" width="170" height="130" rx="8" fill="white"/>
  <text x="325" y="350" text-anchor="middle" font-size="11" fill="#333" font-weight="bold">Interface</text>
  <text x="325" y="370" text-anchor="middle" font-size="9" fill="#666">SQL views</text>
  <text x="325" y="385" text-anchor="middle" font-size="9" fill="#666">API endpoints</text>
  <text x="325" y="400" text-anchor="middle" font-size="9" fill="#666">Snowflake shares</text>
  <text x="325" y="415" text-anchor="middle" font-size="9" fill="#666">Streams/tasks</text>

  <rect x="430" y="330" width="170" height="130" rx="8" fill="white"/>
  <text x="515" y="350" text-anchor="middle" font-size="11" fill="#333" font-weight="bold">Semantics</text>
  <text x="515" y="370" text-anchor="middle" font-size="9" fill="#666">Schema definition</text>
  <text x="515" y="385" text-anchor="middle" font-size="9" fill="#666">Business glossary</text>
  <text x="515" y="400" text-anchor="middle" font-size="9" fill="#666">Lineage docs</text>
  <text x="515" y="415" text-anchor="middle" font-size="9" fill="#666">Usage examples</text>

  <rect x="620" y="330" width="140" height="130" rx="8" fill="white"/>
  <text x="690" y="350" text-anchor="middle" font-size="11" fill="#333" font-weight="bold">Discovery</text>
  <text x="690" y="370" text-anchor="middle" font-size="9" fill="#666">Catalog entry</text>
  <text x="690" y="385" text-anchor="middle" font-size="9" fill="#666">Data contract</text>
  <text x="690" y="400" text-anchor="middle" font-size="9" fill="#666">Version history</text>
  <text x="690" y="415" text-anchor="middle" font-size="9" fill="#666">Contact info</text>
</svg>

Key Concepts

DfData Product

DfDomain Ownership

Domain Database Structure

-- Sales Domain
CREATE DATABASE sales_domain;
CREATE SCHEMA raw_data;
CREATE SCHEMA curated;
CREATE SCHEMA data_products;
CREATE SCHEMA analytics;

-- Marketing Domain
CREATE DATABASE marketing_domain;
CREATE SCHEMA raw_data;
CREATE SCHEMA curated;
CREATE SCHEMA data_products;
CREATE SCHEMA analytics;

Data Product Creation

-- Sales Domain: Revenue Data Product
CREATE OR REPLACE DATA PRODUCT revenue_data_product
  DATABASE = sales_domain
  SCHEMA = data_products
  COMMENT = 'Revenue metrics aggregated by region and time'
  SLA = '99.9% uptime, 15-minute freshness'
  OWNERSHIP = 'Sales Domain Team'
AS (
  SELECT
    region,
    DATE_TRUNC('day', order_date) as date,
    SUM(amount) as total_revenue,
    COUNT(DISTINCT customer_id) as unique_customers,
    AVG(amount) as avg_order_value
  FROM sales_domain.curated.orders
  WHERE order_date >= DATEADD(year, -1, CURRENT_DATE())
  GROUP BY 1, 2
);

Data Sharing Across Domains

-- Create share for cross-domain access
CREATE SHARE revenue_share;
GRANT USAGE ON DATABASE sales_domain TO SHARE revenue_share;
GRANT USAGE ON SCHEMA sales_domain.data_products TO SHARE revenue_share;
GRANT SELECT ON TABLE sales_domain.data_products.revenue_data_product TO SHARE revenue_share;

-- Marketing Domain consumes Sales data
CREATE DATABASE revenue_from_sales FROM SHARE account_org.revenue_share;
SELECT * FROM revenue_from_sales.sales_domain.data_products.revenue_data_product;

Federated Governance

-- Global governance rules
CREATE OR REPLACE ROLE data_product_manager;
CREATE OR REPLACE ROLE data_consumer;

-- Domain-level policies
CREATE OR REPLACE RESOURCE MONITOR domain_monitor
  WITH CREDIT_QUOTA = 1000
  NOTIFY_USERS = ('admin@company.com')
  TRIGGERS ON 80% DO NOTIFY
  TRIGGERS ON 100% DO SUSPEND;

-- Data quality checks
CREATE OR REPLACE TASK validate_data_product
  WAREHOUSE = compute_wh
  SCHEDULE = '1 HOUR'
AS
  BEGIN
    -- Check freshness
    IF (SELECT MAX(updated_at) FROM sales_domain.data_products.revenue_data_product) < 
       DATEADD(hour, -2, CURRENT_TIMESTAMP()) THEN
      RAISE ERROR(10001, 'Data product freshness SLA violated');
    END IF;
  END;

Implement data contracts for each data product defining schema, quality, freshness, and access requirements. Use Snowflake's governance features (masking, tagging, access policies) to enforce compliance automatically.

Data Mesh vs Traditional

AspectTraditionalData MeshBenefit
ArchitectureCentralizedDecentralizedDomain agility
OwnershipData teamDomain teamsBusiness alignment
ScalingVerticalHorizontalCost efficiency
QualityReactiveProactiveReliability
DiscoveryManualSelf-serviceFaster access
  • Data Mesh organizes data around business domains
  • Each domain owns and manages its data products
  • Platform provides self-service discovery and governance
  • Cross-domain sharing uses Snowflake shares and views
  • Federated governance maintains standards while enabling autonomy

Advertisement

←38 Snowflake Change Data Capture40 Snowflake Data Warehouse Automation→

Need Expert Snowflake Help?

Get personalized warehouse optimization, data modeling, or Snowflake platform consulting.

Contact Us →View Services

Advertisement