Azure Data Factory: Pipelines, Datasets & Triggers

Enterprise ETL/ELT orchestration with Azure Data Factory pipelines, activities, and monitoring

ADF Architecture Overview

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    AZURE DATA FACTORY ARCHITECTURE                   │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    ADF FACTORY                                │   │
│  │                                                               │   │
│  │  LINKED SERVICES          DATASETS             PIPELINES     │   │
│  │  ┌──────────────┐       ┌──────────────┐     ┌──────────┐  │   │
│  │  │ ADLS Gen2    │──────>│ Parquet DS   │────>│ Pipeline │  │   │
│  │  │ SQL Server   │──────>│ CSV DS       │     │          │  │   │
│  │  │ Cosmos DB    │──────>│ JSON DS      │     │ Activities│  │   │
│  │  │ Event Hubs   │──────>│ Avro DS      │     │          │  │   │
│  │  │ REST API     │──────>│ Binary DS    │     │ Triggers │  │   │
│  │  └──────────────┘       └──────────────┘     └──────────┘  │   │
│  │                                                               │   │
│  │  INTEGRATION RUNTIMES      MONITORING         GIT INTEGRATION│   │
│  │  ┌──────────────────┐    ┌──────────────┐   ┌──────────┐  │   │
│  │  │ Auto Resolve IR  │    │ Pipeline Runs│   │ Azure    │  │   │
│  │  │ Self-Hosted IR   │    │ Activity Runs│   │ DevOps   │  │   │
│  │  │ Managed VNet IR  │    │ Trigger Runs │   │ GitHub   │  │   │
│  │  │ Spark IR         │    │ Alerts       │   │          │  │   │
│  │  └──────────────────┘    └──────────────┘   └──────────┘  │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  DATA FLOW (Visual ETL):                                            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  Source ──> Filter ──> Derive ──> Join ──> Aggregate ──> Sink │   │
│  │  (ADLS)    (Row     (Add      (Lookup) (Group By)   (ADLS) │   │
│  │            Filter)  Columns)                                │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Pipeline JSON Example

{
  "name": "pl_daily_sales_etl",
  "properties": {
    "activities": [
      {
        "name": "CopySalesData",
        "type": "Copy",
        "typeProperties": {
          "source": {
            "type": "AzureBlobStorageSource",
            "storeSettings": {
              "type": "AzureBlobFSReadSettings",
              "recursive": true
            },
            "formatSettings": {
              "type": "JsonReadSettings"
            }
          },
          "sink": {
            "type": "AzureDataLakeStorageGen2Sink",
            "storeSettings": {
              "type": "AzureDataLakeGen2WriteSettings",
              "copyBehavior": "PreserveHierarchy"
            },
            "formatSettings": {
              "type": "ParquetWriteSettings"
            }
          }
        },
        "inputs": [
          {
            "name": "ds_raw_sales"
          }
        ],
        "outputs": [
          {
            "name": "ds_staging_sales"
          }
        ]
      },
      {
        "name": "TransformAndLoad",
        "type": "DatabricksNotebook",
        "typeProperties": {
          "notebookPath": "/Repos/data_engineering/sales_transformation"
        },
        "dependsOn": [
          {
            "activity": "CopySalesData",
            "dependencyConditions": ["Succeeded"]
          }
        ],
        "policy": {
          "timeout": "0.1:0:0",
          "retry": 1,
          "retryIntervalInSeconds": 30
        }
      },
      {
        "name": "LoadToSynapse",
        "type": "SqlPoolStoredProcedure",
        "typeProperties": {
          "storedProcedureName": "sp_load_fact_sales"
        },
        "dependsOn": [
          {
            "activity": "TransformAndLoad",
            "dependencyConditions": ["Succeeded"]
          }
        ]
      }
    ],
    "parameters": {
      "date": {
        "type": "String",
        "defaultValue": "@formatDateTime(utcNow(), 'yyyy-MM-dd')"
      },
      "source": {
        "type": "String",
        "defaultValue": "sales_api"
      }
    },
    "variables": {
      "retryCount": {
        "type": "Int32",
        "defaultValue": 0
      }
    }
  }
}

Linked Service Configuration

{
  "name": "ls_adls_gen2",
  "properties": {
    "type": "AzureBlobFS",
    "typeProperties": {
      "url": "https://stdatalake001.dfs.core.windows.net",
      "accountKey": {
        "type": "AzureKeyVaultSecret",
        "store": {
          "referenceName": "akv_dataengineering",
          "type": "LinkedServiceReference"
        },
        "secretName": "adls-storage-key"
      }
    },
    "connectVia": {
      "referenceName": "AutoResolveIntegrationRuntime",
      "type": "IntegrationRuntimeReference"
    }
  }
}

Trigger Types

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    ADF TRIGGER TYPES                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  SCHEDULE TRIGGER                                               │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Cron: 0 0 2 * * * (Daily at 2 AM)                       │   │
│  │ Recurrence: Every 1 hour                                │   │
│  │ Time Zone: UTC / Local                                  │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  TUMBLING WINDOW TRIGGER                                        │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Window Size: 1 Day                                      │   │
│  │ Frequency: Day                                          │   │
│  │ Anchor: 2024-01-01                                      │   │
│  │ Parallel: 3 (process 3 windows concurrently)            │   │
│  │ MaxConcurrency: 10                                      │   │
│  │ Retry: 3 attempts, 5 min interval                       │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  EVENT TRIGGER                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Event: Blob Created                                     │   │
│  │ Container: raw                                          │   │
│  │ Blob Path Begins With: sales/                           │   │
│  │ Event Type: Microsoft.Storage.BlobCreated               │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  STORAGE EVENT TRIGGER                                          │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Event: Blob Deleted                                     │   │
│  │ Subject Begins With: /blob/services/blob/containers/     │   │
│  │ Ignore Blob Types: Append Blob                          │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Event Trigger JSON

{
  "name": "tr_blob_arrival",
  "properties": {
    "type": "BlobEventsTrigger",
    "typeProperties": {
      "blobPathBeginsWith": "raw/sales/",
      "blobPathEndsWith": ".json",
      "ignoreEmptyBlobs": true
    },
    "pipelines": [
      {
        "pipelineReference": {
          "referenceName": "pl_daily_sales_etl",
          "type": "PipelineReference"
        },
        "parameters": {
          "date": "@triggerBody().fileName",
          "source": "blob_trigger"
        }
      }
    ]
  }
}

⚠️

Important: Event Triggers require an Event Grid-enabled storage account and a Managed Virtual Network with Event Grid private endpoints for production scenarios.

Self-Hosted Integration Runtime

{
  "name": "ir-selfhosted-onprem",
  "properties": {
    "type": "SelfHosted",
    "typeProperties": {
      "linkedInfo": {
        "type": "LinkedIntegrationRuntimeKey",
        "key": "<EncryptedKey>"
      }
    },
    "hostCaching": "Enabled",
    "nodeCommunicationChannel": "ServiceEndpoint"
  }
}

IR Node Configuration

# Install Self-Hosted IR on Windows
.\DataMovementLibraryRuntimeSetup.exe /quiet /InstallPath:"C:\DI\IR"

# Register IR node
.\RegisterLauncher.exe register -endpoint "https://adf-prod.azure.com" -authKey "<key>" -nodeName "IR-Node-01"

# Check IR status
.\StatusReporter.exe -endpoint "https://adf-prod.azure.com" -authKey "<key>"

Interview Questions

Q1: Explain the difference between Copy Activity and Data Flow in ADF. A: Copy Activity moves data as-is (or with minimal transformation) using optimized engines. Data Flow provides visual ETL with transformations (filter, derive, join, aggregate). Copy is faster for simple moves; Data Flow for complex transformations.

Q2: How do you handle schema changes in ADF pipelines? A: Use mapping data flows with schema drift enabled, or use ADF parameters to dynamically handle column changes. For Copy Activity, use "schema mapping" or "auto mapping" with schema validation.

Q3: What are the best practices for ADF pipeline monitoring? A: 1) Set up alerts for failed runs, 2) Use diagnostic settings to send logs to Log Analytics, 3) Implement custom logging with ADF parameters, 4) Use Power BI dashboards for pipeline metrics, 5) Set up auto-healing with retry policies.