Azure Data Factory Training: Designing and Implementing Data Integration Solutions

• Connectors: Azure services, databases, NoSQL, files, generic protocols, services & apps, custom
• Pipelines
• Activities: data movement, data transformation, control flow
• Datasets: source, sink
• Integration Runtimes: Azure, Self-Hosted, Azure-SSIS

• Creating ADF v2 instance
• Creating a pipeline and associated activities
• Executing the pipeline
• Monitoring execution
• Reviewing results

Copying Tools and SDKS

• Copy Data Tool/Wizard
• Copy activity
• SDKs: Python, .NET
• Automation: PowerShell, REST API, ARM Templates

Copying Considerations

• File formats: Avro, binary, delimited, JSON, ORC, Parquet
• Data store support matrix
• Write behavior: append, upsert, overwrite, write with custom logic
• Schema and data type mapping
• Fault tolerance options

Transformation with Mapping Data Flows

• Introduction to mapping data flows
• Data flow canvas
• Debug mode
• Dealing with schema drift
• Expression builder & language
• Transformation types: Aggregate, Alter row, Conditional split, Derived column, Exists, Filter, Flatten, Join, Lookup, New branch, Pivot, Select, Sink, Sort, Source, Surrogate key, Union, Unpivot, Window

Transformation with External Services

• Databricks: Notebook, Jar, Python
• HDInsight: Hive, Pig, MapReduce, Streaming, Spark
• Azure Machine Learning service
• SQL Stored procedures
• Azure Data Lake Analytics U-SQL
• Custom activities with .NET or R

• Purpose of activity dependencies: branching and chaining
• Activity dependency conditions: succeeded, failed, skipped, completed
• Control flow activities: Append Variable, Azure Function, Execute Pipeline, Filter, ForEach, Get Metadata, If Condition, Lookup, Set Variable, Until, Wait, Web

• Debugging
• Monitoring: visual, Azure Monitor, SDKs, runtime-specific best practices
• Scheduling execution with triggers: event-based, schedule, tumbling window
• Performance, scalability, tuning
• Common troubleshooting scenarios in activities, connectors, data flows and integration runtimes

• Quick introduction to source control with Git
• Integration with GitHub and Azure DevOps platforms
• Environment management: Development, QA, Production
• Iterative development best practices
• Continuous Integration (CI) pipelines
• Continuous Delivery (CD) pipelines

• Templates: out-of-the-box and organizational
• Parameters
• Naming convention

• Data movement security
• Azure Key Vault
• Self-hosted IR considerations
• IP address blocks
• Managed identity