Skip to content

Latest commit

 

History

History
191 lines (103 loc) · 7.98 KB

module06.md

File metadata and controls

191 lines (103 loc) · 7.98 KB

Module 06 - Lineage

< Previous Module - Home - Next Module >

🤔 Prerequisites

📢 Introduction

One of the platform features of Azure Purview is the ability to show the lineage between datasets created by data processes. Systems like Data Factory, Data Share, and Power BI capture the lineage of data as it moves. Custom lineage reporting is also supported via Atlas hooks and REST API.

Lineage in Purview includes datasets and processes.

  • Dataset: A dataset (structured or unstructured) provided as an input to a process. For example, a SQL Table, Azure blob, and files (such as .csv and .xml), are all considered datasets. In the lineage section of Purview, datasets are represented by rectangular boxes.

  • Process: An activity or transformation performed on a dataset is called a process. For example, ADF Copy activity, Data Share snapshot and so on. In the lineage section of Purview, processes are represented by round-edged boxes.

This module steps through what is required for connecting an Azure Data Factory account with an Azure Purview account to track data lineage.

🎯 Objectives

  • Connect an Azure Data Factory account with an Azure Purview account.
  • Trigger a Data Factory pipeline to run so that the lineage metadata can be pushed into Purview.

Table of Contents

  1. Create an Azure Data Factory Connection in Azure Purview
  2. Copy Data using Azure Data Factory
  3. View Lineage in Azure Purview

1. Create an Azure Data Factory Connection in Azure Purview

  1. Open Purview Studio, navigate to Management > Data Factory (under Lineage connections) and click New.

    ⚠️ To view/add/remove Data Factory connections, you need to be assigned the Collection admin role on the root collection.

  2. Select your Azure Data Factory account instance from the drop-down menu (e.g. pvlab-{randomId}-adf) and click OK.

    💡 Did you know?

    Azure Purview can connect to multiple Azure Data Factories but each Azure Data Factory account can only connect to one Azure Purview account.

  3. Once finished, you should see the Data Factory in a connected state.

  4. To confirm that Azure Data Factory has been provided the necessary access, navigate to Data map > Collections > YOUR_ROOT_COLLECTION > Role assignments, within Data curators you should be able to see the Azure Data Factory managed identity.

    💡 Did you know?

    When a user creates an Azure Data Factory connection, behind the scenes the Data Factory managed identity is added to the Data Curator role. This provides Azure Data Factory the necessary access to push lineage to Azure Purview during a pipeline execution. See supported Azure Data Factory activities for more information.

2. Copy Data using Azure Data Factory

  1. Within the Azure Portal, navigate to your Azure Data Factory resource and click Open Azure Data Factory Studio.

  2. Click Ingest.

  3. Select Built-in copy task and then click Next.

  4. Change the Source type to Azure Data Lake Storage Gen2 and then click New connection.

  5. Select your Azure subscription and Storage account (e.g. pvlab{randomId}adls), click Test connection and then click Create.

  6. Click Browse.

  7. Navigate to raw/BingCoronavirusQuerySet/2020/ and click OK.

  8. Confirm your folder path selection and click Next.

  9. Preview the sample data by clicking Preview data, and then click Next.

  10. Change the Target type to Azure Data Lake Storage Gen2, set the Connection to the existing connection (e.g. AzureDataLakeStorage1), and then click Browse.

  11. Navigate to raw/ and click OK.

  12. Confirm your folder path selection, set the file name to 2020_merged.parquet, set the copy behavior to Merge files, and click Next.

  13. Set the file format to Parquet format and click Next.

  14. Leave the default settings and click Next.

  15. Review the summary and proceed by clicking Next.

  16. Once the deployment is complete, click Finish.

  17. Navigate to the Monitoring screen to confirm the pipeline has run successfully.

3. View Lineage in Azure Purview

  1. Open Purview Studio, from the Data catalog screen click Browse assets.

  2. Switch to the By source type tab and then select Azure Data Factory.

  3. Select the Azure Data Factory account instance.

  4. Select the Copy Pipeline and click to open the Copy Activity.

  5. Navigate to the Lineage tab.

  6. You can see the lineage information has been automatically pushed from Azure Data Factory to Purview. On the left are the two sets of files that share a common schema in the source folder, the copy activity sits in the center, and the output file sits on the right.

🎓 Knowledge Check

http://aka.ms/purviewlab/q06

  1. An Azure Purview account can connect to multiple Azure Data Factories?

    A ) True
    B ) False

  2. An Azure Data Factory can connect to multiple Azure Purview accounts?

    A ) True
    B ) False

  3. ETL processes are rendered on the lineage graph with what type of edges?

    A ) Squared edges
    B ) Rounded edges

🎉 Summary

This module provided an overview of how to integrate Azure Purview with Azure Data Factory and how relationships between assets and ETL activities can be automatically created at run time, allowing us to visually represent data lineage and trace upstream and downstream dependencies.

Continue >