Home » Microsoft Azure » Create Datasets in Azure Data Factory

Create Datasets in Azure Data Factory

ADF Datasets

In this blog you will understand all about “How to create datasets in Azure Data Factory?”

Datasets represent data structures within the data stores, which simply point to or reference the data you want to use in your activities as inputs or outputs.

Prerequisites

Create Linked services, see here

Let’s get started

In this procedure, you create two datasets: one for input and other for output, or you can say datasets for source and destination.

Here, our source is Azure SQL Database and destination is Azure Blob Storage, because you want to copy data from Azure SQL Database table to Azure Blob Storage.

These datasets are of type Azure SQL Database & Azure Blob Storage. They refer to the Azure SQL Database & Azure Storage  linked services that you created earlier(see here).

Dataset for input( Azure SQL Database)

Following these steps-

Step-1: Click on Author tab(Pencil icon) > Mouseover the Datasets and click on ellipsis icon(…) >Select New dataset

Create datasets

Create datasets



Step-2: New Dataset window appears > Search Azure SQL Database and Select > Click on Continue.

Add dataset for Azure SQL DB

Add dataset for Azure SQL DB

Step-3: Set Properties window appears > Now assign the name for Dataset > Choose Linked service name for SQL and select Table name which data you want to copy.

Set properties for Azure SQL data base

Set properties for Azure SQL data base

After that test the connection-

Test connection for Linked services

Test connection for Linked services

Dataset for output( Azure Blob Storage)

Prerequisite-

Follow these steps-

Step-1: Click on Author tab(Pencil icon) > Mouseover the Datasets and click on ellipsis icon(…) >Select New dataset

Step-2: New Dataset windows opens > Search Azure Blob storage and Select> Select file format Delimited Text.



Step-3: Assign proper name to output dataset > Select Linked service name > Click on browse folder icon and select container. And Check First row as header.

Set property for output dataset

Set property for output dataset

Step-4: After that set some more connection property for Destination file.

Enter output folder name where you want to copy SQL table data and provide file name.

Here, It will create one folder with name “Destination” automatically inside the same container.

If output folder is already there then choose the complete path till folder.

In Quote Character – Click on Edit and remove double quote(“) and leave it blank.

Dataset Property

Dataset Property

Step-5: After that test the connection-

Test connection for azure blob storage

Test connection for azure blob storage

Step-6: As you can see in below screen shot, we successfully created datasets for Azure SQL Database & Azure Blob Storage.

ADF Datasets

ADF Datasets

Recommend Azure Post: Microsoft Azure Topics

Hope you enjoyed the post. Your valuable feedback, question, or comments about this post are always welcome or you can leave us message on our Contact form , we will revert to you asap.

Leave a Reply