Create Datasets in Azure Data Factory

In this blog you will understand all about “How to create datasets in Azure Data Factory?”

Datasets represent data structures within the data stores, which simply point to or reference the data you want to use in your activities as inputs or outputs.

Prerequisites

Create Linked services, see here

Let’s get started

In this procedure, you create two datasets: one for input and other for output, or you can say datasets for source and destination.

Here, our source is Azure SQL Database and destination is Azure Blob Storage, because you want to copy data from Azure SQL Database table to Azure Blob Storage.

These datasets are of type Azure SQL Database & Azure Blob Storage. They refer to the Azure SQL Database & Azure Storage  linked services that you created earlier(see here).

Dataset for input( Azure SQL Database)

Following these steps-

Step-1: Click on Author tab(Pencil icon) > Mouseover the Datasets and click on ellipsis icon(…) >Select New dataset

Step-2: New Dataset window appears > Search Azure SQL Database and Select > Click on Continue.

Step-3: Set Properties window appears > Now assign the name for Dataset > Choose Linked service name for SQL and select Table name which data you want to copy.

After that test the connection-

Dataset for output( Azure Blob Storage)

Prerequisite-

• Container in Azure Storage, see here- how to create a container in Azure storage.

Follow these steps-

Step-1: Click on Author tab(Pencil icon) > Mouseover the Datasets and click on ellipsis icon(…) >Select New dataset

Step-2: New Dataset windows opens > Search Azure Blob storage and Select> Select file format Delimited Text.

Step-3: Assign proper name to output dataset > Select Linked service name > Click on browse folder icon and select container. And Check First row as header.

Step-4: After that set some more connection property for Destination file.

Enter output folder name where you want to copy SQL table data and provide file name.

Here, It will create one folder with name “Destination” automatically inside the same container.

If output folder is already there then choose the complete path till folder.

In Quote Character – Click on Edit and remove double quote(“) and leave it blank.

Step-5: After that test the connection-

Step-6: As you can see in below screen shot, we successfully created datasets for Azure SQL Database & Azure Blob Storage.

Recommend Azure Post: Microsoft Azure Topics

Hope you enjoyed the post. Your valuable feedback, question, or comments about this post are always welcome or you can leave us message on our Contact form , we will revert to you asap.