How to Create a Temporary Dataset Using PySpark in Microsoft Fabric

In this tutorial, you will learn how to create a temporary or sample dataset using PySpark in a Microsoft Fabric Notebook without connecting to any external data source.

We will define the schema, create a PySpark DataFrame, and display the data for quick analysis.
This approach is very useful when you are learning PySpark, testing your code, preparing a demo, or creating sample data for data analysis.

So, let’s get started.

Open Microsoft Fabric

First, let’s go to Microsoft Fabric workspace.

Screen actions:

1. Open your Microsoft Fabric workspace.
2. Click  +New item.
3. Select Notebook.
4. Give the notebook a name.

Next, add the following PySpark code to the notebook.

# Import PySpark data types
from pyspark.sql.types import *

# Create sample data
data = [
(1, "India", "Laptop", 50000),
(2, "USA", "Monitor", 30000),
(3, "India", "Keyboard", 5000),
(4, "UK", "Laptop", 45000),
(5, "USA", "Laptop", 55000),
(6, "India", "Monitor", 25000),
(7, "UK", "Keyboard", 7000),
(8, "USA", "Keyboard", 8000),
(9, "India", "Laptop", 60000),
(10, "UK", "Monitor", 28000)
]

# Define schema
schema = StructType([
StructField("OrderID", IntegerType(), False),
StructField("Country", StringType(), True),
StructField("Product", StringType(), True),
StructField("Sales", IntegerType(), True)
])

# Create DataFrame
df = spark.createDataFrame(data, schema)

# Display DataFrame
display(df)

# Check DataFrame schema
df.printSchema()

# Check number of rows
print("Total Rows:", df.count())

# Check column names
print("Columns:", df.columns)

You can either execute each cell individually, one at a time, or execute all the cells together by selecting Run all from the notebook menu.

Refer Video: