How to Read CSV, Parquet & Delta Table Data from Microsoft Fabric Lakehouse Using PySpark

In the previous tutorial, we created a PySpark DataFrame and saved the data to a Microsoft Fabric Lakehouse in three different formats: CSV, Parquet, and Delta table.

In this tutorial, we will read that data back from the Lakehouse using PySpark and load it into PySpark DataFrames.

We will cover:

• Reading CSV data from the Lakehouse Files section
• Reading Parquet data from the Lakehouse Files section
• Reading a Delta table from the Lakehouse Tables section

The basic flow is:

Lakehouse Data → PySpark → DataFrame

Read CSV Data from Microsoft Fabric Lakehouse

Our CSV data is stored in the Lakehouse under:

Files/SampleDataCSV

To read the CSV data, we can use spark.read with the CSV format:

df_csv = spark.read \
    .format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("Files/SampleDataCSV")

display(df_csv)

Here is what each part does:

• spark.read is used to read data into a PySpark DataFrame.
• .format("csv") specifies that the source data is in CSV format.
• .option("header", "true") tells Spark that the first row contains column names.
• .option("inferSchema", "true") allows Spark to automatically determine the data types of the columns.
• .load("Files/SampleDataCSV") specifies the location of the CSV data in the Lakehouse.

The display() function allows us to view the loaded DataFrame in the notebook.

Check the CSV Schema

We can use printSchema() to check the structure and data types of the DataFrame:

df_csv.printSchema()

This is useful for verifying whether Spark has correctly identified the data types, such as numeric columns and string columns.

Read Parquet Data from Microsoft Fabric Lakehouse

Next, we can read the Parquet data that was saved in the Lakehouse.

The Parquet data is available at:

Files/SampleDataParquet

We can read it using the following code:

df_parquet = spark.read \
    .format("parquet") \
    .load("Files/SampleDataParquet")

display(df_parquet)

Here, .format("parquet") specifies that the source is a Parquet file, while .load() specifies the location of the data.

Unlike CSV, we don’t need to specify options such as header or inferSchema in this example. Parquet is a columnar storage format that stores schema information with the data.

We can also check the schema using:

df_parquet.printSchema()

This allows us to verify the column names and their corresponding data types.

Read a Delta Table from Microsoft Fabric Lakehouse

Finally, let’s read the Delta table that we created in the previous tutorial.

The table is named:

SampleSales

and is available under the Tables section of the Lakehouse.

We can read the table using spark.table():

df_table = spark.table("SampleSales")

display(df_table)

Here, spark.table("SampleSales") tells Spark to access the existing SampleSales table and return its data as a PySpark DataFrame.

We can also check the table schema:

df_table.printSchema()

Another Way to Read a Delta Table

A Delta table can also be accessed using the DataFrame reader:

df_table2 = spark.read \
    .format("delta") \
    .table("SampleSales")

display(df_table2)

For simply reading an existing Lakehouse table, spark.table() is generally easier to understand:

df_table = spark.table("SampleSales")

CSV vs Parquet vs Delta Table

The main difference is how we access each type of data.

The overall process can be summarized as:

CSV File
   ↓
spark.read
   ↓
PySpark DataFrame

Parquet File
   ↓
spark.read
   ↓
PySpark DataFrame

Delta Table
   ↓
spark.table()
   ↓
PySpark DataFrame

The important thing to remember is:

Files → spark.read() + format + path

Tables → spark.table() + table name

This provides a simple way to read different types of data stored in a Microsoft Fabric Lakehouse and work with them as PySpark DataFrames.

Reference video: