How to do it...

This section walks through the steps to initialize and configure a Spark cluster.

1. Import SparkSession using the following script:

from pyspark.sql import SparkSession

2. Configure SparkSession with a variable named spark using the following script:

spark = SparkSession.builder \
    .master("local[*]") \
    .appName("GenericAppName") \
    .config("spark.executor.memory", "6gb") \
.getOrCreate()