This is my general template for a Spark job. It makes sure we have reasonable Python and Spark versions (i.e. have done the module load on the cluster), and creates a main function that we can easily short-circuit return from if we want to inspect what's going on mid-program.

import sys
from pyspark.sql import SparkSession, functions, types

# add more functions as necessary

def main(inputs, output):
    # main logic starts here

if __name__ == '__main__':
    inputs = sys.argv[1]
    output = sys.argv[2]
    spark = SparkSession.builder.appName('example code').getOrCreate()
    spark.sparkContext.setLogLevel('WARN')
    #sc = spark.sparkContext

    main(inputs, output)