Creating a Neural Network in Spark

In this chapter, the following recipes will be covered:

• Creating a dataframe in PySpark
• Manipulating columns in a PySpark dataframe
• Converting a PySpark dataframe into an array
• Visualizing the array in a scatterplot
• Setting up weights and biases for input into the neural network
• Normalizing the input data for the neural network
• Validating array for optimal neural network performance
• Setting up the activation function with sigmoid
• Creating the sigmoid derivative function
• Calculating the cost function in a neural network
• Predicting gender based on height and weight
• Visualizing prediction scores