Module 12
Big data with Spark
Spark DataFrames, distributed preprocessing, feature engineering at scale, model training, tuning with k-fold cross-validation and grid search.
Outcome
What you will be able to do
You can build and tune a Spark ML pipeline on data that does not fit on one machine.
Lessons
Work through these in order
- 01Spark Crash CourseThe dataset contains bike rental info from 2011 and 2012 in the Capital bikeshare system, plus additional relevant information such as weather. 8 min
- 02DataPreProcessingcat > movies.csv <<EOF name,rating,studio,date Avengers Endgame, 5, marvel,1260759144 Batman Vs Superman,4 ,DC,835355664 The Joker ,, DC,835355681 Frozen,4,Disney,835355604 Hitman,,, EOF %sh ls -alh … 8 min
- 03MLeap-Orange Telecom Customer Churnfrom pyspark.ml.feature import VectorAssembler from pyspark.ml.classification import LogisticRegression from pyspark.ml import Pipeline from pyspark.sql.functions import udf from pyspark.sql.types… 8 min
- 04Capital Bike Rental-Python-1How to Ingest the data into Spark DataFrame. How to clean the Data with DataFrame, SQL Query. How to create a Machine Learning Pipeline. How to train a Machine Learning model. How to save & read the… 8 min
- 05Data IngestionWe begin by loading our data, which is stored in the CSV format. 8 min
- 06Data UnderstandingNow that we have preprocessed our features, we can quickly visualize our data to get a sense of whether the features are meaningful. 8 min
- 07Data Processing (Feature Engineering)For each of the categorical columns, we are going to create one StringIndexer where we 8 min
- 08Train the modelRandom forests and ensembles of decision trees are more powerful than a single decision tree alone. 8 min
- 09Model tuning: Preparing K-fold Cross Validation and Grid Searchfor best model selection and makes sure that there's no overfitting. 8 min
Assessment
Module quiz — 70% to pass
6 questions mixing concept checks and short code-output problems. Graded on the server with per-question explanations afterwards, unlimited retakes, and a badge with a verification code the moment you pass.