Big Data Analysis with PySpark

Lecture Series

Big Data Analysis with PySpark

Two lectures on distributed computing: a code-free introduction to MapReduce, Hadoop & Spark, followed by a fully hands-on implementation lecture that builds four real PySpark programs on the same Olist e-commerce data from Tutorial 3.

Concepts — No Install Required

MapReduce & Distributed Computing Concepts

A gentle, complete introduction: what MapReduce solves and why, the map/shuffle/reduce model, how Hadoop implements it, why Spark replaced raw Hadoop, and the core PySpark vocabulary — all in plain language, nothing to install.

45-60 min Beginner 4 steps

Start Lecture 1

Hands-On Implementation

PySpark Implementation: Step by Step

Install Java & PySpark, then build four real programs on the Olist dataset: Word Count, a key–value RDD job, a DataFrame join, and a trained MLlib regression pipeline.

100-130 min Advanced 6 steps, 5 exercises

Start Lecture 2

Prerequisites

Lecture 1 needs nothing but a browser — it's pure reading, no terminal required
Lecture 2 needs Ubuntu/Debian or Fedora/RHEL Linux, ~1 GB free disk for Java + PySpark
Basic Python familiarity (loops, functions) helps for Lecture 2, but no prior big-data knowledge is assumed
No MongoDB or Docker required for either lecture — this track is fully independent of the MongoDB series

Dataset Used

Olist Brazilian E-Commerce — 100K+ anonymized orders with customers, products, payments, and reviews. Get it from the shared Google Drive folder , or reuse the copy already in data/olist_dataset/ if you completed Tutorial 3.