Big Data / Coursework
Spark / Big Data.
Distributed processing fundamentals with PySpark: RDDs, DataFrames, transformations, actions and partitions.
Problem
Understand distributed processing and Spark programming models.
Context
Academic Big Data work. No large-scale production environment is claimed.
Objective
Work with RDDs and DataFrames, and inspect partitions and jobs.
Dataset / Data
Coursework datasets. Exact sources and sizes remain to be documented.
Architecture
Data → PySpark transformations → actions → Spark Web UI inspection.
Methodology
Practice with transformations, actions and partitions. Study of Hadoop and MapReduce concepts.
My contribution
Project reported in the candidate’s background. Individual scope and team responsibilities remain to be confirmed.
Challenges
Understand lazy evaluation and distributed computation.
Limitations
Evidence of academic fundamentals. No production performance or volume metrics. Runtime version needs confirmation.
What I learned
Distributed processing fundamentals and Spark execution inspection.
Project results
No verified numerical metric supplied. Evidence rests on the documented methods and architecture.