This 8-day hands-on workshop introduces Apache Spark, the open-source cluster computing framework with in-memory processing that makes analytics applications up to 100 times faster compared to technologies in wide deployment today. Developed in the AMPLab at UC Berkeley, Spark can help reduce data interaction complexity, increase processing speed and enhance data-intensive, near-real-time applications with deep intelligence. Highly versatile in many environments, and with a strong foundation in functional programming, Spark is known for its ease of use in creating algorithms that harness insight from complex data. Spark was elevated to a top-level Apache Project in 2014 and continues to expand today.

Between: 25 June 2016 - 31 August 2016 (every Saturday and Sunday)

Private communications will be sent to the selected participants to announce further details (after registration is complete and the list of participants is finalized).

Introduction to Data Analysis with Spark

• What is Apache Spark?
• Introduction to Core Spark Concepts
• Working in the PySpark shell
• Working with PySpark in an iPython notebook
• Standalone Applications

Programming with RDDs

• RDD Basics
• Creating RDDs
• RDD Operations
• Passing Functions to Spark
• Common Transformations and Actions
• Caching RDDs

Working with Key-Value Pairs

• Motivation
• Creating Pairwise RDDs
• Transformations on Pairwise RDDs
• Actions Available on Pairwise RDDs
• Data Partitioning. Key Performance Considerations

Running on a Cluster

• Configuring a Spark Cluster
• Deploying Applications with spark-submit

Structured Data with Spark SQL

• The DataFrame API
• Inner Joins and Left Outer Joins in the RDD API versus in Spark SQL

Building Interactive Data Analytics Apps With Flask and Spark

• A Simple Example - Parameterized CrossFilter Histograms

Spark Streaming

• A Simple Example - Stream of Integers / Rolling Sum
• Streaming data via TCP socket (netcat)
• Streaming data via Kafka topic (Apache Kafka)
• Aggregating streams and storing the results in a NoSQL datastore (Apache Cassandra)

Advanced Spark Programming

• Working on a Per-Partition Basis

Machine Learning with MLlib

• Overview and Terminology
• Machine Learning Basics. What is a Feature
• The LabeledPoint Data Type
• TF-IDF
• Preparing The Data For Analysis / Stemming, Stopword Elimination
• Linear Regression / The Longley Dataset
• Logistic Regression / Filtering Spam
• Decision Trees
• Random Forests

Parallel graph processing with GraphX

• A Simple Example - PageRank

Exercises

• The Complete Works of Shakespeare. Computing Word Counts
• Detecting the 12-01-2001 Anomaly in the CrossFilter Data Set
• Geographical Data - Analysis of City Initials per Country
• Applying PageRank on a Subset of Wikipedia
• Twitter Stream / Sentiment Analysis for Hashtags
• The Brown Corpus (NLTK). Stylistic Classification with Cosine Similarity
• Sensor Data. Detecting Tachycardia and Bradycardia in an ECG Stream

This workshop requires a solid background in functional programming. Knowledge of Python is nice-to-have, but not mandatory.

You can register for the workshop using the online registration form.

If you have any questions, please ask them here.