Apache Spark : Fundamentals

In this post, I would like to share my learning experience in Apache Spark!(In-Memory computation). Trust me, I was amazed to see the working performance of Spark after Map Reduce.

After you read this post, you will be able to explain 

- What is Apache Spark?
- Current Version

- Purpose of Spark?
- List of applications that get benefits from Spark?

- Components of Sparks
- Download & Installation Procedures
- MapReduce Vs Spark Comparison

What is Apache Spark?
-- It is a cluster computing framework for Large-Scale Data Processing.
-- It does not use MapReduce  as a execution Engine.

Current release?
-- The latest release of Spark is Spark 1.5.1, released on October 2, 2015


Purpose of Spark?
-- It is best known for its ability to keep large working datasets in memory between jobs.
-- This allows Spark to Outperform the equivalent MapReduce workflow, where datasets are always loaded from desk.

List of Applications that get benefits from Spark?
-- Batch, Interactive & Streaming
-- Iterative Algorithms
----- can be explained using a function that is applied on specific file or datasets repeatedly until an exit condition is met.

Components of Spark?

 

*Source : Google

Spark Core is the foundation of the overall project. It provides distributed task dispatching, scheduling, and basic I/O functionalities.The fundamental programming abstraction is called Resilient Distributed Datasets (RDDs), a logical collection of data partitioned across machines. 

RDDs can be created in 3 ways ..
> from an in-memory collction of objects
> using a dataset from external storage
> through transforming an existing RDDs


The RDD abstraction is exposed through a language-integrated API in Java, Python, Scala, and R similar to local, in-process collections. This simplifies programming complexity because the way applications manipulate RDDs is similar to manipulating local collections of data.

A Spark cluster is composed of one Driver JVM and one or many Executor JVMs.

Spark SQL:
it's on top of the stack and provides dataframe abstractions, which supports both structured & Semi-structured data.


Spark Streaming: 
> It has fast scheduling capability which helps to perform streaming analytics. 
>It ingest data in mini-batches & perform RDD transformations on those mini-batches of data.

Spark MLib:

Spark MLlib is a distributed machine learning framework on top of Spark Core that, due in large part of the distributed memory-based Spark architecture, is as much as nine times as fast as the disk-based implementation used by Apache Mahout (according to benchmarks done by the MLlib developers against the Alternating Least Squares (ALS) implementations, and before Mahout itself gained a Spark interface), and scales better than Vowpal Wabbit.

Download & Installation Procedures
- Download Apache spark via Click here
--- Note before you install SPark, you need to install Scala. Since Spark is developed using Scala Language. Don't worry if you do not know Scala, but it's easy to learn & apply language. Spark  comes with REPL(Read-Eval-Print-Loop) for both Scala & Python which makes it quick and easy to explore datasets.
- Installation procedures : Click here


MapReduce Vs Spark Comparison

Factor	Hadoop MapRedce	Apache Spark
Data computation	On Disk	In-Memory
Fault Tolerance	Achieves through replication	Uses Data Storage Model(RDD) :
Performance	Need to access disk for each i/o so will take time.	100 times faster than mapreduce, since it uses in-memory execution.
Memory	Can run with available memory	Needs more memory, since it need to cache the data for fast processing.
Iterative Computations	If you do not have enough memory to use spark, then go with MapReduce.	Yes, good deal..
Interactive Analysis	No	Yes, Good deal
Streaming	No	Yes, Good Deal
Batch Processing	Yes	Yes
Native Language	Java	Scala
Ease of use/programming	Difficult to program in java but many tools are available such as PIG, HIVE, etc.	Easier to program & includes an interactive mode.
Cost	Open Source	Open Source
Hardware supports	Commodity	Commodity
Processing Big Data	Best Option	Should be used only if you have as much as memory than data you want to process.

I hope you find this post useful, in next article I will share more on Anatomy of Spark execution & how it works ...


please do not forget to hit like or do comment if you have any questions/suggestions to improve.

MapReduce Part I: How it works?

Most of the time, you wonder what it is all about?, when we hit the training for the very first time.. Students who are not good at writing JAVA program,  really scare about it... To some extent it's true, but if you really understand the concept & how MapReduce works, it's piece of cake..

You just need to apply the logic of data processing, that should work very well.

What if I don't know JAVA?,, don't worry, Hadoop can run MapReduce Programs written in Various Languages; for instance.. Python, R, Ruby, Etc.. We have streaming connectors for all programming that are pre-written.. You just need to use the right streaming for your programming to run MapReduce.

Ok what's MapReduce?? --is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster.

MapReduce how it works? 
MapReduce works by breaking the process into two. Namely, Map Phase & Reduce Phase.
Each Phase has Key-value pairs as input and output.

Input to the Map Phase will be the Raw Data(user feed input file), which reads each and every records using Key/Value Pair concept.

Key -> offset of the beginning of the line
Value -> entire record

We can apply filtering in Map Phase to select only required data that are required for processing.

Output(format) of Map Phase will be the input(formats) to the Reduce Phase. Here both the formats should match..

Let's take an example & see how it works.

I have a raw data : population counts who have participated in survey(sample)

00740211975..+10000+010101010010101010010101010001010100101010001.....
00740211975..+20000+010101010010101010010101010001010100101010001....
00740211980..+50000+010101010010101010010101010001010100101010001....
00740211980..+4000+010101010010101010010101010001010100101010001.....
00740211975..+40000+010101010010101010010101010001010100101010001....
00740211980..+19000+010101010010101010010101010001010100101010001....
00740211975..+12000+010101010010101010010101010001010100101010001...

Now I want to process this using MapReduce Paradigm. As soon as we give this raw data as input to the MapReduce, Map phase will begin execution, but the raw data will be transformed into key-value pairs before it is being taken by Map phase.

(0,    00740211975..+10000+010101010010101010010101010001010100101010001....)
(75,  00740211975..+20000+010101010010101010010101010001010100101010001....)
(150,00740211980..+50000+010101010010101010010101010001010100101010001....)
(225,00740211980..+4000+010101010010101010010101010001010100101010001..... )
....
....
....


Here (0,75,150,225) indicates => Keys which are lines offsets within the file.

Our Purpose is to find the maximum population who have participated in survey in that year. Map Function extracts Year & the population from each records & emits them as output to Reduce Phase.

So after Map Function execution, it would be like this

Map Phase :Output
(1975,10000)
(1975,20000)
(1980,50000)
(1980,4000)
(1975,40000)
(1980,19000)
(1975,12000)

Note, the output of map phase still looks like Key-value pairs, but here, YEARS are acting as Key & population is acting as value.

this o/p will be processed by the MapReduce Framework before being sent to the reduce function. Which is nothing but Shuffle & Sort. We will look about Shuffle phase in details in my next post, but sort phase process the output and groups the Key-Value Pairs by Key. so the above data will be like below after "Shuffle & SORT" process is done

Shuffle&Sort o/p

(1975,[10000,20000,40000,12000])
(1980,[50000,4000,19000])

Now this data will be given to Reduce function, which iterates this list and find out maximum population in that year who have participated in Survey.  O/p of Reduce function will be.

Reduce Phase : o/p

(1975,40000)
(1980,50000)..

We will see in next post, how Hadoop handles this MapReduce. Why this is playing a key role & considered as Core Compoent.

See ya...