Difference Between MapReduce and Spark
Hadoop is a framework which helps us to store Big Data inefficient and distributed manner and also process the data in parallel and distributed manner. Two core components of Hadoop framework are HDFS and Map Reduce in addition we also have YARN(Yet Another Resource Negotiator) which does the resource management for better performance. We have many tools present in Hadoop ecosystem like(Hive, HBase, Pig, Sqoop and Zookeeper etc.)
SPARK is an independent processing engine for real-time processing which can be installed on any Distributed File system like Hadoop. Like YARN we have here a Own Cluster Resource manager commonly know as (Local Resource manager). The resource manager is not as mature as YARN so it is not used in Production environment. SPARK provides a performance which is 10 times faster than Map Reduce on disk and 100 times faster than Map Reduce on a network in memory.
Need For SPARK
- Iterative Analytics: Map reduce is not as efficient as SPARK to solve problems which require iterative analytics as it has to go to disk for every iteration.
- Interactive Analytics: Map reduce is often used to run ad-hoc queries for which it needs to get to on disk memory which again is not as efficient as SPARK because the latter refers in the in-memory which is faster.
- Not Suitable for OLTP: As it works on batch-oriented framework it is not suitable for a large number of the short transaction.
- Not Suitable for Graph: The Apache Graph library processes the graph which adds more complexity to Map Reduce.
- Not suitable for trivial operations: For operations like a filter and joins we might need to rewrite the jobs, which becomes more complex because of the key-value pattern.
Head To Head Comparison Between MapReduce vs Spark (Infographics)
Key Differences Between MapReduce vs Spark
- Spark is suitable for real-time as it process using in memory whereas as MapReduce is limited to batch processing.
- Spark has RDD(Resilient Distributed Dataset) giving us high- level operators but in Map reduce we need to code each and every operation making it comparatively difficult.
- Spark can process graph’s and supports Machine learning tool.
- Below is the difference between MapReduce vs Spark ecosystem.
Example, where MapReduce vs Spark are suitable, are as follows
Spark: Credit Card fraud detection
MapReduce: Making of regular reports which require decision making.
MapReduce vs Spark Comparision Table
| Basis of Comparison | MapReduce | Spark |
| Framework | Open source framework for writing data into HDFS and processing structured and unstructured data present in HDFS. | Open source framework for faster and general purpose data processing |
| Speed | Map-Reduce process the data(reads & write) from disk so the seep is slow as compared to Spark. | Spark is at least 10X faster on disk and 100X faster in memory as that of Map Reduce. |
| Difficulty | We need to code/handle each process. | With the availability of RDD( Resilient Distributed Dataset), it’s easy to program. |
| Real-Time | Not suitable for OLTP transaction only for Batch mode | It can handle the real-time processing. Using SPARK Streaming. |
| Latency | High-level latency computing framework | Low-level latency computing framework. |
| Fault Tolerance | Master daemons check the heartbeat of slave daemons and in case slave daemons fail master daemons reschedule all the pending and in progress operation to another slave. | RDD’s provide fault tolerance to SPARK. They refer to the data set present in external storage like (HDFS, HBase) and operate parallel. |
| Scheduler | In Map Reduce we use an external scheduler like Oozie. | As SPARK work with in-memory computing, it acts as its own scheduler. |
| Cost | Map Reduce is comparatively cheaper as compared to SPARK. | As it works on in memory so it requires a lot of RAM making it comparatively costlier. |
| Platform Developed on | Map Reduce has been developed using Java. | SPARK has been developed using Scala. |
| Language Supported | Map Reduce basically supports C, C++, Ruby, Groovy, Perl, Python. | Spark supports Scala, Java, Python, R, SQL. |
| SQL Support | Map Reduce runs queries using Hive Query Language. | Spark has its own query language known as Spark SQL. |
| Scalability | In Map Reduce we can add up to n number of nodes. The largest Hadoop Cluster has 14000 nodes. | In Spark also we can add n number of nodes. The largest Spark cluster has 8000 nodes. |
| Machine Learning | Map Reduce supports Apache Mahout tool for machine learning. | Spark supports MLlib tool for machine learning. |
| Caching | Map reduce is not able to cache in memory data so its not as fast as compared to Spark. | Spark caches the in-memory data for further iterations so its very fast as compared to Map Reduce. |
| Security | Map Reduce supports more security projects and features in comparison to Spark | Spark security is not yet matured as that of Map Reduce |
Conclusion – MapReduce vs Spark
As per the above differences, it’s pretty clear that SPARK is much more advanced computing engine as compared to Map Reduce. Spark is compatible with any type of file format and also pretty faster than Map Reduce. The spark in addition also has Graph processing and machine learning capabilities.
On one hand, Map Reduce is limited to batch processing and on other Spark is able to do any type of processing (batch, interactive, iterative, streaming, graph). Due to big compatibility Spark is the favorite of Data Scientist and hence its replacing Map Reduce and growing rapidly. But still we need to store the data in HDFS and we also sometime may need HBase. So we need to run both Spark and Hadoop to get the best.
Recommended Articles:
7 Important Things You Must Know About Apache Spark (Guide)
Hadoop vs Apache Spark – Interesting Things you need to know
Apache Hadoop vs Apache Spark |Top 10 Comparisons You Must Know!
Big Data: Confluence of Technology & Business analytics
The post Best 15 Things You Need To Know About MapReduce vs Spark appeared first on eduCBA.
This post first appeared on Best Online Training & Video Courses | EduCBA, please read the originial post: here
