Скачать с ютуб MapReduce Tutorial For Beginners | Hadoop MapReduce Tutorial | MapReduce Training Video в хорошем качестве

MapReduce Tutorial For Beginners | Hadoop MapReduce Tutorial | MapReduce Training Video

MapReduce in Hadoop: MapReduce is a framework using which we can write applications to process huge amounts of data, in parallel, on large clusters of commodity hardware in a reliable manner. What is MapReduce? MapReduce is a processing technique and a program model for distributed computing based on java. The MapReduce algorithm contains two important tasks, namely Map and Reduce. Map takes a set of data and converts it into another set of data, where individual elements are broken down into tuples (key/value pairs). Secondly, reduce task, which takes the output from a map as an input and combines those data tuples into a smaller set of tuples. As the sequence of the name MapReduce implies, the reduce task is always performed after the map job. The major advantage of MapReduce is that it is easy to scale data processing over multiple computing nodes. Under the MapReduce model, the data processing primitives are called mappers and reducers. Decomposing a data processing application into mappers and reducers is sometimes nontrivial. But, once we write an application in the MapReduce form, scaling the application to run over hundreds, thousands, or even tens of thousands of machines in a cluster is merely a configuration change. This simple scalability is what has attracted many programmers to use the MapReduce model. The Algorithm • Generally, MapReduce paradigm is based on sending the computer to where the data resides! • MapReduce program executes in three stages, namely map stage, shuffle stage, and reduce stage. o Map stage: The map or mapper’s job is to process the input data. Generally, the input data is in the form of file or directory and is stored in the Hadoop file system (HDFS). The input file is passed to the mapper function line by line. The mapper processes the data and creates several small chunks of data. o Reduce stage: This stage is the combination of the Shufflestage and the Reduce stage. The Reducer’s job is to process the data that comes from the mapper. After processing, it produces a new set of output, which will be stored in the HDFS. • During a MapReduce job, Hadoop sends the Map and Reduce tasks to the appropriate servers in the cluster. • The framework manages all the details of data-passing such as issuing tasks, verifying task completion, and copying data around the cluster between the nodes. • Most of the computing takes place on nodes with data on local disks that reduces the network traffic. • After completion of the given tasks, the cluster collects and reduces the data to form an appropriate result, and sends it back to the Hadoop server. Terminology • PayLoad - Applications implement the Map and the Reduce functions, and form the core of the job. • Mapper - Mapper maps the input key/value pairs to a set of intermediate key/value pair. • NamedNode - Node that manages the Hadoop Distributed File System (HDFS). • DataNode - Node where data is presented in advance before any processing takes place. • MasterNode - Node where JobTracker runs and which accepts job requests from clients. • SlaveNode - Node where Map and Reduce program runs. • JobTracker - Schedules jobs and tracks the assign jobs to Task tracker. • Task Tracker - Tracks the task and reports status to JobTracker. • Job - A program is an execution of a Mapper and Reducer across a dataset. • Task - An execution of a Mapper or a Reducer on a slice of data. • Task Attempt - A particular instance of an attempt to execute a task on a SlaveNode. For more updates on courses and tips follow us on: Facebook:   / acadgild   Twitter:   / acadgild   LinkedIn:   / acadgild