What Is Figure 2 : Data Blocks Written To HDFS?

1569 Words7 Pages
Figure 2: Data blocks written to HDFS [6]. The above figure shows how the data in hadoop is stored in racks and each rack consists of many distributed blocks of files, where each block is of 64Mb and can be written three times and at least one block is written to a different server rack for redundancy. In the above figure there are three different blocks ie, block1, block2, block3. Each block has been replicated at three different places and at least one replica should be placed in the different server rack for redundancy. In the figure above each block is having one of its replica placed in different rack.block1, block2, block3 are replicated both in rack1 and rack2 for data redundancy [5]. If the nodes in the rack1 have been damaged or…show more content…
Running daemons called task tracker agents monitors status of each task and reports back to job tracker. The data flow in simple map-reduce job will look like this: Figure 3: The data flow in simple map-reduce job. (paul Z) In the above figure at first the data or the files is divided into small blocks of records and are replicated at three different places. After the job tracker receives the job to be performed it will locate where the information is and then allots a task to the task tracker in the slave node then the mapping will be performed first and it produces the key, value pairs which is given as input to the reduce. In between the map and reduce shuffling/sorting will be done where similar data will be gathered together and sorted. Now the structure information ie key, value pairs are given as input to reduce and then it will generate set of key, value pairs as output for the given key value pairs. Here deciding what will be the key and what will be value is developer’s responsibility. A simple map-reduce example which explains the method more elaborately is as follows: (Toronto, 20) (Texas, 30) (New York, 22) (Rome, 33) (Toronto, 18) (Texas, 35) (New York, 27) (Rome, 38) (Toronto, 32) (Texas, 37) (New York, 20) (Rome, 31) (Toronto, 31) (Texas, 33) (New York, 19) (Rome, 30) (Toronto, 30) (Texas, 32) (New York, 25) (Rome, 32)
Get Access