Why Is a Block in Hadoop Distributed Filesystem (HDFS) So Large?

HDFS blocks are large compared to disk blocks, and the reason is to minimize the cost of seeks. By making a block large enough, the time to transfer the data from the disk can be made to be significantly larger than the time to seek to the start of the block.Thus the time to transfer a large file made of multiple blocks operates at the disk transfer rate.
A quick calculation shows that if the seek time is around 10 ms, and the transfer rate is 100 MB/s, then to make the seek time 1% of the transfer time, we need to make the block size around 100 MB. The default is actually 64 MB, although many HDFS installations use 128 MB blocks. This figure will continue to be revised upward as transfer speeds grow with new generations of disk drives.
This argument shouldn’t be taken too far, however. Map tasks in MapReduce normally operate on one block at a time, so if you have too few tasks (fewer than nodes in the cluster), your jobs will run slower than they could otherwise.

Comments

Post a Comment

Popular posts from this blog

MATLAB code for Circular Convolution using Matrix method

Image
% circular convolution using matrix method %clc;x=[1,2,3,1];h=[4,3,2,2];H=zeros(0);h_index=1;N=length(x);y=zeros(0);disp('x(n)=');disp(x);disp('h(n)=');disp(h);subplot(4,1,1);stem(x);title('Signal One : x(n) ');subplot(4,1,2);stem(h);title('Signal Two : h(n)');temp=0;for i=1:NH(i,1)=h(i);end;for i=2:N    h=circshift(h, [0,1]);    for j=1:N     H(j,i)= h(j);   end;end;disp('H =>>');disp(H);for i=1:N    for j=1:N        temp=temp+H(i,j)*x(j);    end;    y(i)=temp;    temp=0;end;disp('Circular Convolution : y(m)==>');disp(y);subplot(4,1,3);stem(y);title('Circular Convolution: y(m)');   Output:
Read more

Positive number pipe in angular 2+

Code: import {Pipe, PipeTransform} from '@angular/core'; @Pipe({   name: 'positive' }) export class PositivePipe implements PipeTransform {   transform(value: number, args?: any): any {     return Math.abs(value);   } } How to use: {{ someNumber| positive }}
Read more