CtrLog

A namenode can eat up memory, since a reference to every block of every file is maintained in memory. It’s difficult to give a precise formula, since memory usage depends on the number of blocks per file, the filename length, and the number of directories in the filesystem; plus it can change from one Hadoop release to another. The default of 1,000 MB of namenode memory is normally enough for a few million files, but as a rule of thumb for sizing purposes you can conservatively allow 1,000 MB per million blocks of storage. You can increase the namenode’s memory without changing the memory allocated to other Hadoop daemons by setting HADOOP_NAMENODE_OPTS in hadoop-env.sh to include a JVM option for setting the memory size. HADOOP_NAMENODE_OPTS allows you to pass extra options to the namenode’s JVM. So, for example, if using a Sun JVM, -Xmx2000m would specify that 2,000 MB of memory should be allocated to the namenode. If you change the namenode’s memory allocation, don’t forge

What Constitutes Progress in MapReduce?

Progress is not always measurable, but nevertheless it tells Hadoop that a task is doing something. For example, a task writing output records is making progress, even though it cannot be expressed as a percentage of the total number that will be written, since the latter figure may not be known, even by the task producing the output. Progress reporting is important, as it means Hadoop will not fail a task that’s making progress. All of the following operations constitute progress: Reading an input record (in a mapper or reducer) Writing an output record (in a mapper or reducer) Setting the status description on a reporter (using Reporter’s setStatus() method) Incrementing a counter (using Reporter’s incrCounter() method) Calling Reporter’s progress() method

Setting User Identity (Hadoop)

The user identity that Hadoop uses for permissions in HDFS is determined by running the whoami command on the client system. Similarly, the group names are derived from the output of running groups. If, however, your Hadoop user identity is different from the name of your user account on your client machine, then you can explicitly set your Hadoop username and group names by setting the hadoop.job.ugi property. The username and group names are specified as a comma-separated list of strings. You can set the user identity that the HDFS web interface runs as by setting dfs.web.ugi using the same syntax. By default, it is webuser,webgroup, which is not a super user, so system files are not accessible through the web interface. Notice that, by default, there is no authentication with this system.

Which Compression Format Should I Use for HDFS?

Which compression format you should use depends on your application. Do you want to maximize the speed of your application or are you more concerned about keeping storage costs down? In general, you should try different strategies for your application, and benchmark them with representative data-sets to find the best approach. For large, unbounded files, like log files, the options are: Store the files uncompressed. Use a compression format that supports splitting, like bzip2 (although bzip2 is fairly slow), or one that can be indexed to support splitting, like LZO. Split the file into chunks in the application and compress each chunk separately using any supported compression format. In this case, you should choose the chunk size so that the compressed chunks are approximately the size of an HDFS block. Use Sequence File, which supports compression and splitting. Use an Avro data file, which supports compression and splitting, just like Sequence File, but has the added advantage

Compilers vs. Interpreters

July 22, 2014

There are two general methods by which a program can be executed. It can be compiled, or it can be interpreted. Although programs written in any computer language can be compiled or interpreted, some languages are designed more for one form of execution than the other. For example, Java was designed to be interpreted, and C was designed to be compiled. However, in the case of C, it is important to understand that it was specifically optimized as a compiled language. Although C interpreters have been written and are available in some environments, C was developed with compilation in mind. Therefore, you will almost certainly be using a C compiler and not a C interpreter when developing your C programs. Since the difference between a compiler and interpreter may not be clear to all readers, the following brief description will clarify matters. In its simplest form, an interpreter reads the source code of your program one line at a time, performing the specific instructions co

Preventing Brute Force Attack

July 16, 2014

A brute force attack is an attempt to log into a secure system by making lots of attempts in the hopes of eventual success.Brute force is also a algorithm design approach/technique in Computer Science. It’s not a sophisticated type of attack, hence the name “brute force.” For example, if you have a login process that requires a username and password, there is a limit as to the possible number of username/password combinations. That limit may be in the billions or trillions, but still, it’s a finite number. Using algorithms and automated processes, a brute force attack repeatedly tries combinations until they succeed. The best way to prevent brute force attacks from succeeding is requiring users to register with good, hard-to-guess passwords: containing letters, numbers, and punctuation; both upper and lowercase; words not in the dictionary; at least eight characters long, etc. Also, don’t give indications as to why a login failed: saying that a username and password combination isn

Auto proxy changer application code in VB .Net

June 25, 2014

The code given here is for making simple app in VB .Net which will read a proxy address from file specified in path field and then automatically change the proxy settings of PC to next proxy address from file after specified time amount. Note that file contains list of proxy addresses with port number in format as given below 0.0.0.0:80 1.1.1.1:8080 …………… ……………. i.e., ip address and port number. Form1.vb ----------------------------------------------------------------------------------Imports SystemImports System.Runtime.InteropServicesImports System.IOImports Microsoft.VisualBasicImports System.TimersPublic Class Form1 Dim FILE_NAME As String = "C:UsersDellDocuments12.txt" Dim label As String Public proxy(2000) As String Public index As Integer = 0 Public max_proxys As Integer=0 Dim a As String Dim start_check As Integer = 0 Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load End Sub Priv

Code for setting webview in android

/*WebViewActivity.java*/package com.shubham.webviewclientdemo;import android.app.Activity;import android.graphics.Bitmap;import android.os.Bundle;import android.view.KeyEvent;import android.view.View;import android.webkit.WebView;import android.webkit.WebViewClient;import android.widget.ProgressBar;/* * Demo of creating an application to open any URL inside the application and clicking on any link from that URlshould not open Native browser but that URL should open in the same screen.- Load WebView with progress bar */public class WebViewClientDemoActivity extends Activity { /** Called when the activity is first created. */ WebView web; ProgressBar progressBar; StartAppAd startAppAd; @Override public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.main); web = (WebView) findViewById(R.id.webview01); progressBar = (ProgressBar) findViewById(R.id.progressBar1); web.setWebViewCl

Matlab code to convert image to binary numbers and convert back to original

% Reading image fileim=imread('C:UsersDellPictures17.jpg'); % converting image to binary numbersc=dec2bin(im,8);% saving result to filefid = fopen('magic5.txt', 'w+');fwrite(fid,c); fclose(fid);disp(c);%converting image back to decimal and displaying it.d=reshape(uint8(bin2dec(c)),size(im) );image(d); [AMAZONPRODUCTS asin=”0470767855″]

PHP code to delete all files in folder

<?php$files = glob('images/*'); // get all file namesforeach($files as $file){ // iterate files if(is_file($file)) unlink($file); // delete file}?>

Code for handling exceptions in C++

#include <iostream>using namespace std;double division(int a, int b){ if( b == 0 ) { throw "Division by zero condition!"; } return (a/b);}int main (){int x,y;double z;cout<<"Enter value of x and yn";cin>>x>>y; try { z = division(x, y); cout << z<<"n"; }catch (const char* msg){ cout << msg << endl; } return 0;}

Program code demonstrating use of abstract class in C++

#include<iostream>using namespace std;class abstractinterface { public: virtual void numbers()=0; void input(); int a, b; };void abstractinterface::input() { cout<< "Enter the numbersn"; cin>>a>>b; }class add : public abstractinterface { public: void numbers() { int sum; sum=a+b; cout<<"sum is "<<sum<<"n"; } };class sub : public abstractinterface { public: void numbers() { int diff; diff=a-b; cout<<"diff is "<<diff<<"n"; } };int main() { add obj1; obj1.input(); obj1.numbers(); sub obj2; obj2.input(); obj2.numbers();return 0; }

Simple visitor counter in php

<?php $file = file_get_contents("count.txt"); $visitors = $file; $visitorsnew = $visitors+1; $filenew=fopen("count.txt","w"); fwrite ($filenew,$visitorsnew); echo "you've had $visitorsnew visitors."; ?>

PHP code for handling file uploads

<?php//properties of the uploaded file$name= $_FILES["myfile"]["name"];$type= $_FILES["myfile"]["type"];$size= $_FILES["myfile"]["size"];$temp= $_FILES["myfile"]["temp_name"];$error= $_FILES["myfile"]["error"];if ($error > 0) die("Error uploading file! code $error.");else { if($type=="image/png" || $size > 2000000)//condition for the file { die("Format not allowed or file size too big!"); } else { move_uploaded_file($temp, "uploaded/" .$name); echo "Upload complete!"; }}?>

Simple user registration code in php using md5 encryption for password encryption.

<?php echo "<h1> Register</h1>"; $submit = $_POST['submit']; //form data $fullname = strip_tags($_POST['fullname']); $username = strtolowe(strip_tags($_POST['username '])); $password = strip_tags($_POST['password ']); $repeatpassword =strip_tags($_POST['repeatpassword ']); $date = date("y-m-d"); if($submit) { $connect = mysql_connect ("localhost","root",""); mysql_select_db("phplogin"); $namecheck = mysql_query("SELECT username FROM users WHERE username ='$username'"); $count = mysql_num_rows($namecheck); if($count!=0) { die("Username already taken!"); } // check for existance if($fullname && $username && $password && $repeatpassword) { if($password == $repea

C program demonstrating use of pipe system call in linux (sending message from child process to parent process)

#include <sys/wait.h>#include <stdio.h>#include <stdlib.h>#include <unistd.h>#include <string.h>intmain(int argc, char *argv[]){int pipefd[2];pid_t cpid;char buf[30];if (pipe(pipefd) == -1) {printf("pipe error");exit(EXIT_FAILURE);}cpid = fork();if (cpid == -1) {printf("fork error");exit(EXIT_FAILURE);}if (cpid == 0) { /* Child reads from pipe */char buffer2[30]="message from child";printf("nnNow in child processn");close(pipefd[0]); /* Close unused write end */write(pipefd[1], buffer2, sizeof(buffer2));close(pipefd[1]);_exit(EXIT_SUCCESS);} else {printf("nnNow in parent process n");wait(NULL);close(pipefd[1]); /* Close unused read end */read(pipefd[0], buf, sizeof(buf));printf("nnNow again in parent process n");printf("nn%snn",buf);close(pipefd[0]);/* Wait for child */exit(EXIT_SUCCESS);}} Output : dell@linuxmint ~ $ gcc pipe.c dell@linuxmint ~ $ ./a.out Now in parent process Now in c

C program to demonstrate use of fork and wait system call in linux

/*Here we read a number from parent process send it to the child process and then child process calculates factorial of that number and returns to the parent process which then prints the final answer*/#include <sys/wait.h>#include <stdlib.h>#include <unistd.h>#include <stdio.h>main( ){int num;int fact=1;int pid, w=1;int status;printf("Enter any number....nn");scanf("%d",&num);printf("n");pid = fork();if (pid == -1) {printf("nError Occured in Forking a Process..n");exit(0);}if (pid==0) { /* Code executed by child */printf("nnChild PID is %ldnn", (long) getpid());/* Wait for signals */int i=0;if(num==0||num==1){fact=1;exit(fact);}else{for(i=1;i<=num;i++){fact = fact * i;//printf("fact= %d",fact);}printf("n Child Execution Completed...n");exit(fact);}}else {/* Code executed by parent */wait(&w);printf("n W= %d",w);int f=WEXITSTATUS(w);printf("nNow in parentnn");pri

C program demonstrating use of signal and alarm system call and signal handler function

#include<stdio.h>#include<signal.h>#include<stdlib.h>typedef void (*sighandler_t)(int);void signal_handler(int signum){printf("nn Sorry Time Outnn");exit(0);}int main(){char uname[20],passwd[20];signal(SIGALRM, signal_handler);printf("nEnter your Username and Password");alarm(10);scanf("%s %s",uname,passwd);printf("nnUsername: %s n Password:%s n You Have Logged In Successfully.. ",uname,passwd);return 0;}

C program demonstrating use of stat system call

#include<stdio.h>#include <sys/types.h>#include <dirent.h>#include <sys/stat.h>#include <unistd.h>#include <string.h>#include <time.h>#include <stdlib.h>#define DIR1 "test/"const char dir[20]="test/";/* struct dirent {ino_t d_ino; // inode numberoff_t d_off; // offset to the next direntunsigned short d_reclen; // length of this recordunsigned char d_type; // type of file; not supportedby all file system typeschar d_name[256]; // filename}; struct stat {dev_t st_dev; // ID of device containing fileino_t st_ino; // inode numbermode_t st_mode; // protectionnlink_t st_nlink; // number of hard linksuid_t st_uid; // user ID of ownergid_t st_gid; // group ID of ownerdev_t st_rdev; // device ID (if special file)off_t st_size; // total size, in bytesblksize_t st_blksize; // blocksize for file system I/Oblkcnt_t st_blocks; // number of 512B blocks allocatedtime_t st_atime; // time of last accesstime_t st_mtime; // time of last

MATLAB code for Circular Convolution using Matrix method

% circular convolution using matrix method %clc;x=[1,2,3,1];h=[4,3,2,2];H=zeros(0);h_index=1;N=length(x);y=zeros(0);disp('x(n)=');disp(x);disp('h(n)=');disp(h);subplot(4,1,1);stem(x);title('Signal One : x(n) ');subplot(4,1,2);stem(h);title('Signal Two : h(n)');temp=0;for i=1:NH(i,1)=h(i);end;for i=2:N h=circshift(h, [0,1]); for j=1:N H(j,i)= h(j); end;end;disp('H =>>');disp(H);for i=1:N for j=1:N temp=temp+H(i,j)*x(j); end; y(i)=temp; temp=0;end;disp('Circular Convolution : y(m)==>');disp(y);subplot(4,1,3);stem(y);title('Circular Convolution: y(m)'); Output:

Circular Convolution using DFT and IDFT ( W4 Matrix )

% W4 circular_convolution_using_dft_idft clc; x1=[1,2,3,4]; x2=[1,1,1,0]; N=4; disp('x1(n)='); disp(x1); disp('x2(n)='); disp(x2); % finding dft of x1 % x1k=zeros(); w4=zeros(); for i=0:N-1 for j=0:N-1 w4(i+1,j+1) = exp(complex(0,(-2*pi*i*j)/4)); end; end; fprintf('W %d Matrix =',N); disp(w4); temp=zeros(); for i=1:N for j=1:N temp=temp+(w4(i,j).*x1(j)); end; x1k(i)=temp; temp=0; end; disp('x1(k) = ') disp(x1k); % finding dft of x2 % x2k=zeros(); temp=zeros(); for i=1:N for j=1:N temp=temp+(w4(i,j).*x2(j)); end; x2k(i)=temp; temp=0; end; disp('x2(k) ='); disp(x2k); % multiplying obtained DFT outputs X1(K)*X2(K) % Y_K=zeros(); for i=1:N Y_K(i)=x1k(i)*x2k(i); end; disp('Y(K) ='); disp(Y_K); %finding IDFT Y(K) that is output of %circular conv. %computing complex conjugate of w8 w4=conj(w4); disp('complex conjugate of w4'); disp(w4); y_m=zeros(); temp=zeros(); for i=1:N for j=1:N temp=temp+(1/4*(w4(i,j).*Y_

Circular convolution using IDFT and DFT ( using W8 matrix )

% W8 circular_convolution_using_dft_idft%clc;x1=[1,1,1,0,0,0,0,0];x2=[0,1,2,2,1,0,0,0];N=8; % finding dft of x1 %x1k=zeros();w8=zeros(); for i=0:N-1for j=0:N-1w8(i+1,j+1) = exp(complex(0,(-2*pi*i*j)/N));end;end;fprintf('W %d Matrix =',N);disp(w8); temp=zeros();for i=1:Nfor j=1:Ntemp=temp+(w8(i,j)*x1(j));end;x1k(i)=temp;temp=0;end; disp('x1(k) = ')disp(x1k); % finding dft of x2 %x2k=zeros();temp=zeros();for i=1:Nfor j=1:Ntemp=temp+(w8(i,j)*x2(j));end;x2k(i)=temp;temp=0;end;disp('x2(k) =');disp(x2k); % multiplying obtained DFT outputs X1(K)*X2(K) %Y_K=zeros();for i=1:NY_K(i)=x1k(i)*x2k(i);end;disp('Y(K) =');disp(Y_K); %finding IDFT Y(K) that is output of circular conv.% %computing complex conjugate of w8w8=conj(w8); disp('complex conjugate of w8');disp(w8); y_m=zeros();temp=zeros();for i=1:Nfor j=1:Ntemp=temp+((1/N)*(w8(i,j)*Y_K(j)));end; y_m(i)=temp;temp=0;end;disp('y_m=');disp(y_m); %plottingsubplot(3,2,1);stem(x1);title('x1(

MATLAB code for making simple calculator with GUI