S3cmd Installation On Ubuntu

's3cmd' is a command utility to manage S3 through command lines. I've given the installation steps for ubuntu linux.

Installation 

1. Import S3tools signing key:

# wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key | sudo apt-key add -


2. Add the repo to sources.list:

# sudo wget -O/etc/apt/sources.list.d/s3tools.list http://s3tools.org/repo/deb-all/stable/s3tools.list

3. Refresh package cache and install the newest s3cmd:

# sudo apt-get update && sudo apt-get install s3cmd

# sudo s3cmd --configure

4. Then it'll ask for  Access Key ID and Secret Access Key which you can get from AWS Account Security Credentials.


Thanks

Voltdb Client

Building Voltdb PHP and CPP Client

1. Download Voltdb Client Php Extension-

# wget http://voltdb.com/downloads/technologies/client/voltdb-client-php-extension-2.8.4.tar.gz

2. Extract it to /opt

cd /opt

tar -xvzf voltdb-client-php-extension-2.8.4.tar.gz

[You'll get voltdb-client-cpp and voltdb-client-php under voltdb-client-php-extension-2.8.4

3. Install dependencies -

# yum install php-dev glibc-static zlib zlib-devel

# cd /opt/voltdb-client-php-extension-2.8.4/voltdb-client-cpp

# make

# cd /opt/voltdb-client-php-extension-2.8.4/voltdb-client-php

# phpize

[ If phpize command not found -

rpm -Uvh http://epel.mirror.net.in/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm

then install php-devel-<version> ]

4. Configure 

# ./configure --with-voltdb=/opt/voltdb-client-php-extension-2.8.4/voltdb-client-cpp/ CXXFLAGS="-m64" LDFLAGS="-m64"

# make

# vi /etc/php.ini

-and edit following line

extension=/opt/voltdb-client-php-extension-2.8.4/voltdb-client-php/modules/voltdb.so

# php -m | grep voltdb

Output - voltdb

If you have any queries or suggestions, please comment.

Thanks

Shell Script : Mail Attachment Linux uuencode

This example shows how to send an email with multiple attachments in linux using uuencode and mail command.

Files to be attached -

/opt/file1.txt
/opt/file2.txt
/opt/file3.txt
/opt/file4.txt

Code -

#/bin/bash

>/tmp/out.mail

echo "Your messae body" >> /tmp/out.mail

for i in `ls /opt/file*.txt`
do
uuencode  $i $i  >> /tmp/out.mail

done

mail -s "Subject" emailid@domain.com < /tmp/out.mail

#######

[ Note - If you've files having different names like abcd.txt outfile.txt sample.csv etc. then you can define the 'for' loop like - for i  in  abcd.txt outfile.txt sample.csv and while uuencode give the full path ]


If you have any queries or suggestions , please comment.



Thanks

Deep DB Mysql

                               Deep DB Mysql - A general purpose database for Big Data

Installation - 

My O/s - Centos 6, 64 bit


* Update Repository -

# wget "http://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/epel-release-6-5.noarch.rpm"
# wget "http://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/ius-release-1.0-11.ius.el6.noarch.rpm"

# rpm -Uvh epel-release-6-5.noarch.rpm
# rpm -Uvh ius-release-1.0-11.ius.el6.noarch.rpm

* Installing Mysql55

# yum install mysql55 mysql55-libs mysql55-server mysql55-devel

# /etc/init.d/mysqld start

* Set root password

# /usr/bin/mysqladmin -u root password 'password'

# /etc/init.d/mysqld stop

# chkconfig mysqld on --level 235

* Install Deepdb Plugin

# yum install yum-utils
# yum-config-manager --add-repo http://repo.cloudtree.net/repo

# yum install --nogpgcheck deepdb-5.5.31-plugin.x86_64

* Start mysql

# /etc/init.d/mysqld start


Thanks

Shell Script : lftp Example

Hi guys, the following scripts you can use to upload and download directories from your ftp server using lftp.


1. Upload Script

#!/bin/bash

### Defining Path

dir1="/source/path"
dir2="/destination/path"

user="ftp-username"
pass="password"

### Upload command

lftp -e "mirror  -R $dir1 $dir2 ;quit" -u $user,$pass yourdomain.com

#### Checking upload failure/success

if [ $? -eq 0 ]
then

echo "Download Successful"
exit 0;
else
echo "Download Not Successful"
exit 1;
fi
####

2. Download Script

In above script change this  command 

lftp -e "mirror  -R $dir1 $dir2 ;quit" -u $user,$pass yourdomain.com

to

lftp -e "mirror  $dir2 $dir1 ;quit" -u $user,$pass yourdomain.com


[ 

Note: You can pass source and destination path (parameters) from command line as run time argument. For this replace source and destination path with $1 and $2 and run your script like below -

sh filename.sh source destination  

]   

If you have any queries and difficulties, please comment. 

  

Thanks

Luarocks and Moonscript Installation

* Get luarocks distribution -

# wget http://luarocks.org/releases/luarocks-2.0.6.tar.gz

# tar xvzf luarocks-2.0.6.tar.gz 
# cd luarocks-2.0.6

* Find lua path and configure with lua

# find / -name lua.h

# ./configure --with-lua-include=/opt/lua/include/ --with-lua=/opt/lua/

# make
# make install

* Now Install moonscript

# luarocks install moonscript


Thanks

Installing redis server

Hi, folks
I am gonna show you how to install 'redis' server and setup master slave on a linux
box. Redis is a key-value-store database and is an open source. Here I'll set up two redis instance one acting as a master and other slave - location /opt and /var

Steps for the Redis Configuration :

Step 1:

Redis Download :

 # wget http://redis.googlecode.com/files/redis-2.6.10.tar.gz

Step 2:

Then extract it :

# tar -zxvf redis-2.6.10.tar.gz# cd redis-2.6.10

Step 3 :

Run make

Step 4 :

make test

Step 5 :  Master configuration

a. ) /opt/redis-2.6.10/redis.conf

b. ) daemonize yes

c. ) port 6379 ( you can change it)

Step 6: Client configuration

a. ) /var/redis-2.6.10/redis.conf

b. ) daemonize yes

c. ) port 6380

d. ) slaveof Master IP 6379

Syntax : slaveof <Master Ip> <Master port no>

  Step 7 :

Now go to master server and type :

# cd /opt/redis-2.6.10/src

Step 8 :

# ./redis-server /opt/redis-2.6.10/redis.conf

Step 9 :

# ./redis-cli -p 6379

Step 10 :Then switch to slave terminal and type :

# cd /var/redis-2.6.10/src

Step 11:

# ./redis-server /var/redis-2.6.10/redis.conf

# ./redis-cli -p 6380

Step 12 :

then to check replication type :

info

Output:

Replication
role:slave
master_host:<Master IP >
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_priority:100
slave_read_only:1
connected_slaves:0

So, this is how we setup redis server master- slave.
Looking forward your comments and suggestions. Thanks !    

How to encrypt and decrypt a file

In Linux there is a utility 'gpg' which is used for encrypting and decrypting of  files.
Syntax -

$ gpg -c filename

[ It will then prompt to enter a pass phrase and it will create a file 'filename.gpg' ] 

Enter passphrase:<YOUR-PASSWORD>
Repeat passphrase:<YOUR-PASSWORD>

To decrypt -
 $ gpg filename.gpg        

[ Now give the same pass phrase to decrypt the file ]

gpg filenme.gpg
gpg: CAST5 encrypted data
Enter passphrase:<YOUR-PASSWORD>

Nginx Openresty

                                                   Lua nginx openresty configuration


* OS - Centos 6, 64bit
* Dependencies - pcre.x86_64, pcre-devel.x86_64
* With module geoip

Building from source distribution - ngx_openresty-1.2.6.6.tar.tgz , GeoIP-1.4.8.tar.gz

Extract and copy to  /opt/

# cd /root/openresty/GeoIP-1.4.8
# ./configure ( if libtool error occure install below)
# make
# make install

# cd /opt/ngx_openresty-1.2.6.6/
# ./configure --with-luajit
# make 
# make install

Testing 

# mkdir /root/luatest
# cd /root/luatest
# mkdir conf logs

Edit nginx.conf and paste following

#
worker_processes  1;
error_log logs/error.log;
events {
            worker_connections 1024;
}
http {
            server {
                            listen 8080;
                                    location / {
                                                        default_type text/html;
                                                                    content_by_lua '
                                                                                    ngx.say("<p>hello, world</p>")
                                                                                                ';
                                                                                                        }
                                                                                                            }
}

#

Start nginx -

# /usr/local/openresty/nginx/sbin/nginx -p /root/luatest -c /root/luatest/conf/nginx.conf

If error comes for lib, execute following and then again start nginx

# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

# curl http://127.0.0.1:8080/  



Thanks

Shell Script : How to use an array to get n number of similar set for different values

In this example we are going to obtain below set (used in nagios config to define a host) using array -

define host{
              use                     linux-server
              host_name              <host>
              alias                        <host>
              address                    <ip>
            }

I have two files, one contains hostname  and second has their IPs.

host.txt           ip.txt 
host1               x.1
host2               x.2
host3               x.3
host4               x.4

Code :

#!/bin/bash

#### Storing the host names in array 'arr', reading from host.txt

j=1
for i in `cat host.txt`
do
  arr[$j]=$i

  j=`expr $j + 1`
done

#### Storing the IP's names in array 'arrp', reading from ip.txt

k=1
for m in `cat ip.txt`
do
  arrp[$k]=$m

  k=`expr $k + 1`
done

#### Taking count of the records

count=`cat host | wc -l`

#### Printing the values stored in the two arrays for making the set

for x in `seq 1 $count`
do
echo "define host{
              use                     linux-server
               host_name           ${arr[$x]}
              alias                      ${arr[$x]}
              address                 ${arrp[$x]}
            }"  
done
##############

Output : 

define host{
              use                     linux-server
               host_name        host1
              alias                   host1
              address              x.1
            }
define host{
              use                     linux-server
               host_name        host2
              alias                   host2
              address               x.2
            }
define host{
              use                     linux-server
               host_name         host3
              alias                    host3
              address                x.3
            }
define host{
              use                     linux-server
               host_name          host4
              alias                     host4
              address                x.4
            }

 Like above we can use array to make similar n number of sets for different values instead of  writing multiple times :)  

Shell Script : How to move all empty files using a shell script

 Here I am giving a simple bash shell script to move all the empty files (zero byte) from current directory to another location-

1. Go to that path where you want to move the files from.
2. Make a text file and edit the following lines (accordingly) -

 #!/bin/bash

for i in `ls`

do 

if [ -s $i ]

then

echo "The file $i has data"

else 

mv $i  <Path where you want to move the empty files>

fi

done

3. Save the file, give execute permission (chmod +x filename) and run.

Hope you've got the desired result.. : ) 

How To Replace A String Recursively

To replace a string in all files recursively with your matching string globally, The following perl utility is very helpful -

Go to the directory and run -

# perl -p -i -e 's/string/replace-with/g' `grep -rl string *`



If your string contains '/' for example you have to replace '/home/sandy' with '/home/mandy' the use it like -

# perl -p -i -e 's/\/home\/sandy/\/home\/mandy/g' `grep -rl sandy *`


If you have any query, please post your comments.
Thanks !

Shell Script : Bash Shell CGI

Shell script with cgi allows you to run linux commands/scripts from web. It can be used to build a monitoring tool for your linux machines/servers or managing your servers from web.

For this you should have cgi integrated with your web service nginx/apache and the user should  have access the cgi document root and  permission to run cgi script.

Here I'm giving an example how you can have some system information by running linux commands/shell script code -


#!/bin/bash

echo "Content-type: text/html"
echo ""
echo "<html><body><head><title>Bash Shell CGI"
echo '</title></head><body bgcolor="#3EA99F">'

### Getting information into veriables

load=`uptime | awk '{print $8,$9,$10}'`

Date=`date +'%Y-%m-%d'`


#### Load Test

### Getting current load avg integer value

la=`uptime | awk '{print $8}' | awk -F '.' '{print $1}'`

## Doing a test

if [ $la -gt 10 ]
then
stat="Critical"
elif [ $la -le 10 ]
then
stat="OK"
fi

####### Printing the information


echo "<h1><font color="blue"><u>System Information</u></font></h1><br><br>"
echo "<font color="black"><h3>Hostname : <font color="white">`hostname`</font></h2></font><br>"
echo "<font color="black"><h3>Load Average : <font color="white">$load - $stat</font></h2></font><br>"
echo "<font color="black"><h3>Date : <font color="white">$Date</font></h2></font><br>"
echo "</body></html>"
 
##### 


Put this file to your cgi-bin directory and run in browser using -


http://your-domain/cgi-bin/filename.cgi


Output : 

Thanks...! If you have any query or suggestions, please comment.
  

How To Setup Voltdb Cluster on AWS Cloud (Part - 2/2)

VoltDb Setup :  (3 nodes cluster ) 

===========================  

Download the voltdb community edition, extract it and move to /opt as /opt/voltb

wget http://voltdb.com/downloads/technologies/server/LINUX-voltdb-3.0.tar.gz

Download and install the java jdk-6u41-linux-x64.bin

http://www.oracle.com/technetwork/java/javase/downloads/jdk6downloads-1902814.html  

Copy to /var , give execute permission and run 

# ./jdk-6u41-linux-x64.bin

Install php-5.3.X

yum install php

Set path variable for java and voltdb (Also put in ~/bashrc)

# export PATH="$PATH:/opt/voltdb/bin" 

# export JAVA_HOME=/var/jdk1.6.0_41 

Node-1 (Leader) -

1. Create a dir /opt/testvoltdb and create a helloworld.sql in it and edit below lines -

CREATE TABLE HELLOWORLD (

HELLO VARCHAR(15),

WORLD VARCHAR(15),

DIALECT VARCHAR(15) NOT NULL,

PRIMARY KEY (DIALECT)

);

[ It will create a table HELLOWORLD ]

2. compile the sql file to create the new catalog

# voltdb compile -o helloworld.jar helloworld.sql

3. Create the deployment.xml file containing the info about how many nodes and partition needs to be used -

# vi deployment.xml

<?xml version="1.0"?>

<deployment>

<cluster hostcount="3"

sitesperhost="4"

kfactor="2"

/>

<paths>

<voltdbroot path="/opt/testvoltdb" />

<snapshots path="/opt/testvoltdb/snapshot" />

</paths>

</deployment>

4. Create and start the database

# voltdb create host <node1-hostname> catalog helloworld.jar deployment deployment.xml &

# sqlcmd

SQL Command :: localhost:21212

1> INSERT INTO HELLOWORLD VALUES(

2> 'You','Are', 'Welcome!');

(1 row(s) affected)

3> SELECT * FROM HELLOWORLD ;

HELLO WORLD DIALECT

------ ------ --------

You Are Welcome!

(1 row(s) affected)

5. Copy the runtime catalog (i.e helloworld.jar ) and deployment.xml to the other nodes on same path /opt/testvoltdb

Node 2 & 3

6. start the database and join the nodes to main cluster node (node-1)

# voltdb create host <node1-hostname> catalog helloworld.jar deployment deployment.xml &
# sqlcmd
1 > SELECT * FROM HELLOWORLD ;
HELLO WORLD DIALECT
------ ------ --------
You Are Welcome!
(1 row(s) affected)   
Thanks....!