Copying files from my remote linux machine

Copying files from my remote linux machine

•  Copying file

Source:  File vs.txt in the location /home/vineeshvs_logged_in/Desktop from the pc I have logged in.
Destination: /home/vineeshvs_remote_pc/Desktop in a remote pc in the same network with ip 10.107.104.1

scp /home/vineeshvs_logged_in/Desktop vineeshvs@10.107.104.1:/home/vineeshvs_remote_pc/Desktop

• Copying folder

Source:  Folder vs in the location /home/vineeshvs_logged_in/Desktop/vs from the pc I have logged in.
Destination: /home/vineeshvs_remote_pc/Desktop in a remote pc in the same network with ip 10.107.104.1

scp -r /home/vineeshvs_logged_in/Desktop/vs vineeshvs@10.107.104.1:/home/vineeshvs_remote_pc/Desktop

{ Read More }

Copying Remote Files with scp

Copying Remote Files with scp

If you have ssh server running on a machine ,you can transfer files to/from it to any other machine connected over a network to it. The open-ssh server suite provides a utility named scp. Here are some examples of useful scp usage

Copy myfile to server
( ubg is my username on server1. You can replace server1 with an IP address or domain name. If the user names on the local machine and the remote server are the same you can omit it. See examples below.)

$ scp myfile ubg@server1:/tmp/
Password: ******

Copy remote myfile to local working dir
$ scp server1:/tmp/myfile /home/ubg/myfile
Password: ******

Use the -p option to preserve permissions and timestamps on the copied files:
$ scp -p myfile server1:/tmp/

If the SSH service is configured to listen on a port other than the default port 22, use
-P to indicate that port on the scp command line:
$ scp -P 12345 myfile server1:/tmp/

To do recursive copies, from a particular point in the remote file system, use the
-r option:

$ scp -r mydir ubg@server1:/tmp/

The above command will copy all mydir to remote /tmp

{ Read More }

Creating Remote APT Package Repositories on Github

Creating Remote APT Package Repositories on Github

Creating remote apt repos on github

If you read my last blog post, you would know that I am really into deploying applications and systems with deb and rpm packages whenever possible. One of the things you will eventualy want to do, to make deb and rpm package deployment easier, is to create a remote apt or yum repo to host your packages. One of the cooler tricks I have come up with in recent times is creating apt/yum repos in obscure locations.

This aritical will go over how to host an apt repo inside of a git repo on github. This little trick has the following benifits:

1. free hosting for you apt repo
2. publicly availability
3. provide a single location for source code and packages.

Yes, I am aware there are free services to host deb and rpm packages, but sometimes, for various reasons, these serices dont fit the bill. While turning github into a remote apt repo is a strange technique, I do find it to be very useful. Keep in mind, you can use this technique on various other free services which you wouldnt expect to funciton as a remote apt repository. Basicly, if you can upload a directory structure and files, you can turn it into an apt or yum repo.

Install Dependancies for Creating our Repo

• First, on our local system, we need to install some dependancies for creating our apt repo which we will upload to github.

• For this step, you will need an Debian or Ubuntu system.

  sudo apt-get install reprepro

Create the Repo Directory Stucture

• Feel free to change "myrepo" to whatever you like. If you do, just remember to also do so in consecutive steps.

  mkdir -p myrepo/{conf,incoming}

Setup The Core Apt Repo Config File

• In this step we will create the main config file for our apt repo. There are many cusomization options in this step.

• Suite is typicaly set to either stable or unstable

• Codename can be anything you wish but is typicaly set to the short name release version of the Debian or Ubuntu distribution such as wheezy, sid, jessie, lucid, precise, trusty, utopic, vivid

• Architecture is typicaly set as one of the following i386, amd64 or armel

• If you want to include multiple architectures in your repo, just include them on the architeres line but seperate each value with a space

• More advanced documentation on conf/distributions files can be found here

  cd myrepo
  cat <<EOF >> conf/distributions
  Origin: Hackgnar
  Label: Hackgnar Github
  Suite: stable
  Codename: wheezy
  Architectures: armel
  Components: main
  Description: Debian ARM packages hosted on hackgnars github repository
  EOF

Add a deb package to your apt repo

• Before we finish up and upload our apt repo to github, we have to add some deb packages to it.

• Make sure the deb packages you use in the following step are of the same architecture you used in the previous step.

  reprepro includedeb wheezy ../my_deb_package.deb

• If you get errors with the above command, try running the following instead:

  reprepro -S utils includedeb wheezy ../my_deb_package.deb

Turn your Apt repo into a git repo and upload it

• This step requires that you have create a new git repo on github. If you have not, you can read how to do so here
• Also make sure that you have setup your github ssh keys so you can properly push your repo to github.

• finally, this example uses git@github.com:hackgnar/myrepo.git but you will replace that with your initialized github repo.

  cd ~/src/myrepo
  git init
  git add -A
  git commit -a -m "initial commit"
  git remote add origin git@github.com:hackgnar/myrepo.git
  git push -u origin master

(Optional) Connect a system to your new apt repo and install something:

• This step shows you how to connect a new system to your newly created apt repo and install a package from it. Note that we have to install and use `apt-transport-https as a dependancy because apt-get does not support https by default.
• make sure to replace hackgnar with the name you use for your github account
• make sure to replace myrepo with the name you used for your github repo

• make sure to replace my-package-name with the name of the deb package you installed into your repo

  sudo apt-get install apt-transport-https
  cat << EOF >> /etc/apt/sources.list
  deb https://raw.githubusercontent.com/hackgnar/myrepo/master main
  EOF
  sudo apt-get update
  sudo apt-get install my-package-name

Summary

If you have read this far, hopefully you found this weird tutorial to be useful. While this apt hosting technique may not work for everyone, I have found it to be very handy. Also, keep in mind that there are limitations to this technique. One such example is github file size limitations which will restrict the size of the deb packages you invlude in your repo. Let me know what you think and what other services you find that allow you to host deb packages using this technique.

{ Read More }

Create a passwordless SSH login to remote computers to automate launch of a zookeeper cluster

Create a passwordless SSH login to remote computers to automate launch of a zookeeper cluster

Passwordless SSH Login can be useful in situations, for example, when one likes to write a bash script that automatically setup or launch a zookeeper cluster. Suppose we want to ssh login to a computer at 192.168.2.4 without a password from a our client computer at 192.168.2.2, we can do so by creating a RSA pair of authentication key at 192.168.2.2 terminal:

> ssh-keygen -t rsa

In the terminal prompt, just press the ENTER key multiple times for each prompt. Once this is done, a public key rsa_id.pub is stored in the .ssh folder under the user root folder. Now ssh login to the 192.168.2.4 (password required at this point) to create the .ssh folder under the user root folder on that computer:

> ssh username@192.168.2.4 mkdir .ssh

The last step is to append the .ssh/rsa_id.pub file (which is the public key) from 192.168.2.2 to the .ssh/authorized_keys in 192.168.2.4 (password required at this point):

> cat .ssh/id_rsa.pub | ssh username@192.168.2.4 cat >> .ssh/authorized_keys

Now we can ssh login to 192.168.2.4 from the client 192.168.2.2 without password. For example, run the following command in the client 192.168.2.2 terminal:

> ssh username@192.168.2.4

Now we are ready to write a bash script to launch the zookeeper cluster which consists of the following networked computers:

192.168.2.1
192.168.2.3
192.168.2.4

First make sure that the zookeeper is setup in the cluster properly (follow instructions from this link: http://czcodezone.blogspot.sg/2014/11/setup-zookeeper-in-cluster.html and for storm at this link http://czcodezone.blogspot.sg/2014/11/setup-storm-in-cluster.html) and make sure the above steps of passwordless SSH login is repeated for 192.168.2.1 and 192.168.2.3 so that user can log into these two computers without password from the client 192.168.2.2. Now on the terminal of 192.168.2.2, create a start_zookeeper_cluster.sh script:

> touch start_zookeeper_cluster.sh
> gedit start_zookeeper_cluster.sh

In the start_zookeeper_cluster.sh that opens, enter the following lines:

ssh username@192.168.2.1 $ZK_HOME/bin/zkServer.sh start
ssh username@192.168.2.3 $ZK_HOME/bin/zkServer.sh start
ssh username@192.168.2.4 $ZK_HOME/bin/zkServer.sh start

Save and close the start_zookeeper_cluster.sh, and run the following command on 192.168.2.2 client terminal to make it executable:

> chmod +x ./start_zookeeper_cluster.sh

Now run the script to start the zookeeper cluster:

> ./start_zookeeper_cluster.sh

We can also write a stop_zookeeper_cluster.sh, start_storm_cluster.sh, stop_storm_cluster.sh in a similar way to automate the process.

{ Read More }