20 *nix commands that I used most often when getting started with DevOps.
In the past month, I was immersed in DevOps starting up various EC2 and RDS instances in AWS, trying to deploy my app consisting of Rails API server, AngularJS frontend and PostgreSQL database in various environments such as staging or production.
It was a challenging time for me as I needed to have a systems view instead of just a single application. Yet, firing up some instances in different operating systems also gave me a slight insight to their similarities and differences.
Below are the list of 20 most used commands. Why is noticing the most used commands important? Because ultimately, we can include them in a script and build automation for various tasks in DevOps.
Here we start!
Logging into the Server
ssh-keygen generate ssh keys
Even before we log in to the remote server, we generate a pair of public and private ssh keys. Ultimately, the contents of the public key
*.pub can go into Github SSH keys or the remote computer's
$ ssh-keygen -t rsa -f project-name -C "email@example.com"
ssh secure shell
Next, to login there are various ways of using the
ssh command. The 3 common ways that I have used are:
- with a config file at
Host projectname HostName website.com Port 8000 IdentityFile ~/.ssh/sshkey IdentitiesOnly=yes User username
And now we can ssh into the server with a much simpler command
$ ssh projectname
- ssh with a username and a public key
<!--emailoff-->$ ssh -i mykey.pem [email protected]<!--/emailoff-->
- ssh directly into a particular directory
$ ssh projectname -t "cd /path/to/app ; /bin/bash"
scp secure copy
scp comes in handy for transferring files from the local machine to the remote machine and vice versa.
- from local to remote
<!--emailoff-->$ scp -i /path/to/key.pem /path/to/file [email protected]<!--/emailoff-->
- from remote to local
<!--emailoff-->$ scp -i ~/.ssh/key.pem /path/to/source/file [email protected]:~/path/to/destination/file<!--/emailoff-->
To copy files or entire folder:
$ cp /path/to/source-folder/filename /path/to/destination-folder $ cp -r /path/to/source-folder/folder /path/to/destination $ cp /path/to/source-folder/filename /path/to/destination . # dot: destination is current folder
To move files/folders is also the same as renaming.
$ mv /path/to/source-folder/filename . # move file to current directory $ mv /path/to/source-folder . # move folder to current directory
To print out simple commands and piping the output to a new file.
$ echo hello $ echo `date` >> currentime.txt # copy to file $ echo `date` > currentime.txt # append
touch update a file
When used without any arguments,
touch will simply create a file.
$ touch filename1 filename2 filename3
Symbolic links for files are useful when we want the exact file contents in 2 different places to be synced up. This can be handy for doing so with executable files if needed.
$ ln -s /path/to/source/filename /path/to/destination/filename
head top of a file
head is very useful to quickly view a certain first few lines of a file with an option
-n to specify how many lines from the top.
$ head -n 5 /path/to/filename
tail end of a file
tail as the name goes, is the opposite of the command
head and this gives the content of the last lines of a file. Using with the option
-f, the output is appended as the file grows and this is used to continuously view server log files while changes are happening.
$ tail -n 5 /path/to/filename $ tail -f log/production.log
cat see the contents of a file
cat is a simple way to view the file contents, but it can also be used to concatenate a few files into a new file.
$ cat /path/to/filename $ cat file1 file2 >> newfile # newfile has contents of file1 and file2
chmod change permissions
File permissions include the ability to read, write or execute the file for users owner, group or others.
chmod can change these permissions with either a number or expression.
$ chmod u+x /path/to/script # user can execute $ chmod 600 ~/.ssh/authorized_keys # user can read, write
sudo act as root
sudo gives the permission to run commands as the root user. It can be used for a single command or a series of commands. This command should definitely be used with caution as root user will have permissions to execute things a normal user might not.
$ sudo !! # execute the previous command with sudo $ sudo cp /path/to/source /path/to/destination # execute as root $ sudo su # switches from current user to root
chown own the folder/file
To quickly change the owner of a folder or file
chown comes in handy in various situations.
$ sudo chown -R username:username ~/path/to/folder
cd change directory
Changing directory is such a common command. Here are some ways to use
$ cd ~ # go to home directory $ cd - # go to the last visited directory $ cd ../path/to/folder # go to relative directory one level up $ cd /path/to/folder # absolute path
ls list files/folders
Another common commands is to list the directory contents. Using
ls with 3 options are useful to view the dotfiles in long format to give a comprehensive look inside that folder.
$ ls -lah . # current folder $ ls -lah ~ # home folder $ ls -lah /path/to/folder # absolute path to a folder
ps running processes
$ ps aux | grep unicorn # find all unicorn processes
ps aux will give a long list of useful information about the processes including the user, CPU usage, path to the command as well as the Process ID (PID) which will come in handy if we want to stop that particular process.
kill terminate a process
Often, we want to find that particular process and kill/stop it so that we can make some change and then run it again. Coupled with the previous command
ps which will give the PID number, we can kill that particular process with a Unix signal 9. Here's an example:
$ ps aux | grep jekyll $ sayanee 1835 0.0 0.8 2509028 68756 s003 S+ 7:59PM 1:00.64 ruby /Users/command $ kill -9 1835
At other times, knowing which process is running in a particular port is also useful. For this
lsof comes in handy!
$ sudo lsof -i:8080 # complete info on the process $ sudo lsof -t -i:8080 # just the PID of the process $ kill -9 `sudo lsof -t -i:8080` # put the PID to kill it
crontab cron jobs for scheduling
Scheduling scripts to run at a certain time or an interval is helpful through CRON jobs. Sometimes CRON jobs are created through a backend application using the application specific variables. To see the output, listing the cron job is useful.
$ crontab -l
Getting started with DevOps
Noticing the most used commands is the first steps towards automation/shortcuts which can be done in the following ways:
- write the commands in a bash script
- include commands in a CRON job
- including commonly used commands as aliases for shorthands
I have also come to realize that there is a change in thought processes when I'm doing development versus devops. But in the end, it was rather rewarding to get to understand the file systems in various linux distributions or operating systems.
What common commands are you using? How do you automate your DevOps processes?
Updates: Some useful tools were suggested: