Peeter Joot's (OLD) Blog.

Math, physics, perl, and programming obscurity.

Shell tips and tricks. The most basic stuff.

Posted by peeterjoot on February 28, 2010

A while back I assembled the following shell tips and tricks notes for an ad-hoc ‘lunch and learn’ session at work. For some reason (probably for colour) I had made these notes in microsoft word instead of plain text. That made them of limited use for reference, not being cut and pastable (since word mucks up the quote characters). Despite a few things that are work centric (references to clearcase and our source code repository directory structure), there’s enough here that is generally applicable that the converted-to-text version makes sense to have available as a blog post.


# a=foo
# b=goo
# echo $a $b

foo goo
# p=/view/joes_view/vbs/engn/sqo
# diff $p/sqlofmga.C .

You will have many predefined variables when you login. Examples could include

$HOME                            home dir.
$EDITOR                          preferred editor.
$VISUAL                          preferred editor.
$REPLYTO                         where mail should be addressed from.
$PS1                             What you want your shell prompt to look like.
$TMPDIR                          Good to set to avoid getting hit as badly when /tmp fills up.
$CDPATH                          good for build tree paths.

CDPATH Example:


one can run: ‘cd sqo’ and go right to that component dir.

$CDPATH                            good for build tree paths.  Example: CDPATH=.:..:/home/hotelz/peeterj:/vbs/engn:/vbs/test/fvt/standalone:/vbs/common:/vbs/common/osse/core ; one can run: 'cd sqo' and go right to that component dir.
$1                                 First argument passed to a shell script (or shell function).
$*                                 All arguments that were passed to a shell script


All files starting with an 'a', and ending with a 'b'
# ls a*b

All files of the form 'a'{char}'b'

# ls a?b


Three different kinds. This is one of the most important things to know for any "shell programming".

Single quotes

Variables and wildcards are NOT expanded when contained in a set of single quotes. Example:

# a=foo
# b='goo boo'
# echo '$a $b'

$a $b

Double quotes

Variables and wildcards (*, ?, $...) are expanded (called globbing and/or interpolation sometimes depending on the context).

# a=foo
# b='goo boo'
# echo "$a $b"
foo goo boo

You don't have to double quote something for this sort of wildcard, and variable expansion, so you could write:

# echo $a $b

and the result will be the same:

foo goo boo

There is a difference though, namely, echo will treat this as three arguments, because the command is expanded before the final execution. This can be important when you want something with spaces to be treated as a single argument. Example:

# gcc x.c | grep ': error'

Back Quotes

Expression is executed as a command.

# cleartool checkin -c 'Please, let this compile and link this time.' `lsco`

Execution of a command in another one can also be done with a variable syntax (sometimes useful for nesting stuff). These would produce the same output:

# echo It once was: `date`
# echo It once was: $(date)

It once was: Mon Jun 18 16:20:28 EDT 2007

The alternate syntax can be useful if you wanted to run a command inside of a command.

Other Special Shell Characters

~              your home dir.
;              command separator
\              backslash (escape).  When you want to use a special character as is, you either have to single quote it, or use an escape character to let the shell know what you want.

Redirect input and output

|                            pipe input from another command
<                            redirect input
>                            redirect output
2>&1                         redirect stderr to stdin
echo hi > hi.out
cat < hi.out
cat /tmp/something | grep ': error' | sort
something_that_may_fail >/tmp/blah 2>&1
something_that_may_fail >/tmp/blah 2>/tmp/blah.err

The for loop.

If you have the quotes and variables mastered, this is probably the next most useful construct for ad-hoc command line stuff. We use computers for repetitive stuff, but it's amazing how little people sometimes take advantage of this.

By example:

# for i in `grep : /tmp/something` ; do echo $i ; done

Here, i is the variable you name, and you can reference it in the loop as $i.

# for i in `cat my_list_of_files` ; do cleartool checkout -nc $i ; done

If the command you want to run is something that accepts multiple arguments then you may even need a for loop. The second example above could be written:

# cleartool checkout -nc `cat my_list_of_files`

It's good to know both ways of doing this, since the backquote method can sometimes hit shell "environment length" limits that force you to split up what has to be done or to do parts individually.

Some common useful programs for command line use.

grep search for an expression or expressions

# gcc x.c | grep ': error'
# grep -n mystring `lsbranch -file` > o ; vim -q o
# grep -e something -e somethingelse
# grep -v '^ *$'                            # all non blank lines.

tr translate characters (one to another, to uppercase, ...)

# echo $PATH | tr : '\n'              # print out path elements on separate lines.
# tr '[a-z]' '[A-Z]'                  # upper case something.

cut Extract parts of a line

# cut -f1 -d:            # extract everything in the line(s) up until the first colon char.
# cut -f3-4              # extract from positions 3 and 4.

sort sort stuff

# sort                 # plain old sort.              
# sort -u              # unique lines only
# sort -n              # numeric sort
# sort -t :            # sort using alternate field separator (default is whitespace).

xargs Run a command on all the items in standard input.

# find . -type f -maxdepth 2 | xargs ls -l

sed search and replace program

# sed 's/string1/string2/g'
# sed 's/#.*//'                                          # remove script "comments"
# sed 's!//.*!!'                                           # remove C++ comments.
# sed 's/[ \t]*$//'                            # strip trailing whitespace
# sed 's/\(.*\):\(.*\)/\2:\1/'              # swap stuff separated by colons.

perl Any of the above and pretty much anything else you can think of.

Explaining perl is a whole different game, and if you don't know it, it probably won't look much different than ASCII barf (much like some of the sed commands above).

Some examples (things done above with a mix of other commands) :

# g++ x.c | perl -ne 'print if /: error/'
# perl -pe 's/string1/string2/g'
# perl -e ' $p = $ENV{PATH}; $p =~ s/:/\n/g ; print "$p\n" '
# perl -pe '$_ = uc($_)'

What's notable here is not the perl itself, but the fact that to run some of these commands required passing a pile of shell special characters. In order to pass these all to perl unaltered, it was required to use single quotes, and not double quotes.

Common to grep, sed, and perl is a concept called a regular expression (or regex). This is an extremely valuable thing to get to know well if you do any programming, since there's often a lot of text manipulation required as a programmer. Going into detail on this topic will require it's own time.

Shell Aliases

These are one liner "shortcuts". ksh/bash example:

alias la='ls -a'

Shell Functions

Multiline shortcuts. ksh/bash example:

function foo
   echo boo
   echo foo

This is similar to putting the commands in their own file and running that as a script, and can be used as helper functions in other scripts or as more complex "alias"es.

The example above could be written as:

alias foo='echo boo ; echo foo'

But functions also allow you do pass arguments. Example:

function debugattach {
    $1 ~/sqllib/adm/db2sysc --pid $(db2pd -edus -dbp $2 | perl -ne 'next unless s/db2sysc PID: // ; print')
alias ddda='debugattach ddd'
alias gdba='debugattach gdb'

calling this with ddda 0 will attach the ddd debugger to the db2sysc process db2pd reports to be node 0.

Except for the perl fragment, which is basically a combined 'grep' and 'sed', this example uses many of the things that have been explained above (variables, embedded command, single quotes to avoid expansion and for grouping arguments).


One Response to “Shell tips and tricks. The most basic stuff.”

  1. […] xargs: 369     xargs for loop    122     xargs loop    89     bash for loop xargs    40     for loop xargs    36     cleartool xargs    30     xargs cleartool    18     bash loop xargs    13     xargs in for loop    11     xargs and for loop    10 […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: