Top bash Questions

List of Tags
702
Jiaaro

How do I get the path of the directory in which a bash script is located FROM that bash script.

For instance, lets say I want to use a bash script as a launcher for another application. I want to change working directory to the one where the bash script is located so I can operate on the files in that directory like so:

$ ./application
Answered By: dogbane ( 838)
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

Is a useful one-liner which will give you the full directory name of the script no matter where it is being called from

These will work as long as the last component of the path used to find the script is not a symlink (directory links are OK). If you want to also resolve any links to the script itself, you need a multi-line solution:

SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
  DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
  SOURCE="$(readlink "$SOURCE")"
  [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
done
DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"

This last one will work with any combination of aliases, source, bash -c, symlinks, etc.

Beware: if you cd to a different directory before running this snippet, the result may be incorrect!

To understand how it works, try running this more verbose form:

#!/bin/bash

SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
  TARGET="$(readlink "$SOURCE")"
  if [[ $SOURCE == /* ]]; then
    echo "SOURCE '$SOURCE' is an absolute symlink to '$TARGET'"
    SOURCE="$TARGET"
  else
    DIR="$( dirname "$SOURCE" )"
    echo "SOURCE '$SOURCE' is a relative symlink to '$TARGET' (relative to '$DIR')"
    SOURCE="$DIR/$TARGET" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
  fi
done
echo "SOURCE is '$SOURCE'"
RDIR="$( dirname "$SOURCE" )"
DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
if [ "$DIR" != "$RDIR" ]; then
  echo "DIR '$RDIR' resolves to '$DIR'"
fi
echo "DIR is '$DIR'"

And it will print something like:

SOURCE './scriptdir.sh' is a relative symlink to 'sym2/scriptdir.sh' (relative to '.')
SOURCE is './sym2/scriptdir.sh'
DIR './sym2' resolves to '/home/ubuntu/dotfiles/fo fo/real/real1/real2'
DIR is '/home/ubuntu/dotfiles/fo fo/real/real1/real2'
260
Bill the Lizard

I've used the following script to see if a file exists:

#!/bin/bash
FILE=$1

if [ -f $FILE ];
then
   echo "File $FILE exists."
else
   echo "File $FILE does not exist."
fi

What's the correct syntax to use if I only want to check if the file does not exist?

#!/bin/bash
FILE=$1

if [ $FILE does not exist ];
then
   echo "File $FILE does not exist."
fi
Answered By: John Feminella ( 422)

Bash has a "not" logical operator, which is the exclamation point (similar to many other languages). Try this:

if [ ! -f /tmp/foo.txt ];
then
    echo "File not found!"
fi

What is the proper way to modify environment variables like PATH in OS X? I've looked on google a little bit and found 3 different files to edit:

  • /etc/paths
  • ~/.profile
  • ~/.tcshrc

I don't even have some of these, and I'm pretty sure that .tcshrc is wrong, since osx uses bash now. Anybody have any idea where these variables, especially PATH, are defined?

Edit: I'm running OS X 10.5

Answered By: Matthew McCullough ( 229)

Bruno is right on track. I've done extensive research and if you want to set variables that are available in all GUI apps, your only option is /etc/launchd.conf

Please note that environment.plist does not work for applications launched via Spotlight. This is documented by Steve Sexton here.

1) Open a terminal prompt

2) Type sudo vi /etc/launchd.conf (note: this file might not yet exist)

3) Put contents like the following into the file

# Set environment variables here so they are available globally to all apps
# (and Terminal), including those launched via Spotlight.
#
# After editing this file run the following command from the terminal to update 
# environment variables globally without needing to reboot.
# NOTE: You will still need to restart the relevant application (including 
# Terminal) to pick up the changes!
# grep -E "^setenv" /etc/launchd.conf | xargs -t -L 1 launchctl
#
# See http://www.digitaledgesw.com/node/31
# and http://stackoverflow.com/questions/135688/setting-environment-variables-in-os-x/
#
# Note that you must hardcode the paths below, don't use enviroment variables.
# You also need to surround multiple values in quotes, see MAVEN_OPTS example below.
#
setenv JAVA_VERSION 1.6
setenv JAVA_HOME /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home
setenv GROOVY_HOME /Applications/Dev/groovy
setenv GRAILS_HOME /Applications/Dev/grails
setenv NEXUS_HOME /Applications/Dev/nexus/nexus-webapp
setenv JRUBY_HOME /Applications/Dev/jruby

setenv ANT_HOME /Applications/Dev/apache-ant
setenv ANT_OPTS -Xmx512M

setenv MAVEN_OPTS "-Xmx1024M -XX:MaxPermSize=512m"
setenv M2_HOME /Applications/Dev/apache-maven

setenv JMETER_HOME /Applications/Dev/jakarta-jmeter

4) Save your changes in VI and reboot your Mac. Or use the grep/xargs command show in the code comment above.

5) Prove that your variables are working by opening a Terminal window and typing export and you should see your new variables. These will also be available in IntelliJ and other GUI apps you launch via Spotlight.

257
Tristan Havelick

In a unix shell, if I want to combine stderr and stdout into the stdout stream for further manipulation, I can append the following on the end of my command:

2>&1

So, if I want to use "head" on the output from g++, I can do something like this:

g++ lots_of_errors 2>&1 | head

so I can see only the first few errors.

I always have trouble remembering this, and I constantly have to go look it up, and it is mainly because I don't fully understand the syntax of this particular trick. Can someone break this up and explain character by character what "2>&1" means?

Answered By: Ayman Hourieh ( 289)

1 is stdout. 2 is stderr.

Here is one way to remember this construct (altough it is not entirely accurate): at first, 2>1 may look like a good way to redirect stderr to stdout. However, it will actually be interpreted as "redirect stderr to a file named 1". & indicates that what follows is a file descriptor and not a filename. So the construct becomes: 2>&1.

247
stefanB

How do I split a string based on a delimiter in Bash?

I have this string stored in a variable:

IN="bla@some.com;john@home.com"

Now I would like to split the strings by ';' delimiter so that I have

ADDR1="bla@some.com"
ADDR2="john@home.com"

I don't necessarily need the ADDR1 and ADDR2 variables. If they are elements of an array that's even better.

Edit: After suggestions from the answers below, I ended up with the following which is what I was after:

#!/usr/bin/env bash

IN="bla@some.com;john@home.com"

arr=$(echo $IN | tr ";" "\n")

for x in $arr
do
    echo "> [$x]"
done

output:

> [bla@some.com]
> [john@home.com]

Edit2: There was a solution involving setting Internal_field_separator (IFS) to ';'. I am not sure what happened with that answer, how do you reset IFS back to default?

Edit3: RE: IFS solution, I tried this and it works, I keep the old IFS and then restore it:

IN="bla@some.com;john@home.com"

OIFS=$IFS
IFS=';'
arr2=$IN
for x in $arr2
do
    echo "> [$x]"
done

IFS=$OIFS

BTW, when I tried

arr2=($IN) 

I only got the first string when printing it in loop, without brackets around $IN it works.

Answered By: Johannes Schaub - litb ( 128)

You can set the internal field separator (IFS) variable, and then let it parse into an array. When this happens in a command, then the assignment to IFS only takes place to that single command's environment (to read ). It then parses the input according to the IFS variable value into an array, which we can then iterate over.

IFS=';' read -ra ADDR <<< "$IN"
for i in "${ADDR[@]}"; do
    # process "$i"
done

It will parse one line of items separated by ;, pushing it into an array. Stuff for processing whole of $IN, each time one line of input separated by ;:

 while IFS=';' read -ra ADDR; do
      for i in "${ADDR[@]}"; do
          # process "$i"
      done
 done <<< "$IN"
242
ibz

I want to get the filename (without extension) and the extension separately.

The best solution I found so far is:

NAME=`echo "$FILE" | cut -d'.' -f1`
EXTENSION=`echo "$FILE" | cut -d'.' -f2`

This is wrong because it doesn't work if the file name contains multiple "." characters. If let's say I have a.b.js it will consider a and b.js, instead of a.b and js.

It can be easily done in Python with

file, ext = os.path.splitext(path)

but I'd prefer not to fire a Python interpreter just for this, if possible.

Any better ideas?

Answered By: Petesh ( 452)

First, get file without path:

filename=$(basename "$fullfile")
extension="${filename##*.}"
filename="${filename%.*}"
215
Muhammad Alkarouri

What are the differences between shell languages like bash, zsh, fish and the scripting languages above that makes them more suitable for the shell?

When using the command line the shell languages seem to be much easier. It feels for me much smoother to use bash for example than to use the shell profile in ipython, despite reports to the contrary. I think most wil agree with me that a large portion of medium to large scale programming is easier in Python than in bash. I use Python as the language I am most familiar with, the same goes for Perl and Ruby.

I have tried to articulate the reason but am unable to, aside from assuming that the treatment of strings differently in both has something to do with it.

The reason of this question is that I am hoping to develop a language usable in both. If you know of such a language, please post it as well.

As S.Lott explains, the question needs some clarification. I am asking about the features of the shell language versus that of scripting languages. So the comparison is not about the characteristics of various interactive (REPL) environments such as history and command line substitution. An alternative expression for the question would be:

Can a programming language that is suitable for design of complex systems be at the same time able to express useful one-liners that can access the file system or control jobs? Can a programming language usefully scale up as well as scale down?

Answered By: J&#246;rg W Mittag ( 334)

There are a couple of differences that I can think of; just thoughtstreaming here, in no particular order:

1. Python & Co. are designed to be good at scripting. Bash & Co. are designed to be only good at scripting, with absolutely no compromise. IOW: Python is designed to be good both at scripting and non-scripting, Bash cares only about scripting.

2. Bash & Co. are untyped, Python & Co. are strongly typed, which means that the number 123, the string 123 and the file 123 are quite different. They are, however, not statically typed, which means they need to have different literals for those, in order to keep them apart. Example:

  • Ruby: 123 (number), Bash: 123
  • Ruby: '123' (string), Bash: 123
  • Ruby: /123/ (regexp), Bash: 123
  • Ruby: File.open('123') (file), Bash: 123
  • Ruby: IO.open('123') (file descriptor), Bash: 123
  • Ruby: URI.parse('123') (URI), Bash: 123
  • Ruby: `123` (command), Bash: 123

3. Python & Co. are designed to scale up to 10000, 100000, maybe even 1000000 line programs, Bash & Co. are designed to scale down to 10 character programs.

4. In Bash & Co., files, directories, file descriptors, processes are all first-class objects, in Python, only Python objects are first-class, if you want to manipulate files, directories etc., you have to wrap them in a Python object first.

5. Shell programming is basically dataflow programming. Nobody realizes that, not even the people who write shells, but it turns out that shells are quite good at that, and general-purpose languages not so much. In the general-purpose programming world, dataflow seems to be mostly viewed as a concurrency model, not so much as a programming paradigm.

I have the feeling that trying to address these points by bolting features or DSLs onto a general-purpose programming language doesn't work. At least, I have yet to see a convincing implementation of it. There is RuSH (Ruby shell), which tries to implement a shell in Ruby, there is rush, which is an internal DSL for shell programming in Ruby, there is Hotwire, which is a Python shell, but IMO none of those come even close to competing with Bash, Zsh, fish and friends.

Actually, IMHO, the best current shell is Microsoft PowerShell, which is very surprising considering that for several decades now, Microsoft has continually had the worst shells evar. I mean, COMMAND.COM? Really? (Unfortunately, they still have a crappy terminal. It's still the "command prompt" that has been around since, what? Windows 3.0?)

PowerShell was basically created by ignoring everything Microsoft has ever done (COMMAND.COM, CMD.EXE, VBScript, JScript) and instead starting from the Unix shell, then removing all backwards-compatibility cruft (like backticks for command substitution) and massaging it a bit to make it more Windows-friendly (like using the now unused backtick as an escape character instead of the backslash which is the path component separator character in Windows). After that, is when the magic happens.

They address problem 1 and 3 from above, by basically making the opposite choice compared to Python. Python cares about large programs first, scripting second. Bash cares only about scripting. PowerShell cares about scripting first, large programs second. A defining moment for me was watching a video of an interview with Jeffrey Snover (PowerShell's lead designer), when the interviewer asked him how big of a program one could write with PowerShell and Snover answered without missing a beat: "80 characters." At that moment I realized that this is finally a guy at Microsoft who "gets" shell programming (probably related to the fact that PowerShell was neither developed by Microsoft's programming language group (i.e. lambda-calculus math nerds) nor the OS group (kernel nerds) but rather the server group (i.e. sysadmins who actually use shells)), and that I should probably take a serious look at PowerShell.

Number 2 is solved by having arguments be statically typed. So, you can write just 123 and PowerShell knows whether it is a string or a number or a file, because the cmdlet (which is what shell commands are called in PowerShell) declares the types of its arguments to the shell. This has pretty deep ramifications: unlike Unix, where each command is responsible for parsing its own arguments (the shell basically passes the arguments as an array of strings), argument parsing in PowerShell is done by the shell. The cmdlets specify all their options and flags and arguments, as well as their types and names and documentation(!) to the shell, which then can perform argument parsing, tab completion, IntelliSense, inline documentation popups etc. in one centralized place. (This is not revolutionary, and the PowerShell designers acknowledge shells like the DIGITAL Command Language (DCL) and the IBM OS/400 Command Language (CL) as prior art. For anyone who has ever used an AS/400, this should sound familiar. In OS/400, you can write a shell command and if you don't know the syntax of certain arguments, you can simply leave them out and hit F4, which will bring a menu (similar to an HTML form) with labelled fields, dropdown, help texts etc. This is only possible because the OS knows about all the possible arguments and their types.) In the Unix shell, this information is often duplicated three times: in the argument parsing code in the command itself, in the bash-completion script for tab-completion and in the manpage.

Number 4 is solved by the fact that PowerShell operates on strongly typed objects, which includes stuff like files, processes, folders and so on.

Number 5 is particularly interesting, because PowerShell is the only shell I know of, where the people who wrote it were actually aware of the fact that shells are essentially dataflow engines and deliberately implemented it as a dataflow engine.

Another nice thing about PowerShell are the naming conventions: all cmdlets are named Action-Object and moreover, there are also standardized names for specific actions and specific objects. (Again, this should sound familar to OS/400 users.) For example, everything which is related to receiving some information is called Get-Foo. And everything operating on (sub-)objects is called Bar-ChildItem. So, the equivalent to ls is Get-ChildItem (although PowerShell also provides builtin aliases ls and dir – in fact, whenever it makes sense, they provide both Unix and CMD.EXE aliases as well as abbreviations (gci in this case)).

But the killer feature IMO is the strongly typed object pipelines. While PowerShell is derived from the Unix shell, there is one very important distinction: in Unix, all communication (both via pipes and redirections as well as via command arguments) is done with untyped, unstructured, ASCII strings. In PowerShell, it's all strongly typed, structured objects. This is so incredibly powerful that I seriously wonder why noone else has thought of it. (Well, they have, but they never became popular.) In my shell scripts, I estimate that up to one third of the commands is only there to act as an adapter between two other commands that don't agree on a common textual format. Many of those adapters go away in PowerShell, because the cmdlets exchange structured objects instead of unstructured text. And if you look inside the commands, then they pretty much consist of three stages: parse the textual input into an internal object representation, manipulate the objects, convert them back into text. Again, the first and third stage basically go away, because the data already comes in as objects.

However, the designers have taken great care to preserve the dynamicity and flexibility of shell scripting through what they call an Adaptive Type System.

Anyway, I don't want to turn this into a PowerShell commercial. There are plenty of things that are not so great about PowerShell, although most of those have to do either with Windows or with the specific implementation, and not so much with the concepts. (E.g. the fact that it is implemented in .NET means that the very first time you start up the shell can take up to several seconds if the .NET framework is not already in the filesystem cache due to some other application that needs it. Considering that you often use the shell for well under a second, that is completely unacceptable.)

The most important point I want to make is that if you want to look at existing work in scripting languages and shells, you shouldn't stop at Unix and the Ruby/Python/Perl/PHP family. For example, Tcl was already mentioned. Rexx would be another scripting language. Emacs Lisp would be yet another. And in the shell realm there are some of the already mentioned mainframe/midrange shells such as the OS/400 command line and DCL. Also, Plan9's rc.

204
davidsheldon

Using bash, I have a string:

string=`echo My string`

How can I test if it contains another string?

if [ $string ?? 'foo' ] then;
  echo "It's there!";
fi;

Where ?? is my unknown operator. Do I use echo and grep?

if [ `echo $string || grep 'foo' ` ] then;
  echo "It's there!";
fi;

That looks a bit clumsy.

Answered By: Adam Bellaire ( 290)

You can use Marcus's answer (* wildcards) outside a case statement, too, if you use double brackets:

string='My string';

if [[ "$string" == *My* ]]
then
  echo "It's there!";
fi

needle='y s'
if [[ "$string" == *"$needle"* ]]; then
  echo "haystack '$string' contains needle '$needle'"
fi

... always quote strings to allow for whitespace

179
CodingWithoutComments

How do I call console/bash commands from inside of a Ruby Program? Also, how do I get output from these commands back into my program?

Answered By: Steve Willard ( 242)

This explanation is based on this commented Ruby script from a friend of mine. If you want to improve the script, feel free to update it at the link.

Ways to execute a shell script

cmd = "echo 'hi'" # Sample string that can be used

1. Kernel#`, commonly called backticks – `cmd`

This is like many other languages, including Bash, PHP, and Perl

Returns the result of the shell command

Docs: http://ruby-doc.org/core/Kernel.html#method-i-60

value = `echo 'hi'`
value = `#{cmd}`

2. Built-in syntax, %x( cmd )

Following the x character is a delimiter, which can be any character. If the delimiter is one of the characters (, [, {, or <, the literal consists of the characters up to the matching closing delimiter, taking account of nested delimiter pairs. For all other delimiters, the literal comprises the characters up to the next occurrence of the delimiter character. String interpolation #{ ... } is allowed.

Returns the result of the shell command, just like the backticks

Docs: http://www.ruby-doc.org/docs/ProgrammingRuby/html/language.html

value = %x( echo 'hi' )
value = %x[ #{cmd} ]

3. Kernel#system

Executes the given command in a subshell,

Return: true if the command was found and ran successfully, false otherwise

Docs: http://ruby-doc.org/core/Kernel.html#method-i-system

wasGood = system( "echo 'hi'" )
wasGood = system( cmd )

4. Kernel#exec

Replaces the current process by running the given external command.

Return: none, the current process is replaced and never continues

Docs: http://ruby-doc.org/core/Kernel.html#method-i-exec

exec( "echo 'hi'" )
exec( cmd ) # Note: this will never be reached beacuse of the line above

Extra Advice

$?, which is the same as $CHILD_STATUS, accesses the status of the last system executed command if you use the backticks, system() or %{}. You can then access the exitstatus and pid properties

$?.exitstatus

More Reading

172
gregh

How would I validate that a program exists? which would then either return an error and exit or continue with the script.

It seems like it should be easy, but it's been stumping me.

Answered By: lhunath ( 267)

Yes; avoid which. Not only is it an external process you're launching for doing very little (meaning builtins like hash, type or command are way cheaper), you can also rely on the builtins to actually do what you want, while the effects of external commands can easily vary from system to system.

Why care?

  • Many operating systems have a which that doesn't even set an exit status, meaning the if which foo won't even work there and will always report that foo exists, even if it doesn't (note that some POSIX shells appear to do this for hash too).
  • Many operating systems make which do custom and evil stuff like change the output or even hook into the package manager.

So, don't use which. Instead use one of these:

$ command -v foo >/dev/null 2>&1 || { echo >&2 "I require foo but it's not installed.  Aborting."; exit 1; }
$ type foo >/dev/null 2>&1 || { echo >&2 "I require foo but it's not installed.  Aborting."; exit 1; }
$ hash foo 2>/dev/null || { echo >&2 "I require foo but it's not installed.  Aborting."; exit 1; }

If your hash bang is /bin/sh then you should care about what POSIX says. type and hash's exit codes aren't terribly well defined by POSIX, and hash is seen to exit successfully when the command doesn't exist (haven't seen this with type yet). command's exit status is well defined by POSIX, so that one is probably the safest to use.

If your script uses bash though, POSIX' rules don't really matter anymore and both type and hash become perfectly safe to use. type now has a -P to search just the PATH and hash has the side-effect that the command's location will be hashed (for faster lookup next time you use it), which is usually a good thing since you probably check for its existence in order to actually use it.

As a simple example, here's a function that runs gdate if it exists, otherwise date:

gnudate() {
    if hash gdate 2>/dev/null; then
        gdate "$@"
    else
        date "$@"
    fi
}