Things you (probably) didn’t know about xargs
If you have spent any amount of
time at a UNIX command line, you have probably already seen
xargs. In case
you have not, xargs is a command
used to execute commands based on arguments from standard input.
Common use cases
I often see xargs
used in combination with
find in order to do something with the list of files
returned by find.
Contrived
examples warning: I needed something simple examples that would
not detract from the topic. This is the best I could do given the time I had.
Recursively find all Python files and count
the number of lines
# find . -name '*.py' | xargs wc –l
|
Recursively find all Emacs backup files and remove them
# find . -name '*~' | xargs rm
|
Recursively find all Python files and search them for the word ‘import’
# find . -name '*.py' | xargs grep ‘import’
|
Handling files or
folders with spaces in the name
One
problem with the above examples is that it does not correctly handle files or
directories with a space in the name. This is because xargs by default will split on any white-space character. A quick
solution to this is to tell find to delimit results with NUL (\0)
characters (by supplying -print0 to
find), and to tell xargs to split the input on NUL characters as well
(-0).
Remove backup files recursively even if they
contain spaces
# find . -name '*~' –print0 | xargs -0 rmg
|
Placement of the
arguments
In
the examples above, xargs reads all non-white-space elements from standard
input and concatenates them into the given command line before executing it.
This alone is very useful in many circumstances. Sometimes however you might
want to insert the arguments into the middle of a command. The -I flag to xargs
takes a string that will be replaced with the supplied input before the command
is executed. A common choice is %.
Move all backup files somewhere else
# find . -name '*~' –print0 | xargs -0 –I % cp % ~/backups
|
Maximum command length
Sometimes
the list of arguments piped to xargs would cause the resulting command line to
exceed the maximum length allowed by the system. You can find this limit with
getconf ARG_MAX
In
order to avoid hitting the system limit, xargs has its own limit to the maximum
length of the resulting command. If the supplied arguments would cause the
invoked command to exceed this built in limit, xargs will split the input and
invoke the command repeatedly. This limit defaults to 4096, which can be
significantly lower than ARG_MAX on modern systems. You can override xargs’s
limit with the -s flag. This will
be particularly important when you are dealing with a large source tree.
Operating on subset of
arguments at a time
You
might be dealing with commands that can only accept 1 or maybe 2 arguments at a
time. For example, the diff command operates on two files at a time. The -n flag to xargs
specifies how many arguments at a time to supply to the given command. The
command will be invoked repeatedly until all input is exhausted. Note that on
the last invocation you might get less than the desired number of arguments if
there is insufficient input. Let’s simply use xargs to break up the input into
2 arguments per line
$
echo {0..9} | xargs -n 2
0
1
2
3
4
5
6
7
8
9
In
addition to running based on a specified number of arguments at time you can
also invoke a command for each line of input at a time with -L 1. You can of course
use an arbitrary number of lines a time, but 1 is most common. Here is how you
might diff every git commit against its parent.
# git log --format="%H %P" | xargs -L 1 git diff
|
Executing commands in
parallel
You
might be using xargs to invoke a compute intensive command for every line of
input. Wouldn’t it be nice if xargs allowed you to take advantage of the
multiple cores in your machine? That is what -P is
for. It allows xargs to invoke the specified command multiple times in
parallel. You might use this for example to run multiple ffmpeg encodes
in parallel. However, I am just going to show you yet another contrived
example.
Parallel sleep
$
time echo {1..5} | xargs -n 1 -P 5 sleep
real 0m5.013s
user 0m0.003s
sys 0m0.014s
Sequential sleep
$
time echo {1..5} | xargs -n 1 sleep
real 0m15.022s
user 0m0.004s
sys 0m0.015s
Examples
Example 1: Copy large number of files to another folder.
Sometimes we required to copy a long list of files,
In that case cp command failed with error
“Argument list too long”. We can use xargs to do that task.
# find /home/linuxman/public_html/tecadmin.net/ -type f | xargs
-n1 -i cp {} /var/www/backup/
|
Example 2: Delete multiple files from a folder.
Sometimes we required to delete a large number of
files from a folder. Below example will delete all .log files from /var/log
directory.
# find /var/www/tmp/ -type f | xargs rm –f
|
Above command will fail to
remove files with spaces in named. Say File name is “Indrajit Bhagat”, to
handle spaces in xargs command try below command.
# find /var/www/tmp/ -type f -print0 | xargs -0 rm -f
|
Example 3: Count number of lines in multiple files.
Below example will count number of lines for each
.txt file in /opt directory and its subdirectory
# find /opt -name "*.txt" | xargs wc –l
|
To handle files having spaces in their name, use
following command.
# find /opt/ -name "*.log" -print0 | xargs -0 wc -l
|
Example 4: Make a backup of all configuration files.
If you want to make a backup of all configuration
files (extension .conf) in your system, use below command.
# find / -name "*.conf" | xargs tar czf config.tar.gz
# ls -l config.tar.gz
-rw-r--r--. 1 root root 193310 Apr 1 13:26 config.tar.gz
|
Example 5: Use custom delimiter with xargs.
We can have also use custom delimiter with xargs command; by default,
it uses space and new line as delimiter. Use -d parameter to define delimiter.
# echo "1,2,3,4,5" | xargs -d, echo
|
Output:
1 2 3 4 5
|
Example
6: Show output in separate line with xargs.
In
example 5 output is showing in single line, we can also specify to show each
output in separate line.
# echo "1,2,3,4,5" | xargs -d, -L 1 echo
|
Output:
1
2
3
4
5
|
Example 7: Handling blank space in filenames or path.
To handle spaces in names use -print0 with find command and -0 with xargs command as parameter.
# find /tmp -print0 |
xargs -0 -L 1 echo
|
No comments:
Post a Comment