Shell Tips: How to recursively archive many folders

In this short tutorial, I’ll show you how to recursively archive (tar.bz2) folders into archives file with same folder names. I’ll use find to list all directories; then archive each folder using tar and parallel.

Scenario

Let’s assume that we have following directories:

drwxr-xr-x 37 macbookair staff 1258 Nov 22 18:16 folder_number_one
-rw-r--r-- 1 macbookair staff 35342662 Nov 22 18:16 foo_file.txt
drwxr-xr-x 9 macbookair staff 306 Nov 22 18:16 folder_number_two
-rw-r--r-- 1 macbookair staff 16505 Nov 22 18:16 foo_bar.txt
drwxr-xr-x 7 macbookair staff 238 Nov 22 18:16 folder_number_three
drwxr-xr-x 7 macbookair staff 238 Nov 22 18:17 folder with space

Objective

The objective is to recursively archive folder_number_one, folder_number_two, folder_number_three into folder_number_one.tar.bz2, folder_number_two.tar.bz2, folder_number_three.tar.bz2 respectively.

Solution

One magic command does ALL:

$ find ./* -type d -print | parallel tar cjvf {}.tar.bz2 {}

How it works?

$ find ./* -type d -print
./folder with space
./folder_number_one
./folder_number_three
./folder_number_two

Next, I used parallel to parse in the list to tar:

find ./* -type d -print | parallel tar cjvf {}.tar.bz2 {}

{} represents every single folder name in the list. You could customize the archive name easily by amending to this {} symbol. For example:

find ./* -type d -print | parallel tar cjvf {}-archived.tar.bz2 {}

which append -achived into the filename to give you:

folder_number_one-archived.tar.bz2
...

Why this solution is better than using ls and awk?

There are many solutions, one is to use ls -l to list all directories then pipe the output to awk to grep the last field start with d (that is directory type). Yet that solution suffers one flaw: directory names’spaces will be omitted in the output.

About Jones Lee

Nothing much about me..

View all posts by Jones Lee →