Shell Tips: How to recursively archive many folders

In this short tutorial, I’ll show you how to recursively archive (tar.bz2) folders into archives file with same folder names. I’ll use find to list all directories; then archive each folder using tar and parallel.

Scenario

Let’s assume that we have following directories:

drwxr-xr-x 37 macbookair staff 1258 Nov 22 18:16 folder_number_one
-rw-r--r-- 1 macbookair staff 35342662 Nov 22 18:16 foo_file.txt
drwxr-xr-x 9 macbookair staff 306 Nov 22 18:16 folder_number_two
-rw-r--r-- 1 macbookair staff 16505 Nov 22 18:16 foo_bar.txt
drwxr-xr-x 7 macbookair staff 238 Nov 22 18:16 folder_number_three
drwxr-xr-x 7 macbookair staff 238 Nov 22 18:17 folder with space

Objective

The objective is to recursively archive folder_number_one, folder_number_two, folder_number_three into folder_number_one.tar.bz2, folder_number_two.tar.bz2, folder_number_three.tar.bz2 respectively.

Solution

One magic command does ALL:

$ find ./* -type d -print | parallel tar cjvf {}.tar.bz2 {}

How it works?

$ find ./* -type d -print
./folder with space
./folder_number_one
./folder_number_three
./folder_number_two

Next, I used parallel to parse in the list to tar:

find ./* -type d -print | parallel tar cjvf {}.tar.bz2 {}

{} represents every single folder name in the list. You could customize the archive name easily by amending to this {} symbol. For example:

find ./* -type d -print | parallel tar cjvf {}-archived.tar.bz2 {}

which append -achived into the filename to give you:

folder_number_one-archived.tar.bz2
...

Why this solution is better than using ls and awk?

There are many solutions, one is to use ls -l to list all directories then pipe the output to awk to grep the last field start with d (that is directory type). Yet that solution suffers one flaw: directory names’spaces will be omitted in the output.

Advertisements

About Jones Lee

Nothing much about me..

2 responses to “Shell Tips: How to recursively archive many folders

  1. If you have dirs that have space in their name use :

    find . -type d -maxdepth 1 | parallel tar cjvf {}-archived.tar.bz2 {}

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: