LINUX

Combine multiple files into one using Bash shell script

---

Background

It is common to need to output its own result to file for each processor in parallel computing environment. Each file has a specified name along with processor id. This method can avoid a lot of communications between the processors and highly encouraged if possible. For time-denpent problems, we could have to output at every time step. A lot of files with similar name but different time step number could be generated. We can use Matlab to read these files and combine them together but not efficient especialy with tons of files. Today I will show you how to combine these files using simple Bash shell scripts under Linux Environment.

---

Pre-check

First check the type of shell the operating system is using by typing "echo $shell" in the terminal.

---

The Problem

Suppose I have ton of files in the pattern listed below and want to combine each row into one file:

A1_1.txt,   A1_2.txt,   A1_3.txt   ⇒ A1.txt

A2_1.txt,   A2_2.txt,   A2_3.txt   ⇒ A2.txt

......

......

An_1.txt,   An_2.txt,   An_3.txt   ⇒ An.txt

---

The Shell Script

The shell script to do this job is pasted below.

#!/bin/bash

header="displacement"
extension=".txt"
underscore="_"
newdir="../combinedresults/"   ## save the combined files to a new folder

for i in {1..5}
do
	filename=$header$i$extension   ##generate a new file name to store the files
	touch $filename               ## create the new file
		for j in {0..3}
		do 
			FILENAME=$header$i$underscore$j$extension   ## generate the existing file names
			cat $FILENAME  >> $newdir$filename          ## concatenate the existing file to the new file
		done
	echo "Finishing $i files"
done

combining_multiple_files.sh