Unix Shell Text Substitution: sed(1)

1. Remove texts:

test:

 #! /bin/bash  

 Hello Hello Hello  
 <world>  
 <world world  

terminal:
First command: 's/<.*//': first 's' is the command meaning substitution( I guess :)).
It firstly pick up all patterns starting with "<", followed by any number of any characters, then replace this pattern with the new one specified at the second place, which is empty here. "/" is the separator in this case. But we still see a lot of empty lines in the output, which is very annoying, we can use grep to filter out it.
Second command: use grep to only pickup the "non-empty" lines. (^$ means empty line), -v means compliment. 

 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed 's/<.*//' ./test  
 #! /bin/bash
  
 Hello Hello Hello  


 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed 's/<.*//' ./test | grep -v "^$"  
 #! /bin/bash  
 Hello Hello Hello  

2. Combine with other commands to create folders

Assume we only have one xxdev folder under Desktop.
terminal:
 1) First command, is trying to find the directory named xxdev, and the ouptut is just "xxdev" since it is at local place.
2) Second command, use pipe to get the result of previous "find" command, replace dev with "fastdev", so we have one new folder name xxfastdev
3) Third command,use pipe to get the result of previous two commands, replace the beginning of text with string "mkdir ", which means insert the mkdir to the beginning
4) Fourth command, use pipe to get the result of previous three commands, and use "sh -x" to run that text as command, -x means turning on the execution trace, so you will see the trace for "mkdir xxfastdev" running.

 aubinxia@aubinxia-VirtualBox:~/Desktop$ find xxdev -type d  
 xxdev  
 aubinxia@aubinxia-VirtualBox:~/Desktop$ find xxdev -type d | sed "s/dev/fastdev/"  
 xxfastdev  
 aubinxia@aubinxia-VirtualBox:~/Desktop$ find xxdev -type d | sed "s/dev/fastdev/" | sed "s/^/mkdir /"
 mkdir xxfastdev  
 aubinxia@aubinxia-VirtualBox:~/Desktop$ find xxdev -type d | sed "s/dev/fastdev/" | sed "s/^/mkdir /" | sh -x  
 + mkdir xxfastdev  
 aubinxia@aubinxia-VirtualBox:~/Desktop$ ls  
 xxdev xxfastdev  

Note:
Combining regular expression with file/directories management in unix is kind of very dangerous, sometimes, any mistake will cause large amounts of files/directories to be created or deleted. We should be highly cautious about this.

3. Use Different Delimiter
1) First command is trying to use ";" as the new delimiter
2) Second command is still using the "/" as the delimiter, for the same character in the patter, we are using "\" to escape it.
terminal:

 aubinxia@aubinxia-VirtualBox:~/Desktop$ find xxdev -type d -print | sed 's;xxdev/xxdev;xxdev/fastdev;'  
 xxdev  
 xxdev/fastdev  
 aubinxia@aubinxia-VirtualBox:~/Desktop$ find xxdev -type d -print | sed 's/xxdev\/xxdev/xxdev\/fastdev/'  
 xxdev  
 xxdev/fastdev  

4. Use Backreferences
sed is based on basic regular expression, so it recognize backreferences

terminal:
It is using "\1" represents the first occurance of the pattern "xxdev"

 aubinxia@aubinxia-VirtualBox:~/Desktop$ find xxdev -type d -print | sed 's;xxdev/\(xxdev\);fastdev/\1;'  
 xxdev  
 fastdev/xxdev  

5. Use "&"
test:

 #! /bin/bash  

 New York
 Hello Hello  

terminal:
1) First command is trying to catch the "New York", and & represent this pattern, right after this pattern, we add ", Financial Capital of US"
2) Second command is trying to catch "Hello" followed with 0 ore more spaces, we aim to find a pattern returning above thing twice, and add " world!" right after it.

 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed 's/New York/&, Financial Capital of US/' ./test  
 #! /bin/bash  
 New York, Financial Capital of US  
 Hello Hello  
 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed 's/\(Hello[[:space:]]*\)\{2\}/& world!/' ./test  
 #! /bin/bash  
 New York  
 Hello Hello world!  

6. Substitute each occurance
test file is same as above:

terminal:
 1) The first command only replace the first occurance of Hello
 2) The second command add 'g' at the end of expression, it replaced all occurances of "Hello"

 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed 's/Hello/world/' ./test  
 #! /bin/bash  
 New York  
 world Hello  
 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed 's/Hello/world/g' ./test  
 #! /bin/bash  
 New York  
 world world  

7. Substitue specific occurnace
test file is same as above

terminal:
1) First command replaced the first occurance of Hello
2) Second command replaced the second occurance of Hello

 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed 's/Hello/world/1' ./test  
 #! /bin/bash  
 New York  
 world Hello  
 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed 's/Hello/world/2' ./test  
 #! /bin/bash  
 New York  
 Hello world  

8. Use multiple commands:
test file is same as above
terminal:
We are using two commands, each staring with option -e. For tool "sed", it will just retrieve each line from the input, and tried to apply each command on the line, output the result, then retrieve the next line.

 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed -e 's/Hello/world/2' -e 's/New York/&, the US Financial Capital/' ./test  
 #! /bin/bash  
 New York, the US Financial Capital  
 Hello world  

We can also separate different commands with semicolon:

 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed -e 's/Hello/world/2;s/New York/&, the US Financial Capital/' ./test  
 #! /bin/bash  
 New York, the US Financial Capital  
 Hello world  

9. Use "Command File"
test file is same as above

command.sed:

 s/Hello/world/2  
 s/New York/&, the US Financial Capital/  

terminal:

 aubinxia@aubinxia-VirtualBox:~/Desktop/xxdev$ sed -f ./command.sed ./test  
 #! /bin/bash  
 New York, the US Financial Capital  
 Hello world