Coding Bash vs. Perl vs. Python (by Evan Liu)

This website shows how to accomplish the same tasks using different scripting languages. I did it while learning bash, perl and python in 2013. My facebook account: liuenmao@hotmail.com.
The website is based on Python and Django.

Topics about regex

Regex: Greedy matching: *, +, ? 

In Bash, egrep,awk and sed can be used for regular expression matching, all of them support [[:digit:]], [[:space:]], \s, \t.
However only egrep supports \d, like in perl

In python, regular expression operations are handled by module re. below is a simple summary.
re.split()  --> splits to array with pattern as delimeter
re.match()  --> matchs the start of the string,and return match object
re.search()  --> matchs anywhere in the string, only finds the first occurrence,  return the match object
re.findall()  --> returns a list of all non-overlapping matches in the string


Tasks:
1.Define a variable holding a multiline String
2.OR condition example in regex
3.AND condition example in regex
4.NOT condition example in regex
Bash Perl Python 
  1. #1.Define a variable holding a multiline String

  2. read -r -d '' poem <<EOD

  3. Lily-like, white as snow,

  4. she hardly knew

  5. she was a woman, so

  6. sweetly she grew.

  7. EOD

  8. 

  9. #2.OR condition example in regex

  10. #  use egrep, awk, sed respectively

  11. echo "$poem" |egrep '^Lily|woman'

  12. echo "$poem" |awk '/^Lily|woman/'

  13. echo "$poem" |sed -n '/^Lily\|woman/p'

  14. echo "$poem" |sed -e '/^Lily/b' -e '/woman/b' -e d

  15. 

  16. #3.AND condition example in regex

  17. #  use egrep, awk, sed respectively

  18. echo "$poem" |egrep '^she.*woman'

  19. echo "$poem" |awk '/^she.*woman/'

  20. echo "$poem" |awk '/^she/ && /woman/'

  21. echo "$poem" |sed '/^she/!d; /woman/!d'

  22. 

  23. #4.NOT condition example in regex

  24. #  use egrep, awk, sed respectively

  25. echo "$poem" |grep -v 'she'

  26. echo "$poem" |awk '!/she/'

  27. echo "$poem" |sed -n '/she/!p'

  28. echo "$poem" |sed '/she/d'

  29. 

  30. -------output----------

  1. #1.Define a variable holding a multiline String

  2. 

  3. #2.OR condition example in regex

  4. #  use egrep, awk, sed respectively

  5. Lily-like, white as snow,

  6. she was a woman, so

  7. Lily-like, white as snow,

  8. she was a woman, so

  9. Lily-like, white as snow,

  10. she was a woman, so

  11. Lily-like, white as snow,

  12. she was a woman, so

  13. 

  14. #3.AND condition example in regex

  15. #  use egrep, awk, sed respectively

  16. she was a woman, so

  17. she was a woman, so

  18. she was a woman, so

  19. she was a woman, so

  20. 

  21. #4.NOT condition example in regex

  22. #  use egrep, awk, sed respectively

  23. Lily-like, white as snow,

  24. Lily-like, white as snow,

  25. Lily-like, white as snow,

  26. Lily-like, white as snow,

  1. #1.Define a variable holding a multiline String

  2. my $poem = <<"EOD";

  3. Lily-like, white as snow,

  4. she hardly knew

  5. she was a woman, so

  6. sweetly she grew.

  7. EOD

  8. my @poem  = split(/\n/,$poem);

  9. 

  10. #2.OR condition example in regex

  11. @result = grep { $_ =~ /^Lily|woman/ } @poem;

  12. print "$_\n" foreach (@result);

  13. 

  14. #3.AND condition example in regex

  15. @result = grep { $_ =~ /^she.*woman/ } @poem;

  16. print "$_\n" foreach (@result);

  17. 

  18. #4.NOT condition example in regex

  19. @result = grep { !/she/ } @poem;

  20. print "$_\n" foreach (@result);

  21. @result = grep ( !/she/ , @poem );

  22. print "$_\n" foreach (@result);

  23. foreach (@poem) { print "$_\n" unless /she/;}

  24. 

  25. -------output----------

  1. #1.Define a variable holding a multiline String

  2. 

  3. #2.OR condition example in regex

  4. Lily-like, white as snow,

  5. she was a woman, so

  6. 

  7. #3.AND condition example in regex

  8. she was a woman, so

  9. 

  10. #4.NOT condition example in regex

  11. Lily-like, white as snow,

  12. Lily-like, white as snow,

  13. Lily-like, white as snow,

  1. import re

  2. #1.Define a variable holding a multiline String

  3. poem='''Lily-like, white as snow,

  4. she hardly knew

  5. she was a woman, so

  6. sweetly she grew.'''

  7. 

  8. #2.OR condition example in regex

  9. print re.findall(r'^Lily.*|.*woman.*', poem, flags=re.M)

  10. 

  11. #3.AND condition example in regex

  12. print re.findall(r'^she.*woman.*', poem, flags=re.M)

  13. 

  14. #4.NOT condition example in regex

  15. print [x for x in poem.split('\n') if not 'she' in x]

  16. 

  17. 

  18. -------output----------

  1. #1.Define a variable holding a multiline String 

  2. 

  3. #2.OR condition example in regex 

  4. ['Lily-like, white as snow,', 'she was a woman, so']

  5. 

  6. #3.AND condition example in regex 

  7. ['she was a woman, so']

  8. 

  9. #4.NOT condition example in regex 

  10. ['Lily-like, white as snow,']

Regex: Lazy matching: *?, +?, ?? 

As a general principle in regex, ?, * , + and {n,m} will match as much of the string as possible while still allowing the whole regexp to match.

In bash, 'grep -P'will make grep use perl syntax 

In python
The '*', '+', and '?' qualifiers are all greedy; they match as much text as possible
The '*?', '+?', and '??' make it match, as few characters as possible, aka lazy mode or minimal mode.

Tasks:
1.Define a variable for matching
2.Perform a greedy match
3.Perform a non-greedy match
Bash Perl Python 
  1. #1.Define a variable for matching

  2. h="<html><head>Neutrino</head></html>"

  3. 

  4. #2.Perform a greedy match

  5. egrep -o  "<h.*>" <<<"$h"

  6. #  or use Perl syntax with '-P'

  7. grep -Po  "<h.*>" <<<"$h"

  8. 

  9. #3.Perform a non-greedy match

  10. grep -Po  "<h.*?>" <<<"$h"

  11. 

  12. -------output----------

  1. #1.Define a variable for matching

  2. 

  3. #2.Perform a greedy match

  4. <html><head>Neutrino</head></html>

  5. #  or use Perl syntax with '-P'

  6. <html><head>Neutrino</head></html>

  7. 

  8. #3.Perform a non-greedy match

  9. <html>

  10. <head>

  1. #1.Define a variable for matching

  2. $h="<html><head>Neutrino</head></html>";

  3. 

  4. #2.Perform a greedy match

  5. $h=~m/(<h.*>)/; print "$1\n";

  6. 

  7. #3.Perform a non-greedy match

  8. $h=~m/(<h.*?>)/; print "$1\n";

  9. 

  10. -------output----------

  1. #1.Define a variable for matching

  2. 

  3. #2.Perform a greedy match

  4. <html><head>Neutrino</head></html>

  5. 

  6. #3.Perform a non-greedy match

  7. <html>

  1. import re

  2. #1.Define a variable for matching

  3. h="<html><head>Neutrino</head></html>"

  4. 

  5. #2.Perform a greedy match

  6. print re.search(r'<h.*>',h).group()

  7. 

  8. #3.Perform a non-greedy match

  9. print re.search(r'<h.*?>',h).group()

  10. 

  11. -------output----------

  1. #1.Define a variable for matching 

  2. 

  3. #2.Perform a greedy match 

  4. <html><head>Neutrino</head></html>

  5. 

  6. #3.Perform a non-greedy match 

  7. <html>

Regex: Substitution 


		  
		  

		  
Tasks:
1.Define a variable holding a multiline String
2.replace 'she' with 'She'
		  

		       
          
          

             
             

		     

                Bash
                Perl
                Python			   
		     

             
              

                
  1. #1.Define a variable holding a multiline String

  2. poem=$(cat <<EOD

  3. Lily-like, white as snow,

  4. she hardly knew

  5. she was a woman, so

  6. sweetly she grew.

  7. EOD

  8. )

  9. 

  10. #2.replace 'she' with 'She'

  11. echo "$poem" |sed 's/^she/She/g'

  12. #  or

  13. echo "${poem//she/She}"

  14. 

  15. -------output----------



  1. #1.Define a variable holding a multiline String

  2. 

  3. #2.replace 'she' with 'She'

  4. Lily-like, white as snow,

  5. She hardly knew

  6. She was a woman, so

  7. sweetly she grew.

  8. #  or

  9. Lily-like, white as snow,

  10. She hardly knew

  11. She was a woman, so

  12. sweetly She grew.



                
  1. #1.Define a variable holding a multiline String

  2. my $poem = <<"EOD";

  3. Lily-like, white as snow,

  4. she hardly knew

  5. she was a woman, so

  6. sweetly she grew.

  7. EOD

  8. 

  9. #2.replace 'she' with 'She'

  10. my @poem  = split(/\n/,$poem);

  11. foreach (@poem){

  12.  s/^she/She/;

  13. }

  14. print "$_\n" foreach (@poem);

  15. 

  16. -------output----------



  1. #1.Define a variable holding a multiline String

  2. 

  3. #2.replace 'she' with 'She'

  4. Lily-like, white as snow,

  5. She hardly knew

  6. She was a woman, so

  7. sweetly she grew.



                
  1. #1.Define a variable holding a multiline String

  2. import re

  3. poem='''Lily-like, white as snow,

  4. she hardly knew

  5. she was a woman, so

  6. sweetly she grew.'''

  7. 

  8. #2.replace 'she' with 'She'

  9. print re.sub(r'^she', r'She', poem, flags=re.M)

  10. 

  11. -------output----------



  1. #1.Define a variable holding a multiline String 

  2. 

  3. #2.replace 'she' with 'She' 

  4. Lily-like, white as snow,

  5. She hardly knew

  6. She was a woman, so

  7. sweetly she grew.



              

             
            
 
          

          

		  

        
          

		  
Regex: Parentheses capture: () 

		  
In Bash, command sed supports capture variables \1, \2, \3 by using parentheses
In Perl, capture variables $1,$2, $3 are used to retrieve values in the parentheses
In Python, one additional feature for capturing values are named group: the format for named regular expression group is "(?P<group_name>regexp)", P means placeholder

		  
		  

		  
Tasks:
1.Define a string for capturing
2.Capture values by parentheses
		  

		       
          
          

             
             

		     

                Bash
                Perl
                Python			   
		     

             
              

                
  1. #1.Define a string for capturing

  2. line1="Alpha Centauri-4.3650-1689"

  3. 

  4. #2.Capture values by parentheses

  5. #  use sed

  6. sed -r 's/(.*)-(.*)-(.*)/star->\1 distance->\2 year->\3/' <<<$line1

  7. #  or use awk

  8. awk -F'-' '{print "star->"$1" distance->"$2" year->"$3 }' <<<$line1

  9. 

  10. -------output----------



  1. #1.Define a string for capturing

  2. 

  3. #2.Capture values by parentheses

  4. #  use sed

  5. star->Alpha Centauri distance->4.3650 year->1689

  6. #  or use awk

  7. star->Alpha Centauri distance->4.3650 year->1689



                
  1. #1.Define a string for capturing

  2. $line1="Alpha Centauri-4.3650-1689";

  3. 

  4. #2.Capture values by parentheses

  5. $line1 =~ /(.*)-(.*)-(.*)/;

  6. print "star->$1; distance->$2; year->$3\n";

  7. ($star, $distance, $year) = $line1 =~ /(.*)-(.*)-(.*)/;

  8. print "start->$star; distance->$distance; year->$year";

  9. 

  10. 

  11. -------output----------



  1. #1.Define a string for capturing

  2. 

  3. #2.Capture values by parentheses

  4. star->Alpha Centauri; distance->4.3650; year->1689

  5. start->Alpha Centauri; distance->4.3650; year->1689



                
  1. import re

  2. #1.Define a string for capturing

  3. line1="Alpha Centauri-4.3650-1689"

  4. 

  5. #2.Capture values by parentheses

  6. #  note: group(0) returns the entire match

  7. match=re.search(r'(.*)-(.*)-(.*)', line1)

  8. print "start->%s; distance->%s; year->%s" %(match.group(1), match.group(2), match.group(3))

  9. #  or use re.findall

  10. print re.findall(r'(.*)-(.*)-(.*)', line1)

  11. print re.findall(r'([^-]+?)(?=$|-)', line1)

  12. 

  13. -------output----------



  1. #1.Define a string for capturing 

  2. 

  3. #2.Capture values by parentheses 

  4. #  note: group(0) returns the entire match 

  5. start->Alpha Centauri; distance->4.3650; year->1689

  6. #  or use re.findall 

  7. [('Alpha Centauri', '4.3650', '1689')]

  8. ['Alpha Centauri', '4.3650', '1689']



              

             
            
 
          

          

		  

        
          

		  
Regex: Parentheses capture with named group: (?P<>) 

		  
In Bash, if regex match operator =~ is used, the results of the match are saved to array ${BASH_REMATCH[@]}
In Perl, the format for named capture is (?<group_name>regexp), then use the %+ hash to retrieve them.
In Python, the format for named group is "(?P<group_name>regexp)", P means placeholder

		  
		  

		  
Tasks:
1.Define a string for capturing
2.Capture values by parentheses
		  

		       
          
          

             
             

		     

                Bash
                Perl
                Python			   
		     

             
              

                
  1. #1.Define a string for capturing

  2. line1="Alpha Centauri-4.3650-1689"

  3. 

  4. #2.Capture values by parentheses

  5. [[ "$line1" =~ (.*)-(.*)-(.*) ]]

  6. echo "star->${BASH_REMATCH[1]}; distance->${BASH_REMATCH[2]}; year->${BASH_REMATCH[3]}"

  7. 

  8. 

  9. -------output----------



  1. #1.Define a string for capturing

  2. 

  3. #2.Capture values by parentheses

  4. star->Alpha Centauri; distance->4.3650; year->1689



                
  1. #1.Define a string for capturing

  2. $line1="Alpha Centauri-4.3650-1689";

  3. 

  4. #2.Capture values by parentheses

  5. $line1 =~ /(?<star>.*)-(?<distance>.*)-(?<year>.*)/;

  6. print "start->$+{star}; distance->$+{distance}; year->$+{year}";

  7. 

  8. -------output----------



  1. #1.Define a string for capturing

  2. 

  3. #2.Capture values by parentheses

  4. start->Alpha Centauri; distance->4.3650; year->1689



                
  1. import re

  2. #1.Define a string for capturing

  3. line1="Alpha Centauri-4.3650-1689"

  4. 

  5. #2.Capture values by parentheses

  6. match = re.search ('(?P<star>.*)-(?P<distance>.*)-(?P<year>.*)',line1)

  7. print "start->%s; distance->%s; year->%s" %(match.group('star'), match.group('distance'), match.group('year'))

  8. 

  9. -------output----------



  1. #1.Define a string for capturing 

  2. 

  3. #2.Capture values by parentheses 

  4. start->Alpha Centauri; distance->4.3650; year->1689



              

             
            
 
          

          

		  

        
          

		  
Regex: Single line mode by mode modifier: (?s) 

		  
(?s) or DOTALL mode
makes . match all characters, including newline.
makes ^ and $ match at the start and end of the string only.
In Perl, use /s, or (?s)
In Python, use re.DOTALL or re.S, or(?s)

		  
		  

		  
Tasks:
1.Define a multiline string
2.Match all before substirng 'two'by DOT MATCH ALL mode
		  

		       
          
          

             
             

		     

                Bash
                Perl
                Python			   
		     

             
              

                
  1. #1.Define a multiline string

  2. read -r -d '' mstr <<'EOD';

  3. one

  4. two

  5. three

  6. EOD

  7. 

  8. #2.Match all before substirng 'two'by DOT MATCH ALL mode

  9. m="$(grep -Pzo '(?s)^.*two'<<<"$mstr")"

  10. echo -n "->$m<-"

  11. 

  12. -------output----------



  1. #1.Define a multiline string

  2. 

  3. #2.Match all before substirng 'two'by DOT MATCH ALL mode

  4. ->one

  5. two<-



                
  1. #1.Define a multiline string

  2. $mstr = <<'EOD';

  3. one

  4. two

  5. three

  6. EOD

  7. 

  8. #2.Match all before substirng 'two'by DOT MATCH ALL mode

  9. $mstr =~ /^(.*two)/s;

  10. print "->$1<-\n";

  11. #  or

  12. $mstr =~ /(?s)^(.*two)/;

  13. print "->$1<-\n";

  14. 

  15. 

  16. -------output----------



  1. #1.Define a multiline string

  2. 

  3. #2.Match all before substirng 'two'by DOT MATCH ALL mode

  4. ->one

  5. two<-

  6. #  or

  7. ->one

  8. two<-



                
  1. #1.Define a multiline string

  2. import re

  3. mstr='''one

  4. two

  5. three'''

  6. 

  7. #2.Match all before substirng 'two'by DOT MATCH ALL mode

  8. print re.findall(r'(?s)^(.*two)', mstr)

  9. #  or

  10. print re.findall(r'^(.*two)', mstr, re.S)

  11. 

  12. 

  13. -------output----------



  1. #1.Define a multiline string 

  2. 

  3. #2.Match all before substirng 'two'by DOT MATCH ALL mode 

  4. ['one\ntwo']

  5. #  or 

  6. ['one\ntwo']



              

             
            
 
          

          

		  

        
          

		  
Regex: Multiline mode by Mode modifier: (?m) 

		  
(?m) or MULTILINE mode make ^ and $ match at the start and end of each line in the string

		  
		  

		  
Tasks:
1.Define a multiline string
2.Match all before substirng 'two'by MULTILINE mode
		  

		       
          
          

             
             

		     

                Bash
                Perl
                Python			   
		     

             
              

                
  1. #1.Define a multiline string

  2. read -r -d '' mstr <<'EOD';

  3. one

  4. two

  5. three

  6. EOD

  7. 

  8. #2.Match all before substirng 'two'by MULTILINE mode

  9. m="$(grep -Pzo '(?m)^.*two'<<<"$mstr")"

  10. echo -n "->$m<-"

  11. 

  12. -------output----------



  1. #1.Define a multiline string

  2. 

  3. #2.Match all before substirng 'two'by MULTILINE mode

  4. ->two<-



                
  1. #1.Define a multiline string

  2. $mstr = <<'EOD';

  3. one

  4. two

  5. three

  6. EOD

  7. 

  8. #2.Match all before substirng 'two'by MULTILINE mode

  9. $mstr =~ /(?m)^(.*two)/;

  10. print "->$1<-\n";

  11. 

  12. 

  13. -------output----------



  1. #1.Define a multiline string

  2. 

  3. #2.Match all before substirng 'two'by MULTILINE mode

  4. ->two<-



                
  1. #1.Define a multiline string

  2. import re

  3. mstr='''one

  4. two

  5. three'''

  6. 

  7. #2.Match all before substirng 'two'by MULTILINE mode

  8. print re.findall(r'(?m)^(.*two)', mstr)

  9. #  or

  10. print re.findall(r'^(.*two)', mstr, re.M)

  11. 

  12. 

  13. -------output----------



  1. #1.Define a multiline string 

  2. 

  3. #2.Match all before substirng 'two'by MULTILINE mode 

  4. ['two']

  5. #  or 

  6. ['two']



              

             
            
 
          

          

		  

        
          

		  
Regex: Lookahead assertion: (?=...), (?!...) 

		  

		  
		  

		  
Tasks:
1.Define a string
2.Positive lookahead assertion: find all dog colors
3.Negative lookahead assertion: find all animal colors, that is not a dog
		  

		       
          
          

             
             

		     

                Bash
                Perl
                Python			   
		     

             
              

                
  1. #1.Define a string

  2. str="blackcat,yellowdog,whitecat,browndog,redrooster";

  3. 

  4. #2.Positive lookahead assertion: find all dog colors

  5. grep -Po '([a-z]+)(?=dog)' <<<"$str"

  6. 

  7. #3.Negative lookahead assertion: find all animal colors, that is not a dog

  8. grep -Po '(black|yellow|white|brown|red)(?!dog)' <<<"$str"

  9. 

  10. 

  11. -------output----------



  1. #1.Define a string

  2. 

  3. #2.Positive lookahead assertion: find all dog colors

  4. yellow

  5. brown

  6. 

  7. #3.Negative lookahead assertion: find all animal colors, that is not a dog

  8. black

  9. white

  10. red



                
  1. #1.Define a string

  2. $str="blackcat,yellowdog,whitecat,browndog,redrooster";

  3. 

  4. #2.Positive lookahead assertion: find all dog colors

  5. @a=($str =~ m/([a-z]+)(?=dog)/g); print "@a\n";

  6. 

  7. #3.Negative lookahead assertion: find all animal colors, that is not a dog

  8. @a=($str =~ m/(black|yellow|white|brown|red)(?!dog)/g); print "@a";

  9. 

  10. -------output----------



  1. #1.Define a string

  2. 

  3. #2.Positive lookahead assertion: find all dog colors

  4. yellow brown

  5. 

  6. #3.Negative lookahead assertion: find all animal colors, that is not a dog

  7. black white red



                
  1. #1.Define a string

  2. import re

  3. str="blackcat,yellowdog,whitecat,browndog,redrooster"

  4. 

  5. #2.Positive lookahead assertion: find all dog colors

  6. print re.findall('[a-z]+(?=dog)', str)

  7. 

  8. #3.Negative lookahead assertion: find all animal colors, that is not a dog

  9. print re.findall('(black|yellow|white|brown|red)(?!dog)', str)

  10. 

  11. -------output----------



  1. #1.Define a string 

  2. 

  3. #2.Positive lookahead assertion: find all dog colors 

  4. ['yellow', 'brown']

  5. 

  6. #3.Negative lookahead assertion: find all animal colors, that is not a dog 

  7. ['black', 'white', 'red']



              

             
            
 
          

          

		  

        
          

		  
Regex: Lookbehind assertion: (?<=...), (?<!...) 

		  

		  
		  

		  
Tasks:
1.Define a string
2.Positive lookbehind assertion: find all animals that is white
3.Negative lookbehind assertion: find all animals that is not white
		  

		       
          
          

             
             

		     

                Bash
                Perl
                Python			   
		     

             
              

                
  1. #1.Define a string

  2. str="blackcat,yellowdog,whitecat,browndog,redrooster";

  3. 

  4. #2.Positive lookbehind assertion: find all animals that is white

  5. grep -Po '(?<=white)([a-z]+)' <<<"$str"

  6. 

  7. #3.Negative lookbehind assertion: find all animals that is not white

  8. grep -Po '(?<!white)(cat|dog|rooster)' <<<"$str"

  9. 

  10. 

  11. -------output----------



  1. #1.Define a string

  2. 

  3. #2.Positive lookbehind assertion: find all animals that is white

  4. cat

  5. 

  6. #3.Negative lookbehind assertion: find all animals that is not white

  7. cat

  8. dog

  9. dog

  10. rooster



                
  1. #1.Define a string

  2. $str="blackcat,yellowdog,whitecat,browndog,redrooster";

  3. 

  4. #2.Positive lookbehind assertion: find all animals that is white

  5. @a=($str =~ m/(?<=white)([a-z]+)/g); print "@a\n";

  6. 

  7. #3.Negative lookbehind assertion: find all animals that is not white

  8. @a=($str =~ m/(?<!white)(cat|dog|rooster)/g); print "@a\n";

  9. 

  10. 

  11. -------output----------



  1. #1.Define a string

  2. 

  3. #2.Positive lookbehind assertion: find all animals that is white

  4. cat

  5. 

  6. #3.Negative lookbehind assertion: find all animals that is not white

  7. cat dog dog rooster



                
  1. #1.Define a string

  2. import re

  3. str="blackcat,yellowdog,whitecat,browndog,redrooster"

  4. 

  5. #2.Positive lookbehind assertion: find all animals that is white

  6. print re.findall('(?<=white)[a-z]+',str)

  7. 

  8. #3.Negative lookbehind assertion: find all animals that is not white

  9. print re.findall('(?<!white)(cat|dog|rooster)',str)

  10. 

  11. -------output----------



  1. #1.Define a string 

  2. 

  3. #2.Positive lookbehind assertion: find all animals that is white 

  4. ['cat']

  5. 

  6. #3.Negative lookbehind assertion: find all animals that is not white 

  7. ['cat', 'dog', 'dog', 'rooster']