CS/CE 218 Lecture -*-Outline-*-

note: I added more examples of regular expressions that weren't covered in
class; these would have really helped...

connection: still continuing our discussion of manipulating text
(e.g. programs) under program control;
we've seen the shape of a standard Unix comamnd,
and how to hook commands together into pipelines;
we've also seen how to control commands and pipelines that take a long time
to finish.
We saw that the most important command for pipelining was a filter.
Today we'll look at filters that work on (ascii) text files;
these are all ways to "observe" a file.

* text-manipulation

advert: allow to to treat your program source files as a database,
	to make major changes quickly, to get "into" the files
	and look at them in unplanned ways

	examples of sort, wc, grep, tr, sed, cut, paste, diff, ...

** wc (word count) section 6.4
	counts lines, words, and characters in a document
	Q: How would you tell how many lines are in gigantic.c using wc?
		wc -l gigantic.c
		or wc -l <gigantic.c
			latter useful for getting a number all by itself

	Q: What does ``wc -l <big.c | wc -c'' do?
		should give the order of magnitude,
		but it always counts the padding blanks

** character by character manipulation
*** tr (translates characters)
	copies stdin to stdout,
		translating on a per character basis as directed

	Q: How (using tr) would you translate all blanks to newlines?
		tr ' ' '\012'
		this is important, as we'll see later that newlines are
			used by almost all the other tools.
		better: tr -s ' ' | tr ' ' '\012'

	Q: Is there any way to use tr to capitalize all the words in a
		document?  I.e., to change only the First Chars To Upper Case?
		no, tr is not context sensitive (mostly)

	also can't change every other newline to a blank (joining lines)

	Q: What does ``wc -l <big.c | tr -d ' \012' | wc -c'' do?
		gives order of magnitude
		tr -d deletes all of the blanks and newlines

*** cmp (compares files) section 6.13
	tells which characters differ
	used mainly for it's return code
		returns 0 (ok) if same
			1 if differ
			2 for errors
		can be tested by shell scripts...

** line by line manipulation
	most programs that manipulate text in Unix use line boundaries
		delimited by newlines (ascii 12)

*** grep (global regular expression print) section 6.5
	copies lines matching a RE to standard output
	
	---------------------
	grep 'lost_function (' *.c
	grep '#define [A-Z]X.*' *.c
	---------------------

	Q: What does the -v option do for grep?
		prints non-matching lines (complements)
	Q: What does the -l option do?

**** regular expressions (pages 163-168)
	grep, sed, some parts of emacs all work with regular expressions
		patterns that match text
		not exactly shell wild cards.

	for details see regexp(5)

***** matching characters
	Q: Can you fill in the following table
		shell wild card	| equivalent regular expression
			?	|
			*	|
			[a-z]	|
			[!a-z]	|				?

	-------------------
	$ echo "" | grep .
	$ echo "a" | grep .
	a
	$ echo "abcd" | grep .
	abcd
	$ echo "abcd" | grep z
	$ echo "abcd" | grep bc
	abcd
	$ echo "abcd" | grep '.*'
	abcd
	$ echo "abcd" | grep 'a*'
	abcd
	$ echo "abcd" | grep 'aa*'
	abcd
	$ echo "abcd" | grep 'aaa*'
	$ echo "abcd" | grep 'xy.*'
	$ echo "abcd" | grep 'bc.*'
	abcd
	$ echo "abcd" | grep '[d-f]'
	abcd
	$ echo "abcd" | grep '[f-z]'
	$ echo "abcd" | grep '[^a-d]'
	$ echo "abcd" | grep '[a-d]'
	abcd
	$ echo "abcd" | grep 'a[bc]d'
	$ echo "abcd" | grep 'a[bc].d'
	abcd
	$ echo "abcd" | grep 'a[bc].*d'
	abcd
	$ echo "abcd" | grep 'a[bc]c.*d'
	abcd
	-------------------
	escaping the special meaning of a metacharacter is done with backslash
		to include a ] in a character set: []a-d]
		trick: [---xyz] same as [-xyz]

	Q: What does the regular expression [1-30] mean?
		same as [1230]

	-------------
	$ echo 9 | grep '[0-9]'
	9
	$ echo 9 | grep '[1-44]'
	$ echo 9 | grep '[1234]'
	-------------
***** Anchoring	an expression
	usually a pattern is matched anywhere on a line
	------------
	^ = beginning of line
	$ = end of line
	------------
		$ is the only thing that almost matches a newline
		don't confuse two uses of ^

	---------------
	$ echo "abcd" | grep '^abcd$'
	abcd
	$ echo "abcd" | grep '^b'
	$ echo "abcd" | grep '^a'
	abcd
	$ echo "abcd" | grep '^[b-d]'
	$ echo "abcd" | grep 'd$'
	abcd
	$ echo "abcd" | grep '^[b-d].*[a-c]$'
	$ echo "abcd" | grep '^.*$'
	abcd
	$ echo "abcd" | grep '^bb*'
	$ echo "x abcd abcd" | grep '^abcd*'
	$ echo "x abcd abcd" | grep 'abcd*$'
	x abcd abcd
	$ echo "x abcd abcd" | grep 'x[^xyz]*d*$'
	x abcd abcd
	---------------
***** Closures (repeating patterns)
	----------
	* = {0,}
	+ = {1,}
	? = {0,1}
	----------
		+, ? and the ones on the right hand side don't work with grep
			or sed, grep doesn't tell you!
		also + and ? often don't work...

	Q: What lines of "file" are printed by "grep a* file"?
		all of them!

	.* matches
		""
		"abc"
	a* matches
		""
		"aa"
		but not "ab" (it only matches the "a" from "ab", not the whole)

	match is as long as possible (important for sed)

*** sed (section 6.8, also page 283-284)
	can not only find stuff, but can change it
		this is perhaps the command I use most in shell programming
		(grep for sed in ~leavens/bin/*)

	------------
	$ echo "abcd" | sed -e 's/ab.d/q/'
	q
	$ echo "abcd" | sed -e 's/c\(.\)/x\1/'
	abxcd
	$ echo "abcd abcd" | sed -e 's/c\(.\)/x\1/'
	abxcd abcd
	$ echo "abcd abcd" | sed -e 's/\(ab\)cd/\1/'
	ab abcd
	$ echo "abcd abcd" | sed -e 's/^\(..\)cd/\1/'
	ab abcd
	$ echo "abcd abcd" | sed -e 's/\(..\)cd$/\1/'
	abcd ab
	$ echo "abcd wxyz" | sed -e 's/^\(..\)cd  *\(.*\)$/\2, \1/'
	wxyz, ab
	------------

	an example from my unit-files program (simplified)
	----------------------
	sed -n -e 's!^\*\** \([^A-Z].*\)!\1.txt!p' \
		Outline-files.txt
	----------------------
		-n only prints lines explicitly asked for
		matches lines beginning with a *, no caps
			matched stuff following the blank is output
				followed by ".txt"

		changes
			** file
		to
			file.txt
			
		sed doesn't understand \*+ as the same as \*\**
			(neither does grep)
*** manipulation of tables (relations)
	can mimic database relational operators with these commands
	some commands specialized to edit fields of a file
		separated by tabs (or blanks) not by position usually

	for example a relation showing on what line a identifier appears
		in a program...

Q: Can you give the output of the following examples, assuming the following is
	the data in the file id-index (there are tabs, not blanks)?

-------------------
ch	5
ch	3
xyz	4
i	5
i	4
-------------------

**** cut (project) section 6.10 and uniq (page 157)
	leaves out some fields
-------------
$ cut -f1 id-index | uniq
-------------
ch
xyz
i

**** grep (forms of restriction and division) and sort (section 6.3)
-------------
$ grep ch id-index |sort -u
-------------
ch      3
ch      5

-------------
$ grep '	5' id-index | sort -u
-------------
ch      5
i       5

**** paste and pr -t -m (very limited form of composition) section 6.10
	Q: After executing "cut -f1 id-index >names"
	   and "cut -f2 id-index >lines", how do you use paste to put
		id-index back together?  How do use use pr to do it?

		paste names lines
		pr -t -m -w10 names lines

*** diff (comparing files) section 6.12
	diff finds the smallest set of differences between 2 files
	Q: If you have two files, f.c and f.c~, what does "echo f.c*" print?
			f.c, then f.c~
		so diff f.c* gives f.c as 1st arg, f.c~ (old) as 2nd.
	Q: Does diff check to see which file is older?
		contrary to book no, just uses order.

		1st arg	| 2nd arg
		<	| >
		
------------------
$ diff .profile*
14a15,18
> ENV="$HOME/.kshrc"
> . "$ENV"
> export ENV
> 
29,36c33
< # set path variables
< . "$HOME/.path.sh"
< 
< ENV="$HOME/.kshrc"
< . "$ENV"
< export ENV
< 
< . "$HOME/.startup.tty"
---
> . $HOME/.startup.tty
--------------------
	first arg is .profile~ (the old file)
		so I moved the ENV lines from after . $HOME/.path.sh
		to before that (after line 14 in the old file)

*** sdiff and comm
	sdiff would be useful if displays were wider;
		shows differences side by side,
		useful for merging variants
	comm is the opposite of diff in a sense,
		but also shows differences
		output may be confusing if files have initial tabs