Two new command line programs in ~/tools: scmp ...

Two new command line programs in ~/tools: scmp and tokens.

1) scmp: will need a bit more fleshing out to be super useful, but here
is what you can do right now:

  // This means run the given command line first with the options
  // given to p1 and then without, and show the diff in output.
  % tools/scmp --p1 '-no-specialization -nowarn' scalac -Ydebug
  src/library/scala/Function1.scala

Upcoming features will involve seeing diffs of such things as the
pickled signatures of generated files and the javap disassembly.

2) tokens: tokenizes all the scala files found under any given paths and
prints one token per line.

Example: the five most frequently used tokens under scala/util.

  % tools/tokens src/library/scala/util |sort | uniq -c | sort -r | head
  -5 598 ')' 598 '(' 347 ; 294 '=' 278 ,

Good to see those parens are balanced.

Example: number of appearances of an identifier called x:

  % tools/tokens src/library/scala/util | grep ^x$ | wc 137

Way to go, x.  Review by community.

2 files changed, 8 insertions, 0 deletions

diff --git a/tools/scmp b/tools/scmp
new file mode 100755
index 0000000000..f6acea5ab1
--- /dev/null
+++ b/tools/scmp
@@ -0,0 +1,4 @@
+#!/bin/sh
+#
+
+scala scala.tools.cmd.program.Scmp "$@"
diff --git a/tools/tokens b/tools/tokens
new file mode 100755
index 0000000000..b910fb29cc
--- /dev/null
+++ b/tools/tokens
@@ -0,0 +1,4 @@
+#!/bin/sh
+#
+
+scala scala.tools.cmd.program.Tokens "$@"