One of Mill's core abstractions is it's *Task Graph*: this is how Mill defines,
orders and caches work it needs to do, and exists independently of any support
for building Scala.
The following is a simple self-contained example using Mill to compile Java:
```scala
import ammonite.ops._, mill._
// sourceRoot -> allSources -> classFiles
// |
// v
// resourceRoot ----> jar
def sourceRoot = T.sources{ pwd / 'src }
def resourceRoot = T.sources{ pwd / 'resources }
def allSources = T{ sourceRoot().flatMap(p => ls.rec(p.path)).map(PathRef(_)) }
def classFiles = T{
mkdir(T.ctx().dest)
import ammonite.ops._
%("javac", sources().map(_.path.toString()), "-d", T.ctx().dest)(wd = T.ctx().dest)
PathRef(T.ctx().dest)
}
def jar = T{ Jvm.createJar(Loose.Agg(classFiles().path) ++ resourceRoot().map(_.path)) }
def run(mainClsName: String) = T.command{
%%('java, "-cp", classFiles().path, mainClsName)
}
```
Here, we have two `T.source`s, `sourceRoot` and `resourceRoot`, which act as the
roots of our task graph. `allSources` depends on `sourceRoot` by calling
`sourceRoot()` to extract it's value, `classFiles` depends on `allSources` the
same way, and `jar` depends on both `classFiles` and `resourceRoot`.
Filesystem o1perations in Mill are done using the
[Ammonite-Ops](http://ammonite.io/#Ammonite-Ops) library.
The above build defines the following task graph:
```
sourceRoot -> allSources -> classFiles
|
v
resourceRoot ----> jar
```
When you first evaluate `jar` (e.g. via `mill jar` at the command line), it will
evaluate all the defined targets: `sourceRoot`, `allSources`, `classFiles`,
`resourceRoot` and `jar`.
Subsequent `mill jars` will evaluate only as much as is necessary, depending on
what input sources changed:
- If the files in `sourceRoot` change, it will re-evaluate `allSources`,
compiling to `classFiles`, and building the `jar`
- If the files in `resourceRoot` change, it will only re-evaluate `jar` and use
the cached output of `allSources` and `classFiles`
Apart from the `foo()` call-sites which define what each targets depend on, the
code within each `T{...}` wrapper is arbirary Scala code that can compute an
arbitrary result from it's inputs.
## Different Kinds of Tasks
There are four primary kinds of *Tasks* that you should care about:
- [Targets](#targets), defined using `T{...}`
- [Sources](#sources), defined using `T.source{...}`
- [Commands](#commands), defined using `T.command{...}`
### Targets
```scala
def allSources = T{ ls.rec(sourceRoot().path).map(PathRef(_)) }
```
`Target`s are defined using the `def foo = T{...}` syntax, and dependencies on
other targets are defined using `foo()` to extract the value from them. Apart
from the `foo()` calls, the `T{...}` block contains arbitrary code that does
some work and returns a result.
Each target e.g. `classFiles` is assigned a path on disk as scratch space & to
store it's output files at `out/classFiles/dest/`, and it's returned metadata is
automatically JSON-serialized and stored at `out/classFiles/meta.json`. The
return-value of targets has to be JSON-serializable via
[uPickle](https://github.com/lihaoyi/upickle).
If you want to return a file or a set of files as the result of a `Target`,
write them to disk within your `T.ctx().dest` available through the
[Task Context API](#task-context-api) and return a `PathRef` to the files you
wrote.
If a target's inputs change but it's output does not, e.g. someone changes a
comment within the source files that doesn't affect the classfiles, then
downstream targets do not re-evaluate. This is determined using the `.hashCode`
of the Target's return value. For target's returning `ammonite.ops.Path`s that
reference files on disk, you can wrap the `Path` in a `PathRef` (shown above)
whose `.hashCode()` will include the hashes of all files on disk at time of
creation.
The graph of inter-dependent targets is evaluated in topological order; that
means that the body of a target will not even begin to evaluate if one of it's
upstream dependencies has failed. This is unlike normal Scala functions: a plain
old function `foo` would evaluate halfway and then blow up if one of `foo`'s
dependencies throws an exception.
Targets cannot take parameters and must be 0-argument `def`s defined directly
within a `Module` body
### Sources
```scala
def sourceRootPath = pwd / 'src
def sourceRoots = T.sources{ sourceRootPath }
```
`Source`s are defined using `T.source{ ... }`, taking one-or-more
`ammonite.ops.Path`s as arguments. A `Source` is a subclass of
`Target[Seq[PathRef]]`: this means that it's build signature/`hashCode` depends
not just on the path it refers to (e.g. `foo/bar/baz`) but also the MD5 hash of
the filesystem tree under that path.
`T.source` also has an overload which takes `Seq[PathRef]`, to let you
override-and-extend source lists the same way you would any other `T{...}`
definition:
```scala
def additionalSources = T.sources{ pwd / 'additionalSources }
def sourceRoots = T.sources{ super.sourceRoots() ++ additionalSources() }
```
### Commands
```scala
def run(mainClsName: String) = T.command{
%%('java, "-cp", classFiles().path, mainClsName)
}
```
Defined using `T.command{ ... }` syntax, `Command`s can run arbitrary code, with
dependencies declared using the same `foo()` syntax (e.g. `classFiles()` above).
Commands can be parametrized, but their output is not cached, so they will
re-evaluate every time even if none of their inputs have changed.
Like [Targets](#targets), a command only evaluates after all it's upstream
dependencies have completed, and will not begin to run if any upstream
dependency has failed.
Commands are assigned the same scratch/output directory `out/run/dest/` as
Targets are, and it's returned metadata stored at the same `out/run/meta.json`
path for consumption by external tools.
Commands can only be defined directly within a `Module` body.
## Task Context API
There are several APIs available to you within the body of a `T{...}` or
`T.command{...}` block to help your write the code implementing your Target or
Command:
### mill.util.Ctx.DestCtx
- `T.ctx().dest`
- `implicitly[mill.util.Ctx.DestCtx]`
This is the unique `out/classFiles/dest/` path or `out/run/dest/` path that is
assigned to every Target or Command. It is cleared before your task runs, and
you can use it as a scratch space for temporary files or a place to put returned
artifacts. This is guaranteed to be unique for every `Target` or `Command`, so
you can be sure that you will not collide or interfere with anyone else writing
to those same paths.
### mill.util.Ctx.LogCtx
- `T.ctx().log`
- `implicitly[mill.util.Ctx.LogCtx]`
This is the default logger provided for every task. While your task is running,
`System.out` and `System.in` are also redirected to this logger. The logs for a
task are streamed to standard out/error as you would expect, but each task's
specific output is also streamed to a log file on disk e.g. `out/run/log` or
`out/classFiles/log` for you to inspect later.
## Other Tasks
- [Anonymous Tasks](#anonymous-tasks), defined using `T.task{...}`
- [Persistent Targets](#persistent-targets)
- [Inputs](#inputs)
- [Workers](#workers)
### Anonymous Tasks
```scala
def foo(x: Int) = T.task{ ... x ... bar() ... }
```
You can define anonymous tasks using the `T.task{ ... }` syntax. These are not
runnable from the command-line, but can be used to share common code you find
yourself repeating in `Target`s and `Command`s.
```scala
def downstreamTarget = T{ ... foo() ... }
def downstreamCommand = T.command{ ... foo() ... }
```
Anonymous tasks's output does not need to be JSON-serializable, their output is
not cached, and they can be defined with or without arguments. Unlike
[Targets](#targets) or [Commands](#commands), anonymous tasks can be defined
anywhere and passed around any way you want, until you finally make use of them
within a downstream target or command.
While an anonymous task `foo`'s own output is not cached, if it is used in a
downstream target `bar` and the upstream targets's `baz` `qux` haven't changed,
`bar`'s cached output will be used and `foo`'s evaluation will be skipped
altogether.
### Persistent Targets
```scala
def foo = T.persistent{ ... }
```
Identical to [Targets](#targets), except that the `dest/` directory is not
cleared in between runs.
This is useful if you are running external incremental-compilers, such as
Scala's [Zinc](https://github.com/sbt/zinc), Javascript's
[WebPack](https://webpack.js.org/), which rely on filesystem caches to speed up
incremental execution of their particular build step.
Since Mill no longer forces a "clean slate" re-evaluation of `T.persistent`
targets, it is up to you to ensure your code (or the third-party incremental
compilers you rely on!) are deterministic. They should always converge to the
same outputs for a given set of inputs, regardless of what builds and what
filesystem states existed before.
### Inputs
```scala
def foo = T.input{ ... }
```
A generalization of [Sources](#sources), `T.input`s are tasks that re-evaluate
*every time* (Unlike [Anonymous Tasks](#anonymous-tasks)), containing an
arbitrary block of code.
Inputs can be used to force re-evaluation of some external property that may
affect your build. For example, if I have a [Target](#targets) `bar` that makes
use of the current git version:
```scala
def bar = T{ ... %%("git", "rev-parse", "HEAD").out.string ... }
```
`bar` will not know that `git rev-parse` can change, and will
not know to re-evaluate when your `git rev-parse HEAD` *does* change. This means
`bar` will continue to use any previously cached value, and `bar`'s output will
be out of date!
To fix this, you can wrap your `git rev-parse HEAD` in a `T.input`:
```scala
def foo = T.input{ %%("git", "rev-parse", "HEAD").out.string }
def bar = T{ ... foo() ... }
```
This makes `foo` will always re-evaluate every build; if `git rev-parse HEAD`
does not change, that will not invalidate `bar`'s caches. But if `git rev-parse
HEAD` *does* change, `foo`'s output will change and `bar` will be correctly
invalidated and re-compute using the new version of `foo`.
Note that because `T.input`s re-evaluate every time, you should ensure that the
code you put in `T.input` runs quickly. Ideally it should just be a simple check
"did anything change?" and any heavy-lifting can be delegated to downstream
targets.
### Workers
```scala
def foo = T.worker{ ... }
```
Most tasks dispose of their in-memory return-value every evaluation; in the case
of [Targets](#targets), this is stored on disk and loaded next time if
necessary, while [Commands](#commands) just re-compute them each time. Even if
you use `--watch` or the Build REPL to keep the Mill process running, all this
state is still discarded and re-built every evaluation.
Workers are unique in that they store their in-memory return-value between
evaluations. This makes them useful for storing in-memory caches or references
to long-lived external worker processes that you can re-use.
Mill uses workers to managed long-lived instances of the
[Zinc Incremental Scala Compiler](https://github.com/sbt/zinc) and the
[Scala.js Optimizer](https://github.com/scala-js/scala-js). This lets us keep
them in-memory with warm caches and fast incremental execution.
Like [Persistent Targets](#persistent-targets), Workers inherently involve
mutable state, and it is up to the implementation to ensure that this mutable
state is only used for caching/performance and does not affect the
externally-visible behavior of the worker.
## Cheat Sheet
The following table might help you make sense of the small collection of
different Task types:
| | Target | Command | Source/Input | Anonymous Task | Persistent Target | Worker |
|:-------------------------------|:-------|:--------|:-------------|:---------------|:------------------|:-------|
| Cached on Disk | X | X | | | X | |
| Must be JSON Writable | X | X | | | X | |
| Must be JSON Readable | X | | | | X | |
| Runnable from the Command Line | X | X | | | X | |
| Can Take Arguments | | X | | X | | |
| Cached between Evaluations | | | | | | X |
