From c0c84ce814bb535fd770521dc1acf16b2c172c4e Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Thu, 15 Feb 2018 23:07:37 -0800 Subject: generate a reasonable-looking docsite using my blog code --- docs/tasks.md | 322 ---------------------------------------------------------- 1 file changed, 322 deletions(-) delete mode 100644 docs/tasks.md (limited to 'docs/tasks.md') diff --git a/docs/tasks.md b/docs/tasks.md deleted file mode 100644 index 73695bb8..00000000 --- a/docs/tasks.md +++ /dev/null @@ -1,322 +0,0 @@ -One of Mill's core abstractions is it's *Task Graph*: this is how Mill defines, -orders and caches work it needs to do, and exists independently of any support -for building Scala. - -The following is a simple self-contained example using Mill to compile Java: - -```scala -import ammonite.ops._, mill._ - -// sourceRoot -> allSources -> classFiles -// | -// v -// resourceRoot ----> jar - -def sourceRoot = T.sources{ pwd / 'src } - -def resourceRoot = T.sources{ pwd / 'resources } - -def allSources = T{ sourceRoot().flatMap(p => ls.rec(p.path)).map(PathRef(_)) } - -def classFiles = T{ - mkdir(T.ctx().dest) - import ammonite.ops._ - %("javac", sources().map(_.path.toString()), "-d", T.ctx().dest)(wd = T.ctx().dest) - PathRef(T.ctx().dest) -} - -def jar = T{ Jvm.createJar(Loose.Agg(classFiles().path) ++ resourceRoot().map(_.path)) } - -def run(mainClsName: String) = T.command{ - %%('java, "-cp", classFiles().path, mainClsName) -} -``` - -Here, we have two `T.source`s, `sourceRoot` and `resourceRoot`, which act as the -roots of our task graph. `allSources` depends on `sourceRoot` by calling -`sourceRoot()` to extract it's value, `classFiles` depends on `allSources` the -same way, and `jar` depends on both `classFiles` and `resourceRoot`. - -Filesystem o1perations in Mill are done using the -[Ammonite-Ops](http://ammonite.io/#Ammonite-Ops) library. - -The above build defines the following task graph: - -``` -sourceRoot -> allSources -> classFiles - | - v - resourceRoot ----> jar -``` - -When you first evaluate `jar` (e.g. via `mill jar` at the command line), it will -evaluate all the defined targets: `sourceRoot`, `allSources`, `classFiles`, -`resourceRoot` and `jar`. - -Subsequent `mill jars` will evaluate only as much as is necessary, depending on -what input sources changed: - -- If the files in `sourceRoot` change, it will re-evaluate `allSources`, - compiling to `classFiles`, and building the `jar` - -- If the files in `resourceRoot` change, it will only re-evaluate `jar` and use - the cached output of `allSources` and `classFiles` - -Apart from the `foo()` call-sites which define what each targets depend on, the -code within each `T{...}` wrapper is arbirary Scala code that can compute an -arbitrary result from it's inputs. - -## Different Kinds of Tasks - -There are four primary kinds of *Tasks* that you should care about: - -- [Targets](#targets), defined using `T{...}` -- [Sources](#sources), defined using `T.source{...}` -- [Commands](#commands), defined using `T.command{...}` - -### Targets - -```scala -def allSources = T{ ls.rec(sourceRoot().path).map(PathRef(_)) } -``` - -`Target`s are defined using the `def foo = T{...}` syntax, and dependencies on -other targets are defined using `foo()` to extract the value from them. Apart -from the `foo()` calls, the `T{...}` block contains arbitrary code that does -some work and returns a result. - -Each target e.g. `classFiles` is assigned a path on disk as scratch space & to -store it's output files at `out/classFiles/dest/`, and it's returned metadata is -automatically JSON-serialized and stored at `out/classFiles/meta.json`. The -return-value of targets has to be JSON-serializable via -[uPickle](https://github.com/lihaoyi/upickle). - -If you want to return a file or a set of files as the result of a `Target`, -write them to disk within your `T.ctx().dest` available through the -[Task Context API](#task-context-api) and return a `PathRef` to the files you -wrote. - -If a target's inputs change but it's output does not, e.g. someone changes a -comment within the source files that doesn't affect the classfiles, then -downstream targets do not re-evaluate. This is determined using the `.hashCode` -of the Target's return value. For target's returning `ammonite.ops.Path`s that -reference files on disk, you can wrap the `Path` in a `PathRef` (shown above) -whose `.hashCode()` will include the hashes of all files on disk at time of -creation. - -The graph of inter-dependent targets is evaluated in topological order; that -means that the body of a target will not even begin to evaluate if one of it's -upstream dependencies has failed. This is unlike normal Scala functions: a plain -old function `foo` would evaluate halfway and then blow up if one of `foo`'s -dependencies throws an exception. - -Targets cannot take parameters and must be 0-argument `def`s defined directly -within a `Module` body - -### Sources - -```scala -def sourceRootPath = pwd / 'src - -def sourceRoots = T.sources{ sourceRootPath } -``` - -`Source`s are defined using `T.source{ ... }`, taking one-or-more -`ammonite.ops.Path`s as arguments. A `Source` is a subclass of -`Target[Seq[PathRef]]`: this means that it's build signature/`hashCode` depends -not just on the path it refers to (e.g. `foo/bar/baz`) but also the MD5 hash of -the filesystem tree under that path. - -`T.source` also has an overload which takes `Seq[PathRef]`, to let you -override-and-extend source lists the same way you would any other `T{...}` -definition: - -```scala -def additionalSources = T.sources{ pwd / 'additionalSources } -def sourceRoots = T.sources{ super.sourceRoots() ++ additionalSources() } -``` - -### Commands - -```scala -def run(mainClsName: String) = T.command{ - %%('java, "-cp", classFiles().path, mainClsName) -} -``` - -Defined using `T.command{ ... }` syntax, `Command`s can run arbitrary code, with -dependencies declared using the same `foo()` syntax (e.g. `classFiles()` above). -Commands can be parametrized, but their output is not cached, so they will -re-evaluate every time even if none of their inputs have changed. - -Like [Targets](#targets), a command only evaluates after all it's upstream -dependencies have completed, and will not begin to run if any upstream -dependency has failed. - -Commands are assigned the same scratch/output directory `out/run/dest/` as -Targets are, and it's returned metadata stored at the same `out/run/meta.json` -path for consumption by external tools. - -Commands can only be defined directly within a `Module` body. - -## Task Context API - -There are several APIs available to you within the body of a `T{...}` or -`T.command{...}` block to help your write the code implementing your Target or -Command: - -### mill.util.Ctx.DefCtx - -- `T.ctx().dest` -- `implicitly[mill.util.Ctx.DefCtx]` - -This is the unique `out/classFiles/dest/` path or `out/run/dest/` path that is -assigned to every Target or Command. It is cleared before your task runs, and -you can use it as a scratch space for temporary files or a place to put returned -artifacts. This is guaranteed to be unique for every `Target` or `Command`, so -you can be sure that you will not collide or interfere with anyone else writing -to those same paths. - -### mill.util.Ctx.LogCtx - -- `T.ctx().log` -- `implicitly[mill.util.Ctx.LogCtx]` - -This is the default logger provided for every task. While your task is running, -`System.out` and `System.in` are also redirected to this logger. The logs for a -task are streamed to standard out/error as you would expect, but each task's -specific output is also streamed to a log file on disk e.g. `out/run/log` or -`out/classFiles/log` for you to inspect later. - -## Other Tasks - -- [Anonymous Tasks](#anonymous-tasks), defined using `T.task{...}` -- [Persistent Targets](#persistent-targets) -- [Inputs](#inputs) -- [Workers](#workers) - - -### Anonymous Tasks - -```scala -def foo(x: Int) = T.task{ ... x ... bar() ... } -``` - -You can define anonymous tasks using the `T.task{ ... }` syntax. These are not -runnable from the command-line, but can be used to share common code you find -yourself repeating in `Target`s and `Command`s. - -```scala -def downstreamTarget = T{ ... foo() ... } -def downstreamCommand = T.command{ ... foo() ... } -``` -Anonymous tasks's output does not need to be JSON-serializable, their output is -not cached, and they can be defined with or without arguments. Unlike -[Targets](#targets) or [Commands](#commands), anonymous tasks can be defined -anywhere and passed around any way you want, until you finally make use of them -within a downstream target or command. - -While an anonymous task `foo`'s own output is not cached, if it is used in a -downstream target `bar` and the upstream targets's `baz` `qux` haven't changed, -`bar`'s cached output will be used and `foo`'s evaluation will be skipped -altogether. - -### Persistent Targets -```scala -def foo = T.persistent{ ... } -``` - -Identical to [Targets](#targets), except that the `dest/` directory is not -cleared in between runs. - -This is useful if you are running external incremental-compilers, such as -Scala's [Zinc](https://github.com/sbt/zinc), Javascript's -[WebPack](https://webpack.js.org/), which rely on filesystem caches to speed up -incremental execution of their particular build step. - -Since Mill no longer forces a "clean slate" re-evaluation of `T.persistent` -targets, it is up to you to ensure your code (or the third-party incremental -compilers you rely on!) are deterministic. They should always converge to the -same outputs for a given set of inputs, regardless of what builds and what -filesystem states existed before. - -### Inputs - -```scala -def foo = T.input{ ... } -``` - -A generalization of [Sources](#sources), `T.input`s are tasks that re-evaluate -*every time* (Unlike [Anonymous Tasks](#anonymous-tasks)), containing an -arbitrary block of code. - -Inputs can be used to force re-evaluation of some external property that may -affect your build. For example, if I have a [Target](#targets) `bar` that makes -use of the current git version: - -```scala -def bar = T{ ... %%("git", "rev-parse", "HEAD").out.string ... } -``` - -`bar` will not know that `git rev-parse` can change, and will -not know to re-evaluate when your `git rev-parse HEAD` *does* change. This means -`bar` will continue to use any previously cached value, and `bar`'s output will -be out of date! - -To fix this, you can wrap your `git rev-parse HEAD` in a `T.input`: - -```scala -def foo = T.input{ %%("git", "rev-parse", "HEAD").out.string } -def bar = T{ ... foo() ... } -``` - -This makes `foo` will always re-evaluate every build; if `git rev-parse HEAD` -does not change, that will not invalidate `bar`'s caches. But if `git rev-parse -HEAD` *does* change, `foo`'s output will change and `bar` will be correctly -invalidated and re-compute using the new version of `foo`. - -Note that because `T.input`s re-evaluate every time, you should ensure that the -code you put in `T.input` runs quickly. Ideally it should just be a simple check -"did anything change?" and any heavy-lifting can be delegated to downstream -targets. - -### Workers - -```scala -def foo = T.worker{ ... } -``` - -Most tasks dispose of their in-memory return-value every evaluation; in the case -of [Targets](#targets), this is stored on disk and loaded next time if -necessary, while [Commands](#commands) just re-compute them each time. Even if -you use `--watch` or the Build REPL to keep the Mill process running, all this -state is still discarded and re-built every evaluation. - -Workers are unique in that they store their in-memory return-value between -evaluations. This makes them useful for storing in-memory caches or references -to long-lived external worker processes that you can re-use. - -Mill uses workers to managed long-lived instances of the -[Zinc Incremental Scala Compiler](https://github.com/sbt/zinc) and the -[Scala.js Optimizer](https://github.com/scala-js/scala-js). This lets us keep -them in-memory with warm caches and fast incremental execution. - -Like [Persistent Targets](#persistent-targets), Workers inherently involve -mutable state, and it is up to the implementation to ensure that this mutable -state is only used for caching/performance and does not affect the -externally-visible behavior of the worker. - -## Cheat Sheet - -The following table might help you make sense of the small collection of -different Task types: - -| | Target | Command | Source/Input | Anonymous Task | Persistent Target | Worker | -|:-------------------------------|:-------|:--------|:-------------|:---------------|:------------------|:-------| -| Cached on Disk | X | X | | | X | | -| Must be JSON Writable | X | X | | | X | | -| Must be JSON Readable | X | | | | X | | -| Runnable from the Command Line | X | X | | | X | | -| Can Take Arguments | | X | | X | | | -| Cached between Evaluations | | | | | | X | - -- cgit v1.2.3