From 665183a76d25d241fdd9df9e12b8008fb2c80cbe Mon Sep 17 00:00:00 2001 From: Christopher Vogt Date: Sun, 18 Sep 2016 23:06:44 +0000 Subject: Re-wrote docs and some error messages --- README.md | 497 ++++++++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 357 insertions(+), 140 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 34eb57c..e16602d 100644 --- a/README.md +++ b/README.md @@ -1,184 +1,361 @@ -Join the chat at https://gitter.im/cvogt/cbt +(For a tutorial scroll down.) -Welcome to Chris' Build Tool (CBT) for Scala +Chris' Build Tool (CBT) for Scala ============================================ -Fun, fast, intuitive, composable and statically checked builds written in Scala. +Easy to learn and master, lightning fast and backed by a thriving community of enthusiasts and contributors. [Join us on gitter](https://gitter.im/cvogt/cbt). For talks, development roadmap, projects using cbt, etc see the wiki. + +What is CBT? +---------------------------------------------- +CBT is a build tool meaning it helps orchestrating compilation, code and +documentation generation, packaging, deployment and custom tooling for +your project. It mainly targets Scala projects but is not exclusive to them. + +CBT builds are full programs written using vanilla Scala code. +Familiar concepts make you feel right at home - build files are classes, +tasks are defs, you customize using method overrides. You already know these +things and everything behaves as expected. That way implementing any +build related requirement becomes as easy as writing any other Scala code. + +CBT is simple in the sense that it uses very few concepts. +A single build uses classes, defs and inheritance. +Builds and binary dependencies can be composed to model +modules depending on each other. + +CBT believes good integration with existing tools to be very +helpful. In that spirit CBT aims for excellent integration with +the command line and your shell. + +CBT considers source files to be an excellent way to distribute code +and has first class support for source and git dependencies. + +How is CBT different from other build tools? +---------------------------------------------- +Not all build tools allow you to write builds in a full programming language. +CBT is based on the assumption that builds are complex enough problems to +warrant this and abstraction and re-use is better handled through libraries +rather than some restricted, declarative DSL. CBT shares this philosophy with SBT. +(This also means that integration with external tools such as an IDE better happens +programmatically through an api rather than a static data representation such as xml.) + +Like SBT, CBT chooses Scala as it's language of choice, trying to appeal to +Scala programmers allowing them to re-use their knowledge and the safety of the language. + +Unlike SBT 0.11 and later, CBT maps task execution to JVM method invocations. +SBT implements it's own self-contained task graph model and interpreter. +This allows SBT to have it's model exactly fit the requirements. +CBT instead uses existing JVM concepts for the solution and adds +custom concepts only when necessary. CBT assumes this to lead to better +ease of use due to familarity and better integration with existing tools +such as interactive debuggers or stack traces because CBT's task call stack +IS the JVM call stack. + +SBT 0.7 shared this design decision as many may have forgotten. +However, CBT is still quite a bit simpler than even SBT 0.7 as +CBT gets away with fewer concepts. SBT 0.7 had notions of +main vs test sources, multi-project builds, task-dependencies +which weren't invocations and other concepts. CBT maps all of +these to def invocations and build composition instead. + +System requirements +------------------- + +CBT is best tested under OSX. People are also using it also under +Ubuntu and Windows via cygwin. It should be easy to port CBT to +other systems or drop the cygwin requirement. You will only +have to touch the launcher bash or .bat scripts. +Please contribute back if you fixed something :). + +You currently need javac and realpath or gcc installed. +nailgun is optional for speedup. gpg is required only for publishing maven artifacts. + +Features +-------- + +CBT supports the basic needs for Scala builds right now: +Compiling, running, testing, packaging, publishing local and to sonatype, +scaladoc, maven dependencies, source dependencies (e.g. for modularized projects), +triggering tasks on file changes, cross-compilation, reproducible builds. + +There is also a growing number of plugins in `plugins/` and `stage2/plugins/`, +but some things you'd like may still be missing. Consider writing +a plugin in that case. It's super easy, just a trait. Share it :). + + +Tutorial +-------- + +This section explains how to get started with cbt step-by-step. +There are also example projects with build files in examples/ and test/. + +### Installation + +If you haven't cloned cbt yet, clone it now. Cloning is how you install cbt. +We know that's a bit unusual, but roll with it, there are good reasons :). +Open a shell, cd to the directory where you want to install cbt and execute: -Currently CBT has been tested in OSX only. Adding support for Unix and Windows -should not be hard. Please contribute back if you mange :). +``` +$ git clone git@github.com:cvogt/cbt.git +``` + +There are a bash script `cbt` and a `cbt.bat` in the checkout directory. +Add one to your `$PATH`, e.g. symlink it from `~/bin/cbt`. + +Check that it works by calling `cbt`. You should see CBT compiling itself +and showing a list of built-in tasks. + +Great, you're all set up. Now, let's use cbt for an example project. + +### Creating your first project + +Create a new directory and cd into it. E.g. `my-project`. + +``` +$ mkdir my-project +$ cd my-project +``` + +Let's create a tiny sample app. CBT can generate it for you. Just run: + +``` +$ cbt tools createMain +``` + +### Running your code + +Now there should be a file `Main.scala`, which prints `Hello World` when run. +So let's run it: + +``` +$ cbt run +``` + +You should see how CBT first compiles your project, then runs it and prints +`Hello World`. CBT created the file `Main.scala` top-level in your directory. +You can alternatively place `.scala` or `.java` files in `src/` +or any of it's subdirectories. + +### Creating a build file + +Without a build file, CBT just uses some default build settings. +Let's make the build more concrete by creating a build file. -CBT supports the basic needs for Scala builds right now. -Composing, compiling, running, testing, packaging, publishing. -Tasks outside of these, such as building manuals will require -easy custom code. If you integrate something, consider -doing it as traits that you make available as a library that -other builds can depend on and mix in. +CBT can help you with that. Execute: -Slides and video from CBT talk from NEScala 2016: -https://github.com/cvogt/talks/raw/master/2016-03-04_NEScala-2016_A-Vision-For-Scala-Builds.pdf -https://www.youtube.com/watch?v=5HfKw3hgdOM +``` +$ cbt tools createBuild +``` -Getting started ---------------- +Now there should be a file `build/build.scala` with a sample `Build class. -### Dependencies -You currently need javac, nailgun, gpg and realpath or gcc installed. +Btw., a build file can have it's own build and so on recursively like in SBT. +When you create a file `build/build/build.scala` and change `Build` class in there +to extend `BuildBuild`, it will be used to build your `build/build.scala`. You can +add built-time dependencies like plugins this way. -### Installing -CBT bootstraps from source. To install, just clone the repository. +### Adding dependencies -### Using -To use, just call the `cbt` bash script. You will see CBT first -building up itself, then trying to start your build. +In the generated `build/build.scala` there are +several examples for dependencies. We recommend using the constructor syntax +`ScalaDependency` (for automatically adding the scala version to the artifact id) +or `MavenDependency` (for leaving the artifact id as is). The SBT-Style `%`-DSL +syntax is also supported for copy-and-paste convenience, but discouraged. -The easiest CBT build requires no script. It compiles source files from the -current directory and from `src/` into the target folder. Try calling `cbt compile`. -If you have a class called Main with a main method, try `cbt run`. -Put a source file with a Main class into `test/` in order for -`cbt test` to run it. It will see your main code. +Alright, let's enable the `override def dependencies`. Make sure to include +`super.dependencies`, which currently only includes the Scala standard library. +Add a dependency of your choice, start using it from `Main.scala` and `cbt run` again. -If you need more than this, like dependencies, create a scala file in `build/` -that describes your build. Here is an example +As you can see CBT makes choice of the maven repository explicit. It does so for clarity. -```scala -// build/build.scala -class Build(val context: cbt.Context) extends PackageJars { - override def defaultVersion = "0.6.1" +### Calling other tasks - override def name = "play-json-extensions" +Tasks are just defs. You can call any public zero-arguments method of your +`Build class or its parents straight from the command line. To see how it works +let's call the compile task. - override def groupId = "org.cvogt" +``` +$ cbt compile +``` - override def dependencies = - super.dependencies ++ - Resolver(mavenCentral).bind( - // encouraged way to declare dependencies - ScalaDependency("com.typesafe.play", "play-json", "2.4.4"), - MavenDependency("joda-time", "joda-time", "2.9.2") - ) +### Creating custom tasks - override def compile = { - println("Compiling...") - super.compile - } +In order to create a custom task, simply add a new def to your Build class, e.g. - def foo = "Hello World" +``` +class Build...{ + ... + def foo = "asdf" } ``` -Dependencies could also be declared using SBT style. +Now call the def from the command line: -```scala -class Build(val context: cbt.Context) extends PackageJars { -... -// sbt compatible dependencies definition -override def dependencies = - super.dependencies ++ - Resolver(mavenCentral).bind( - "com.typesafe.play" %% "play-json" % "2.4.4", - "joda-time" % "joda-time" % "2.9.2" - ) -... -} +``` +$ cbt foo ``` -Now you can call methods of this class through cbt. Try `cbt foo`. -You can see how your build is configured via overrides. +As you can see it prints `asdf`. Adding tasks is that easy. -call `cbt` to see a full list of available commands for this build. +### Triggering tasks on file-changes -Look into the class PackageBuild (and it's super class BasicBuild) in CBT's source code to see their -details. The source code is really simple. Don't shy away from -looking, even as a beginner. No crazy stuff, I promise ;). You -can find the relevant code in CBT's stage2/BasicBuild.scala +When you call a task, you can prefix it with `loop`. +CBT then watches the source files, the build files and even CBT's own +source code and re-runs the task when anything changes. If necessary, +this forces CBT to re-build itself, the project's dependencies and the project itself. -I order to keep executing the same command triggered by file changes use `cbt loop `. +Let's try it. Let's loop the run task. Call this from the shell: -You can find example builds in CBT's own `test/` folder. -Not all of them have a build file, in which case CBT uses the default -cbt.BasicBuild. +``` +$ cbt loop run +``` -A folder `build/` can have its own folder `build/` inside in order -to add source or maven dependencies to your build. Eventually -you'll be able to also choose the CBT and Scala versions for -target builds. Make sure you extend cbt.BuilBuild instead of -cbt.Build, in order to automatically trigger building of the -target build. +Now change `Main.scala and see how cbt picks it up and re-runs it. +CBT is fast. It may already be done re-compiling and re-running before +you managed to change windows back from your editor to the shell. -cbt is fast. It uses Nailgun to keep the JVM hot. It uses the Java -WatchService (respectively a fast OSX port of it) for instant triggering -re-compilation on file changes. Use `cbt loop compile`. +Try changing the build file and see how CBT reacts to it as well. -CBT concepts ------------- +### Adding tests -There two essential primitives available in build scripts for composing -modular projects: +The simplest way to add tests is putting a few assertions into the previously +created Main.scala and be done with it. Alternatively you can add a test +framework plugin to your build file to use something more sophisticated. - 1. Dynamically compiling and loading Build scripts in other - directories and calling methods (aka tasks) on them to compile, - get the classpath, ask for version numbers, etc. +This however means that the class files of your tests will be included in the +jar should you create one. If that's fine, you are done :). If it is not you +need to create another project, which depends on your previous project. This +project will be packaged separately or you can disable packaging there. Let's create +such a project now. - This allows to do a lot of things just like that: - Multi-project builds, source dependencies, builds of builds and - allowing tests simply as dependent projects of the main project, etc. +Your project containing tests can be anywhere but a recommended location is a +sub-folder called `test/` in your main project. Let's create it and create a +Main class and build file: - 2. Maven dependencies +``` +$ mkdir test +$ cd test +$ rm ../Main.scala +$ cbt tools createMain +$ cbt tools createBuild +``` - I wrote my own 50 LOC Maven resolver. It's super quick and I have - yet see it not to being able to handle something. I know cases - exist, but seem rare. - alexarchambault's Coursier can be used as a more complete drop-in. +We also deleted the main projects Main.scala, because now that we created a new one +we would have two classes with the same name on the classpath which can be very confusing. -Build scripts also have access to a small unsurprising library for -- triggering dependencies to build / download and get the classpath -- compiling Java / Scala code using zinc with given class paths -- running code -- packaging jars -- signing / publishing to sonatype/maven +Now that we have a Main file in our test project, we can add some assertions to it. +In order for them to see the main projects code, we still need to do one more thing - +add a `DirectoryDependency` to your test project's build file. There is a similar example +in the generated build.scala. What you need is this: -Scala.js support ----------------- +``` +override def dependencies = super.dependencies ++ Seq( + DirectoryDependency( projectDirectory ++ "/.." ) +) +``` -CBT supports cross-project Scala.js build. -It preserves same structure as in SBT (https://www.scala-js.org/doc/project/cross-build.html) +This successfully makes your test project's code see the main projects code. +Add some class to your main project, e.g. `case class Foo(i: Int = 5)`. Now +put an assertion into the Main class of your test project, e.g. +`assert(Foo().i == 5)` and hit `cbt run` inside your test project. - 1. Example for user scalajs project is in: `$CBT_HOME/cbt/examples/build-scalajs` - 2. `$CBT_HOME/cbt compile` - Will compile JVM and JS sources - `$CBT_HOME/cbt jsCompile` - Will compile JS sources - `$CBT_HOME/cbt jvmCompile` - Will compile JVM sources - 3. `$CBT_HOME/cbt fastOptJS` and `$CBT_HOME/cbt fullOptJS` - Same as in Scala.js sbt project +Make sure you deleted your main projects class Main when running your tests. - Note: Scala.js support is under ongoing development. +Congratulations, you successfully created a dependent project and ran your tests. - Currently missing features: - * No support for jsDependencies: - It means that all 3rd party dependencies should added manually, see scalajs build example - * No support for test +### Multi-projects Builds -Missing features in comparison with SBT ---------------------------------------- +A single build only handles a single project in CBT. So there isn't exactly +such a things as a Multi-project Build. Instead you can simply write multiple +projects that depend on each other. We have already done that with tests above, +but you can do the exact same thing to modularize your project into multiple ones. -Not implemented yet, but rather easily possible without API changes or -major refactors is concurrently building dependencies and -running tests. Right now it is sequential. -Maven downloads already happen in parallel as well as some tasks like packaging. +### Reproducible builds -Another edge case that may need a solution is dynamically overwriting -tasks. SBT allows that. Classes and traits are static. The only use -cases I know are debugging, cross builds and the sbt release plugin. A -solution could be code generating traits at build-time and mixing them -in ad-hoc. It's a build-tool after all. Build-time code-generation and -class loading is not rocket science. But there may be simpler solutions -for the cases at hand. And they are edge cases anyways. +To achieve reproducible builds, you'll need to tie your build files to a particular +CBT-version. It doesn't matter what version of CBT you are actually running, +as long as the `BuildInterface` is compatible (which should be true for a large number +of versions and we may find a better solution long term. If you see a compile error +during compilation of CBT itself that some method in BuildInterface was not +implemented or incorrectly implemented, you may be running an incompatible CBT +version. We'll try to fix that later, but for now you might have to checkout +the required hash of CBT by hand.). -Bash completions ----------------- +When you specify a particular version, CBT will use that one instead of the installed one. + +You can specify one by adding one line right before `class Build`. It looks like this: + +``` +// cbt:https://github.com/cvogt/cbt.git#75c32537cd8f29f9d12db37bf06ad942806f02393 +class Build... +``` + +The URL points to any git repository containing one of CBT's forks. You currently +have to use a stable reference - i.e. a hash or tag. (Checkouts are currently not +updated. If you refer to a branch or tag which is moved on the remote, CBT +will not realize that and keep using the old version). + +### Using CBT like a boss + +Do you own your Build Tool or does your Build Tool own you? CBT makes it easy for YOU +to be in control. We try to work on solid documentation, but even good +documentation never tells the whole truth. Documentation can tell how to use +something and why things are happening, but only the code can tell all the +details of what exactly is happening. Reading the code can be intimidating for +many Scala libraries, but not so with CBT. The source code is easy to read +to the point that even Scala beginners will be able to understand it. So +don't be afraid to actually look under the hood and check out what's happening. + +And guess what, you already have the source code on your disk, because +you installed CBT by cloning it's git repository. You can even debug CBT and +your build files in an interactive debugger like IntelliJ after some minor setup. + +Finally, you can easily change CBT's code. Then CBT re-builds itself when you try +to use it the next time. This means any changes you make are instantly reflected. +This and the simple code make it super easy to fix bugs or add features yourself +and feed them back into main line CBT. + + +When debugging things, it can help to enable CBT's debug logging by passing +`-Dlog=all` to CBT (or a logger name instead of `all`). + + +Other design decisions +-------------------- + +CBT tries to couple it's code very loosely. OO is used for configuration in build files. +Interesting logic is in simple supporting library classes/objects, which can be used +independently. You could even build a different configuration api than OO on top of them. + + +Known limitations +-------------------------- +- currently CBT supports no generic task scoping. A solution is known, but not implemented. + For now manually create intermediate tasks which serve as scoping for + known situations and encode the scope in the name, e.g. fastOptScalaJsOptions +- currently CBT supports no dynamic overrides of tasks. A solution is known, but not implemented. + scalaVersion and version are passed through the context instead for dynamic overrides. +- there is currently no built-in support for resources being added to a jar. + Should be simple to add, consider a PR +- test framework support is currently a bit spotty but actively being worked on +- concurrent task and build execution is currently disabled +- CBT uses it's own custom built maven resolver, which is really fast, + but likely does not work in some edge cases. Those may or may not be easy to fix. + We should add optional coursier integration back for a more complete solution. + +Known bugs +------------- +- currently there is a bug in CBT where dependent builds may miss changes in the things + they depend on. Deleting all target directories and starting from scratch helps. +- There are some yet unknown bugs which can be solved by killing the nailgun background process + and/or re-running the desired cbt task multiple times until it succeeds. +- if you ever see no output from a command but expect some, make sure you are in the right directory + and try to see if any of the above recommendations help + + +Shell completions +------------------- + +### Bash completions To auto-complete cbt task names in bash do this: ``` @@ -193,13 +370,16 @@ for f in ~/.bash_completion.d/*; do done ``` -Fish shell completions ----------------- -copy this line into your fish configuration, on OSX: /.config/fish/config.fish +### Fish shell completions + +copy this line into your fish configuration, on OSX: `/.config/fish/config.fish` + +``` complete -c cbt -a '(cbt taskNames)' +``` + +### Zsh completions -Zsh completions ---------------- ##### Manual installation Add the following to your `.zshrc` ``` @@ -215,3 +395,40 @@ Then enable it in your `.zshrc`: ``` plugins=( ... cbt) ``` + +Plugin-author guide +-------------------- + +A CBT plugin is a trait that is mixed into a Build class. +Only use this trait only for wiring things together. +Don't put logic in there. Instead simply call methods +on a separate class or object which serves as a library +for your actual logic. It should be callable and testable +outside of a Build class. This way the code of your plugin +will be easier to test and easier to re-use. Feel free +to make your logic rely on CBT's logger. + + See `plugins/` for examples. + +Scala.js support +---------------- + +CBT supports cross-project Scala.js builds. +It preserves same structure as in SBT (https://www.scala-js.org/doc/project/cross-build.html) + + 1. Example for user scalajs project is in: `$CBT_HOME/cbt/examples/build-scalajs` + 2. `$CBT_HOME/cbt compile` + Will compile JVM and JS sources + `$CBT_HOME/cbt jsCompile` + Will compile JS sources + `$CBT_HOME/cbt jvmCompile` + Will compile JVM sources + 3. `$CBT_HOME/cbt fastOptJS` and `$CBT_HOME/cbt fullOptJS` + Same as in Scala.js sbt project + + Note: Scala.js support is under ongoing development. + + Currently missing features: + * No support for jsDependencies: + It means that all 3rd party dependencies should added manually, see scalajs build example + * No support for test -- cgit v1.2.3