Generate documentation

13 files changed, 349 insertions, 145 deletions

diff --git a/Documentation/developer.md b/Documentation/developer.md
new file mode 100644
index 0000000..27e64f4
--- /dev/null
+++ b/Documentation/developer.md
@@ -0,0 +1,68 @@
+---
+layout: page
+title: Developer Guide
+---
+# Building from Source
+A complete build of flow involves two parts
+
+1. Building Scala sources (the front-end), resulting in a platform independent artifact (i.e. a jar file).
+
+2. Building C sources (the back-end), yielding a native library that may only be used on systems resembling the platform for which it was compiled.
+
+Both steps are independent, their only interaction being a header file generated by the JDK utility `javah` (see `sbt javah` for details), and may therefore be built in any order.
+
+## Building Scala Sources
+Run `sbt core/package` in the base directory. This simply compiles Scala sources as with any standard sbt project and packages the resulting class files into a jar.
+
+## Building Native Sources
+The back-end is managed by CMake and all relevant files are contained in `flow-native/src`.
+
+### Build Process
+Several steps are involved in producing the native library:
+
+1. Bootstrap the build (run this once, if `Makefile` does not exist).
+
+	1. Required dependencies: CMake (2.6 or higher), JDK (1.8 or above)
+    2. Run `cmake .`
+
+2. Compile
+
+    1. Run `make`.
+       *Note: should you encounter an error about a missing "jni.h" file, try setting the JAVA_HOME environment variable to point to the base path of your JDK installation.*
+
+3. Install
+
+    The native library is now ready and can be:
+
+	- copied to a local directory: `DESTDIR=$(pwd)/<directory> make install`
+
+    - installed system-wide: `make install`
+
+    - put into a "fat" jar, useful for dependency management with sbt (see next section)
+
+### Creating a Fat Jar
+The native library produced in the previous step may be bundled into a "fat" jar so that it can be included in SBT projects through its regular dependency mechanisms. In this process, sbt basically acts as a wrapper script around CMake, calling the native build process and packaging generated libraries. Running `sbt native/package` produces the fat jar in `flow-native/target`.
+
+Note: an important feature of fat jars is to include native libraries for several platforms. To copy binaries compiled on other platforms to the fat jar, place them in a subfolder of `flow-native/lib_native`. The subfolder should have the name `$(arch)-$(kernel)`, where `arch` and `kernel` are, respectively, the lower-case values returned by `uname -m` and `uname -s`.
+
+### Note About Versioning
+The project and package versions follow a [semantic](http://semver.org/) pattern: `M.m.p`, where
+
+- `M` is the major version, representing backwards incompatible changes to the public API
+
+- `m` is the minor version, indicating backwards compatible changes such as new feature additions
+
+- `p` is the patch number, representing internal modifications such as bug-fixes
+
+Usually (following most Linux distribution's conventions), shared libraries produced by a project `name` of version `M.m.p` are named `libname.so.M.m.p`. However, since when accessing shared libraries through the JVM, only the `name` can be specified and no particular version, the convention adopted by flow is to append `M` to the library name and always keep the major version at zero. E.g. `libflow.so.3.1.2` becomes `libflow3.so.0.1.2`.
+
+# Publishing and Releasing
+The release process is managed with the `sbt-release` plugin. See 'project/Release.scala' for a description of the various steps involved.
+
+Here are some important notes on creating a release:
+
+- During a release, only readily available libraries in `lib_native` are packaged into the fat jar, no local native compilation is performed. The rationale behind this is that while native libraries rarely change, they are still tied to the version of libc of the compiling system. Since the releases are mostly done on a development machine (cutting-edge OS), compiling native libraries locally could break compatibility with older systems.
+
+- Currently, the release script does not handle uploading the native libraries archive (don't confuse this with the fat jar, which is uploaded). If creating a release that changed the native libraries or added support for more platforms, creating and uploading a new native archive must be done manually.
+
+- Don't forget to update the website after creating a new release.
diff --git a/Documentation/manual.md b/Documentation/manual.md
new file mode 100644
index 0000000..416b316
--- /dev/null
+++ b/Documentation/manual.md
@@ -0,0 +1,224 @@
+---
+layout: page
+title: User Guide
+---
+# Content
+* TOC
+{:toc}
+
+# Getting Started
+Flow uses sbt as build system. To get started, add the Bintray jcenter resolver and include a dependency to flow in your project:
+
+~~~scala
+resolvers += Resolver.jcenterRepo
+
+libraryDependencies += "com.github.jodersky" %% "flow-core" % "@version@"
+~~~
+
+Next, you need to include flow's native library that supports communication for serial devices.
+
+## Including Native Library
+There are two options to include the native library:
+
+1. Using a pre-packaged dependency, available only for certain OSes but easily included.
+
+2. Including the library manually for maximum portability.
+
+It is recommended that you use the first option for testing purposes or end-user applications. The second option is recomended for libraries, since it leaves more choice to the end-user.
+
+### The Easy Way
+In case your kernel/architecture combination is present in the "supported platforms" table in the [downloads section]({{site.url}}/downloads/), add a second dependency to your project:
+
+~~~scala
+libraryDependencies += "com.github.jodersky" % "flow-native" % "@version@" % "runtime"
+~~~
+
+This will add a jar to your classpath containing native libraries for various platforms. At run-time, the correct library for the current platform is selected, extracted and loaded. This solution enables running applications seamlessly, as if they were pure JVM applications.
+
+### Maximum Portability
+Start by obtaining a copy of the native library, either by [building flow](./developer) or by [downloading]({{site.url}}/downloads/) a native archive. In order to work with this version of flow, native libraries need to be of major version @native_major@ and minor version greater or equal to @native_minor@.
+
+Then, for every end-user application that relies on flow, manually add the native library for the current platform to the JVM's library path. This can be achieved through various ways, notably:
+
+- Per application:
+
+    Run your program with the command-line option ```-Djava.library.path=".:<folder containing libflow@native_major@.so>"```. E.g. ```java -Djava.library.path=".:/home/<folder containing libflow@native_major@.so>" -jar your-app.jar```
+
+- System- or user-wide:
+
+	Copy the native library to a place that is on the default Java library path and run your application normally. Such places usually include `/usr/lib` and `/usr/local/lib`.
+
+---
+
+# Communication Protocol
+The following is a general guide on the usage of flow. If you prefer a complete example, check out the code contained in the [flow-samples](https://github.com/jodersky/flow/tree/v@version@/flow-samples) directory.
+
+Flow's API follows that of an actor based system, where each actor is assigned specific functions involved in serial communication. The two main actor types are:
+
+1. Serial "manager". The manager is a singleton actor that is instantiated once per actor system, a reference to it may be obtained with `IO(Serial)`. It is typically used to open serial ports (see following section).
+
+2. Serial "operators". Operators are created once per open serial port and serve as an intermediate between client code and native code dealing with serial data transmission and reception. They isolate the user from threading issues and enable the reactive dispatch of incoming data. A serial operator is said to be "associated" to its underlying open serial port.
+
+The messages understood by flow's actors are all contained in the `com.github.jodersky.flow.Serial` object. They are well documented and should serve as the entry point when searching the API documentation.
+
+## Opening a Port
+A serial port is opened by sending an `Open` message to the serial manager. The response varies on the outcome of opening the underlying serial port.
+
+1. In case of failure, the serial manager will respond with a `CommandFailed` message to the original sender. The message contains details on the reason to why the opening failed.
+
+2. In case of success, the sender is notified with an `Opened` message. This message is sent from an operator actor, spawned by the serial manager. It is useful to capture the sender (i.e. the operator) of this message as all further communication with the newly opened port must pass through the operator.
+
+~~~scala
+import com.github.jodersky.flow.{ Serial, SerialSettings, AccessDeniedException }
+
+val port = "/dev/ttyXXX"
+val settings = SerialSettings(
+  baud = 115200,
+  characterSize = 8,
+  twoStopBits = false,
+  parity = Parity.None
+)
+
+IO(Serial) ! Serial.Open(port, settings)
+
+def receive = {
+  case Serial.CommandFailed(cmd: Serial.Open, reason: AccessDeniedException) =>
+    println("You're not allowed to open that port!")
+  case Serial.CommandFailed(cmd: Serial.Open, reason) =>
+	println("Could not open port for some other reason: " + reason.getMessage)
+  case Serial.Opened(settings) => {
+    val operator = sender
+    //do stuff with the operator, e.g. context become opened(op)
+  }
+}
+~~~
+
+## Writing Data
+Writing data is as simple as sending a `Write` message to an operator. The data to send is an instance of `akka.util.ByteString`:
+
+~~~scala
+operator ! Serial.Write(data)
+~~~
+
+Optionally, an acknowledgement for sent data can be requested by adding an `ack` parameter to a `Write` message. The `ack` parameter is of type `Int => Serial.Event`, i.e. a function that takes the number of actual bytes written and returns an event. Note that "bytes written" refers to bytes enqueued in a kernel buffer; no guarantees can be made on the actual transmission of the data.
+
+~~~scala
+
+case class MyPacketAck(wrote: Int) extends Serial.Event
+
+operator ! Serial.Write(data, MyPacketAck(_))
+operator ! Serial.Write(data, n => MyPacketAck(n))
+
+def receive = {
+  case MyPacketAck(n) => println("Wrote " + n + " bytes of data")
+}
+
+~~~
+
+## Receiving Data
+The actor that opened a serial port (referred to as the client), exclusively receives incomming messages from the operator. These messages are in the form of `akka.util.ByteString`s and wrapped in a `Received` object.
+
+~~~scala
+def receive = {
+  case Serial.Received(data) => println("Received data: " + data.toString)
+}
+~~~
+
+## Closing a Port
+A port is closed by sending a `Close` message to its operator:
+~~~scala
+operator ! Serial.Close
+~~~
+The operator will close the underlying serial port and respond with a final `Closed` message before terminating.
+
+
+## Resources and Error Handling
+The operator has a deathwatch on the client actor that opened the port, this means that if the latter crashes, the operator closes the port and equally terminates, freeing any allocated resources.
+
+The opposite is not true by default, i.e. if the operator crashes (this can happen for example on IO errors) it dies silently and the client is not informed. Therefore, it is recommended that the client keep a deathwatch on the operator.
+
+---
+
+# Watching Ports
+As of version 2.2.0, flow can watch directories for new files. On most unix systems this can be used for watching for new serial ports in `/dev/`.
+Watching happens through a message-based, publish-subscribe protocol as explained in the sections below.
+
+## Subscribing
+A client actor may watch -- i.e subscribe to notifications on -- a directory by sending a `Watch` command to the serial manager.
+
+Should an error be encountered whilst trying to obtain the watch, the manager will respond with a `CommandFailed` message.
+Otherwise, the client may be considered "subscribed" to the directory and the serial manager will thenceforth notify
+the client on new files.
+
+~~~scala
+IO(Serial) ! Serial.Watch("/dev/")
+
+def receive = {
+  case Serial.CommandFailed(w: Watch, reason) =>
+    println(s"Cannot obtain a watch on ${w.directory}: ${reason.getMessage}")
+}
+
+~~~
+
+## Notifications
+Whilst subscribed to a directory, a client actor is informed of any new files in said directory by receiving
+`Connected` messages from the manager.
+
+~~~scala
+def receive = {
+  case Serial.Connected(port) if port matches "/dev/ttyUSB\\d+" =>
+    // do something with the available port, e.g.
+    // IO(Serial) ! Open(port, settings)
+}
+~~~
+
+## Unsubscribing
+Unsubscribing from events on a directory is done by sending an `Unsubscribe` message to the serial manager.
+
+~~~scala
+IO(Serial) ! Unwatch("/dev/")
+~~~
+
+## Resource Handling
+Note that the manager has a deathwatch on every subscribed client. Hence, should a client die, any underlying resources will be freed.
+
+---
+
+# Stream Support
+Flow provides support for Akka streams and thus can be interfaced with reactive-streams. Support is implemented in a separate module, which needs to be added as a library dependency:
+
+~~~scala
+libraryDependencies += "com.github.jodersky" %% "flow-stream" % "@version@"
+~~~
+
+The main entry point for serial streaming is `com.github.jodersky.flow.stream.Serial`. It's API is also well documented and should serve as the starting point when searching documentation on serial streaming.
+
+## Opening a Port
+Connection is established by materializing a `Flow[ByteString, ByteString, Future[Connection]]` obtained by calling `Serial().open()`
+
+~~~scala
+val serial: Flow[ByteString, ByteString, Future[Connection]] = Serial().open("/dev/ttyUSB0", settings)
+
+val source: Source[ByteString, _] = // some source
+val sink: Sink[ByteString, _] = // some sink
+
+source.viaMat(serial)(Keep.right).toMat(sink)(Keep.left).run() onComplete {
+  case Success(connection) => // a serial connection has been established
+  case Failure(error) => // connection could not be established due to error
+}
+~~~
+
+The materialized future will be completed with a `Success` in case the port is opened or a `Failure` in case an error is encountered whilst opening.
+
+## Communication
+Any data pushed to the `Flow`'s inlet will be sent to the serial port and any data received by the port will be emitted by the `Flow`'s outlet.
+
+Note that backpressure is only available for writing, to add backpressure on the receiving side a higher-level protocol needs to be implemented on top of serial communication.
+
+## Closing a Port
+The underlying serial port is closed when its materialized serial flow is closed.
+
+## Errors and Resource Handling
+Any errors described in flow-core can also be encountered in flow-streaming. When thrown, they will be wrapped as the cause of a `StreamSerialException` and cause the the serial `Flow` stage to fail.
+
+As with flow-core, native resources are handled by underlying Akka mechanisms and any crashes in user code will automatically case the resources to be freed.
diff --git a/README.md b/README.md
index 6e9b41b..60e5966 100644
--- a/README.md
+++ b/README.md
@@ -26,9 +26,9 @@ flow/
 ```
 
 ## Build
-Detailed documentation on building flow is available on the website (or, equivalently, in [developer.md](site/jekyll/documentation/developer.md)).
+Detailed documentation on building flow is available on the website (or, equivalently, in [developer.md](Documentation/developer.md)).
 
-Since flow integrates into the Akka-IO framework, a good resource on its general design is the framework's [documentation](http://doc.akka.io/docs/akka/2.4.2-RC1/scala/io.html).
+Since flow integrates into the Akka-IO framework, a good resource on its general design is the framework's [documentation](http://doc.akka.io/docs/akka/current/scala/io.html).
 
 This project is also an experiment on working with JNI and automating build infrastructure.
 
diff --git a/build.sbt b/build.sbt
new file mode 100644
index 0000000..15ca193
--- /dev/null
+++ b/build.sbt
@@ -0,0 +1,39 @@
+import flow.{FlowBuild, Release}
+
+FlowBuild.commonSettings
+
+Release.settings
+
+/* Settings related to publishing */
+publishArtifact := false
+publish := ()
+publishLocal := ()
+// make sbt-pgp happy
+publishTo := Some(Resolver.file("Unused transient repository", target.value / "unusedrepo"))
+
+/* Generate documentation */
+enablePlugins(PreprocessPlugin)
+sourceDirectory in Preprocess := (baseDirectory in ThisBuild).value / "Documentation"
+preprocessVars in Preprocess := Map(
+  "version" -> version.value,
+  "native_major" -> "3",
+  "native_minor" -> "0"
+)
+
+/* Add scaladoc to documentation */
+enablePlugins(SiteScaladocPlugin)
+unidocSettings
+scalacOptions in (ScalaUnidoc, doc) ++= Seq(
+  "-groups", // Group similar methods together based on the @group annotation.
+  "-diagrams", // Show classs hierarchy diagrams (requires 'dot' to be available on path)
+  "-implicits", // Add methods "inherited" through implicit conversions
+  "-sourcepath", baseDirectory.value.getAbsolutePath
+) ++ {
+  val latestTag: String = "git describe --abbrev=0".!!
+  Opts.doc.sourceUrl(
+  s"https://github.com/jodersky/flow/blob/$latestTag€{FILE_PATH}.scala"
+  )
+}
+siteMappings ++= (mappings in (ScalaUnidoc, packageDoc)).value.map{ case (file, path) =>
+  (file, "api/" + path)
+}
diff --git a/flow-samples/README.md b/flow-samples/README.md
index aa6e7d4..5460b4d 100644
--- a/flow-samples/README.md
+++ b/flow-samples/README.md
@@ -1 +1,3 @@
-This directory contains fully functional application examples of flow. To run an example, change to the base directory of flow (this parent's directory) and run `sbt flow-samples-<sample_name>/run`.
+This directory contains fully functional application examples of flow. To run an example, change to the base directory of flow (this parent's directory) and run `sbt samples<SampleName>/run`.
+
+All projects, including samples, can be listed by running `sbt projects`.
diff --git a/ghpages.sbt b/ghpages.sbt
deleted file mode 100644
index 5ade34e..0000000
--- a/ghpages.sbt
+++ /dev/null
@@ -1,3 +0,0 @@
-ghpages.settings
-
-git.remoteRepo := "git@github.com:jodersky/flow.git"
diff --git a/project/Bintray.scala.notyet b/project/Bintray.scala.notyet
deleted file mode 100644
index 5ed9953..0000000
--- a/project/Bintray.scala.notyet
+++ /dev/null
@@ -1,68 +0,0 @@
-package flow
-
-import sbt._
-import sbt.Keys._
-import ch.jodersky.sbt.jni.plugins.JniPackaging
-import ch.jodersky.sbt.jni.plugins.JniPackaging.autoImport._
-import bintray._
-import bintray.BintrayPlugin.autoImport._
-
-/** Custom bintray tasks. */
-object CustomBintray extends AutoPlugin {
-
-  override def requires = JniPackaging && BintrayPlugin
-  override def trigger = allRequirements
-
-  object autoImport {
-
-    val unmanagedNativeZip = taskKey[File](
-      "Packages unmanaged native libraries in a zip file."
-    )
-
-    val publishNativeZip = taskKey[Unit](
-      "Signs and publishes native zip files to a generic bintray repository."
-    )
-
-  }
-  import autoImport._
-
-  lazy val settings: Seq[Setting[_]] = Seq(
-
-    unmanagedNativeZip := {
-      val out = target.value / (name.value + "-native.zip")
-
-      val files: Seq[File] = unmanagedNativeDirectories.value flatMap {dir =>
-        (dir ** "*").get.filter(_.isFile)
-      }
-      val baseDirectories: Seq[File] = unmanagedNativeDirectories.value
-
-      val mappings: Seq[(File,String)] = files pair Path.relativeTo(baseDirectories)
-
-      IO.zip(mappings, out)
-      out
-    },
-
-
-    publishNativeZip := {
-      val credsFile = bintrayCredentialsFile.value
-      val btyOrg = bintrayOrganization.value
-      val repoName = "generic"
-
-      val zip = unmanagedNativeZip.value
-
-      Bintray.withRepo(credsFile, btyOrg, repoName, prompt = false) { repo =>
-        repo.upload(
-          "flow",
-          version.value,
-          zip.name,
-          zip,
-          streams.value.log
-        )
-      }
-    }
-
-  )
-
-  override def projectSettings = inConfig(Compile)(settings)
-
-}
diff --git a/project/FlowBuild.scala b/project/FlowBuild.scala
index 05f2161..8600e49 100644
--- a/project/FlowBuild.scala
+++ b/project/FlowBuild.scala
@@ -30,50 +30,23 @@ object FlowBuild extends Build {
     }
   )
 
-  lazy val root: Project = (
-    Project("root", file("."))
+  lazy val root = (project in file(".")).
     aggregate(core, native, stream)
-    settings(commonSettings: _*)
-    settings(
-      publishArtifact := false,
-      publish := (),
-      publishLocal := (),
-      publishTo := Some(Resolver.file("Unused transient repository", target.value / "unusedrepo")) // make sbt-pgp happy
-    )
-  )
 
-  lazy val core = Project(
-    id = "flow-core",
-    base = file("flow-core")
-  )
+  lazy val core = (project in file("flow-core"))
 
-  lazy val native = Project(
-    id = "flow-native",
-    base = file("flow-native")
-  )
+  lazy val native = (project in file("flow-native"))
 
-  lazy val stream = Project(
-    id = "flow-stream",
-    base = file("flow-stream"),
-    dependencies = Seq(core)
-  )
+  lazy val stream = (project in file("flow-stream")).
+    dependsOn(core)
 
-  lazy val samplesTerminal = Project(
-    id = "samples-terminal",
-    base = file("flow-samples") / "terminal",
-    dependencies = Seq(core, native % Runtime)
-  )
+  lazy val samplesTerminal = (project in file("flow-samples") / "terminal").
+    dependsOn(core, native % Runtime)
 
-  lazy val samplesTerminalStream = Project(
-    id = "samples-terminal-stream",
-    base = file("flow-samples") / "terminal-stream",
-    dependencies = Seq(stream, native % Runtime)
-  )
+  lazy val samplesTerminalStream = (project in file("flow-samples") / "terminal-stream").
+    dependsOn(stream, native % Runtime)
 
-  lazy val samplesWatcher = Project(
-    id = "samples-watcher",
-    base = file("flow-samples") / "watcher",
-    dependencies = Seq(core, native % Runtime)
-  )
+  lazy val samplesWatcher = (project in file("flow-samples") / "watcher").
+    dependsOn(core, native % Runtime)
 
 }
diff --git a/project/Release.scala b/project/Release.scala
index aef3ec8..3a1a670 100644
--- a/project/Release.scala
+++ b/project/Release.scala
@@ -68,7 +68,6 @@ object Release {
       //Push all changes (commits and tags) to GitHub
       pushChanges
 
-      //TODO: release artifact on bintray
     )
   )
 
diff --git a/project/plugins.sbt b/project/plugins.sbt
index ff1f6d5..031aa44 100644
--- a/project/plugins.sbt
+++ b/project/plugins.sbt
@@ -10,15 +10,11 @@ addSbtPlugin("ch.jodersky" % "sbt-jni" % "0.4.4")
 /*
  * Utility plugins, can be disabled during plain build
  */
-
 // Generate documentation for all sources
 addSbtPlugin("com.eed3si9n" % "sbt-unidoc" % "0.3.3")
 
-// Build website
-addSbtPlugin("com.typesafe.sbt" % "sbt-site" % "0.8.2")
-
-// Integrate website with GitHub pages
-addSbtPlugin("com.typesafe.sbt" % "sbt-ghpages" % "0.5.4")
+// Generate website content
+addSbtPlugin("com.typesafe.sbt" % "sbt-site" % "1.0.0")
 
 // Automate release process
 addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.0")
diff --git a/release.sbt b/release.sbt
deleted file mode 100644
index a7294c9..0000000
--- a/release.sbt
+++ /dev/null
@@ -1 +0,0 @@
-flow.Release.settings
diff --git a/site.sbt b/site.sbt
deleted file mode 100644
index d6fd3d5..0000000
--- a/site.sbt
+++ /dev/null
@@ -1,10 +0,0 @@
-import com.typesafe.sbt.site.JekyllSupport._
-import com.typesafe.sbt.site.PamfletSupport._
-
-site.settings
-
-site.addMappingsToSiteDir(mappings in (ScalaUnidoc, packageDoc), "latest/api")
-
-sourceDirectory in Jekyll := file("site")
-
-site.jekyllSupport()
diff --git a/unidoc.sbt b/unidoc.sbt
deleted file mode 100644
index 9e471d0..0000000
--- a/unidoc.sbt
+++ /dev/null
@@ -1,15 +0,0 @@
-import sbtunidoc.Plugin.UnidocKeys._
-
-unidocSettings
-
-scalacOptions in (ScalaUnidoc, unidoc) ++= Seq(
-  "-groups", // Group similar methods together based on the @group annotation.
-  "-diagrams", // Show classs hierarchy diagrams (requires 'dot' to be available on path)
-  "-implicits", // Add methods "inherited" through implicit conversions
-  "-sourcepath", baseDirectory.value.getAbsolutePath
-) ++ {
-  val latestTag: String = "git describe --abbrev=0".!!
-  Opts.doc.sourceUrl(
-  s"https://github.com/jodersky/flow/blob/$latestTag€{FILE_PATH}.scala"
-  )
-}