diff options
| author | Aaron Davidson <aaron@databricks.com> | 2014-03-07 10:22:27 -0800 |
|---|---|---|
| committer | Aaron Davidson <aaron@databricks.com> | 2014-03-07 10:22:27 -0800 |
| commit | dabeb6f160f7ad7df1c54b1b8b069700dd4b74dd (patch) | |
| tree | 1abbe46bae3890ec2924e87366d36d3e8d7e22cb /docker | |
| parent | 33baf14b04bcb5cb8dc39ae0773b9e0ef79ef9cf (diff) | |
| download | spark-dabeb6f160f7ad7df1c54b1b8b069700dd4b74dd.tar.gz spark-dabeb6f160f7ad7df1c54b1b8b069700dd4b74dd.tar.bz2 spark-dabeb6f160f7ad7df1c54b1b8b069700dd4b74dd.zip | |
SPARK-1136: Fix FaultToleranceTest for Docker 0.8.1
This patch allows the FaultToleranceTest to work in newer versions of Docker.
See https://spark-project.atlassian.net/browse/SPARK-1136 for more details.
Besides changing the Docker and FaultToleranceTest internals, this patch also changes the behavior of Master to accept new Workers which share an address with a Worker that we are currently trying to recover. This can only happen when the Worker itself was restarted and got the same IP address/port at the same time as a Master recovery occurs.
Finally, this adds a good bit of ASCII art to the test to make failures, successes, and actions more apparent. This is very much needed.
Author: Aaron Davidson <aaron@databricks.com>
Closes #5 from aarondav/zookeeper and squashes the following commits:
5d7a72a [Aaron Davidson] SPARK-1136: Fix FaultToleranceTest for Docker 0.8.1
Diffstat (limited to 'docker')
| -rw-r--r-- | docker/README.md | 4 | ||||
| -rwxr-xr-x | docker/spark-test/master/default_cmd | 8 | ||||
| -rwxr-xr-x | docker/spark-test/worker/default_cmd | 8 |
3 files changed, 17 insertions, 3 deletions
diff --git a/docker/README.md b/docker/README.md index bf59e77d11..40ba9c3065 100644 --- a/docker/README.md +++ b/docker/README.md @@ -2,4 +2,6 @@ Spark docker files =========== Drawn from Matt Massie's docker files (https://github.com/massie/dockerfiles), -as well as some updates from Andre Schumacher (https://github.com/AndreSchumacher/docker).
\ No newline at end of file +as well as some updates from Andre Schumacher (https://github.com/AndreSchumacher/docker). + +Tested with Docker version 0.8.1. diff --git a/docker/spark-test/master/default_cmd b/docker/spark-test/master/default_cmd index a5b1303c2e..5a7da3446f 100755 --- a/docker/spark-test/master/default_cmd +++ b/docker/spark-test/master/default_cmd @@ -19,4 +19,10 @@ IP=$(ip -o -4 addr list eth0 | perl -n -e 'if (m{inet\s([\d\.]+)\/\d+\s}xms) { print $1 }') echo "CONTAINER_IP=$IP" -/opt/spark/spark-class org.apache.spark.deploy.master.Master -i $IP +export SPARK_LOCAL_IP=$IP +export SPARK_PUBLIC_DNS=$IP + +# Avoid the default Docker behavior of mapping our IP address to an unreachable host name +umount /etc/hosts + +/opt/spark/bin/spark-class org.apache.spark.deploy.master.Master -i $IP diff --git a/docker/spark-test/worker/default_cmd b/docker/spark-test/worker/default_cmd index ab6336f70c..31b06cb0eb 100755 --- a/docker/spark-test/worker/default_cmd +++ b/docker/spark-test/worker/default_cmd @@ -19,4 +19,10 @@ IP=$(ip -o -4 addr list eth0 | perl -n -e 'if (m{inet\s([\d\.]+)\/\d+\s}xms) { print $1 }') echo "CONTAINER_IP=$IP" -/opt/spark/spark-class org.apache.spark.deploy.worker.Worker $1 +export SPARK_LOCAL_IP=$IP +export SPARK_PUBLIC_DNS=$IP + +# Avoid the default Docker behavior of mapping our IP address to an unreachable host name +umount /etc/hosts + +/opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker $1 |