aboutsummaryrefslogtreecommitdiff
path: root/repl
Commit message (Collapse)AuthorAgeFilesLines
* [maven-release-plugin] prepare release v1.1.0-rc4v1.1.0Patrick Wendell2014-09-031-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.1.0-rc3"Patrick Wendell2014-09-021-1/+1
| | | | This reverts commit b2d0493b223c5f98a593bb6d7372706cc02bebad.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-09-021-1/+1
| | | | This reverts commit 865e6f63f63f5e881a02d1a4e3b4c5d0e86fcd8e.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-301-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-rc3Patrick Wendell2014-08-301-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.1.0-rc3"Patrick Wendell2014-08-301-1/+1
| | | | This reverts commit 2b2e02265f80e4c5172c1e498aa9ba2c6b91c6c9.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-301-1/+1
| | | | This reverts commit 8b5f0dbd8d32a25a4e7ba3ebe1a4c3c6310aeb85.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-301-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-rc3Patrick Wendell2014-08-301-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.1.0-rc2"Patrick Wendell2014-08-291-1/+1
| | | | This reverts commit 711aebb329ca28046396af1e34395a0df92b5327.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-291-1/+1
| | | | This reverts commit a4a7a241441489a0d31365e18476ae2e1c34464d.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-291-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-rc2Patrick Wendell2014-08-291-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.1.0-rc1"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit f07183249b74dd857069028bf7d570b35f265585.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit f8f7a0c9dce764ece8acdc41d35bbf448dba7e92.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-281-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-rc1Patrick Wendell2014-08-281-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.1.0-rc1"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit 58b0be6a29eab817d350729710345e9f39e4c506.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit 78e3c036eee7113b2ed144eec5061e070b479e56.
* Revert "[maven-release-plugin] prepare release v1.1.0-rc1"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit 79e86ef3e1a3ee03a7e3b166a5c7dee11c6d60d7.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit a118ea5c59d653f5a3feda21455ba60bc722b3b1.
* Revert "Revert "[maven-release-plugin] prepare for next development iteration""Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit 71ec0140f7e121bdba3d19e8219e91a5e9d1e320.
* Revert "Revert "[maven-release-plugin] prepare release v1.1.0-rc1""Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit 56070f12f455bae645cba887a74c72b12f1085f8.
* Revert "[maven-release-plugin] prepare release v1.1.0-rc1"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit da4b94c86c9dd0d624b3040aa4b9449be9f60fc3.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit 96926c5a42c5970ed74c50db5bd9c68cacf92207.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-281-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-rc1Patrick Wendell2014-08-281-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.1.0-rc1"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit 79e86ef3e1a3ee03a7e3b166a5c7dee11c6d60d7.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-281-1/+1
| | | | This reverts commit a118ea5c59d653f5a3feda21455ba60bc722b3b1.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-281-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-rc1Patrick Wendell2014-08-281-1/+1
|
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-271-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-rc1Patrick Wendell2014-08-271-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.1.0-snapshot2"Patrick Wendell2014-08-271-1/+1
| | | | This reverts commit e1535ad3c6f7400f2b7915ea91da9c60510557ba.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-271-1/+1
| | | | This reverts commit 9af3fb7385d1f9f221962f1d2d725ff79bd82033.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-211-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-snapshot2Patrick Wendell2014-08-211-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.1.0-snapshot1"Patrick Wendell2014-08-201-1/+1
| | | | This reverts commit d428d88418d385d1d04e1b0adcb6b068efe9c7b0.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-08-201-1/+1
| | | | This reverts commit c204a742a9eb9d3fd318e0f059bd00cbfb8b2c14.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-08-071-1/+1
|
* [maven-release-plugin] prepare release v1.1.0-snapshot1Patrick Wendell2014-08-071-1/+1
|
* Updating versions for Spark 1.1.0Patrick Wendell2014-08-061-1/+1
|
* [SPARK-2157] Enable tight firewall rules for SparkAndrew Or2014-08-061-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The goal of this PR is to allow users of Spark to write tight firewall rules for their clusters. This is currently not possible because Spark uses random ports in many places, notably the communication between executors and drivers. The changes in this PR are based on top of ash211's changes in #1107. The list covered here may or may not be the complete set of port needed for Spark to operate perfectly. However, as of the latest commit there are no known sources of random ports (except in tests). I have not documented a few of the more obscure configs. My spark-env.sh looks like this: ``` export SPARK_MASTER_PORT=6060 export SPARK_WORKER_PORT=7070 export SPARK_MASTER_WEBUI_PORT=9090 export SPARK_WORKER_WEBUI_PORT=9091 ``` and my spark-defaults.conf looks like this: ``` spark.master spark://andrews-mbp:6060 spark.driver.port 5001 spark.fileserver.port 5011 spark.broadcast.port 5021 spark.replClassServer.port 5031 spark.blockManager.port 5041 spark.executor.port 5051 ``` Author: Andrew Or <andrewor14@gmail.com> Author: Andrew Ash <andrew@andrewash.com> Closes #1777 from andrewor14/configure-ports and squashes the following commits: 621267b [Andrew Or] Merge branch 'master' of github.com:apache/spark into configure-ports 8a6b820 [Andrew Or] Use a random UI port during tests 7da0493 [Andrew Or] Fix tests 523c30e [Andrew Or] Add test for isBindCollision b97b02a [Andrew Or] Minor fixes c22ad00 [Andrew Or] Merge branch 'master' of github.com:apache/spark into configure-ports 93d359f [Andrew Or] Executors connect to wrong port when collision occurs d502e5f [Andrew Or] Handle port collisions when creating Akka systems a2dd05c [Andrew Or] Patrick's comment nit 86461e2 [Andrew Or] Remove spark.executor.env.port and spark.standalone.client.port 1d2d5c6 [Andrew Or] Fix ports for standalone cluster mode cb3be88 [Andrew Or] Various doc fixes (broken link, format etc.) e837cde [Andrew Or] Remove outdated TODOs bfbab28 [Andrew Or] Merge branch 'master' of github.com:apache/spark into configure-ports de1b207 [Andrew Or] Update docs to reflect new ports b565079 [Andrew Or] Add spark.ports.maxRetries 2551eb2 [Andrew Or] Remove spark.worker.watcher.port 151327a [Andrew Or] Merge branch 'master' of github.com:apache/spark into configure-ports 9868358 [Andrew Or] Add a few miscellaneous ports 6016e77 [Andrew Or] Add spark.executor.port 8d836e6 [Andrew Or] Also document SPARK_{MASTER/WORKER}_WEBUI_PORT 4d9e6f3 [Andrew Or] Fix super subtle bug 3f8e51b [Andrew Or] Correct erroneous docs... e111d08 [Andrew Or] Add names for UI services 470f38c [Andrew Or] Special case non-"Address already in use" exceptions 1d7e408 [Andrew Or] Treat 0 ports specially + return correct ConnectionManager port ba32280 [Andrew Or] Minor fixes 6b550b0 [Andrew Or] Assorted fixes 73fbe89 [Andrew Or] Move start service logic to Utils ec676f4 [Andrew Or] Merge branch 'SPARK-2157' of github.com:ash211/spark into configure-ports 038a579 [Andrew Ash] Trust the server start function to report the port the service started on 7c5bdc4 [Andrew Ash] Fix style issue 0347aef [Andrew Ash] Unify port fallback logic to a single place 24a4c32 [Andrew Ash] Remove type on val to match surrounding style 9e4ad96 [Andrew Ash] Reformat for style checker 5d84e0e [Andrew Ash] Document new port configuration options 066dc7a [Andrew Ash] Fix up HttpServer port increments cad16da [Andrew Ash] Add fallover increment logic for HttpServer c5a0568 [Andrew Ash] Fix ConnectionManager to retry with increment b80d2fd [Andrew Ash] Make Spark's block manager port configurable 17c79bb [Andrew Ash] Add a configuration option for spark-shell's class server f34115d [Andrew Ash] SPARK-1176 Add port configuration for HttpBroadcast 49ee29b [Andrew Ash] SPARK-1174 Add port configuration for HttpFileServer 1c0981a [Andrew Ash] Make port in HttpServer configurable (cherry picked from commit 09f7e4587bbdf74207d2629e8c1314f93d865999) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* [SPARK-2454] Do not ship spark home to WorkersAndrew Or2014-08-021-3/+0
| | | | | | | | | | | | | | | | | When standalone Workers launch executors, they inherit the Spark home set by the driver. This means if the worker machines do not share the same directory structure as the driver node, the Workers will attempt to run scripts (e.g. bin/compute-classpath.sh) that do not exist locally and fail. This is a common scenario if the driver is launched from outside of the cluster. The solution is to simply not pass the driver's Spark home to the Workers. This PR further makes an attempt to avoid overloading the usages of `spark.home`, which is now only used for setting executor Spark home on Mesos and in python. This is based on top of #1392 and originally reported by YanTangZhai. Tested on standalone cluster. Author: Andrew Or <andrewor14@gmail.com> Closes #1734 from andrewor14/spark-home-reprise and squashes the following commits: f71f391 [Andrew Or] Revert changes in python 1c2532c [Andrew Or] Merge branch 'master' of github.com:apache/spark into spark-home-reprise 188fc5d [Andrew Or] Avoid using spark.home where possible 09272b7 [Andrew Or] Always use Worker's working directory as spark home
* SPARK-2632, SPARK-2576. Fixed by only importing what is necessary during ↵Prashant Sharma2014-07-315-5/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | class definition. Without this patch, it imports everything available in the scope. ```scala scala> val a = 10l val a = 10l a: Long = 10 scala> import a._ import a._ import a._ scala> case class A(a: Int) // show case class A(a: Int) // show class $read extends Serializable { def <init>() = { super.<init>; () }; class $iwC extends Serializable { def <init>() = { super.<init>; () }; class $iwC extends Serializable { def <init>() = { super.<init>; () }; import org.apache.spark.SparkContext._; class $iwC extends Serializable { def <init>() = { super.<init>; () }; val $VAL5 = $line5.$read.INSTANCE; import $VAL5.$iw.$iw.$iw.$iw.a; class $iwC extends Serializable { def <init>() = { super.<init>; () }; import a._; class $iwC extends Serializable { def <init>() = { super.<init>; () }; class $iwC extends Serializable { def <init>() = { super.<init>; () }; case class A extends scala.Product with scala.Serializable { <caseaccessor> <paramaccessor> val a: Int = _; def <init>(a: Int) = { super.<init>; () } } }; val $iw = new $iwC.<init> }; val $iw = new $iwC.<init> }; val $iw = new $iwC.<init> }; val $iw = new $iwC.<init> }; val $iw = new $iwC.<init> }; val $iw = new $iwC.<init> } object $read extends scala.AnyRef { def <init>() = { super.<init>; () }; val INSTANCE = new $read.<init> } defined class A ``` With this patch, it just imports only the necessary. ```scala scala> val a = 10l val a = 10l a: Long = 10 scala> import a._ import a._ import a._ scala> case class A(a: Int) // show case class A(a: Int) // show class $read extends Serializable { def <init>() = { super.<init>; () }; class $iwC extends Serializable { def <init>() = { super.<init>; () }; class $iwC extends Serializable { def <init>() = { super.<init>; () }; case class A extends scala.Product with scala.Serializable { <caseaccessor> <paramaccessor> val a: Int = _; def <init>(a: Int) = { super.<init>; () } } }; val $iw = new $iwC.<init> }; val $iw = new $iwC.<init> } object $read extends scala.AnyRef { def <init>() = { super.<init>; () }; val INSTANCE = new $read.<init> } defined class A scala> ``` This patch also adds a `:fallback` mode on being enabled it will restore the spark-shell's 1.0.0 behaviour. Author: Prashant Sharma <scrapcodes@gmail.com> Author: Yin Huai <huai@cse.ohio-state.edu> Author: Prashant Sharma <prashant.s@imaginea.com> Closes #1635 from ScrapCodes/repl-fix-necessary-imports and squashes the following commits: b1968d2 [Prashant Sharma] Added toschemaRDD to test case. 0b712bb [Yin Huai] Add a REPL test to test importing a method. 02ad8ff [Yin Huai] Add a REPL test for importing SQLContext.createSchemaRDD. ed6d0c7 [Prashant Sharma] Added a fallback mode, incase users run into issues while using repl. b63d3b2 [Prashant Sharma] SPARK-2632, SPARK-2576. Fixed by only importing what is necessary during class definition.
* [SPARK-2762] SparkILoop leaks memory in multi-repl configurationsTimothy Hunter2014-07-311-19/+20
| | | | | | | | | | This pull request is a small refactor so that a partial function (hence a closure) is not created. Instead, a regular function is used. The behavior of the code is not changed. Author: Timothy Hunter <timhunter@databricks.com> Closes #1674 from thunterdb/closure_issue and squashes the following commits: e1e664d [Timothy Hunter] simplify closure
* [SPARK-2452] Create a new valid for each instead of using lineId.Prashant Sharma2014-07-222-2/+17
| | | | | | | | | Author: Prashant Sharma <prashant@apache.org> Closes #1441 from ScrapCodes/SPARK-2452/multi-statement and squashes the following commits: 26c5c72 [Prashant Sharma] Added a test case. 7e8d28d [Prashant Sharma] SPARK-2452, create a new valid for each instead of using lineId, because Line ids can be same sometimes.
* [SPARK-1776] Have Spark's SBT build read dependencies from Maven.Prashant Sharma2014-07-101-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Patch introduces the new way of working also retaining the existing ways of doing things. For example build instruction for yarn in maven is `mvn -Pyarn -PHadoop2.2 clean package -DskipTests` in sbt it can become `MAVEN_PROFILES="yarn, hadoop-2.2" sbt/sbt clean assembly` Also supports `sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 clean assembly` Author: Prashant Sharma <prashant.s@imaginea.com> Author: Patrick Wendell <pwendell@gmail.com> Closes #772 from ScrapCodes/sbt-maven and squashes the following commits: a8ac951 [Prashant Sharma] Updated sbt version. 62b09bb [Prashant Sharma] Improvements. fa6221d [Prashant Sharma] Excluding sql from mima 4b8875e [Prashant Sharma] Sbt assembly no longer builds tools by default. 72651ca [Prashant Sharma] Addresses code reivew comments. acab73d [Prashant Sharma] Revert "Small fix to run-examples script." ac4312c [Prashant Sharma] Revert "minor fix" 6af91ac [Prashant Sharma] Ported oldDeps back. + fixes issues with prev commit. 65cf06c [Prashant Sharma] Servelet API jars mess up with the other servlet jars on the class path. 446768e [Prashant Sharma] minor fix 89b9777 [Prashant Sharma] Merge conflicts d0a02f2 [Prashant Sharma] Bumped up pom versions, Since the build now depends on pom it is better updated there. + general cleanups. dccc8ac [Prashant Sharma] updated mima to check against 1.0 a49c61b [Prashant Sharma] Fix for tools jar a2f5ae1 [Prashant Sharma] Fixes a bug in dependencies. cf88758 [Prashant Sharma] cleanup 9439ea3 [Prashant Sharma] Small fix to run-examples script. 96cea1f [Prashant Sharma] SPARK-1776 Have Spark's SBT build read dependencies from Maven. 36efa62 [Patrick Wendell] Set project name in pom files and added eclipse/intellij plugins. 4973dbd [Patrick Wendell] Example build using pom reader.
* [SPARK-1199][REPL] Remove VALId and use the original import style for ↵Prashant Sharma2014-07-043-11/+31
| | | | | | | | | | | | defined classes. This is an alternate solution to #1176. Author: Prashant Sharma <prashant.s@imaginea.com> Closes #1179 from ScrapCodes/SPARK-1199/repl-fix-second-approach and squashes the following commits: 820b34b [Prashant Sharma] Here we generate two kinds of import wrappers based on whether it is a class or not.
* [SPARK-1841]: update scalatest to version 2.1.5witgo2014-06-061-2/+4
| | | | | | | | | | | | | | | | | | | | Author: witgo <witgo@qq.com> Closes #713 from witgo/scalatest and squashes the following commits: b627a6a [witgo] merge master 51fb3d6 [witgo] merge master 3771474 [witgo] fix RDDSuite 996d6f9 [witgo] fix TimeStampedWeakValueHashMap test 9dfa4e7 [witgo] merge bug 1479b22 [witgo] merge master 29b9194 [witgo] fix code style 022a7a2 [witgo] fix test dependency a52c0fa [witgo] fix test dependency cd8f59d [witgo] Merge branch 'master' of https://github.com/apache/spark into scalatest 046540d [witgo] fix RDDSuite.scala 2c543b9 [witgo] fix ReplSuite.scala c458928 [witgo] update scalatest to version 2.1.5
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-12 21:20:57 +0000