aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* a few left over document changePrashant Sharma2014-01-023-4/+4
|
* pyspark -> bin/pysparkPrashant Sharma2014-01-025-19/+19
|
* run-example -> bin/run-examplePrashant Sharma2014-01-0219-31/+31
|
* spark-shell -> bin/spark-shellPrashant Sharma2014-01-029-15/+15
|
* Merge branch 'scripts-reorg' of github.com:shane-huang/incubator-spark into ↵Prashant Sharma2014-01-0241-96/+90
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | spark-915-segregate-scripts Conflicts: bin/spark-shell core/pom.xml core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala core/src/test/scala/org/apache/spark/DriverSuite.scala python/run-tests sbin/compute-classpath.sh sbin/spark-class sbin/stop-slaves.sh
| * deprecate "spark" script and SPAKR_CLASSPATH environment variableAndrew xia2013-10-127-99/+4
| |
| * refactor $FWD variableAndrew xia2013-09-295-7/+7
| |
| * Merge branch 'reorgscripts' into scripts-reorgshane-huang2013-09-2740-87/+175
| |\
| | * rm bin/spark.cmd as we don't have windows test environment. Will added it ↵shane-huang2013-09-261-27/+0
| | | | | | | | | | | | | | | | | | later if needed Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * fix paths and change spark to use APP_MEM as application driver memory ↵shane-huang2013-09-263-35/+10
| | | | | | | | | | | | | | | | | | instead of SPARK_MEM, user should add application jars to SPARK_CLASSPATH Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * fix pathshane-huang2013-09-261-1/+1
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * add scripts in binshane-huang2013-09-2312-17/+163
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * moved user scripts to bin foldershane-huang2013-09-2311-0/+0
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * add admin scripts to sbinshane-huang2013-09-2314-47/+47
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * added spark-class and spark-executor to sbinshane-huang2013-09-2314-22/+16
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
* | | Merge pull request #309 from mateiz/conf2Patrick Wendell2014-01-01140-941/+1731
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SPARK-544. Migrate configuration to a SparkConf class This is still a work in progress based on Prashant and Evan's code. So far I've done the following: - Got rid of global SparkContext.globalConf - Passed SparkConf to serializers and compression codecs - Made SparkConf public instead of private[spark] - Improved API of SparkContext and SparkConf - Switched executor environment vars to be passed through SparkConf - Fixed some places that were still using system properties - Fixed some tests, though others are still failing This still fails several tests in core, repl and streaming, likely due to properties not being set or cleared correctly (some of the tests run fine in isolation). But the API at least is hopefully ready for review. Unfortunately there was a lot of global stuff before due to a "SparkContext.globalConf" method that let you set a "default" configuration of sorts, which meant I had to make some pretty big changes.
| * | | Fix Python code after change of getOrElseMatei Zaharia2014-01-012-7/+14
| | | |
| * | | Fixed two uses of conf.get with no default value in MesosMatei Zaharia2014-01-012-2/+2
| | | |
| * | | Miscellaneous fixes from code review.Matei Zaharia2014-01-0149-189/+206
| | | | | | | | | | | | | | | | | | | | | | | | Also replaced SparkConf.getOrElse with just a "get" that takes a default value, and added getInt, getLong, etc to make code that uses this simpler later on.
| * | | Merge remote-tracking branch 'apache/master' into conf2Matei Zaharia2014-01-0119-37/+52
| |\ \ \ | |/ / / |/| | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
* | | | Merge pull request #312 from pwendell/log4j-fix-2Patrick Wendell2014-01-0119-37/+52
|\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SPARK-1008: Logging improvments 1. Adds a default log4j file that gets loaded if users haven't specified a log4j file. 2. Isolates use of the tools assembly jar. I found this produced SLF4J warnings after building with SBT (and I've seen similar warnings on the mailing list).
| * \ \ \ Merge remote-tracking branch 'apache-github/master' into log4j-fix-2Patrick Wendell2014-01-0138-465/+803
| |\ \ \ \ | |/ / / / |/| | | | | | | | | | | | | | Conflicts: streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
| * | | | Adding outer checkout when initializing loggingPatrick Wendell2013-12-311-3/+5
| | | | |
| * | | | Tiny typo fixPatrick Wendell2013-12-311-2/+2
| | | | |
| * | | | Removing use in testPatrick Wendell2013-12-311-2/+0
| | | | |
| * | | | Minor fixesPatrick Wendell2013-12-302-3/+3
| | | | |
| * | | | Removing initLogging entirelyPatrick Wendell2013-12-3017-32/+21
| | | | |
| * | | | Response to Shivaram's reviewPatrick Wendell2013-12-302-15/+18
| | | | |
| * | | | SPARK-1008: Logging improvmentsPatrick Wendell2013-12-294-13/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Adds a default log4j file that gets loaded if users haven't specified a log4j file. 2. Isolates use of the tools assembly jar. I found this produced SLF4J warnings after building with SBT (and I've seen similar warnings on the mailing list).
| | * | | Merge remote-tracking branch 'apache/master' into conf2Matei Zaharia2014-01-0112-211/+457
| | |\ \ \ | |_|/ / / |/| | | | | | | | | | | | | | Conflicts: project/SparkBuild.scala
* | | | | Merge pull request #314 from witgo/masterReynold Xin2013-12-312-1356/+240
|\ \ \ \ \ | | | | | | | | | | | | | | | | | | restore core/pom.xml file modification
| * | | | | restore core/pom.xml file modificationliguoqiang2014-01-012-1356/+240
|/ / / / /
* | | | | Merge pull request #73 from falaki/ApproximateDistinctCountReynold Xin2013-12-3112-233/+1595
|\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Approximate distinct count Added countApproxDistinct() to RDD and countApproxDistinctByKey() to PairRDDFunctions to approximately count distinct number of elements and distinct number of values per key, respectively. Both functions use HyperLogLog from stream-lib for counting. Both functions take a parameter that controls the trade-off between accuracy and memory consumption. Also added Scala docs and test suites for both methods.
| * | | | | Made the code more compact and readableHossein Falaki2013-12-313-23/+8
| | | | | |
| * | | | | minor improvementsHossein Falaki2013-12-312-4/+5
| | | | | |
| * | | | | Added Java unit tests for countApproxDistinct and countApproxDistinctByKeyHossein Falaki2013-12-301-0/+32
| | | | | |
| * | | | | Added Java API for countApproxDistinctHossein Falaki2013-12-301-0/+11
| | | | | |
| * | | | | Added Java API for countApproxDistinctByKeyHossein Falaki2013-12-301-0/+36
| | | | | |
| * | | | | Added stream 2.5.1 jar depenencyHossein Falaki2013-12-301-1/+2
| | | | | |
| * | | | | Renamed countDistinct and countDistinctByKey methods to include ApproxHossein Falaki2013-12-305-15/+15
| | | | | |
| * | | | | Using origin versionHossein Falaki2013-12-30374-8424/+19051
| |\ \ \ \ \
| * | | | | | Removed superfluous abs call from test cases.Hossein Falaki2013-12-101-2/+2
| | | | | | |
| * | | | | | Made SerializableHyperLogLog Externalizable and added Kryo testsHossein Falaki2013-10-182-5/+10
| | | | | | |
| * | | | | | Added stream-lib dependency to Maven buildHossein Falaki2013-10-182-0/+9
| | | | | | |
| * | | | | | Improved code style.Hossein Falaki2013-10-174-15/+19
| | | | | | |
| * | | | | | Fixed document typoHossein Falaki2013-10-172-4/+4
| | | | | | |
| * | | | | | Added dependency on stream-lib version 2.4.0 for approximate distinct count ↵Hossein Falaki2013-10-171-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | support.
| * | | | | | Added countDistinctByKey to PairRDDFunctions that counts the approximate ↵Hossein Falaki2013-10-172-0/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | number of unique values for each key in the RDD.
| * | | | | | Added a countDistinct method to RDD that takes takes an accuracy parameter ↵Hossein Falaki2013-10-172-1/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | and returns the (approximate) number of distinct elements in the RDD.
| * | | | | | Added a serializable wrapper for HyperLogLogHossein Falaki2013-10-171-0/+44
| | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-11 07:02:31 +0000