| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rendering markdown
# What changes were proposed in this pull request?
It seems there are several non-breaking spaces were inserted into several `.md`s and they look breaking rendering markdown files.
These are different. For example, this can be checked via `python` as below:
```python
>>> " "
'\xc2\xa0'
>>> " "
' '
```
_Note that it seems this PR description automatically replaces non-breaking spaces into normal spaces. Please open a `vi` and copy and paste it into `python` to verify this (do not copy the characters here)._
I checked the output below in Sapari and Chrome on Mac OS and, Internal Explorer on Windows 10.
**Before**
![2017-04-03 12 37 17](https://cloud.githubusercontent.com/assets/6477701/24594655/50aaba02-186a-11e7-80bb-d34b17a3398a.png)
![2017-04-03 12 36 57](https://cloud.githubusercontent.com/assets/6477701/24594654/50a855e6-186a-11e7-94e2-661e56544b0f.png)
**After**
![2017-04-03 12 36 46](https://cloud.githubusercontent.com/assets/6477701/24594657/53c2545c-186a-11e7-9a73-00529afbfd75.png)
![2017-04-03 12 36 31](https://cloud.githubusercontent.com/assets/6477701/24594658/53c286c0-186a-11e7-99c9-e66b1f510fe7.png)
## How was this patch tested?
Manually checking.
These instances were found via
```
grep --include=*.scala --include=*.python --include=*.java --include=*.r --include=*.R --include=*.md --include=*.r -r -I " " .
```
in Mac OS.
It seems there are several instances more as below:
```
./docs/sql-programming-guide.md: │ ├── ...
./docs/sql-programming-guide.md: │ │
./docs/sql-programming-guide.md: │ ├── country=US
./docs/sql-programming-guide.md: │ │ └── data.parquet
./docs/sql-programming-guide.md: │ ├── country=CN
./docs/sql-programming-guide.md: │ │ └── data.parquet
./docs/sql-programming-guide.md: │ └── ...
./docs/sql-programming-guide.md: ├── ...
./docs/sql-programming-guide.md: │
./docs/sql-programming-guide.md: ├── country=US
./docs/sql-programming-guide.md: │ └── data.parquet
./docs/sql-programming-guide.md: ├── country=CN
./docs/sql-programming-guide.md: │ └── data.parquet
./docs/sql-programming-guide.md: └── ...
./sql/core/src/test/README.md:│ ├── *.avdl # Testing Avro IDL(s)
./sql/core/src/test/README.md:│ └── *.avpr # !! NO TOUCH !! Protocol files generated from Avro IDL(s)
./sql/core/src/test/README.md:│ ├── gen-avro.sh # Script used to generate Java code for Avro
./sql/core/src/test/README.md:│ └── gen-thrift.sh # Script used to generate Java code for Thrift
```
These seems generated via `tree` command which inserts non-breaking spaces. They do not look causing any problem for rendering within code blocks and I did not fix it to reduce the overhead to manually replace it when it is overwritten via `tree` command in the future.
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #17517 from HyukjinKwon/non-breaking-space.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
## What changes were proposed in this pull request?
According to the notice of the following Wiki front page, we can remove the obsolete wiki pointer safely in `README.md` and `docs/index.md`, too. These two lines are the last occurrence of that links.
```
All current wiki content has been merged into pages at http://spark.apache.org as of November 2016.
Each page links to the new location of its information on the Spark web site.
Obsolete wiki content is still hosted here, but carries a notice that it is no longer current.
```
## How was this patch tested?
Manual.
- `README.md`: https://github.com/dongjoon-hyun/spark/tree/remove_wiki_from_readme
- `docs/index.md`:
```
cd docs
SKIP_API=1 jekyll build
```
![screen shot 2016-12-09 at 2 53 29 pm](https://cloud.githubusercontent.com/assets/9700541/21067323/517252e2-be1f-11e6-85b1-2a4471131c5d.png)
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #16239 from dongjoon-hyun/remove_wiki_from_readme.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
## What changes were proposed in this pull request?
"Useful Developer Tools" link inside [README.md](https://github.com/apache/spark/blob/master/README.md#building-spark) doesn't work on master branch. This pr corrects this Markdown link.
## How was this patch tested?
[README.md](https://github.com/linbojin/spark/blob/fix-markdown-link-in-readme/README.md#building-spark) on this branch
![image](https://cloud.githubusercontent.com/assets/5894707/20864124/4c83499e-ba1e-11e6-9948-07b4627f516f.png)
srowen
Author: linbojin <linbojin203@gmail.com>
Closes #16132 from linbojin/fix-markdown-link-in-readme.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
## What changes were proposed in this pull request?
Updates links to the wiki to links to the new location of content on spark.apache.org.
## How was this patch tested?
Doc builds
Author: Sean Owen <sowen@cloudera.com>
Closes #15967 from srowen/SPARK-18073.1.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
and some warnings in PULL_REQUEST_TEMPLATE
## What changes were proposed in this pull request?
Link to contributing wiki in PR template, README.md
## How was this patch tested?
Doc-only change, tested by Jekyll
Author: Sean Owen <sowen@cloudera.com>
Closes #15429 from srowen/SPARK-17840.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
## What changes were proposed in this pull request?
We should mention that users can build Spark using multiple threads to decrease build times; either here or in "Building Spark"
## How was this patch tested?
Built on machines with between one core to 192 cores using mvn -T 1C and observed faster build times with no loss in stability
In response to the question here https://issues.apache.org/jira/browse/SPARK-15821 I think we should suggest this option as we know it works for Spark and can result in faster builds
Author: Adam Roberts <aroberts@uk.ibm.com>
Closes #13562 from a-roberts/patch-3.
|
|
|
|
|
|
|
|
| |
These links make it easier for new developers to work with Spark in their IDE.
Author: kaklakariada <kaklakariada@users.noreply.github.com>
Closes #10104 from kaklakariada/readme-developing-ide-gettting-started.
|
|
|
|
|
|
|
|
|
|
| |
Remove Hadoop third party distro page, and move Hadoop cluster config info to configuration page
CC pwendell
Author: Sean Owen <sowen@cloudera.com>
Closes #9298 from srowen/SPARK-11305.
|
|
|
|
|
|
|
|
|
|
|
|
| |
YARN, -master yarn --deploy-mode x vs -master yarn-x'.
Recommend `--master yarn --deploy-mode {cluster,client}` consistently in docs.
Follow-on to https://github.com/apache/spark/pull/8385
CC nssalian
Author: Sean Owen <sowen@cloudera.com>
Closes #8968 from srowen/SPARK-9570.
|
|
|
|
|
|
|
|
| |
… main README.
Author: Stephen Hopper <shopper@shopper-osx.local>
Closes #8646 from enragedginger/master.
|
|
|
|
|
|
|
|
|
|
| |
Also cut trailing whitespaces.
Author: Reynold Xin <rxin@databricks.com>
Closes #6548 from rxin/readme and squashes the following commits:
630efc3 [Reynold Xin] Update README to include DataFrames and zinc.
|
|
|
|
|
|
|
|
| |
Author: Sean Owen <sowen@cloudera.com>
Closes #6063 from srowen/FixRunningTestsLink and squashes the following commits:
db62018 [Sean Owen] Fix the link to test building info on the wiki
|
|
|
|
|
|
|
|
|
|
|
|
| |
The link to "Specifying the Hadoop Version" currently points to http://spark.apache.org/docs/latest/building-with-maven.html#specifying-the-hadoop-version.
The correct link is: http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version
Author: Theodore Vasiloudis <tvas@sics.se>
Closes #4999 from thvasilo/patch-1 and squashes the following commits:
c34aea8 [Theodore Vasiloudis] Fix dead link in Readme
|
|
|
|
|
|
|
|
| |
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #4312 from nchammas/patch-2 and squashes the following commits:
9d943aa [Nicholas Chammas] [Docs] Fix Building Spark link text
|
|
|
|
|
|
|
|
|
|
| |
Corrected link to the Building Spark with Maven page from its original (http://spark.apache.org/docs/latest/building-with-maven.html) to the current page (http://spark.apache.org/docs/latest/building-spark.html)
Author: Denny Lee <denny.g.lee@gmail.com>
Closes #3802 from dennyglee/patch-1 and squashes the following commits:
15f601a [Denny Lee] Update README.md
|
|
|
|
|
|
|
|
|
|
| |
This is a trivial change to add links to the wiki from `README.md` and the main docs page. It is already linked to from spark.apache.org.
Author: Sean Owen <sowen@cloudera.com>
Closes #3169 from srowen/SPARK-971 and squashes the following commits:
dcb84d0 [Sean Owen] Add link to wiki from README, docs home page
|
|
|
|
|
|
|
|
|
|
|
|
| |
seems like `building-spark.html` was renamed to `building-with-maven.html`?
Is Maven the blessed build tool these days, or SBT? I couldn't find a building-with-sbt page so I went with the Maven one here.
Author: Ryan Williams <ryan.blake.williams@gmail.com>
Closes #2859 from ryan-williams/broken-links-readme and squashes the following commits:
7692253 [Ryan Williams] fix broken links in README.md
|
| |
|
|
|
|
|
|
|
|
| |
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2414 from nchammas/patch-1 and squashes the following commits:
14664bf [Nicholas Chammas] [Docs] minor punctuation fix
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Here's my crack at Bertrand's suggestion. The Github `README.md` contains build info that's outdated. It should just point to the current online docs, and reflect that Maven is the primary build now.
(Incidentally, the stanza at the end about contributions of original work should go in https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark too. It won't hurt to be crystal clear about the agreement to license, given that ICLAs are not required of anyone here.)
Author: Sean Owen <sowen@cloudera.com>
Closes #2014 from srowen/SPARK-3069 and squashes the following commits:
501507e [Sean Owen] Note that Zinc is for Maven builds too
db2bd97 [Sean Owen] sbt -> sbt/sbt and add note about zinc
be82027 [Sean Owen] Fix additional occurrences of building-with-maven -> building-spark
91c921f [Sean Owen] Move building-with-maven to building-spark and create a redirect. Update doc links to building-spark.html Add jekyll-redirect-from plugin and make associated config changes (including fixing pygments deprecation). Add example of SBT to README.md
999544e [Sean Owen] Change "Building Spark with Maven" title to "Building Spark"; reinstate tl;dr info about dev/run-tests in README.md; add brief note about building with SBT
c18d140 [Sean Owen] Optionally, remove the copy of contributing text from main README.md
8e83934 [Sean Owen] Add CONTRIBUTING.md to trigger notice on new pull request page
b1c04a1 [Sean Owen] Refer to current online documentation for building, and remove slightly outdated copy in README.md
|
|
|
|
|
|
|
|
|
|
| |
Also make the list of features consistent in style.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2278 from nchammas/patch-1 and squashes the following commits:
56df319 [Nicholas Chammas] [Docs] fix minor MLlib case typo
|
|
|
|
|
|
|
|
|
|
|
|
| |
The Contributing to Spark guide [recommends](https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-AutomatedTesting) running tests by calling `./dev/run-tests`. The README should, too.
`./sbt/sbt test` does not cover Python tests or style tests.
Author: nchammas <nicholas.chammas@gmail.com>
Closes #2149 from nchammas/patch-2 and squashes the following commits:
2b3b132 [nchammas] [Docs] Run tests like in contributing guide
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
and Thrift JDBC server is absent in proper document -
The most important things I mentioned in #1885 is as follows.
* People who build Spark is not always programmer.
* If a person who build Spark is not a programmer, he/she won't read programmer's guide before building.
So, how to build for using CLI and JDBC server is not only in programmer's guide.
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes #2080 from sarutak/SPARK-2963 and squashes the following commits:
ee07c76 [Kousuke Saruta] Modified regression of the description about building for using Thrift JDBC server and CLI
ed53329 [Kousuke Saruta] Modified description and notaton of proper noun
07c59fc [Kousuke Saruta] Added a description about how to build to use HiveServer and CLI for SparkSQL to building-with-maven.md
6e6645a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2963
c88fa93 [Kousuke Saruta] Added a description about building to use HiveServer and CLI for SparkSQL
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently we have a separate profile called hive-thriftserver. I originally suggested this in case users did not want to bundle the thriftserver, but it's ultimately lead to a lot of confusion. Since the thriftserver is only a few classes, I don't see a really good reason to isolate it from the rest of Hive. So let's go ahead and just include it in the same profile to simplify things.
This has been suggested in the past by liancheng.
Author: Patrick Wendell <pwendell@gmail.com>
Closes #2006 from pwendell/hiveserver and squashes the following commits:
742ea40 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into hiveserver
034ad47 [Patrick Wendell] SPARK-3092: Always include the thriftserver when -Phive is enabled.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
and CLI for SparkSQL
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes #1885 from sarutak/SPARK-2963 and squashes the following commits:
ed53329 [Kousuke Saruta] Modified description and notaton of proper noun
07c59fc [Kousuke Saruta] Added a description about how to build to use HiveServer and CLI for SparkSQL to building-with-maven.md
6e6645a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2963
c88fa93 [Kousuke Saruta] Added a description about building to use HiveServer and CLI for SparkSQL
|
| |
|
|
|
|
|
| |
(cherry picked from commit 401083be9f010f95110a819a49837ecae7d9c4ec)
Signed-off-by: Reynold Xin <rxin@apache.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now, we should use -Pyarn instead of SPARK_YARN when building but README says as follows.
For Apache Hadoop 2.2.X, 2.1.X, 2.0.X, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions
with YARN, also set `SPARK_YARN=true`:
# Apache Hadoop 2.0.5-alpha
$ sbt/sbt -Dhadoop.version=2.0.5-alpha -Pyarn assembly
# Cloudera CDH 4.2.0 with MapReduce v2
$ sbt/sbt -Dhadoop.version=2.0.0-cdh4.2.0 -Pyarn assembly
# Apache Hadoop 2.2.X and newer
$ sbt/sbt -Dhadoop.version=2.2.0 -Pyarn assembly
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes #1382 from sarutak/SPARK-2457 and squashes the following commits:
e7b2d64 [Kousuke Saruta] Replaced "SPARK_YARN=true" with "-Pyarn" in README
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Look for JARs in the right place
- Launch examples the same way as on Unix
- Load datanucleus JARs if they exist
- Don't attempt to parse local paths as URIs in SparkSubmit, since paths with C:\ are not valid URIs
- Also fixed POM exclusion rules for datanucleus (it wasn't properly excluding it, whereas SBT was)
Author: Matei Zaharia <matei@databricks.com>
Closes #819 from mateiz/win-fixes and squashes the following commits:
d558f96 [Matei Zaharia] Fix comment
228577b [Matei Zaharia] Review comments
d3b71c7 [Matei Zaharia] Properly exclude datanucleus files in Maven assembly
144af84 [Matei Zaharia] Update Windows scripts to match latest binary package layout
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Gives a nicely formatted message to the user when `run-example` is run to
tell them to use `spark-submit`.
Author: Patrick Wendell <pwendell@gmail.com>
Closes #704 from pwendell/examples and squashes the following commits:
1996ee8 [Patrick Wendell] Feedback form Andrew
3eb7803 [Patrick Wendell] Suggestions from TD
2474668 [Patrick Wendell] SPARK-1565 (Addendum): Replace `run-example` with `spark-submit`.
|
|
|
|
|
|
|
|
|
| |
Author: Reynold Xin <rxin@apache.org>
Closes #443 from rxin/readme and squashes the following commits:
16853de [Reynold Xin] Updated SBT and Scala instructions.
3ac3ceb [Reynold Xin] README update
|
|
|
|
|
|
|
|
| |
Author: Reynold Xin <rxin@apache.org>
Closes #1 from rxin/readme and squashes the following commits:
b3a77cd [Reynold Xin] Removed reference to incubation in README.md.
|
|
|
| |
The link does not work otherwise.
|
| |
|
| |
|
| |
|
|\
| |
| |
| |
| |
| | |
Conflicts:
core/src/test/scala/org/apache/spark/DriverSuite.scala
docs/python-programming-guide.md
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| | |
Closes #316
|
| | |
|
|/ |
|
| |
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
core/pom.xml
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
pom.xml
project/SparkBuild.scala
streaming/pom.xml
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
|
| |
| |
| |
| |
| |
| | |
This is misleading because the build doesn't source that file. IMO
it's better to force people to specify build environment variables
on the command line always, like we do in every example.
|
| | |
|
|\| |
|