diff options
| author | Alex Bozarth <ajbozart@us.ibm.com> | 2016-08-30 16:33:54 -0500 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-08-30 16:33:54 -0500 |
| commit | f7beae6da02e6b70a34c714e93136becbde7679b (patch) | |
| tree | 18c0c1d197da796e636a5479e6347cf5d3480357 /core/src/test/scala | |
| parent | 02ac379e8645ce5d32e033f6683136da16fbe584 (diff) | |
| download | spark-f7beae6da02e6b70a34c714e93136becbde7679b.tar.gz spark-f7beae6da02e6b70a34c714e93136becbde7679b.tar.bz2 spark-f7beae6da02e6b70a34c714e93136becbde7679b.zip | |
[SPARK-17243][WEB UI] Spark 2.0 History Server won't load with very large application history
## What changes were proposed in this pull request?
With the new History Server the summary page loads the application list via the the REST API, this makes it very slow to impossible to load with large (10K+) application history. This pr fixes this by adding the `spark.history.ui.maxApplications` conf to limit the number of applications the History Server displays. This is accomplished using a new optional `limit` param for the `applications` api. (Note this only applies to what the summary page displays, all the Application UI's are still accessible if the user knows the App ID and goes to the Application UI directly.)
I've also added a new test for the `limit` param in `HistoryServerSuite.scala`
## How was this patch tested?
Manual testing and dev/run-tests
Author: Alex Bozarth <ajbozart@us.ibm.com>
Closes #14835 from ajbozarth/spark17243.
Diffstat (limited to 'core/src/test/scala')
| -rw-r--r-- | core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala index 631a7cd9d5..ae3f5d9c01 100644 --- a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala +++ b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala @@ -100,6 +100,7 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers "minDate app list json" -> "applications?minDate=2015-02-10", "maxDate app list json" -> "applications?maxDate=2015-02-10", "maxDate2 app list json" -> "applications?maxDate=2015-02-03T16:42:40.000GMT", + "limit app list json" -> "applications?limit=3", "one app json" -> "applications/local-1422981780767", "one app multi-attempt json" -> "applications/local-1426533911241", "job list json" -> "applications/local-1422981780767/jobs", |