aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorAndrew Or <andrew@databricks.com>2015-07-09 13:25:11 -0700
committerAndrew Or <andrew@databricks.com>2015-07-09 13:25:11 -0700
commitebdf58538058e57381c04b6725d4be0c37847ed3 (patch)
treea2fdad7fdc56cc74fa05d7e5dafb2896c3e733fa
parent88bf430331eef3c02438ca441616034486e15789 (diff)
downloadspark-ebdf58538058e57381c04b6725d4be0c37847ed3.tar.gz
spark-ebdf58538058e57381c04b6725d4be0c37847ed3.tar.bz2
spark-ebdf58538058e57381c04b6725d4be0c37847ed3.zip
[SPARK-2017] [UI] Stage page hangs with many tasks
(This reopens a patch that was closed in the past: #6248) When you view the stage page while running the following: ``` sc.parallelize(1 to X, 10000).count() ``` The page never loads, the job is stalled, and you end up running into an OOM: ``` HTTP ERROR 500 Problem accessing /stages/stage/. Reason: Server Error Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2367) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) ``` This patch compresses Jetty responses in gzip. The correct long-term fix is to add pagination. Author: Andrew Or <andrew@databricks.com> Closes #7296 from andrewor14/gzip-jetty and squashes the following commits: a051c64 [Andrew Or] Use GZIP to compress Jetty responses
-rw-r--r--core/src/main/scala/org/apache/spark/ui/JettyUtils.scala10
1 files changed, 8 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
index 06e616220c..f413c1d37f 100644
--- a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
+++ b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
@@ -210,10 +210,16 @@ private[spark] object JettyUtils extends Logging {
conf: SparkConf,
serverName: String = ""): ServerInfo = {
- val collection = new ContextHandlerCollection
- collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
+ val collection = new ContextHandlerCollection
+ val gzipHandlers = handlers.map { h =>
+ val gzipHandler = new GzipHandler
+ gzipHandler.setHandler(h)
+ gzipHandler
+ }
+ collection.setHandlers(gzipHandlers.toArray)
+
// Bind to the given port, or throw a java.net.BindException if the port is occupied
def connect(currentPort: Int): (Server, Int) = {
val server = new Server(new InetSocketAddress(hostName, currentPort))