1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
|
<!DOCTYPE html>
<!--[if IE 6]>
<html id="ie6" dir="ltr" lang="en-US">
<![endif]-->
<!--[if IE 7]>
<html id="ie7" dir="ltr" lang="en-US">
<![endif]-->
<!--[if IE 8]>
<html id="ie8" dir="ltr" lang="en-US">
<![endif]-->
<!--[if !(IE 6) | !(IE 7) | !(IE 8) ]><!-->
<html dir="ltr" lang="en-US">
<!--<![endif]-->
<head>
<link rel="shortcut icon" href="favicon.ico" />
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width" />
<title>Documentation | Spark</title>
<link rel="stylesheet" type="text/css" media="all" href="/css/style.css" />
<link rel="stylesheet" href="/css/pygments-default.css">
<script type="text/javascript">//<![CDATA[
// Google Analytics for WordPress by Yoast v4.2.8 | http://yoast.com/wordpress/google-analytics/
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-32518208-1']);
_gaq.push(['_trackPageview']);
(function () {
var ga = document.createElement('script');
ga.type = 'text/javascript';
ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ga, s);
})();
//]]></script>
<link rel='canonical' href='/index.html' />
<style type="text/css">
#site-title,
#site-description {
position: absolute !important;
clip: rect(1px 1px 1px 1px); /* IE6, IE7 */
clip: rect(1px, 1px, 1px, 1px);
}
</style>
<style type="text/css" id="custom-background-css">
body.custom-background { background-color: #f1f1f1; }
</style>
</head>
<!--body class="page singular"-->
<body class="page singular">
<div id="page" class="hfeed">
<header id="branding" role="banner">
<hgroup>
<h1 id="site-title"><span><a href="/" title="Spark" rel="home">Spark</a></span></h1>
<h2 id="site-description">Lightning-Fast Cluster Computing</h2>
</hgroup>
<a href="/">
<img src="/images/spark-project-header1.png" width="1000" height="220" alt="Spark: Lightning-Fast Cluster Computing" title="Spark: Lightning-Fast Cluster Computing" />
</a>
<nav id="access" role="navigation">
<h3 class="assistive-text">Main menu</h3>
<div class="menu-main-menu-container">
<ul id="menu-main-menu" class="menu">
<li class="menu-item menu-item-type-post_type menu-item-object-page ">
<a href="/index.html">Home</a>
</li>
<li class="menu-item menu-item-type-post_type menu-item-object-page ">
<a href="/downloads.html">Downloads</a>
</li>
<li class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item">
<a href="/documentation.html">Documentation</a>
</li>
<li class="menu-item menu-item-type-post_type menu-item-object-page ">
<a href="/examples.html">Examples</a>
</li>
<li class="menu-item menu-item-type-post_type menu-item-object-page ">
<a href="/mailing-lists.html">Mailing Lists</a>
</li>
<li class="menu-item menu-item-type-post_type menu-item-object-page ">
<a href="/research.html">Research</a>
</li>
<li class="menu-item menu-item-type-post_type menu-item-object-page ">
<a href="/faq.html">FAQ</a>
</li>
</ul></div>
</nav><!-- #access -->
</header><!-- #branding -->
<div id="main">
<div id="primary">
<div id="content" role="main">
<article class="page type-page status-publish hentry">
<h2>Spark Documentation</h2>
<p>Setup instructions, programming guides, and other documentation are available for each version of Spark below:</p>
<ul>
<li><a href="/docs/latest/">Spark 0.7.3 (latest release)</a></li>
<li><a href="/docs/0.6.2/">Spark 0.6.2</a></li>
<li><a href="https://github.com/mesos/spark/wiki/Spark-0.5-Documentation">Spark 0.5.x</a> (hosted on GitHub)</li>
</ul>
<p>Read these documents to get started with Spark. In addition, this page lists a few other external resources for learning Spark.</p>
<h3>Video Tutorials</h3>
<ul>
<li><a href="/screencasts/1-first-steps-with-spark.html">Screencast 1: First Steps with Spark</a></li>
<li><a href="/screencasts/2-spark-documentation-overview.html">Screencast 2: Spark Documentation Overview</a></li>
</ul>
<h3>Hands-On Exercises</h3>
<ul>
<li><a href="http://ampcamp.berkeley.edu/big-data-mini-course/">Hands-on exercises from Strata 2013</a> are available online. These exercises let you launch a small EC2 cluster, load a dataset, and query it with Spark, Shark, and Spark Streaming.</li>
</ul>
<h3>AMP Camp Slides and Videos</h3>
<ul>
<li>The AMP Lab hosted a <a href="http://ampcamp.berkeley.edu/agenda-2012/">two-day training camp</a> on Spark, Shark and "big data" analysis in August 2012. <a href="http://ampcamp.berkeley.edu/agenda-2012/">Slides and videos</a> are available online for free.
</li>
</ul>
<h3>Presentations</h3>
<ul>
<li>
<a href="/talks/overview.pptx">Overview of Spark</a> (<a href="/talks/overview.pptx">pptx</a>) (<a href="/talks/overview.pdf">pdf</a>)
</li>
<li>Programming Spark tutorial at AMP Camp: <a href="http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/matei-zaharia-part-1-amp-camp-2012-spark-intro.pdf">part 1 (pdf)</a>, <a href="http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/matei-zaharia-part-2-amp-camp-2012-standalone-programs.pdf">part 2 (pdf)</a>, <a href="http://www.youtube.com/watch?v=7k4yDKBYOcw">video</a>.
</li>
<li><a href="/talks/strata_spark_streaming.ppt">Overview of Spark Streaming</a> (<a href="/talks/strata_spark_streaming.ppt">ppt</a>) (<a href="/talks/strata_spark_streaming.pdf">pdf</a>)
<h3>External Tutorials, Development Blogs, and Talks</h3>
<ul>
<li><a href="http://www.slideshare.net/EvanChan2/cassandra2013-spark-talk-final">Real-time Analytics with Cassandra, Spark, and Shark</a> -- presentation by Evan Chan from Ooyala at the 2013 Cassandra Summit</li>
<li><a href="http://syndeticlogic.net/?p=311">Getting Spark Setup in Eclipse</a> -- Developer blog post by Spark community member James Percent</li>
<li><a href="http://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923">Run Spark and Shark on Amazon Elastic MapReduce</a> -- Article by Amazon AWS Elastic MapReduce team member Parviz Deyhim</li>
<li><a href="http://blog.quantifind.com/posts/spark-unit-test/">Unit testing with Spark</a> -- Quantifind tech blog post by Imran Rashid</li>
<li><a href="http://blog.quantifind.com/posts/logging-post/">Configuring Spark logs</a> -- Quantifind tech blog by Imran Rashid</li>
<li><a href="http://www.ibm.com/developerworks/library/os-spark/">Spark, an alternative for fast data analytics</a> -- IBM Developer Works article by M. Tim Jones</li>
</ul>
<h3>Research Papers</h3>
<ul>
<li>
<a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-214.pdf">Shark: SQL and Rich Analytics at Scale</a>. Reynold Xin, Joshua Rosen, Matei Zaharia, Michael J. Franklin, Scott Shenker, Ion Stoica. <em>Technical Report UCB/EECS-2012-214</em>. November 2012.
</li>
<li>
<a href="http://www.cs.berkeley.edu/~matei/papers/2012/hotcloud_spark_streaming.pdf">Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters</a>. Matei Zaharia, Tathagata Das, Haoyuan Li, Scott Shenker, Ion Stoica. <em>HotCloud 2012</em>. June 2012.
</li>
<li>
<a href="http://www.cs.berkeley.edu/~matei/papers/2012/sigmod_shark_demo.pdf">Shark: Fast Data Analysis Using Coarse-grained Distributed Memory</a> (demo). Cliff Engle, Antonio Lupher, Reynold Xin, Matei Zaharia, Haoyuan Li, Scott Shenker, Ion Stoica. <em>SIGMOD 2012</em>. May 2012. <b>Best Demo Award</b>.
</li>
<li>
<a href="http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf">Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing</a>. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica. <em>NSDI 2012</em>. April 2012. <b>Best Paper Award</b> and <b>Honorable Mention for Community Award</b>.
</li>
<li>
<a href="http://www.cs.berkeley.edu/~matei/papers/2011/tr_spark.pdf">Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing</a>. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica. <em>Technical Report UCB/EECS-2011-82</em>. July 2011.</li>
<li>
<a href="http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf">Spark: Cluster Computing with Working Sets</a>. Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, Ion Stoica. <em>HotCloud 2010</em>. June 2010.
</li>
</ul>
</li></ul>
</article><!-- #post -->
</div><!-- #content -->
<footer id="colophon" role="contentinfo">
<div id="site-generator">
<p>Apache Spark is an effort undergoing incubation at The Apache Software Foundation.
<img src="/images/incubator-logo.png" alt="Apache Incubator" title="Apache Incubator" /></p>
</div>
</footer><!-- #colophon -->
</div><!-- #primary -->
</div><!-- #main -->
</div><!-- #page -->
</body>
</html>
|
