diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-03-17 09:18:57 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-03-17 09:18:57 -0700 |
| commit | 0f673c21f68ee3d5df3c01ae405709d3c1f4909b (patch) | |
| tree | 3d637c661ea26d97ffa1754833f83d792ee15ef2 /docs/configuration.md | |
| parent | e9f22c61290348c58af54c0ae3c6226105126a8d (diff) | |
| download | spark-0f673c21f68ee3d5df3c01ae405709d3c1f4909b.tar.gz spark-0f673c21f68ee3d5df3c01ae405709d3c1f4909b.tar.bz2 spark-0f673c21f68ee3d5df3c01ae405709d3c1f4909b.zip | |
[SPARK-3266] Use intermediate abstract classes to fix type erasure issues in Java APIs
This PR addresses a Scala compiler bug ([SI-8905](https://issues.scala-lang.org/browse/SI-8905)) that was breaking some of the Spark Java APIs. In a nutshell, it seems that methods whose implementations are inherited from generic traits sometimes have their type parameters erased to Object. This was causing methods like `DoubleRDD.min()` to throw confusing NoSuchMethodErrors at runtime.
The fix implemented here is to introduce an intermediate layer of abstract classes and inherit from those instead of directly extends the `Java*Like` traits. This should not break binary compatibility.
I also improved the test coverage of the Java API, adding several new tests for methods that failed at runtime due to this bug.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #5050 from JoshRosen/javardd-si-8905-fix and squashes the following commits:
2feb068 [Josh Rosen] Use intermediate abstract classes to work around SPARK-3266
d5f3e5d [Josh Rosen] Add failing regression tests for SPARK-3266
Diffstat (limited to 'docs/configuration.md')
0 files changed, 0 insertions, 0 deletions