| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
| |
This reverts commit 72a4fdbe82203b962fe776d0edaed7f56898cb02.
|
|
|
|
| |
This reverts commit 685bdd2b7e584c84e7d39e40de2d5f30c5388cb5.
|
| |
|
| |
|
|
|
|
| |
This reverts commit 3f9e073ff0bb18b6079fda419d4e9dbf594545b0.
|
|
|
|
| |
This reverts commit 6de888129fcfe6e592458a4217fc66140747b54f.
|
| |
|
| |
|
|
|
|
| |
This reverts commit 7029301778895427216f2e0710c6e72a523c0897.
|
|
|
|
| |
This reverts commit db22a9e2cb51eae2f8a79648ce3c6bf4fecdd641.
|
| |
|
| |
|
|
|
|
| |
This reverts commit 837deabebf0714e3f3aca135d77169cc825824f3.
|
| |
|
|
|
|
|
|
|
| |
This reverts commit f3e62ffa4ccea62911207b918ef1c23c1f50467f.
Conflicts:
pom.xml
|
|
|
|
| |
This reverts commit 5c0032a471d858fb010b1737ea14375f1af3ed88.
|
|
|
|
|
|
|
| |
This reverts commit 78887f94a0ae9cdcfb851910ab9c7d51a1ef2acb.
Conflicts:
pom.xml
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Author: Michael Armbrust <michael@databricks.com>
Author: ravipesala <ravindra.pesala@huawei.com>
Author: scwf <wangfei1@huawei.com>
Closes #3199 from marmbrus/backport1.1 and squashes the following commits:
019a0dd [Michael Armbrust] Drop incorrectly ported test cases
4c9f3e6 [ravipesala] [SPARK-3708][SQL] Backticks aren't handled correctly is aliases
064750d [scwf] [SPARK-3704][SQL] Fix ColumnValue type for Short values in thrift server
f4e17cd [ravipesala] [SPARK-3834][SQL] Backticks not correctly handled in subquery aliases
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR backports #2843 to branch-1.1. The key difference is that this one doesn't support Hive 0.13.1 and thus always returns `0.12.0` when `spark.sql.hive.version` is queried.
6 other commits on which #2843 depends were also backported, they are:
- #2887 for `SessionState` lifecycle control
- #2675, #2823 & #3060 for major test suite refactoring and bug fixes
- #2164, for Parquet test suites updates
- #2493, for reading `spark.sql.*` configurations
Author: Cheng Lian <lian@databricks.com>
Author: Cheng Lian <lian.cs.zju@gmail.com>
Author: Michael Armbrust <michael@databricks.com>
Closes #3113 from liancheng/get-info-for-1.1 and squashes the following commits:
d354161 [Cheng Lian] Provides Spark and Hive version in HiveThriftServer2 for branch-1.1
0c2a244 [Michael Armbrust] [SPARK-3646][SQL] Copy SQL configuration from SparkConf when a SQLContext is created.
3202a36 [Michael Armbrust] [SQL] Decrease partitions when testing
7f395b7 [Cheng Lian] [SQL] Fixes race condition in CliSuite
0dd28ec [Cheng Lian] [SQL] Fixes the race condition that may cause test failure
5928b39 [Cheng Lian] [SPARK-3809][SQL] Fixes test suites in hive-thriftserver
faeca62 [Cheng Lian] [SPARK-4037][SQL] Removes the SessionState instance created in HiveThriftServer2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
SparkSql crashes on selecting tables using custom serde.
Example:
----------------
CREATE EXTERNAL TABLE table_name PARTITIONED BY ( a int) ROW FORMAT 'SERDE "org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer" with serdeproperties("serialization.format"="org.apache.thrift.protocol.TBinaryProtocol","serialization.class"="ser_class") STORED AS SEQUENCEFILE;
The following exception is seen on running a query like 'select * from table_name limit 1':
ERROR CliDriver: org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
at org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer.initialize(ThriftDeserializer.java:68)
at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:80)
at org.apache.spark.sql.hive.execution.HiveTableScan.addColumnMetadataToConf(HiveTableScan.scala:86)
at org.apache.spark.sql.hive.execution.HiveTableScan.<init>(HiveTableScan.scala:100)
at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$14.apply(HiveStrategies.scala:188)
at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$14.apply(HiveStrategies.scala:188)
at org.apache.spark.sql.SQLContext$SparkPlanner.pruneFilterProject(SQLContext.scala:364)
at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$.apply(HiveStrategies.scala:184)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54)
at org.apache.spark.sql.execution.SparkStrategies$BasicOperators$.apply(SparkStrategies.scala:280)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59)
at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:402)
at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:400)
at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:406)
at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:406)
at org.apache.spark.sql.hive.HiveContext$QueryExecution.stringResult(HiveContext.scala:406)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:59)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:291)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:226)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
Author: chirag <chirag.aggarwal@guavus.com>
Closes #2674 from chiragaggarwal/branch-1.1 and squashes the following commits:
370c31b [chirag] SPARK-3807: Add a test case to validate the fix.
1f26805 [chirag] SPARK-3807: SparkSql does not work for tables created using custom serde (Incorporated Review Comments)
ba4bc0c [chirag] SPARK-3807: SparkSql does not work for tables created using custom serde
5c73b72 [chirag] SPARK-3807: SparkSql does not work for tables created using custom serde
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Do not use TestSQLContext in JavaHiveQLSuite, that may lead to two SparkContexts in one jvm and enable JavaHiveQLSuite
Author: scwf <wangfei1@huawei.com>
Closes #2652 from scwf/fix-JavaHiveQLSuite and squashes the following commits:
be35c91 [scwf] enable JavaHiveQLSuite
(cherry picked from commit 58f5361caaa2f898e38ae4b3794167881e20a818)
Signed-off-by: Michael Armbrust <michael@databricks.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
SPARK-3039: Adds the maven property "avro.mapred.classifier" to build spark-assembly with avro-mapred with support for the new Hadoop API. Sets this property to hadoop2 for Hadoop 2 profiles.
I am not very familiar with maven, nor do I know whether this potentially breaks something in the hive part of spark. There might be a more elegant way of doing this.
Author: Bertrand Bossy <bertrandbossy@gmail.com>
Closes #1945 from bbossy/SPARK-3039 and squashes the following commits:
c32ce59 [Bertrand Bossy] SPARK-3039: Allow spark to be built using avro-mapred for hadoop2
(cherry picked from commit c243b21a8ba2610266702e00d7d4b5443cb1f687)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
constructor
Please refer to the JIRA ticket for details.
**NOTE** We should check all test suites that do similar initialization-like side effects in their constructors. This PR only fixes `ParquetMetastoreSuite` because it breaks our Jenkins Maven build.
Author: Cheng Lian <lian.cs.zju@gmail.com>
Closes #2375 from liancheng/say-no-to-constructor and squashes the following commits:
0ceb75b [Cheng Lian] Moves test suite setup code to beforeAll rather than in constructor
(cherry picked from commit 6d887db7891be643f0131b136e82191b5f6eb407)
Signed-off-by: Michael Armbrust <michael@databricks.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Logically, we should remove the Hive Table/Database first and then reset the Hive configuration, repoint to the new data warehouse directory etc.
Otherwise it raised exceptions like "Database doesn't not exists: default" in the local testing.
Author: Cheng Hao <hao.cheng@intel.com>
Closes #2352 from chenghao-intel/test_hive and squashes the following commits:
74fd76b [Cheng Hao] eliminate the error log
(cherry picked from commit 8194fc662c08eb445444c207264e22361def54ea)
Signed-off-by: Michael Armbrust <michael@databricks.com>
|
| |
|
| |
|
|
|
|
| |
This reverts commit b2d0493b223c5f98a593bb6d7372706cc02bebad.
|
|
|
|
| |
This reverts commit 865e6f63f63f5e881a02d1a4e3b4c5d0e86fcd8e.
|
| |
|
| |
|
|
|
|
| |
This reverts commit 2b2e02265f80e4c5172c1e498aa9ba2c6b91c6c9.
|
|
|
|
| |
This reverts commit 8b5f0dbd8d32a25a4e7ba3ebe1a4c3c6310aeb85.
|
| |
|
| |
|
|
|
|
| |
This reverts commit 711aebb329ca28046396af1e34395a0df92b5327.
|
|
|
|
| |
This reverts commit a4a7a241441489a0d31365e18476ae2e1c34464d.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
":" is not allowed to appear in a file name of Windows system. If file name contains ":", this file can't be checked out in a Windows system and developers using Windows must be careful to not commit the deletion of such files, Which is very inconvenient.
Author: qiping.lqp <qiping.lqp@alibaba-inc.com>
Closes #2191 from chouqin/querytest and squashes the following commits:
0e943a1 [qiping.lqp] rename golden file
60a863f [qiping.lqp] TestcaseName in createQueryTest should not contain ":"
(cherry picked from commit 634d04b87c2744d645e9c26e746ba2006371d9b5)
Signed-off-by: Michael Armbrust <michael@databricks.com>
|
| |
|
| |
|
|
|
|
| |
This reverts commit f07183249b74dd857069028bf7d570b35f265585.
|
|
|
|
| |
This reverts commit f8f7a0c9dce764ece8acdc41d35bbf448dba7e92.
|
| |
|
| |
|
|
|
|
| |
This reverts commit 58b0be6a29eab817d350729710345e9f39e4c506.
|
|
|
|
| |
This reverts commit 78e3c036eee7113b2ed144eec5061e070b479e56.
|
|
|
|
| |
This reverts commit 79e86ef3e1a3ee03a7e3b166a5c7dee11c6d60d7.
|
|
|
|
| |
This reverts commit a118ea5c59d653f5a3feda21455ba60bc722b3b1.
|