Basic Read Works in spark-shell. Fails in maven Project. ClassNotFoundException: Failed to find data source: memsql

Help
1

With Spark Shell: While using jars memsql-spark-connector_2.11-3.0.5-spark-2.3.4.jar, mariadb-java-client-2.7.2.jar and spray-json_2.11-1.3.6.jar.
I am able to do basic read.

But same command, via a maven spark-java project gives the below error on read.

java.lang.ClassNotFoundException: Failed to find data source: memsql. Please find packages at http://spark.apache.org/third-party-projects.html

Read Command:
spark.read.format("memsql")
.option("ddlEndpoint", "<host:port>").option("user", "<user>")
.option("password", "<password>").option("database", "<DB>")
.load("<table>")

Maven dependencies:

<dependency>
    <groupId>com.memsql</groupId>
    <artifactId>memsql-spark-connector_2.11</artifactId>
    <version>3.0.5-spark-2.3.4</version>
</dependency>
<dependency>
    <groupId>org.mariadb.jdbc</groupId>
    <artifactId>mariadb-java-client</artifactId>
    <version>2.7.2</version>
</dependency>
<dependency>
    <groupId>io.spray</groupId>
    <artifactId>spray-json_2.11</artifactId>
    <version>1.3.5</version>
</dependency>
2

Hi @naman.jain4,
seems like spark-connector dependency hasn’t been connected to the project, otherwise, I can’t see another reason why it can’t find a data source.
Can you please also try with com.memsql.spark format? like: spark.read.format("com.memsql.spark")...

3

@iblinov-ua, I have added the maven dependency appropriately.
Isn’t it ? -> memsql-spark-connector_2.11

Also, I tried to read the way you mentioned. Still got same error: java.lang.ClassNotFoundException: Failed to find data source: com.memsql.spark.

4

@naman.jain4, yes, you’ve added maven dependency correctly, but the only reason your project can’t see the memsql data source is that maven has not included it in the project (it may be for some other reasons).
Can you please check External libraries in your project and confirm that you are able to see the memsql-spark-connector library there.
Thanks!