Content of main.py:
spark.conf.set("spark.datasource.memsql.ddlEndpoint", "hostname")
spark.conf.set("spark.datasource.memsql.dmlEndpoints", "hostname,hostname:3306")
spark.conf.set("spark.datasource.memsql.user", "username")
spark.conf.set("spark.datasource.memsql.password", "password")
df_rules = spark.read.format("memsql").option("ddlEndpoint", "hostname").option("username", "password").load("table_name")
df_rules.show()
Spark-submit command. (spark v2.4.0) :
spark-submit --master local[4] --jars valid_path/mariadb-java-client-2.4.0.jar main.py
Error:
java.lang.ClassNotFoundException: Failed to find data source: memsql. Please find packages at http://spark.apache.org/third-party-projects.html
What could be the possible reason for this error ?