Liquibase Can't Create DatabaseChangeLog Table in Databricks

General Discussion
1

What I am trying to do
I am trying to test the use of Liquibase with Databricks by creating a docker compose file.

The error
I am able to connect to Databricks but get the following error when Liquibase tries to create the DATABASECHANGELOG table:

Caused by: org.apache.spark.sql.catalyst.parser.ParseException: 
popeyes_spinach  | [UNSUPPORTED_DATATYPE] Unsupported data type "DATETIME". SQLSTATE: 0A000
popeyes_spinach  | == SQL (line 1, position 134) ==
popeyes_spinach  | ...HAR(255) NOT NULL, DATEEXECUTED datetime NOT NULL, ORDEREXECUTED INT NOT...

. [Failed SQL: (500051) CREATE TABLE DATABASECHANGELOG (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED datetime NOT NULL, ORDEREXECUTED INT NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10))]

What I have tried

  • I have tried mounting the databricks extension file to the classpath folder
  • I have tried mounting the databricks extension file to the lib folder
  • I have tried mounting the databricks extension file to the lib folder and explicitly specifying the path to the jar file using the --classpath parameter

Code in docker compose file below

Thank you for your help!


services:
  
  liquibase_poc:
    environment:
      - JAVA_OPTS=--add-opens=java.base/java.nio=ALL-UNNAMED"
    container_name: popeyes_spinach
    image: liquibase/liquibase
    command: liquibase update --url='jdbc:databricks://adb-redacted.3.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/redacted;ConnCatalog=188_edap_dev;ConnSchema=databricks_cicd_scratch;' --username='redacted' --password='redacted' --driver='com.databricks.client.jdbc.Driver' --classpath=/liquibase/lib/liquibase-databricks-1.4.0-javadoc.jar  --changelog-file='changelog.xml'
    volumes:
    #download databricks driver and extension and put in proper directory so liquibase can find it; this needs to be baked into our docker image for the POC
      - ./liquibase/scripts:/liquibase/scripts
      - ./liquibase/changelog.xml/:/liquibase/changelog.xml
       - ./liquibase/lib:/liquibase/lib
1 Like
2

I figured out what I did wrong. I was downloading the javadoc.jar file, it needs to be the file without the javadoc in the file name: liquibase-databricks-1.4.0.jar

2 Likes