Our company is selecting DataBricks product DeltaBricks as a lake-house tool. We wish to manage schema evolution using Liquibase if possible. Please may anyone advise if Liquibase can support DeltaBricks/DataBricks?
Our raw ingest will be Parquet files on s3.
DeltaBricks format would be used for the next layers in the data lake for ‘silver’ (trusted de-duplicated events) and ‘gold’ (transformed) layers.
DeltaBricks is an open-course format, which under-the-hood is similar to Parquet but with additional metadata that allows more efficient queries than against Parquet, and supports ACID compliant operations.
We do not need to manage schemas in Parquet Format.
We Do wish to manage schemas in DeltaBricks Format.
DeltaBricks does support some SQL operations such as ALTER TABLE (ALTER TABLE | Databricks on AWS)
If we cannot directly use Liquibase to modify DeltaBricks schemas, may we reverse-engineer the schema from DeltaBricks on say a daily basis for version control? I know this might seem an odd requirement.
Many Thanks and sorry for asking such as basic question, I can see that none of these technologies are actively listed as a supported database.