Share your ideas for the 2022 Liquibase Roadmap

General Discussion
25

@kevin a bit of all of the above – but mostly thinking about what types of tests/conditions we should test for that would give contributors more confidence to submit PRs; especially interested in learning more about use cases/conditions that folks using the tool regularly run into that we may not see on this end as frequently with our heads down in building/testing/building/testing it.

26

There have been a few conditions, especially when it comes to multiple teams/developers working on the same database, that have caused some conflicts with the DATABASECHANGELOG. For example, there was a recent issue that was partially resolved related to time zones. There are a few others that we have hit that present themselves almost as race conditions, but are likely more common as teams scale. Would be interested in hearing what others in the community may have run into as related to this.

27

Thanks for the ideas @fbiville . I hope you will be able to join us at the Meetup on Feb 10th to discuss more. Sounds like you are working on an extension or set of extensions of your own? Is that the case? Also sounds like you are definitely working with Liquibase with graph databases. Can you share a bit more about the specifics of the use case behind the request for parameterized statements in the core APIs, especially as it relates to your work with Neo4j?

28

Hi, I just registered for the meetup and will attend for sure.

I am working on the Neo4j plugin for Liquibase (I’m a Neo4j employee). I am in regular contact with Robert R. and Nathan V., this has helped me a lot :slight_smile:

The need for parameterized statements in the core API is definitely not a blocker, but it would be useful for the plugin internal queries that are run often. Currently, the Neo4j runtime will re-create an execution plan for every of these query, even if they just vary by one parameter and they are fundamentally just the same. Another minor this would solve is SQL (in Neo4j case: Cypher) injection, but that’s not necessarily very relevant in Liquibase context since users already need privileges to run migrations in the first place.

29

One thing we (as a team successfully (and thankfully) using Liquibase for 3+ years) consider a neat improvement would be a “template/validator” for the id, author and filename (and optionally comments/remarks).

Let me detail:
So every project has some development rules and guidelines that want to be consistent across time and developers group changes. In our case, specifically, with Liquibase, for example, when using Liquibase, you (as a team) decide that:

  • id should be numeric, incremental or random generated number etc (or even Id1, Id2, Id3 etc)
  • author can be either developer’s name, or the developer’s company id or even an issue number (that is mapped that development effort and/or issue the particular changeset(s) addresses).
  • filename can be of a given format (name of the file and/or suffix+prefix, some sort of numbering, the possibilities are only limited by imagination and OS file name length limitations :smiley: ).
  • (Optional) - comments/remarks structure per changeset have a particular form to be consumed at a later point by a different tool/process.

Since Liquibase engine “chews” all the changelogs (so for the filename level) and changesets (for the rest), it would be awesome to compare and validate the properties mentioned above (of course, using regex mechanisms), with a user defined template (like an xsd validation schema for those properties, but of course available for JSON or YAML - our case for example) and deliver warnings to the console/log where the rules and guidelines are not met (separate discussion, if warnings and/or errors should be raised (or even have option on this), as I don’t think we should consider these “development rules and guidelines violations” as true errors).

Thanks,
Eduard

PS: We ended up building an in-house tool for this (using Powershell) since it’s a pretty important thing for us but I am sure an integrated solution/validator would be x10 better for addressing this scenario.
PS2: This feedback process here is a nice and constructive initiative.

30

Does the community have any recommendations for database to add to the Liquibase Test Harness: GitHub - liquibase/liquibase-test-harness: This is a Harness of Integration Tests designed to validate Liquibase behavior? You can see the ones we have already added on the main page.

Thanks!

Robert

31

Thanks @EduardUta !! This is such a great, well-thought out idea. I hope you will also be able to make it to the Meetup on the 10th to talk about it more. Have you had a chance to also try out Quality Checks at all? It’s not quite the exact same as what you discuss here, but it’s something that might be able to help a bit and is designed for a similar set of use cases. It’s still pretty new, so feedback/ideas on it specifically are very welcome!

32

@fbiville of course just after I sent that message to you I talked with Robert and Nathan and Caroline and realized exactly who you are! It’s nice to “meet” you here. I am personally schooling myself more so thanks in advance for your patience as my brain catches up!

33

It would be great to have Changelog Parameters (i.e. substitution variables) accessible/usable in a changeset change type of ‘sql’. Example use: you create many Postgres Schemas (for which there is no native Liquibase change type, so we must use sql or sql scripts), one for each microservice, and each one gets an identical set of grants.

You could create a changelog file to ‘include’ in the root changelog that acts like a ‘template’ and contains commands like:

  • sql: CREATE SCHEMA IF NOT EXISTS ${newschema} AUTHORIZATION ${appowner}
  • sql: GRANT USAGE ON SCHEMA ${newschema} TO ${appowner}
  • sql: GRANT USAGE ON SCHEMA ${newschema} TO ${appuser}
  • sql: ALTER DEFAULT PRIVILEGES IN SCHEMA ${newschema} GRANT SELECT,INSERT,UPDATE,DELETE ON TABLES TO ${read_write_role}
  • sql: ALTER DEFAULT PRIVILEGES IN SCHEMA ${newschema} GRANT SELECT ON TABLES TO ${read_only_role}
  • etc.

Then in the root changelog:

databaseChangeLog:
# Create product_catalog schema
- property:
    name: appowner
    value: someapp_owner
- property:
    name: appuser
    value: someapp_user
- property:
    name: newschema
    value: product_catalog
- include:
    file: common.create_schema.yaml
# Create sales_history schema
- property:
    name: appowner
    value: someapp_owner
- property:
    name: appuser
    value: someapp_user
- property:
    name: newschema
    value: sales_history
- include:
    file: common.create_schema.yaml

This would be extremely useful.

Also, the ability to use changelog parameters in scripts called via the sqlFile attribute would be equally useful, if not even more so.

34

@ktaggart thanks for the reply and info on Quality Checks! Haven’t tried these so far (first time I hear about it) but I will definitely have a look and see if this can also cover our needs (with regard to my post).
I will definitely try to make it to the Feb meetup.
Thanks,
Eduard

35

There is a BIG Liquibase user that has created a REST API wrapper for Liquibase. This is because they had built an integration with on CI/CD and then had to move to another one. Using the REST API made life a lot easier for them.

36

How about an option to do post validation of a changeset?
For instance if you add function, insert_user
Have a post validation that the function was indeed added.
SELECT has_function_privilege(‘mikeo.insert_user(text, text)’, ‘execute’);

37

Adding this to our release checklist. Thanks for the suggestion @rakhi

38

Hello,

I have a few ideas (improvements) [my Liquibase version is 4.6.1]

1.Improve working of “context” with "includeAll"

A)

  • in my changelog I have:
 <includeAll path="database\tables" context="TEST_CONTEXT" />
  • inside *\tables* folder I have SQL changelog:
--changeset rgrzegorczyk:CREATE_TAB
CREATE TABLE TEST (id number);

Actual behavior:

  • after running “Liquibase update”, my newly inserted row is having a column

DATABASECHANGELOG.CONTEXTS = NULL

Expected behavior:

DATABASECHANGELOG.CONTEXTS = “TEST_CONTEXT”

Context in parent changelog should be default for changesets without individually defined contexts.

B)

  • in my changelog I have:
<includeAll path="database\tables" context="TEST_CONTEXT" />
  • inside /tables/ folder I have SQL changelog:
--changeset rgrzegorczyk:CREATE_TAB context:MY_CONTEXT
CREATE TABLE TEST (id number);

Actual behavior:

  • after running “Liquibase update”, my newly inserted row is having column

DATABASECHANGELOG.CONTEXTS = ‘TEST_CONTEXT AND MY_CONTEXT’

Expected behavior:

If changeset has its own context defined then it’s the only one that should be used. So the value should be:

DATABASECHANGELOG.CONTEXTS = ‘MY_CONTEXT’

2. Enable “labels” working with "includeAll"

Behavior should be analogical to those described in 1.A and 1.B

3. Normalizing “context” and "contexts"

E.g. in XML changesets or SQL changesets valid tag is “context” but the column in DATABASECHANGELOG table is called contextS

It made me crazy many times when misspelled “context” with “contexts” when writing changesets

Could you make both “context” and “contexts” valid syntax when writing changesets?

4. Liquibase checks

As I understand, the main idea is: to check if every check passed. If not then show it in a report, if yes then remove it from the report.

What about creating a syntax, sth like

<ValidCheckSum>ANY</validCheckSum>"?

I would like to mark some changesets or whole changelogs and tell Liquibase:

Don’t run ANY or SOME (listed) checks with this changeset/ changelogs?

5. Normalizing LABELS and do not LOWER() my values in labels

When I define label in changeset like this:

label:TEST or labels="TEST"

then the final insert into DATABSECHANGELOGTABLE always go with LOWER(), so the final value in column LABELS is “test” with lower letters.

It would be great to make it case-sensitive, and insert exact value I provided in my changeset.

39

Please add a sidebar to your documentation. Something similar to this Prisma Client (Reference) | Prisma Docs

Your docs are incredibly difficult to search or find stuff. I always resort to googling stuff instead of searching in the docs.

Also, you should rethink the format for a single documentation page.

  • The table format is hard to read.
  • Code is often not formatted the same
  • Provide more description about things or point the user to the relevant resource

Just more info in general.

It’s a great product with poor documentation that often leaves me wanting to know more.

40

@vuki656 Thank you!! Great feedback and better docs would help everyone.

41

Another thing that would be helpful for Liquibase plugin authors is an automated way to deploy custom XML schemas.

42

Thanks to all who attended yesterday and posted here. Ideas, voting, commenting has moved to the Idea Board – please keep them coming!

43

One of the purported benefits of using model-based Liquibase change types (i.e. not using change types ‘sql’ or ‘sqlFile’) is the ability to rollback changes. In my liquibase prototype I have created a scenario similar to one I’ve experienced in real life where as part of the changes, data must be populated (details on that below), and the use of a 'sql change type in that set of changes means the changes cannot be rolled back.

My suggestion is to add an attribute to the ‘sql’ and ‘sqlFile’ change types (or it could apply to all change types perhaps) such as ‘skipOnRollback’. This could inform liquibase that the sql change type can be ignored in a rollback, and thus allow the rest of the changes to be rolled back.

My specific example is:

  • create table parent with columns id, event_type, event_summary and single-column primary key ‘id’
  • create table child with 2-column primary key id and event_timestamp
  • create foreign key ‘child_to_parent_fk’ from child to parent referencing column ‘id’
  • populate both parent and child table with data

Later on it is realized that the ‘event_type’ column should have been included in the primary key for the parent table. The changes needed are:

  • drop foreign key child_to_parent_fk
  • drop PK from parent and child
  • create PK on parent with columns ‘id, event_type’
  • add column event_type to table child
  • Use a ‘sql’ change type to update child c set event_type = (select p.event_type from parent p where p.id = c.id)
  • create PK on child with columns ‘id, event_type, event_timestamp’

In this case, the single ‘sql’ change prevents rollback. But in fact, in this case, the ‘sql’ change type doesn’t logically impact the ability to rollback because the column we populated via SQL is actually being dropped by the rollback.

Obviously use of this ‘skipOnRollback’ would be suggested for “Use at your own risk, only when you know it is safe to do so, and only after first testing the rollback.”

Just a thought! If there is a way to accomplish this rollback with liquibase that I’m not aware of, please let me know.

44

I would like to be able to execute SQL scripts using liquibase “sql-executer” - programmatically - without storing the execution in DATABASECHANGELOG