The same md5sum is being recorded for all files

1

When deploying with Liquibase, the same md5sum is being recorded for all files for some reason. As a result, running the deployment with the --liquibase clear-checksums option allows the deployment to complete successfully. However, when redeploying the same code, the process fails during validation with the following error:

liquibase.exception.ValidationFailedException: Validation Failed:
9056 changesets check sum

This indicates that the checksum validation fails due to incorrect or identical checksums being generated for different changesets.

This issue occurs when there are more than 10,000 scripts. However, with smaller deployments of around 100+ scripts, the md5sum is calculated correctly and uniquely for each file.

image
image1864×393 54.2 KB

2

I am using Liquibase from the Docker container version 4.29.1.

3

It’s hard to troubleshoot without file examples.

I’m suspecting this has something to do with using sqlFile to include the files instead of include, and possibly the .ddl file extension. I would recommend experimenting with using include.

4

I am using YAML for the changelog. However, since there are too many files (more than 10,000), it’s not practical to place them all in a single changelog. Therefore, the file I provide to Liquibase looks like this:

databaseChangeLog:
    - include:
        file: changelog-part0-dwh.yaml
    - include:
        file: changelog-part1-dwh.yaml
    - include:
        file: changelog-part2-dwh.yaml
    - include:
        file: changelog-part3-dwh.yaml
	...
	...
	...

Each part contains around 100 scripts and looks like this:

databaseChangeLog:
    - changeSet:
        id: set-role-adb_deploy_cdjks
        author: cdjks
        runAlways: true
        runOnChange: true
        changes:
            - sql: 
                dbms: 'postgresql'
                endDelimiter: ';'
                splitStatements: false
                sql: set role adb_deploy_cdjks;
                stripComments: false
    - changeSet:
        id: dwh-bdv.schm.ddl
        author: cdjks
        runAlways: True
        runOnChange: true
        changes:
            - sqlFile: 
                dbms: 'postgresql'
                endDelimiter: ';'
                splitStatements: false
                path: dwh-bdv.schm.ddl
                stripComments: false
            
            - sql:
                dbms: 'postgresql'
                endDelimiter: ';'
                splitStatements: false
                sql:  GRANT ALL ON SCHEMA bdv TO adb_exec_dwh;
                stripComments:  false
                comment:  Set True ALL for bdv 
	...
	...
	...

As a result, the fullupdate file currently includes about 100 file inclusions, each containing 100 scripts. At the moment, everything works fine with Liquibase version 4.20.0, and no such issues arise.
I tried to switch to the latest version and encountered the problem.
An interesting observation is that if I run the deployment on a completely empty database (where no files have been applied and there are no records in the databasechangelog), the md5sum is calculated correctly.
I also tried removing the entries related to part0 from the databasechangelog. Liquibase applied the files and calculated a unique md5sum for each. However, the issue reappeared with the next batch, part1, where the same md5sum was generated again.

image
image1793×549 62.9 KB

5

I tried different versions.
With version 4.24 (docker pull liquibase/liquibase:4.24), everything works fine.
However, starting from version 4.25 (docker pull liquibase/liquibase:4.25), the behavior changes to what I have described.

6

Did anything change regarding the change validation when moving to version 4.25?
Please take a look at the debug log:

...
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.changelog] Updating null or out of date checksum on changeSet changelog-part0-dwh.yaml::dwh-bdv.schm.ddl::cdjks to correct value
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.util] Computed checksum for  as d41d8cd98f00b204e9800998ecf8427e
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.executor] UPDATE liquibase.databasechangelog SET MD5SUM = '9:d41d8cd98f00b204e9800998ecf8427e' WHERE ID = 'dwh-bdv.schm.ddl' AND AUTHOR = 'cdjks' AND FILENAME = 'changelog-part0-dwh.yaml'
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.executor] 1 row(s) affected
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.changelog] Updating null or out of date checksum on changeSet changelog-part0-dwh.yaml::dwh-bdv_rdm.schm.ddl::cdjks to correct value
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.util] Computed checksum for  as d41d8cd98f00b204e9800998ecf8427e
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.executor] UPDATE liquibase.databasechangelog SET MD5SUM = '9:d41d8cd98f00b204e9800998ecf8427e' WHERE ID = 'dwh-bdv_rdm.schm.ddl' AND AUTHOR = 'cdjks' AND FILENAME = 'changelog-part0-dwh.yaml'
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.executor] 1 row(s) affected
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.changelog] Updating null or out of date checksum on changeSet changelog-part0-dwh.yaml::dwh-bdv_rdm_tmp.schm.ddl::cdjks to correct value
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.util] Computed checksum for  as d41d8cd98f00b204e9800998ecf8427e
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.executor] UPDATE liquibase.databasechangelog SET MD5SUM = '9:d41d8cd98f00b204e9800998ecf8427e' WHERE ID = 'dwh-bdv_rdm_tmp.schm.ddl' AND AUTHOR = 'cdjks' AND FILENAME = 'changelog-part0-dwh.yaml'
[2024-08-29T14:56:55.301Z] [2024-08-29 14:56:55] FINE [liquibase.executor] 1 row(s) affected
...

and then:

...
[2024-08-29T14:57:19.301Z] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv.schm.ddl::cdjks
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] INFO [liquibase.ui] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv.schm.ddl::cdjks
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.changelog] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv.schm.ddl::cdjks
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Changeset changelog-part0-dwh.yaml::dwh-bdv.schm.ddl::cdjks
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.changelog] Reading ChangeSet: changelog-part0-dwh.yaml::dwh-bdv.schm.ddl::cdjks
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.configuration] No configuration value for liquibase.fileEncoding found
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.configuration] Configuration liquibase.fileEncoding is using the default value of UTF-8
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Executing Statement: --bdv SCHEMA
[2024-08-29T14:57:19.301Z] --1
[2024-08-29T14:57:19.301Z] DO
[2024-08-29T14:57:19.301Z] $do$
[2024-08-29T14:57:19.301Z] BEGIN
[2024-08-29T14:57:19.301Z]     IF NOT EXISTS(
[2024-08-29T14:57:19.301Z]         SELECT schema_name
[2024-08-29T14:57:19.301Z]           FROM information_schema.schemata
[2024-08-29T14:57:19.301Z]           WHERE schema_name = 'bdv'
[2024-08-29T14:57:19.301Z]       )
[2024-08-29T14:57:19.301Z]     THEN
[2024-08-29T14:57:19.301Z]       EXECUTE 'CREATE SCHEMA bdv';
[2024-08-29T14:57:19.301Z]     END IF;
[2024-08-29T14:57:19.301Z] END
[2024-08-29T14:57:19.301Z] $do$;
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] --bdv SCHEMA
[2024-08-29T14:57:19.301Z] --1
[2024-08-29T14:57:19.301Z] DO
[2024-08-29T14:57:19.301Z] $do$
[2024-08-29T14:57:19.301Z] BEGIN
[2024-08-29T14:57:19.301Z]     IF NOT EXISTS(
[2024-08-29T14:57:19.301Z]         SELECT schema_name
[2024-08-29T14:57:19.301Z]           FROM information_schema.schemata
[2024-08-29T14:57:19.301Z]           WHERE schema_name = 'bdv'
[2024-08-29T14:57:19.301Z]       )
[2024-08-29T14:57:19.301Z]     THEN
[2024-08-29T14:57:19.301Z]       EXECUTE 'CREATE SCHEMA bdv';
[2024-08-29T14:57:19.301Z] 
[2024-08-29T14:57:19.301Z]     END IF;
[2024-08-29T14:57:19.301Z] END
[2024-08-29T14:57:19.301Z] $do$;
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] SQL in file dwh-bdv.schm.ddl executed
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Executing Statement: GRANT ALL ON SCHEMA bdv TO team_exec
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] GRANT ALL ON SCHEMA bdv TO team_exec
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] Custom SQL executed
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] ChangeSet changelog-part0-dwh.yaml::dwh-bdv.schm.ddl::cdjks ran successfully in 62ms
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] UPDATE liquibase.databasechangelog SET COMMENTS = '', CONTEXTS = NULL, DATEEXECUTED = NOW(), DEPLOYMENT_ID = '4943438530', DESCRIPTION = 'sqlFile path=dwh-bdv.schm.ddl; sql', EXECTYPE = 'RERAN', LABELS = NULL, LIQUIBASE = '4.29.1', MD5SUM = '9:d41d8cd98f00b204e9800998ecf8427e', ORDEREXECUTED = 15479 WHERE ID = 'dwh-bdv.schm.ddl' AND AUTHOR = 'cdjks' AND FILENAME = 'changelog-part0-dwh.yaml'
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 1 row(s) affected
[2024-08-29T14:57:19.301Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Changelog query completed.
[2024-08-29T14:57:19.558Z] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv_rdm.schm.ddl::cdjks
[2024-08-29T14:57:19.558Z] [2024-08-29 14:57:19] INFO [liquibase.ui] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv_rdm.schm.ddl::cdjks
[2024-08-29T14:57:19.558Z] [2024-08-29 14:57:19] FINE [liquibase.changelog] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv_rdm.schm.ddl::cdjks
[2024-08-29T14:57:19.558Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Changeset changelog-part0-dwh.yaml::dwh-bdv_rdm.schm.ddl::cdjks
[2024-08-29T14:57:19.558Z] [2024-08-29 14:57:19] FINE [liquibase.changelog] Reading ChangeSet: changelog-part0-dwh.yaml::dwh-bdv_rdm.schm.ddl::cdjks
[2024-08-29T14:57:19.558Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Executing Statement: --bdv_rdm SCHEMA
[2024-08-29T14:57:19.558Z] --1
[2024-08-29T14:57:19.558Z] DO
[2024-08-29T14:57:19.558Z] $do$
[2024-08-29T14:57:19.558Z] BEGIN
[2024-08-29T14:57:19.558Z]     IF NOT EXISTS(
[2024-08-29T14:57:19.559Z]         SELECT schema_name
[2024-08-29T14:57:19.559Z]           FROM information_schema.schemata
[2024-08-29T14:57:19.559Z]           WHERE schema_name = 'bdv_rdm'
[2024-08-29T14:57:19.559Z]       )
[2024-08-29T14:57:19.559Z]     THEN
[2024-08-29T14:57:19.559Z]       EXECUTE 'CREATE SCHEMA bdv_rdm';
[2024-08-29T14:57:19.559Z] 
[2024-08-29T14:57:19.559Z]     END IF;
[2024-08-29T14:57:19.559Z] END
[2024-08-29T14:57:19.559Z] $do$;
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] --bdv_rdm SCHEMA
[2024-08-29T14:57:19.559Z] --1
[2024-08-29T14:57:19.559Z] DO
[2024-08-29T14:57:19.559Z] $do$
[2024-08-29T14:57:19.559Z] BEGIN
[2024-08-29T14:57:19.559Z]     IF NOT EXISTS(
[2024-08-29T14:57:19.559Z]         SELECT schema_name
[2024-08-29T14:57:19.559Z]           FROM information_schema.schemata
[2024-08-29T14:57:19.559Z]           WHERE schema_name = 'bdv_rdm'
[2024-08-29T14:57:19.559Z]       )
[2024-08-29T14:57:19.559Z]     THEN
[2024-08-29T14:57:19.559Z]       EXECUTE 'CREATE SCHEMA bdv_rdm';
[2024-08-29T14:57:19.559Z] 
[2024-08-29T14:57:19.559Z]     END IF;
[2024-08-29T14:57:19.559Z] END
[2024-08-29T14:57:19.559Z] $do$;
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] SQL in file dwh-bdv_rdm.schm.ddl executed
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Executing Statement: GRANT ALL ON SCHEMA bdv_rdm TO team_exec
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] GRANT ALL ON SCHEMA bdv_rdm TO team_exec
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] Custom SQL executed
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] ChangeSet changelog-part0-dwh.yaml::dwh-bdv_rdm.schm.ddl::cdjks ran successfully in 34ms
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] UPDATE liquibase.databasechangelog SET COMMENTS = '', CONTEXTS = NULL, DATEEXECUTED = NOW(), DEPLOYMENT_ID = '4943438530', DESCRIPTION = 'sqlFile path=dwh-bdv_rdm.schm.ddl; sql', EXECTYPE = 'RERAN', LABELS = NULL, LIQUIBASE = '4.29.1', MD5SUM = '9:d41d8cd98f00b204e9800998ecf8427e', ORDEREXECUTED = 15480 WHERE ID = 'dwh-bdv_rdm.schm.ddl' AND AUTHOR = 'cdjks' AND FILENAME = 'changelog-part0-dwh.yaml'
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 1 row(s) affected
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Changelog query completed.
[2024-08-29T14:57:19.559Z] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv_rdm_tmp.schm.ddl::cdjks
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] INFO [liquibase.ui] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv_rdm_tmp.schm.ddl::cdjks
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.changelog] Running Changeset: changelog-part0-dwh.yaml::dwh-bdv_rdm_tmp.schm.ddl::cdjks
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Changeset changelog-part0-dwh.yaml::dwh-bdv_rdm_tmp.schm.ddl::cdjks
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.changelog] Reading ChangeSet: changelog-part0-dwh.yaml::dwh-bdv_rdm_tmp.schm.ddl::cdjks
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Executing Statement: --bdv_rdm_tmp SCHEMA
[2024-08-29T14:57:19.559Z] --1
[2024-08-29T14:57:19.559Z] DO
[2024-08-29T14:57:19.559Z] $do$
[2024-08-29T14:57:19.559Z] BEGIN
[2024-08-29T14:57:19.559Z]     IF NOT EXISTS(
[2024-08-29T14:57:19.559Z]         SELECT schema_name
[2024-08-29T14:57:19.559Z]           FROM information_schema.schemata
[2024-08-29T14:57:19.559Z]           WHERE schema_name = 'bdv_rdm_tmp'
[2024-08-29T14:57:19.559Z]       )
[2024-08-29T14:57:19.559Z]     THEN
[2024-08-29T14:57:19.559Z]       EXECUTE 'CREATE SCHEMA bdv_rdm_tmp';
[2024-08-29T14:57:19.559Z] 
[2024-08-29T14:57:19.559Z]     END IF;
[2024-08-29T14:57:19.559Z] END
[2024-08-29T14:57:19.559Z] $do$;
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] --bdv_rdm_tmp SCHEMA
[2024-08-29T14:57:19.559Z] --1
[2024-08-29T14:57:19.559Z] DO
[2024-08-29T14:57:19.559Z] $do$
[2024-08-29T14:57:19.559Z] BEGIN
[2024-08-29T14:57:19.559Z]     IF NOT EXISTS(
[2024-08-29T14:57:19.559Z]         SELECT schema_name
[2024-08-29T14:57:19.559Z]           FROM information_schema.schemata
[2024-08-29T14:57:19.559Z]           WHERE schema_name = 'bdv_rdm_tmp'
[2024-08-29T14:57:19.559Z]       )
[2024-08-29T14:57:19.559Z]     THEN
[2024-08-29T14:57:19.559Z]       EXECUTE 'CREATE SCHEMA bdv_rdm_tmp';
[2024-08-29T14:57:19.559Z] 
[2024-08-29T14:57:19.559Z]     END IF;
[2024-08-29T14:57:19.559Z] END
[2024-08-29T14:57:19.559Z] $do$;
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] SQL in file dwh-bdv_rdm_tmp.schm.ddl executed
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Executing Statement: GRANT ALL ON SCHEMA bdv_rdm_tmp TO team_exec
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] GRANT ALL ON SCHEMA bdv_rdm_tmp TO team_exec
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 0 row(s) affected
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] Custom SQL executed
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] INFO [liquibase.changelog] ChangeSet changelog-part0-dwh.yaml::dwh-bdv_rdm_tmp.schm.ddl::cdjks ran successfully in 21ms
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] UPDATE liquibase.databasechangelog SET COMMENTS = '', CONTEXTS = NULL, DATEEXECUTED = NOW(), DEPLOYMENT_ID = '4943438530', DESCRIPTION = 'sqlFile path=dwh-bdv_rdm_tmp.schm.ddl; sql', EXECTYPE = 'RERAN', LABELS = NULL, LIQUIBASE = '4.29.1', MD5SUM = '9:d41d8cd98f00b204e9800998ecf8427e', ORDEREXECUTED = 15481 WHERE ID = 'dwh-bdv_rdm_tmp.schm.ddl' AND AUTHOR = 'cdjks' AND FILENAME = 'changelog-part0-dwh.yaml'
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] 1 row(s) affected
[2024-08-29T14:57:19.559Z] [2024-08-29 14:57:19] FINE [liquibase.executor] Changelog query completed.
...

When recalculating the hashes, it replaces all of them with the md5sum of an empty string (d41d8cd98f00b204e9800998ecf8427e).
However, I cannot understand what this means. If it did not have access to the file, how does it correctly compute the hash when some entries are removed from the databasechangelog?

7

Someone from Liquibase will need to chime in on this, or I’d recommend opening a Github issue:

1 Like
8

@eldalex if this is still an issue for you, it would be best to file an issue in our repo so we can take a look. Thanks!

9

Hello! Sorry, I was on vacation.
No problem! I will prepare a small archive with the code that can be checked and write about the issue in the repo.
Thank you!