Memory consumption for large files

General Discussion
1

Thanks for the troubleshooting, that helps a lot. I’d like to try to keep the normalization process if possible and I think it will be possible to do a more memory-friendly manner, possibly working on a line by line basis instead of the whole file at a time.


I’m trying to finish up a new revision of the hibernate integration, but should be able to get to this in the next day or two.


Nathan

2

I created https://liquibase.jira.com/browse/CORE-1509 to track the issue. 


A separate option would be to create an extension (liquibase.org/extensions) that subclasses sqlFile and overrides Nathan

3

Our application hardly relies on Liquibase for database migration.
Today we were trying to add a migration file which has 22Mb of SQL instructions with data and eventually got OutOfMemoryException.

Playing a bit with the profiler I saw ~250MB of allocation done by Liquibase, which is a bit huge even for 22Mb file.
Allocation of ~200MB happened inside AbstractSQLChange#prepareSqlForChecksum.
Another ~45MB came from AbstractSQLChange#setSql

These methods are pretty simple and just do string manipulation in memory. The whole chain of manipulations like replaceAll and trim. It’s really painful for the large files.

I think it might be reasonable to add an attribute like “useRawQueries” declared for changeset which will cause Liquibase to skip reprocessing of the strings if the user set it to true on purpose.

Actually I can also provide a patch if it makes sense for anyone else except for me. :slight_smile:

4

Thank you for being very responsive.
I’ll take a look into extensions direction.

5

The fix from 1509 is in 3.1.1 as well. I haven’t seen any memory issues with 3.1.1 lately.


Would you be able to provide a memory dump I could look at to determine where the problem is?

Nathan

6

I’m running into a very similar situation. One of the liquibase files is large (initializes a database to a certain release level). We run liquibase from the command line with 256 MB of heap and it will error out with an OutOfMemory exception. We’re using the latest 3.1.1 version. The JIRA referenced above appears to have been pulled into the 3.0.7 snapshot. Is it in 3.1.1?


Are there any known, unresolved memory issues with version 3.1.1?


Thank you.