"Migration Failed: Java heap space" error when generateChangeLog with "data"

General Discussion
1

Hi all

I am trying to create a baseline on one of my development database, here’s my environment

  • Database, -> Oralce 10g, with 500+ Tables with lots of configuration data, the oracle export dump file is about 70mb;
  • Java - Java 6
  • Oracle JDBC Driver - ojdbc14.jar (downloaded from Oracle web site)
  • Command line execute
    liquibase --changeLogFile=base.changelog.data.xml --diffTypes=“data”  generateChangeLog

Execute result:

  • Liquibase is configure to run with “-Xmx512m -Xms256m” jvm parameters, fail, error message - "Migration Failed: Java heap space
    "
  • configure to run with “-Xmx1024m -Xms512m”, same error occurs
  • configure to run with “-Xmx2048 -Xms512m”, same error

What other options I have in oder to create base line for my development projects?

Appreciate ur advise, thanks!

James

2

I commented on stackoverflow on your question as well.  You could try the latest builds from 2.0.  There have been some improvements in performance.  The diff tool is scheduled for more extensive improvements in the 2.1 timeframe. 

Nathan

3

Hi Nathan

First of all, thanks for your advise (previously I run liquebase 1.9.5).

I downloaded “LiquiBase 2.0 RC1 Binary (tar.gz)”, and try to run the generateChangeLog with “diffTypes=data”, I still encounter the same problem.  ???
Please note that generateChangeLog without diffType=data, run fine.

Although I didn’t check the liquibase code on how it generate the change Log for Database Data, I presume it will

  1. first scan the database, and find out table dependencies to determine SQL insert order (i.e which table data to be populate first)
  2. Append the ChangeSet XML to ChangeSet DOm in memory
  3. and finally write it to a file
    ?

I think above will work fine for small database, but when run against larger database, it will hit heap space problem. One way to resolve this, is one the order of SQL insert is define, liquibase should write multiple SQL insert scripts in a temporary folder, and later merge it back…

Just my suggestion…

Cheers and thanks for great product!

4

Hi

Any update on this? … I will appreciate if anyone could recommends me any tools/approach to help me resolve my issue…i.e creating baseline of my development database, so that we could version control it.

Thanks!
James

5

So it sounds like a problem with the performance out outputting the data.  You are about right on the order.  It doesn’t check for insertion order, it just creates foreign keys after all the tables. 

There have been a lot of improvements since the RC1 release, although our build server is currently down so you would have to build the newest version from trunk yourself. 

There is the ability to have all the inserts as CSV files instead of XML nodes which would help considerably with the memory usage.  It looks like in RC1 there wasn’t the abiltity to set that property in the command line liquibase.  I added a dataOutputDirectory flag just now. 

Would you be able to build from source yourself?  Or should I make a build manually for you to test it out?

Nathan

6

Depending on how large and complex your database is, you may want to use an oracle-specific tool to backup and restore your database instead.  Our diff tool is not the primary use of liquibase, and is intended mainly to allow you to get a starting point description of your database that will probably need modification due to objects we do not always correctly handle (stored procs, complex index descriptions, etc). 

If your database is very large, it may work best to just create a snapshot using a tool that you know will back up everything and always use that as your starting database and have liquibase manage changes from now on.

Nathan

7

Hi Nathan

Thanks for ur advise, I did try to check out liquibase from trunk and try to build from the trunk, but I fail to build from “liquibase-core”, I type  “mvn package”

/liquibase-core/src/main/java/liquibase/lockservice/LockService.java:[31,22] generics are not supported in -source 1.3"

Is there any additional configuration I needs to set to build, FYI, I am using Sun JDK 6

Cheers
James

8
Originally posted by: coolboykl
Hi Nathan

Thanks for ur advise, I did try to check out liquibase from trunk and try to build from the trunk, but I fail to build from “liquibase-core”, I type  “mvn package”

/liquibase-core/src/main/java/liquibase/lockservice/LockService.java:[31,22] generics are not supported in -source 1.3"

Is there any additional configuration I needs to set to build, FYI, I am using Sun JDK 6

Cheers
James

It looks to me like the pom.xmls are set up wrong.

There is a top-level pom.xml which defines all sorts of defaults, including the source level for the maven-compiler-plugin.  Then, one level down, there is a pom.xml under liquibase-core that is a module of the first pom.xml, but which does not designate that pom.xml as its parent.

The result is that the pom.xml under liquibase-core uses the defaults for maven-compiler-plugin–which among other things means -source 1.3–and not the settings from the pom.xml that should be, I think, its parent.  Nathan or someone will have to stitch this up before the build will work again.  In the meantime, try modifying that sub-pom.xml so that it has a stanza in it that points to the top-level pom.xml.

Best,
Laird

9

Yes, I’m working through changes to the pom files, it must still have problems. 

Make the change Laird suggested until I get a new version committed.

Nathan