Resource Accessor / Classpath Duplicates / ClassCastException

General Discussion
1

Hey,

A company I’m working with has an older project with Liquibase and GWT and they’ve upgraded GWT and are encountering some problems with the Liquibase migrations, so I’m looking for suggestions on how to work around the problem.

Background
The project was on GWT v2.7.x and Liquibase v3.5.3, neither of which are close to current. Liquibase is invoked using a custom servlet listener (not using the one that comes with liquibase, but a simpler and customized one).

They’re looking to make some dependency upgrades and moved up to GWT v2.8.2 which went pretty smoothly.

The Problem

Liquibase still works fine in a production configuration, but it doesn’t run properly when you run GWT SuperDevMode in JetBrains IntelliJ. Seems like something has changed with the GWT launcher which results in the liquibase file appearing in multiple classpath locations simultaneously, which results in this error:

liquibase.exception.ChangeLogParseException: Error Reading Migration File: Found 3 files that match db-schema/installer.xml
	at liquibase.parser.core.xml.XMLChangeLogSAXParser.parseToNode(XMLChangeLogSAXParser.java:118) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.parser.core.xml.AbstractChangeLogParser.parse(AbstractChangeLogParser.java:15) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.Liquibase.getDatabaseChangeLog(Liquibase.java:217) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:190) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:179) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:175) ~[liquibase-core-3.8.9.jar:na]

(That error shows liquibase 3.8.9 because I’ve been trying different versions of Liquibase to see if they all behave the same way, as well as digging into the code a little).

If I check the classpath myself, I can see that the class loader has three URLs for three filesystem paths and each of them seems to be resolving the “installer.xml”, so “getResources” shows three different URL for the same classpath name.

I haven’t been able to figure out why the classpath is behaving differently with GWT 2.8.2 launched through IntelliJ so far, so I was trying to see if I could somehow tweak the invocation of liquibase to avoid the problem.

The ResourceAccessor being used is this:

	ResourceAccessor accessor = new ClassLoaderResourceAccessor( Thread.currentThread().getContextClassLoader() );

Nice and simple, but if the file appears multiple times has duplicates and Liquibase (understandably) doesn’t like that, I thought I’d try a FileSystemResourceAccessor (or a CompositeResourceAccessor starting with the FileSystemResourceAccessor) to see if that solved the problem, but instead I get this error:

Caused by: liquibase.exception.UnexpectedLiquibaseException: java.lang.ClassCastException: com.somecompany.somepackage.SecurityChange cannot be cast to liquibase.change.custom.CustomChange
	at liquibase.change.custom.CustomChangeWrapper.customLoadLogic(CustomChangeWrapper.java:337) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.change.AbstractChange.load(AbstractChange.java:708) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.change.custom.CustomChangeWrapper.load(CustomChangeWrapper.java:312) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.changelog.ChangeSet.toChange(ChangeSet.java:468) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.changelog.ChangeSet.handleChildNode(ChangeSet.java:401) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.changelog.ChangeSet.load(ChangeSet.java:325) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.changelog.DatabaseChangeLog.createChangeSet(DatabaseChangeLog.java:606) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.changelog.DatabaseChangeLog.handleChildNode(DatabaseChangeLog.java:336) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.changelog.DatabaseChangeLog.load(DatabaseChangeLog.java:305) ~[liquibase-core-3.8.9.jar:na]
	at liquibase.parser.core.xml.AbstractChangeLogParser.parse(AbstractChangeLogParser.java:23) ~[liquibase-core-3.8.9.jar:na]

I’m puzzled by this one – is this the result of changing the ResourceAcccessor? Is there way other way I can work around the fact that the installer.xml seems to be shows up in multiple classpath roots? I’ve dug a little into newer versions of Liquibase and so far it looks like I see the same problem with them.

2

Looks like searchPath on liquibase 4.13+ might be an option, although after reading the docs I’m a little fuzzy on whether or not it would help. We’re loading from the classpath, the filesystem path does vary depending on local development, deployed environments etc, so its not clear to me if I can use searchPath to resolve it … but possibly?

3

Tried liquibase.duplicateFileMode=WARN briefly, but it looks like that is triggered multiple times and it’s extremely spammy, so I’ll try working with searchPath to see if that gets me to a better place.

4

Hi Geoffrey,

I’d recommend trying to upgrade to 4.20, as we have worked through a number issues related to classloading vs. file lookup. Using FileSystemResourceAccessor looks like the right direction.

The subsequent issue related to the ClassCastException is something different. Do you have an example from a ChangeLog file of how you’re using CustomChange via your class com.somecompany.somepackage.SecurityChange. And does SecurityChange implement liquibase.change.custom.CustomChange?

- PJ

5

Yeah, I suspect that moving to v4.13+ may well give me the tools to solve the issue, but it’s also very definitely going to create more work, and even then we might run into more problems. :wink:

Checksums

Checksum calculation has changed:

799 changesets check sum
          release-1.0.xml::0::author was: 7:d41d8cd98f00b204e9800998ecf8427e but is now: 8:d41d8cd98f00b204e9800998ecf8427e
...

Using validChecksum on 799 changesets sounds like a bad idea; could clear checksums but even that will require some environment-by-environment work since it’s not something covered by the servlet filter currently.

Validation Changes

Some errors on a quick test of a blank database that aren’t present on the earlier version:

[ERROR] Failed to execute goal org.liquibase:liquibase-maven-plugin:4.20.0:update (default-cli) on project iams:
[ERROR] Error setting up or running Liquibase:
[ERROR] liquibase.exception.ValidationFailedException: Validation Failed:
[ERROR]      1 changes have validation failures
[ERROR]           This version of mysql does not support non-literal default values, release-1.0.xml::63::javier

Might mean there’s always been something slightly off about that changeset and the new version of Liquibase is better at detecting it, but it’s also evidence that there’s work to be done. Not surprising, but not a simple drop-in replacement either.

SecurityChange

I’m not yet sure if that’s going to be an issue with v4.20.0, but, yeah, there doesn’t seem anything particularly surprising about the class. It’s doing some encryption of PII using a Java lib. Looks fairly normal on a quick scan, although I don’t think I’ve done any CustomChange work before.

public class SecurityChange implements CustomTaskChange {

	@Override
	public void execute( Database database ) throws CustomChangeException {
		Properties properties = new Properties();
		try {
			properties.load( this.getClass().getResourceAsStream( PROPERTIES_FILE ) );
		} catch ( IOException e ) {
			e.printStackTrace();
			throw new CustomChangeException( e );
		}
		Connection connection = ( (JdbcConnection)database.getConnection() ).getUnderlyingConnection();
		try {
			PreparedStatement preparedStatement = connection.prepareStatement( "select id, piiColumn from TABLE;" );
			ResultSet resultSet = preparedStatement.executeQuery();
			while ( resultSet.next() ) {
				int id = resultSet.getInt( "id" );
				int piiColumn = resultSet.getInt( "piiColumn" );
				if ( piiColumn != 0 ) {
					PreparedStatement updateStatement = connection.prepareStatement(
							"update TABLE set piiColumn = ? where id = ?" );
					updateStatement.setString( 1, AESUtils.encrypt( String.valueOf( piiColumn ), PASSPHRASE ) );
					updateStatement.setInt( 2, id );
					updateStatement.executeUpdate();
					updateStatement.close();
				}
			}
			resultSet.close();
			preparedStatement.close();
			connection.commit();
		} catch ( SQLException e ) {
			e.printStackTrace();
		}
	}

	@Override
	public String getConfirmationMessage() {
		return "Successfully applied security patch";
	}

	@Override
	public void setUp() throws SetupException {
	}

	@Override
	public void setFileOpener( ResourceAccessor resourceAccessor ) {
		//do nothing
	}

	@Override
	public ValidationErrors validate( Database database ) {
		return null;
	}
}

Obfuscated the code slightly, but that’s roughly the code. There’s some things that could be better – re-using the prepared update statement, for instance, but nothing that seems likely to cause class loader issues.

6

As a temporary workaround, I’ve made a subclass of ClassPathResourceAccessor that is aware of the resource path for Liquibase resources and filters out the duplicates – a bit like setting the duplicate file mode in later versions of Liquibase.

Seems to work without triggering whatever’s causing the ClassCastException with the FileSystemResourceAccessor. Would still like to bring the Liquibase version forward, but this at least takes the pressure off the timing of it.