I am trying to load data from a CSV file. It works when I provide a fully qualified filename with the full path to the CSV file. Since this application is used by several people to load data I need some way to make the path to the file relative. I tried a bunch of different takes on the CSV file location but no luck with a relative path. I have the following:
<?xml version="1.0" encoding="UTF-8"?>
http://www.liquibase.org/xml/ns/dbchangelog/1.9"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd">
I was able to get change logs to be relative but no luck with CSV files.
Is there a solution?
Scott
It should look up the files relative to the classpath, so file=“com/testapp/domain/changelog/data/sub/csv/setting.csv” should work
Nathan
That is not a classpath but just a file location. Is there a way to specify a file location relative to something like the changefile like you can do in the change files themselves? When other users run these updates they will not have the files in the classpath as they change very often and might just be a CSV file they want to upload to the database.
The file is not in the classpath of the current application.
Thanks for your continued support and a great Open Source project.
Scot
Hi,
I’ve just posted a new topic about which seems to expect only classpath files. Logically, if it works the same way, loadData “file” only expects classpath files… I hope there’s another way to do this.
Anyway, when you say “the files in the classpath as they change very often and might just be a CSV file they want to upload to the database”, does it really work? Don’t you have problems with checksums like this :
Caused by: liquibase.exception.ValidationFailedException: Validation Failed:
2 change sets check sum
sirius_base_dml_changelog.xml::id1::author::(MD5Sum: b060d1da4ae1ahvhjvljvhvhvhvvhvhv)
sirius_base_dml_changelog.xml::id2::author::(MD5Sum: 5412aec98788azujsdbfjsdbfjfbjsbfbff)
at liquibase.DatabaseChangeLog.validate(DatabaseChangeLog.java:119)
at liquibase.Liquibase.update(Liquibase.java:106)
at liquibase.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:214)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1368)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1334)
… 177 more
This error produces when the csv changes for a changeSet that already exists in your databasechangelog table. I suppose the good practice would be to create a new changeSet with a new CSV. I agree this it not very flexible for the user that needs to buil and deply the project each time he wants to modify a reference data. Has anybody a idea about that?
Thanks for the reply. What I meant when i said that the files are changing is that we are constantly adding files and recombining them to support different customers. WE have a grails app we have developed to support our upgrades of the 1000 or so client databases we have. It is similar to liquibase IDE but more flexible to meet our needs and runs on 2.x. We would like to be able to add new clients easily by adding new XML files and CSV files with that clients data and then load the data using our web tool. If the data has to be in the classpath then we have to reload the application each time we add new files and point to a fixed spot on disk rather than a variable. Also if we need to load emergency data we want to just update the changelog files and add new csv files and load the data.
We are able to keep our changelog xml files in a directory and point to that directory for model changes and other changes but not for CSV loads. It would be tough to have a dynamic environment and have to always have the csv files in the classpath.
I hope there is a solution.
The reason that liquibase throws a checksum error if you change the csv file is because we only insert data based on the csv file, not update existing data. So if you executed the changeset at one point against a database, then change the CSV, it looks like you expect more data in the database than was originally loaded.
What do you expect should happen if you change the csv file? There is a onValidationFail=“MARK_RAN” attribute you can set on the changeSet tag to supress the checksum errors as you change data.
Nathan
I generally use classpaths myself and therefore liquibase best supports classpath-based file lookups because most of the time absolute paths are bad because they get stored in the database. Relative paths are a good option as well, but they are newer and not as well supported and tested, although they should be. Classpaths generally seem nicer because they are like relative paths, but are more packaged with your application. There are definitely use cases where they don’t make as much sense, though.
The way liquibase loads files is by using ResourceAccessor implementations (2.0 codebase terminology). There are many ResourceAccessors built into liquibase (AntResourceAccessor, ClassLoaderResourceAccessor, FileSystemResourceAccessor, and CompositeResourceAccessor that lets you combine multiple ones). The ResourceAccessor to is passed to the contstructor for the liquibase.Liquibase object and is used for finding changelog files and loading csv files. If you have a custom way to lookup resources you can create your own ResourceAccessor implementation and use that.
Scryan, how are you starting liquibase? Do you know what resource accessor is being used?
Nathan
I like the current functionality. Once I run the CSV file I should never be able to run that CSV file against the same database again unless I do a rollback. If I need to add more data I should create a new CSV file with some version strategy so that everything is in sync.
- I need to check to see how to rollback an entire CSV file if that is possible.
My main issue is being able to use relative file path names to the CSV file. For example if I have the following:
global-change-set
changeset-1 with csv-1 and csv-2 (This is in global)
Now I add changeset-2 with csv-3 and csv-4(This is in global)
The application can find the global change set and the changeset-1 and changeset-2 without them being in the classpath but the csv files are not found. It would be great if I could reference them like I do my changesets.
Here is an example for my model side of the house
MASTER CHANGE SET
<?xml version="1.0" encoding="UTF-8"?>
http://www.liquibase.org/xml/ns/dbchangelog/1.9"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd">
It would be cool if I had the same capability to set the csv relative to the changeset
I am using the command line to test with but my application code uses a File based resource accessor. The problem with the classpath is that when I run inside a server like tomcat I don’t have access to the classpath or the directories in the classpath. This is why I need to point to a set of directories that are more public.
I am having luck with the file accessing and relative paths but they are not currently supported for CSV files and I don’t have access to change them as that code is internal to the liquibase core.
Scott
Hi,
In my opinion, it’s a good thing that the checksum validation fails when you modify your CSV. I discovered it during the development and agreed.
As far as the other subject is concerned, our “run” team gives us a constraint to put our property files in a special folder of a special partition of our servers. This enable to change properties without re-deliver a re-built package, with all the risks we have with a delivery.
This folder is defined in an environment variable, therefore, your solution of a relative path wouldn’t be usable in the case of property files.
I tried because this “file:” is used by Spring to read property files (PropertyPlaceHolderConfigurer).
Anyway, for the moment, I put my property file in the classpath and it works correctly!
Thanks!
I created an issue for it (http://liquibase.jira.com/browse/CORE-656) and will get it fixed in 2.0 final.
Nathan
From what i can tell, the csv files are loaded by the ResourceAccessor configured when liquibase started, and all the build-in liquibase runners use a compositeResourceAccessor that includes the FileOpener. How are running liquibase?
Nathan
