searchPath? resourcePath? What should it be called?

Hello amazing Liquibase users! We are working on some improvements related to helping resolve some of the confusion related to classPath and how to identify in your properties/parameters the locations of changelogs, csv files, etc. and would like your input.

Currently in the CLI (and other places) we have a single argument of “classpath” which we use to configure BOTH the classpath for looking up java drivers and extensions AND the root locations to search for “resources” (changelog files, csv files, etc.). This is not clear to users and causes confusion. To help clear up the confusion effort, will are working on splitting things out into:

  • searchPath(final name TBD) will used to look up files, to set the base/root for relative pathing to resources, like changelogs.

  • classpath is used to look up classes

QUESTION: What should we call this new path? searchPath? resourcePath? Some other name we aren’t thinking of? What would be short, sweet, descriptive, and help us more intuitively understand the purpose of the path?

NOTE: We will be making the change backwards compatible, so if you are currently using classpath for looking up files like changelogs, it should still work with the upcoming change.

Thank you all!

Karen
Product Owner, Community
DevOps + database + open source = <3

2 Likes

@EduardUta @premierrange what do you think? What name do you prefer?

I kind of like resourcePath, but searchPath works too.
Good improvement BTW

1 Like

Hi Karen, Team,

First of all, what a nice idea to “ask around”, really nice to involve the community in things like these. Plus, it raises awarness about this future coming change!

Now we all know that one of the hardest things a developer has to do is to name a variable or parameter so I understand the struggle here :smiley:
Joking aside, here are my votes, in the order of preference:

  1. basePath or rootPath
    • pro: clean, simple names. Covers a good variety of potential objects that can be found in that place.
    • con: can be a little misleading because it seems a little too generic (but just at a very minimal level).
  2. sourcePath
    • same as above, but as a second option for no reason I can explain now.
  3. resourcesRootPath
    • pro: more “verbose” / detailed name; leaves little room for misinterpretation.
    • con: more words to type, longer command.
    • con: can make you think the location isn’t actually user related one (for user defined resources) but more for Liquibase framework scripts/objects.
  4. searchPath
    • con: is misleading regarding what it represents I think; resonates more with a search / filtering functionality. (Yes, indeed, the actual thing Liquibase does is “crawl” (read search) through that entire location for things to “chew” but that I think is assumed to be a known fact; just thinking from this perspective: a name to represent the destination (as the name ends with Path) and not the sub-process/functionality behind that parameter - if this makes sense).

Thanks and best regards,
Eduard

PS: Other names: assetsPath, artifactsPath, componentsPath, projectRootPath.

1 Like

I feel Liquibase is rooted in the Java ecosystem which I haven’t worked in for years so am not really up with the terminology people use.

resourcePath seems best to me, as with @almccr . A close second would be rootPath, but that seems a little bossy, or basePath, also good and I think often used in build projects (webpack?) to denote the path from which paths will be resolved. However, to avoid naming conflict with these build tools, resourcePath is distinct and doesn’t imply the files found may be transformed.

sourcePath is misleading as it seems to denote a location of source files that will be transformed into output files. searchPath seems too vague as it seems to indicate a specific thing is being searched for without specifying what it is, in comparison resourcePath denotes resources being searched for which I think is specific enough while leaving the exact type of resource vague, a good compromise.

@premierrange partly what we’re trying to help with this new term is to better avoid Java ecosystem terminology. So the view from outside that world is greatly appreciated.

The arguments against “source” related names like sourcePath make sense, since for developers “source” means something often different than what should go here. For example in maven, the source directories are NOT included in this only the target directories.

One worry we had internally was that “resourcePath” makes it sound like you are giving a path to a resource. Like that is where you specify the root changelog file or something. It didn’t seem well descriptive of “list of locations we look up any given paths to resources from”?

It is a good question, @EduardUta whether “searchPath” implies a different sort of searching/filtering than is happening. What I like about searchPath is that it is focusing on the fact that it is for searching since that is what it’s for. But maybe the first thought people have for “search” will be wrong?

What I worry about with something like “projectRoot” or “baseDir” or something which is built around the idea of a single location or “project” location, is that there are many ways of running liquibase which don’t have a “project” type location. Like running embedded in an application. And there is always the ability to have multiple locations in this, and we search them all. Any sort of “project root” type name will be an addition to the overall list of places we look through.

In the end we need to be able to talk about a global “When you reference a file, you are not specifying the actual file path. Instead, Liquibase looks up that path in the X which is a list of places which that file could be.” Something like “project root” or “base dir” won’t work for that X. While we could have different terms, like “your projectRoot gets included in the X” it seems nicest to be able to be more consistent like “LIquibase has a default searchPath defined by the specific integration, but you can add to it with the searchPath parameter”

Nathan

2 Likes

Hi Nathan,

Thanks for the additional details on this, makes more sense now with regard of “search” term and how things are/need to be in this “area”.
Can think of similar terms, if it helps the discussion: lookupPath or explorePath (?).

Regards,
Eduard

Nathan, you’ve said

“One worry we had internally was that “resourcePath” makes it sound like you are giving a path to a resource. Like that is where you specify the root changelog file or something.”

but the original question said the path was for

" * searchPath(final name TBD) will used to look up files, to set the base/root for relative pathing to resources, like changelogs."

I’m left a bit confused as to what this new config parameter will do :slight_smile: I confess I’ve always kept things super simple with Liquibase, running it from the CLI with the changelog.xml and all changelog XML files in the current directory. Maybe a more specific example would help here, showing why and how you would distribute files in different locations, and how the proposed config would allow you to point Liquibase at them.

For example, is it common to have your changelog.xml file in a different location to the XML files that it refers to? My changelog.xml has a list of “” lines with each file in the same location.

I’m left a bit confused as to what this new config parameter will do… Maybe a more specific example would help here, showing why and how you would distribute files in different locations, and how the proposed config would allow you to point Liquibase at them.

(I’ll use the “search path” term here as a placeholder. Also maybe starting too basic but building it all up for whoever is following along)

Changelog files need to have paths coded into them. But we don’t want to be putting absolute paths in there, because the full paths are often different on different systems.

For example, if you had a root changelog like:

<databaseChangeLog>
    <include file="/home/example/my-project/src/dbchangelog/changelog-1.0.xml"/>
    <include file="/home/example/my-project/src/dbchangelog/changelog-1.1.xml"/>
    <include file="/home/example/my-project/src/dbchangelog/changelog-1.2.xml"/>
</databaseChangeLog>

If you reorganize your machine so it’s not checked out to /home/example/my-project anymore those need to be changed. Or when another developer checks it out to a different location on their system. Or when you check them out to the build server. Or if you package them up into a spring application to run on startup. Etc.

So instead, you want those files to be the same for everyone like:

<databaseChangeLog>
    <include file="dbchangelog/changelog-1.0.xml"/>
    <include file="dbchangelog/changelog-1.1.xml"/>
    <include file="dbchangelog/changelog-1.2.xml"/>
</databaseChangeLog>

and leave it up to a runtime configuration (searchPath) to specify where to look for those files. On your machine, you can set searchPath to /home/example/my-project/src/ On the build server, it might be /runner/checkout/src. When it runs packaged in an application, those paths are looked for in the application archive.

To make things easy, the CLI (and other Liquibase interfaces) will set up a default searchPath of “normal” places to look. The CLI for example, includes the current working directory in the search path, so as long as you run liquibase from /home/example/my-project/src/ you don’t have to set anything.

Besides include/includeAll, we also use the searchPath for finding data files in loadData/loadUpdateData as well as looking for liquibase.properties and other configuration files.

But… sometimes you aren’t running the CLI from /home/example/my-project/src/. What do you do then? You need to add that directory to your searchPath with --search-path=/home/example/my-project/src/. With that setting, it doesn’t matter where you run Liquibase, it will know where to look.

Or, maybe your changelogs get packaged into a changelogs.zip file on the build server. By setting --search-path=/path/to/changelogs.zip, Liquibase will find the files in that zip. Or your changelog files are stored remotely in an s3 bucket you can use --search-path=s3:/changelogs/latest.

The “searchPath” setting is what lets you specify those non-default places Liquibase should be looking for files in.

A bit more complex is a more componentized setup. For example, you may have an application which is divided up into a cart and an accounts sub-module, where each sub-module has it’s own changelog file. So your overall project structure looks like:

src/main/resources/dbchangelog/changelog-root.xml

accounts/src/main/resources/dbchangelog/accounts/changelog-root.xml
accounts/src/main/resources/dbchangelog/accounts/changelog-1.0.xml

cart/src/main/resources/dbchangelog/cart/changelog-root.xml
cart/src/main/resources/dbchangelog/cart/changelog-1.0.xml
cart/src/main/resources/dbchangelog/cart/changelog-1.1.xml

If you ran liquibase from the root of that directory structure, you’d run liquibase --changelog-file=src/main/resources/dbchangelog/changelog-root.xml update and your overall root changelog would look like:

<databaseChangeLog>
    <include file="accounts/src/main/resources/dbchangelog/accounts/changelog-root.xml"/>
    <include file="cart/src/main/resources/dbchangelog/cart/changelog-root.xml"/>
</databaseChangeLog>

which works, but is baking that current project structure into the changelog files. Since liquibase uses those paths for identifying changeSets, if you reorganize your project, want to reference the “compiled” versions in your project, or want to run from a built artifact you are going to hit problems.

Instead, you’d like to have your root changelog file look like:

<databaseChangeLog>
    <include file="dbchangelog/accounts/changelog-root.xml"/>
    <include file="dbchangelog/cart/changelog-root.xml"/>
</databaseChangeLog>

and set search-path=src/main/resources,accounts/src/main/resources,cart/src/main/resources

With that setting, when liquibase sees the reference to dbchangelog/accounts/changelog-root.xml, it searches for it in each of the specified searchPath locations. If you ever reorganize your project, you just update your searchPath and liquibase continues to identify all the files with a consistent path.

All that behavior is how Liquibase has always worked, but we used the “classpath” setting for it because early versions were more focused on running embedded in an application and the classpath is what provides that “search path” functionality in a java application. As we added a CLI etc., we just kept that “classpath” term (and originally implemented it relying on that java feature) but it continues to cause confusion so we’re looking to improve that terminology. And so we want to find the right word that describes that “list of base locations we look in for all the later specified paths in”. Ideally something that works for that overall concept like “Any time you are defining a path to a resource (like include’s ‘file’ or includeAll’s ‘path’ or the changelogFile setting) it is looked in the search path. Each interface will configure the search path with reasonable defaults, but they can be added to with the searchPath setting”

Does that help?

Nathan

2 Likes

Very helpful, @nvoxland , thank you. The name searchPath seems good and natural, better than resourcePath, when you describe it like that, though to be honest it’s a bit of a coin toss :slight_smile:

1 Like

Great idea, the property name “classpath” threw me off initially as well. In some scratch test automation I used to test a series of different scenarios, I was using the name “changelogRoot” which is definitely too narrow since the property denotes the paths to search for changelogs, CSVs, etc. and not just changelogs.

To me, “basePath” makes the most sense. It denotes the base directory in which to look for all specified items defined in changelogs. Which are all relative paths, right? Well except for ‘*’ entries, but those are also relative to the “base directory” defined by the property.

So “basePath” really ticks all the boxes for me.

we actually JUST decided to move forward with searchPath. I like basePath too. Thanks for the suggestion and it’s good to hear from you!

1 Like