What does loadData utility uses for Oracle?

Liquibase Development
, ,
1

Hello,

I need to load big amount of data into Oracle tables and I am trying to use loadData functionality (changeset type).
My question is: what does this Liquibase functionality has “under the hood” ?

  • is it Oracle Datapump?
  • is it Oracle SQL Loader?
  • is it a new Liquibase in-house built functionality?
  • Other?

I am asking because I am having some issues with the way I am using/referencing the files (in a container setup) and I need to see what options I have (the plan is to not have the input file(s) on the server/container of course, but access/reference them from other location, which is apparently not possible (at least on Windows)).

Thanks,
Eduard

2

It’s in-house built functionality. We want something that works consistently and similarly across all the different databases, so it’s really just:

  1. Generate an insert statement based on the loadData config
  2. Parse the csv file
  3. For each line in the csv file, run the insert statement with the correct data

It’s not going to be as performant as datapump etc, but Liquibase is also generally used more for seed data management than heavy duty data manipulation. And if there is always if you really want to run another tool.

Regarding the file loading, we have a separation between the “search path” which is the set of base locations where liquibase should look for files, and then the file path you put in the changelog which is a path within one of the search path entries.

We have that separation because the physical locations of the files will likely differ in different environments, but you want the exact same changelog file and therefore can’t have the complete path in there.

The default searchpath depends on the way you are running liquibase. The CLI uses the current working directory for example. But it can be set to whatever you need.

I don’t know what you mean by " plan is to not have the input file(s) on the server/container", do you mean that you are running liquibase from a docker container which doesn’t have the files built into the container? Are those mounted into the container? Or how would you ideally be accessing them?

Nathan

1 Like
3

Thanks Nathan, that really helps / clears out my questions. That makes sense.
Will also re-visit the search path functionality for sure.

Wr are running Liquibase from outside the container and need the files to be outside the container as well.
Based on the reply, for our “particular” case, given performance and large volume of data are “decisive” factors, we’ll probably end up using Ora SQL loader for this particular load.

Thanks again, Liquibase rules :smiley:
Eduard

1 Like
4

Makes sense, right tool for the right job.

For reference, if you or anyone is ever needing support for search path entries that are remote systems, it is possible to write custom extensions that add support for new URL types within your searchPath configuration. And we are looking to add support out of the box for non-local locations in Liquibase Pro in the relatively near future.

Nathan

2 Likes
5

if liquibase generate insert statement , the process will be very slow and consuming more memory . Is not good option

6

I think this was accepted in Nathan’s original answer:

It’s not going to be as performant as datapump etc, but Liquibase is also generally used more for seed data management than heavy duty data manipulation. And if there is always if you really want to run another tool.

1 Like
7

Yes, it was never trying to be super-performant. There are certainly opportunities to make it more efficient, but we’ll probably not look at them for a bit. For very-performant needs it will likely always be faster to make a call to database-specific native tools, and you can do that currently.

Nathan