Can I Load Metadata with DataLoad

in Liquibase Users • 2 months ago

I am wanting to import a CSV of data that contains data attributes from varying datasets. I also have accompanying metadata that I would like to include in the LoadData. Is there a way to accomplish this? I have tried and failed to do so. If you look below there are several column I would like to set a default value for when each record of the CSV is added. For instance, for this change set when pulling in the "column__name", "data_type", and "length" from the CSV I would like to set "dataset_name" equal to "ZONES", "collection_effort" to equal "Q22018 Data Inventory", ect.....

For example:

    <loadData encoding="UTF-8"
              file="file/path/to/ZONES.csv"
              quotchar='"'
              schemaName="test"
              separator=","
              tableName="data_schema">
        <column name="id" type="SKIP"/>
        <column name="dataset_name" defaultValue="ZONES" type="STRING"/>
        <column name="collection_effort" defaultValue="Q22018 Data Inventory" type="STRING"/>
        <column name="information_source" defaultValue="Person 1" type="STRING"/>
        <column name="reported_by" defaultValue="Person 2" type="STRING"/>
        <column name="reported_date" defaultValue="2018-05-03" type="DATE"/>
        <column name="captured_date" defaultValue="2018-05-07" type="DATE"/>
        <!--<column name="name" header="Column Name" type="STRING"/>-->
        <column name="column_name" header="Column Name" type="STRING"/>
        <column name="data_type" header="name" type="STRING"/>
        <column name="length" header="Max Length in Bytes" type="STRING"/>
        <column name="operational_status" defaultValue="Active" type="STRING"/>
        <column name="operational_datetime" defaultValue="2018-05-07" type="DATE"/>
        <column header="DDL name" type="SKIP"/>
        <column header="precision" type="SKIP"/>
        <column header="scale" type="SKIP"/>
        <column header="is_nullable" type="SKIP"/>
        <column header="Primary Key" type="SKIP"/>
    </loadData>
</changeSet>

Replies(4)

Steve Fro..

Re: Can I Load Metadata with DataLoad

2 months ago

It is not clear from your question what metadata you want to load, what you have tried, what your expected results were or what the actual results were. "I have tried and failed" just isn't descriptive enough.

Steve Donie
Principal Software Engineer
Datical, Inc. http://www.datical.com/

Cameronee..

Re: Re: Can I Load Metadata with DataLoad

2 months ago

Sorry about that. I have updated my question with a little more detail. Please let me know if you can now understand what I am asking.

Thanks.

Steve Fro..

Re: Can I Load Metadata with DataLoad

2 months ago

Thanks - that helps a little. You still haven't mentioned the actual results you got, but I'll take a guess. I think that you are expecting that when the loadData change is executed, the columns that have a defaultValue attribute will be populated with that defaultValue if the entry in the CSV file is null. I also think that your actual results are that the rows where those values are null in the CSV file are still null in the database, and that no error messages were shown.

I think that the issue is that the column tag probably does not support the defaultValue attribute when used inside a loadData change. The documentation is unclear on this since in the docs for loadData only says "see the column tag documentation", which does say that there is a defaultValue attribute that can be set.

I see two possible approaches to this:

1 - when you create the table itself you could set the default value for the columns. It isn't clear if you are using Liquibase to manage your schema already or not. If there is currently no defaultValue set for those columns and the table already exists, you could create a changeset to set that up.
2 - you could ensure that the CSV file has the default values where needed.

Personally, I would probably use option #2, but it would depend on whether this is a recurring task or a one-time only task, whether the default values are always the same in a recurring scenario, and other similar considerations.

Steve Donie
Principal Software Engineer
Datical, Inc. http://www.datical.com/

Cameronee..

Re: Re: Can I Load Metadata with DataLoad

2 months ago

Yes I forgot to do that....you assumed correctly. When I executed the LoadData changeset, none of the columns with defaultValue Attributes were inserted into the table but instead contained a null value. The documentation is not clear on this and now we know the column tag does not support defaultValue.

I went ahead and used option two and it worked great....albeit time consuming. Really with there was a value attribute for loadData like the insert tag has. Thanks for your help.

Top Reply

Response title

Attachments

New to this Portal?