character encoding and sql output

Hi to all,

i have some character encoding problems. I want to insert some data via csv and loadData into my database. In case of updating the database, everything works fine. But if i want to see the sql changes before inserting data (via outputFile), then every special character (umlaut) is replaced by ‘?’. I tried different encodings of my .csv file (utf8, iso) with setting encoding parameter inside loadData-tag. But every time the sql output of the database update task contains (a real) ‘?’ instead of the correct character (umlaut).

Does anybody know what to do for solving this issue? What encoding should i use for my csv file?

Thanks.

  • liquibase 1.9.5
  • mysql (charset latin1)

Can you try it with the latest 2.0 RC build from http://liquibase.org/ci/latest ?  Since it is just testing the 2.0 code against your changeset, run it against a backup of your database.  It shouldn’t modify your databasechangelog table or anything else since you are running updateSQL mode, but it’s good to be sure.

Nathan

The latest build of 2.0 RC doesn’t solve the problem.

I made some tests with an utf8 encoded test database. Running updateSQL (with the command line or ant) will produce the same issue: missing umlauts (only ‘?’). The encoding of the database makes no differences in creating output files.

Updating the database still works fine, but i want to check the created sql statements before updating…

Any other idea? Let me know, thanks.

I’ve tested with current trunk and also add two integration tests about encoding and it works correctly when updating to database or to a SQL file.
The only bug about encoding that I found is that by default if you don’t specify an encoding UTF-8 is not the default one as documentation says (Added isue CORE-605).

Der Schade, if you used the last trunk version and specified the encoding manually it has to work. Can you double check that the encoding of the csv file is correct? Can you check with an editor like Notepad++ http://notepad-plus.sourceforge.net/es/site.htm or similar?
Also take note that by now the output encoding of the generated SQL file can’t be specified and is always the java system default encoding (look at System.getProperty(“file.encoding”):wink:

What we can possibly do as a new feature request is a param to specify the output encoding.

Can you recheck with the indications given?

That’s it. The java system default encoding is: ANSI_X3.4-1968 (i can not change these settings). No chance to map characters correctly.

What we can possibly do as a new feature request is a param to specify the output encoding.
-that would be great-

Thanks for your help.

I created an issue in the jira http://liquibase.jira.com/browse/CORE-628 to track that.
In the meantime I think that if you use the commandline tool you can use -Dfile.encoding to set the default encoding for the program execution.

Thanks for creating an issue for that.
I’m using ant. So I changed my update database target to something like that:

                           

It works great.

What will be syntax of using “updateSQL” with maven?
How can I specify a file or stdout for pom to give me SQL generated for change logs.