June 25, 2009 Meetup Followup/Transcript

Thanks to everyone who made it to the meetup today.  I think it went well and hopefully it was useful to you. 

What feedback do you have on how it went?  Chat app work well for you? Discussion flow good or bad? Other thoughts?  I’m thinking that the chat app had too much of a delay typing and sending and we should switch to a real IRC server, or something else next time.  It worked well if it wasn’t for the typing delay. 

Is there any additional comments on topics discussed?

The transcript is too large for the 20k character limit, so I will post the first half here, and the second half in a follow-up post.

(11:56:49) Nathan_Voxland: Hello everyone (11:57:02) sduncan: hello (11:57:04) tvial: Hi! (11:57:07) kkrikor: Hello (11:57:13) upul: Hi (11:57:21) taranenko: morning (11:57:39) Nathan_Voxland: Good to see you, It will be interesting to see how well this works :) (11:57:44) otatop: testing 1.. 2... 3... (11:58:23) Nathan_Voxland: We'll wait a couple mintutes to really get started (11:58:51) Nathan_Voxland: A couple things to think about: afterwords I would like some feedback on what went well, what didn't with this so we can improve them in the future (11:59:11) Nathan_Voxland: Related to chat technology, discussion management, etc. (11:59:27) tvial: Will there be a transcript of the conversation ? (I won't attend the whole thing :() (11:59:56) Nathan_Voxland: I am hoping to get a trascript.  It may just be copy/paste, but I thinki ti will work (12:00:18) tvial: great! (12:00:37) Nathan_Voxland: How many of you have actually look at/worked with the liquibase code before, vs. simply using it? (12:00:46) Nathan_Voxland: (as a guide for talking) (12:01:18) kkrikor: Looked at the code very briefly (12:01:36) tvial: only used it so far, and not in a real project yet (12:02:02) otatop: We are going to be using liquibase in my current project but haven't done much with it yet. (12:02:24) taranenko: I just spending 7-8 full days to working with codebase. Basically to make JUnit tests working and some new feature implementing (12:02:39) kkrikor: We do use it in our project, including maven and spring integrations (12:03:09) upul: I am a user (12:03:50) taranenko: additionally i'm actively using liquibase in 4 different projects (12:04:09) Nathan_Voxland: Great, thanks for letting me know.  Lets start it off with an open question of "is there anything in particular you would like to cover?" (12:04:53) tvial: here's a hard one ;) I've heard that Visual Studio's DB designer and Rail's db:migrate tool can compare pretty much with LiquiBase wrt DB refactoring, and are more mature. I don't know those tools; what is your opinion on them? (12:05:28) kkrikor: rolling back changeSets using their ids (12:06:07) taranenko: i'm strongly need to sync table's data (12:06:07) otatop: I'm really interested in what's the current priorities, any really big tips, and finally I'm still really hoping for some enhancments for hibernate integration (there's a feature request out there for it) (12:06:25) kkrikor: that would specifically be useful when the changeSet is breaking something (12:06:45) Nathan_Voxland: I'll keep queuing up questions, so even if we are on something else, keep asking (12:07:09) Nathan_Voxland: I'll start with the first one... (12:07:50) samawon: is it possible to generate full database scripts without connecting to the database? thinking of using it inside the build system (12:08:14) Nathan_Voxland: tvial: A hard one, but luckly I'm very opinionated on the subject :) I haven't used visual studio's db designer to really know what it does, but from what I understand it is a diff based system. (12:08:49) Nathan_Voxland: It compares your local database with the "standard" database to figure out what changes need to be made. (12:08:54) kkrikor: Define strategies on how to write changeSets for multiple dbs (12:09:32) Nathan_Voxland: I think in general doing a database diff is not a safe way to track chages.  See http://blog.liquibase.org/2007/06/the-problem-with-database-diffs.html. (12:10:12) tvial: I agree with that. Also, merging two SQL scripts by hand is not a trivial task (12:10:18) Nathan_Voxland: The trouble comes down to just because you can compare two points in time, you loose important information if you don't understand what happened in between. (12:11:50) tvial: well you've got commit comments from source control, but they can be very coarse or simply missing (12:11:55) Nathan_Voxland: rail's db:migrate I also think is inherently broken (12:11:55) otatop: Agreed. My dream would would be liquibase comparing the current liquibase xml with the hibernate annotations and appending to the liquibase xml what additional changes are necessary. That would rid us of needing a real db to gen the liquibase file. (12:13:01) Nathan_Voxland: My rails blog post: http://blog.liquibase.org/2007/06/the-problem-with-rails-active-migrations.html (12:13:31) Nathan_Voxland: The trouble with the hibernate diff is that you still can get wrong changeSets. (12:14:20) Nathan_Voxland: With databases, it is just as important HOW you get from one state to the next compared to THAT you get to the final state. (12:15:09) Nathan_Voxland: The example I use is renaming a column from "firstname" to "first_name", a diff program would see "there is no longer a firstname column and we are missing the 'first_name' column.  drop firstname, add first_name' (12:15:27) Nathan_Voxland: it gets you to where you need to be, but you loose your valuable data in the process (12:15:54) tvial: that's a very good example & argument (12:16:11) Nathan_Voxland: That is the main reason why hbm2ddl with hibernate isn't safe to use directly either.  It is going to do the diff and you could loose data. (12:16:43) Nathan_Voxland: The goal of the diff support we have is to help out creating changesets, but should not replace them. (12:17:51) Nathan_Voxland: comparing the hibernate mapping with your database and appending to the changelog is a good example, because 90% of the time it is right and so easy to do, but you need to actually read what is being added and change it as nessisary before applying it. (12:18:29) otatop: As long as the tool can figure out the final state from the liquibase file I think it'd be good to have a generate from hibernate that compared the xml and the hibernate annotations. If the tool generates a drop "first_name" and then an add "firstname" I'm ok with that, as long as I can modify it afterwords and on subsequent runs it can figure out that I've already handled the rename to "firstname" myself. (12:18:41) taranenko: another technical question: is there a plans to change building from ant to maven? (12:18:48) Nathan_Voxland: Comparing hibernate.xml to liquibase.xml is non-trivial when you allow SQL and end-user extensions, however, so I doubt we will ever get to that point, unfortuantely otatop (12:18:52) markfarnsworth: can liqua base produce a sql script in the target dialect from a change set? (this would be helpful to hand off to a DBA for formal review) (12:19:07) tvial: I have to go now, thanks a lot for your time and your answers. I'll be looking forward to the transcript :) (12:19:12) otatop: You can't 100% rely on it without reviewing, but it can save a significant amount of leg work to gen then review, vs. just doing it all on your own. I definately understand that it's quite the challange though. (12:19:46) tvial: just another question if time permits -- a list of best practices: when to start from a fresh changelog, how to tie it with the releases of your application, ... (12:20:35) Nathan_Voxland: Ok tvial, thanks for coming.  A bit more on the rails question: They use a more simple version number to control ordering and what is to be applied and that breaks down quickly with muiltiple devs and branches.  That is why we have the separate id/author/filename tracking and why we apply changesets by changelog order, not id order. (12:21:51) Nathan_Voxland: Next question: kkrikor: rolling back changeSets using their ids (12:22:21) Nathan_Voxland: You are asking for the ability to be able to pass in the id/author of a changeset to roll just it back? (12:22:39) Nathan_Voxland: My concern with that is that what happened after that changeset can make rolling back an earlier one impossible. (12:23:15) Nathan_Voxland: for example, changelog1 adds table "person", changelog2 renames "person"->"users", you now cannot rollback "person" by dropping the table because it no longer exists as "person". (12:23:53) Nathan_Voxland: There is a rollbackCount command that lets you undo the last N changes to help troubleshoot why things are breaking. (12:24:37) kkrikor: ok (12:24:45) markfarnsworth: I assum LQB tracks everything in tracking tables but does it also inspect the state of the schema? (12:24:46) Nathan_Voxland: I think there is an updateCount command as well to let you step through changesets.  I suually use those during development to watch what is going on. (12:25:11) markfarnsworth: i.e. does LQB assume that others are not changing the schema? (12:25:41) markfarnsworth: i had been thinking of walking the schema and calculate an MD5 to be sure that no others had mondified the schema (12:25:47) Nathan_Voxland: In a large existing changeset that is failing on a far past changeset, I will usually add the tag temporarily to get it to a particular point and step through it then, or add precondtions to handle new, unexpected cases. (12:25:48) markfarnsworth: but hard to do that in DB neutral way (12:26:12) kkrikor: We run into this issue because we primarily deploy on postgres and then every now and then deploy on oracle (12:26:43) kkrikor: i think it is our approach that is the wrong one (12:26:55) Nathan_Voxland: markfarnsorth: yes, we don't track the state of the schema, just the applied changes.  If something is done to the database outside of liquibase, we would not know it at all. (12:27:19) Nathan_Voxland: kkrinkor: I have ran into the same problem with supporting multiple database (12:27:37) Nathan_Voxland: multi-database support is built in for that reason, however. (12:28:14) Nathan_Voxland: I think the trick is to have all your supported database updated by liquibase on your build server.  That way you will catch any non-compliant changes right away. (12:29:09) markfarnsworth: so basically make sure the user assigned to LQB is the only user with DDL priv (12:29:15) kkrikor: yes that is they way to go (12:29:29) Nathan_Voxland: yes, that is best.  Not always possible, but best :) (12:30:06) Nathan_Voxland: You may be able to do with with using the the generateChangeLog command to make a dump of the schema and doing an md5sum on it. (12:30:27) Nathan_Voxland: WOuld make a good addition to the liquibase extension portal :) (12:31:07) kkrikor: thanks (12:31:14) kkrikor: for the response (12:31:28) Nathan_Voxland: Question: taranenko: i'm strongly need to sync table's data: Unfortunately I think it is a difficult problem to solve in general, so nothing we hve plans to implement in the near future. (12:31:58) taranenko: sad (12:32:15) kkrikor: Another question let's say you have your liquibase scripts creating postgres and oracle schemas and then you get a requirement that you need to support mysql what is the strategy to get that done ? (12:33:15) Nathan_Voxland: you could probably get something going with the data export capabilities, but it will not be overly robust.  I would guess that any data sync support that performs well and is powerful enough is probably going to have to be heavily database dependent. (12:34:16) Nathan_Voxland: I recently had a case where we added MSSQL support to a product that had previouly just been mysql.  What we ended up dong was using generateChangeLog on our fully-updated mysql database and restarted our changelog from there. (12:35:19) Nathan_Voxland: 1/2 hour in.  I do plan to post the transcript to the user forum, and we can continue any threads even after time is up.


(12:35:46) Nathan_Voxland: Question: otatop: I'm really interested in what's the current priorities: The priorities for 2.0 is primarily around extension/integration (12:37:31) Nathan_Voxland: I have been finding that many of the feature requests are falling around one-off, database specific, and things that are easy to solve in particular constrainted environments, but not in general.  THe goal of the 3rd party extension system is to allow those custom enhancements to more easily be added. Possibly with the goal of adding them into the liquibase core once they are feature complete (12:38:45) Nathan_Voxland: Beyond that in 2.0, there is general code cleanup with the 2.0 release, and new feature additions as well, although what is added has not been determined yet.  I would like to get 2.0 out by september, so the actual features that get in will depend on how long the extension work goes. (12:39:10) markfarnsworth: What I am looking for from LQB is an API that allows me to hand LQB a jdbc connection and the path to a change set file.  I would then expect LQB to apply the change set (only if needed) and report success or failure. (12:39:27) markfarnsworth: is such an API already part of the system? (12:39:55) Nathan_Voxland: Question: otatop: hibernate integration: That is something we would like to add, but will need to be prioritized (or have someone prioritize it for themselves and send a patch) (12:40:33) Nathan_Voxland: markfarnsworth: there is an API for that.  See http://liquibase.jira.com/wiki/display/CONTRIB/Liquibase+Facade.  Still improving docs around those apis as part of 2.0. (12:40:45) albatross: Nathan, in order to make it easier for non-Java developers to create enhancements/extensions, it would be nice if these could be created in a scripting language such as Groovy. (12:42:09) Nathan_Voxland: I have been looking and thinking about non-java scripting of parts of liquibase, but have not acted on it yet.  You can call the liquibase apis as needed from groovy, but I don't know what we can do to make it easier, not being a big groovy guy. (12:42:42) Nathan_Voxland: "Error: connection status 0": does that mean this message isn't coming through? (12:43:36) Nathan_Voxland: albatross, if you have any suggestions on what you would like to see as a liquibase scripting option, option please let me know. (12:43:50) albatross: Your message is coming thru (12:44:06) Nathan_Voxland: great, thanks. (12:44:59) Nathan_Voxland: question: samawon: is it possible to generate full database scripts without connecting to the database? thinking of using it inside the build system.  The trouble with that is that you need the databasechangelog table to know what has been applied and what has not. (12:45:16) albatross: Where can we read up on the design of LB 2.0 ? Just to have a starting point to be a beter discussion partner (12:46:03) Nathan_Voxland: without a database you could only generate all SQL which I don't think is what is normally wanted. (12:46:10) albatross: in particular the extension system (12:46:35) Nathan_Voxland: The best 2.0 docs is currenly in the "resources" section of http://liquibase.jira.com/wiki/display/CONTRIB/LiquiBase+Extensions+Portal (12:46:57) samawon: we are using liquibase to generate hsql scripts in development for unit tests (12:47:02) otatop: Is there a way to generate all SQL from say a hibernate config without comparing against a blank db schema? (12:47:28) samawon: and then generate oracle scripts for deployment (12:48:22) Nathan_Voxland: It would be possible to do that, I don't think the feature is in there now as a single command.  Especially with 2.0, there is less dependence on actually having the database there. (12:48:56) Nathan_Voxland: There are changes, howveever, that may query the state of the database to determine what to do. (12:49:36) samawon: ok (12:50:08) Nathan_Voxland: Even if I move on to another question, continue to follow up on past ones if you need.  I want to get through as much as I can and cannot tell when when a topic is done :) (12:50:39) samawon: the other thing is the reverse engineering an existing database misses certain things, for instance Oracle check constraints (12:51:50) Nathan_Voxland: Question: kkrikor: Define strategies on how to write changeSets for multiple dbs: There is some built in support for this in particular.  THere is a dbms attribute on changeSet to control if a changeset is supposed to run on a database, you can use changelog parameter to abstract values (like datatypes) that change from one database to the next, and there is built-in smarts about datatype conversion etc. (12:51:51) otatop: what is the top couple common problems people run into with the current release of liquibase (12:52:26) Nathan_Voxland: kkrikor: is there any specific questions on multi-db changesets you have? (12:53:20) Nathan_Voxland: reverse enginerring a database does miss certain things, and with a cross-database tool there is no way we can get everything.  Another reason why I see the diff tool as a support tool/sanity check tool, rather than something to rely on. (12:54:38) Nathan_Voxland: Question: tvial: a list of best practices: when to start from a fresh changelog, how to tie it with the releases of your application.  There has been discussion on this in the past.  The answer was "it depends" :)  I'll try to pull together some links afterwords (12:56:29) otatop: A solid diff tool is a crazy feature that I see as being a major challange, but still something that should be ultimately strived for. Even if it takes 5-10 years to get right the end benfit would be amazing. Liquibase combined with hibernate and a dependable diff tool would essentially eliminate 99% of developers interaction with the DB. (12:57:06) kkrikor: Yes, Is it important to preserver the count of changeLogs cross DB's ? (12:57:06) otatop: Sorry for bing a bit overzelous on the hibernate issue 8) (12:57:28) Nathan_Voxland: question: taranenko: another technical question: is there a plans to change building from ant to maven? Not currently.  I have yet to be sold on the merits of maven over ant :) (12:57:54) markfarnsworth: i use maven and ant, some +/- with each (12:58:01) markfarnsworth: maven does good for distribution (12:58:10) markfarnsworth: but ant is the better IMHO build tool (12:58:34) Nathan_Voxland: I've generally found taht ant is similar and does what we need it to, so that has been good enough for me (12:58:47) otatop: Maven does some things amazingly, and makes others amazingly painful. (12:59:02) Nathan_Voxland: maven seems to add complexity that is useful in other systems, but not ours.  Especially since we don't have any dependencies to speak of (12:59:54) markfarnsworth: I still am new to LiquiBase and really have not yet built it or run it (13:00:10) markfarnsworth: but i tried to pull down the project into my eclipse workspace (13:00:24) markfarnsworth: it seemed like it was not finding jars and such that it needed (13:00:33) markfarnsworth: do you have a mvn eclipse:eclipse? (13:00:43) otatop: Maven does modularity, ~dependancies, & reports really really well, but the second you need to do something non-standard it can be quite painful. (13:00:53) Nathan_Voxland: otatop, I agree that the diff tool is something I would really like to get going, especially the hibernate integration.  It is 80% of the way there and it is cool to see it in action as it is.  I'm just concerned how hard the last 20% will be, and we have limited developer resources and need to focus on the core update process first. (13:01:13) taranenko: ok, i see. I just want to know what about other guy that was involved in liquibase development? Haven they going to keep the developing? (13:01:27) Nathan_Voxland: kkrikor: "Yes, Is it important to preserver the count of changeLogs cross DB's ?"  What do you mean by that? (13:02:13) Nathan_Voxland: I personally use intellij, and so don't keep up on the eclipse project config as well as I should. (13:02:24) otatop: I agree with that. The core functionality is definately the priority. I just like seeing it on the list and perhaps some minor continuous work. That 20% to get right I would expect would take a huge amount of effort. (13:02:36) Nathan_Voxland: Are you trying to pull it in from the src.zip file or from svn? (13:03:11) Nathan_Voxland: I think I noticed a problem with the source.zip file lastly that needs to be fixed up.  If you check out the code from svn you may have better luck. (13:03:18) markfarnsworth: i pulled from svn (13:03:19) otatop: The guys are demanding I leave for lunch. Thanks for putting out a great product! Keep it up! (13:03:42) Nathan_Voxland: thanks for coming by.  Have a good lunch :) (13:03:46) markfarnsworth: i got distracted and did not chase the issues very far (13:04:01) Nathan_Voxland: I understand :) (13:04:02) markfarnsworth: so i will check the build again this weekend (13:04:06) Nathan_Voxland: Are there any questions I have missed so far? (13:04:31) Nathan_Voxland: And if you have any questions on getting the build going, please let me know.  nathan@voxland.net. (13:04:55) kkrikor: thanks nathan (13:05:00) Nathan_Voxland: Like I said, 2.0 is focused on opening up the codebase to additional users, so I want it to be as seamless and easy as it can be (13:06:32) Nathan_Voxland: Well, we hit the 1 hour mark.  Is there anything else you want to make sure we cover? (13:07:14) samawon: please make it easy to merge multiple files into a giant file which would be useful for the project moving foward (13:08:01) Nathan_Voxland: There is the tag that lets you combine files into one, but if you have a large number of them it would be difficult.  And it's not just one file, it is still several. (13:08:10) Nathan_Voxland: I can add it as a feature request so it doesn't get lost. (13:08:55) Nathan_Voxland: Since it is quiet, I am assuming we are done.  Thanks for coming everyone, and for all the feedback you've given. (13:09:10) kkrikor: thank you , keep up the good work (13:09:27) Nathan_Voxland: I plan on doing something similar to this on a regular basis (maybe every 2 months) (13:09:39) taranenko: Nathan, thank for the good job (13:09:42) kkrikor: that would be great (13:09:59) Nathan_Voxland: I will try to get the transcript up on the forum later today. (13:10:17) Nathan_Voxland: Any feedback you have on how it went would be great. (13:10:33) Nathan_Voxland: And if you prefer, you can always email me directly about anything. (13:11:07) Nathan_Voxland: Now if only I had some cleaver sign-off phrase... :) (13:11:25) upul: commit (13:11:57) kkrikor: thanks bye