Liquibase IDE Development

A belated followup to this post: http://liquibase.org/forum/index.php?topic=564.msg1930#msg1930

I would like to participate in the development of the IDE Mk II (or even to lead the charge) and am interested in hearing more about the ideas Nathan mentioned in the linked post.

I’m passionate about database change facilitation and I think any tools in this space have massive potential to make the lives of developers much easier and allow them to concentrate on the more important / fun aspects of the job. I also believe Liquibase offers a fantastic foundation on which to build such tools.

Nathan mentioned the importance of having a person or group who can offer long-term support for such a project. Insofar as my crystal ball allows, I believe I am such a person. I would be using this tool every day in my job, which is the best incentive I can think of for continued improvement. Any other takers?

cheers,
Anthony

Great to hear.  I definitely agree that it would be hugely helpful.  Here are some thoughts I’ve had during and since development of the first version of the IDE.  It’s hopefully not too rambling, but I’ll try to lay out some architecture and design ideas/thoughts I’ve had

Target Market
I’m not sure the demographics of who uses liquibase, but I would guess that it is generally agile-practicing developers who use java.  There are some non-java developers who use liquibase, and there are some non-developers (i.e. DBAs, release managers, etc) who use liquibase, but I think that they are a minority.  Should the liquibase IDE cater to our current largest group? Or does the core liquibase library meet their needs to a large extent, and the IDE should have a goal of giving liquibase a larger reach?  I think an IDE would make it much easier to quickly understand how liquibase can help you without necessarily exposing you to the Java/XML implementation details and so a DBA used to using TOAD or a .Net developer wanting to manage their database would find the Liquibase IDE to be a good tool while they would never have considered the core liquibase library.  At the same time, we have an existing market, and we certainly do not want to increase the time develop, complexity, or usefulness of the IDE for that existing java developer market for the hope of expanding liquibase to people who may still not be interested.

Stand alone IDE vs Plugin
The old IDE version was built first on top of Eclipse, and then also as an IntelliJ IDEA plugin.  There are many good IDE base packages to build on, and we definitely want to take advantage of one of them so that we can focus on the liquibase logic, and not on all the other logic that goes into building an IDE.  One question we need to answer is if we want to go the plugin route, or the stand alone IDE route.  Developers certainly like having everything in a single IDE, but if you build a plugin for one IDE you are cutting out users of different editors.  I was going to go the route of abstracting the plugin code so that it was easy to write eclipse, intellij, netbeans, etc. adapters but since that effort fizzled due to a lack of time, I’ve been starting to wonder if that is unnecessary work.  Each IDE works differently and has different UI requirements.  If we spread our small resources over too many editors, they will all suffer.  Plus, while developers do like having everything in their IDE, they are fine using different tools, especially with regards to databases.  They are used to having the MySQL workbench or sqlserver management studio open in addition to their IDE, and so if they can replace that window with the liquibase IDE, that may not be a deal breaker.  So I would suggest concentrating on creating a stand-alone IDE and not worrying about supporting plugins to multiple IDEs.  If we choose a base platform that is an IDE that supports it as a plugin, great, but it shouldn’t be a requirement.  What is a requirement, is that there is a low-overhead version of the platform so the final liquibase IDE isn’t a huge, bloated app because it also supports php competition and ejb deployment.

What base to use  
Assuming we are doing the stand-alone IDE, not a plugin, we would want a base toolkit that supports:

  1. A user-friendly database schema browser
  2. XML editing support
  3. Multiple database types
  4. Easy installation
  5.  Ability to include only functionality we need (see above)
  6. Royalty-free
  7. Looks like a standard application, even to DBAs and non-developers

From the IDE world, I think that means: eclipse, netbeans, intellij community edition, visual studio shell.  From the database-tool world there may be options as well including SQuirreL.

A potentially obvious choice would be eclipse: it is open source, it is very common among java developers, so it would fit as a plugin and just not stand alone, and plugin support is a strong part of their architecture.  I haven’t looked at eclipse for a while, so my opinions on it may be outdated.  I found the database tool support to be very not user-friendly.  It consisted of dealing with drivers, url strings, and confusing connection management to open a connection, then the schema browser was limited and not as extensible as we would need.  I also found the support for stand-alone IDEs built on eclipse rather than plugins was less documented and thought out, and creating a single codebase to support both options was overly difficult.  I also feel that the eclipse UI in general takes getting used to and so would be a deal-breaker for non-java+eclipse developers such as DBAs.  

At this point, netbeans concerns me to use as a base, given Oracle’s purchase of Sun.  I don’t think it will go away, but it has never had the reach or user base of eclipse, and I’m not sure how much more it will be advanced.  When I last looked, their stand-alone IDE support was also very poor.

Jetbrain’s Intellij community edition is a new entry.  I personally find their UI much nicer looking and more user-friendly than eclipse (which is why I use intellj and perhaps why I am biased).  They open sourced their base IDE, but it does not (from what I could tell) include their database browser.  I’m not sure if it includes a datasource connection manager at least, or not.  Creating a base platform that Jetbrains can build on is a very good business move for them (they have several IDEs built off it now), but I don’t know how well they will treat the open source version.  Is it the same as they use? Or a fork that needs to be maintained separately?  Will the decide it’s not worth keeping open source and move development internal at some point?  Those are questions we would need to answer.

Visual Studio Shell is the surprising option to throw out, but is worth considering.  It is certainly the most native-feeling application as a stand-alone IDE and with VS2010 arguably the prettiest.  It is not open source, but it is royalty free.  It would definitely help make liquibase appeal to the .net crew (and possibly the DBA crew as well).  Would it alienate our large java developer users?  I would think not.  To them, it should just look like a stand-alone app that they can use to modify their changelog files that are then ran on server startup or deployment like always.  The two biggest issues in my mind is:

  1. IDE cost for contributors.  If we had a .net app, the devs would already have VS and it wouldn’t be an issue.  For a java developer looking to help, however, there would be no reason they would want to buy VS so they can volunteer time.  I’m not sure if MS has any open source project licensing options.
  2. .Net port of liquibase.  I did a proof of concept port and it looks like it would not be overly difficult using ikvmc.  However, to use liquibase from inside of VS, it would be much easier to have a .net version running to use.  Adding this as a requirement, probably makes the chance of ever getting to the IDE near zero.
    Beyond those issues, I’m also unsure as to the support for multiple database types and/or the ease of use of their database browsing tool.  I wasn’t able to figure out how to get it to connect to non-sqlserver databases, but I am also not a .net developer.  Still, VS is a option worth strongly considering, I think.

Database Browser UI (aka Not a Code IDE)
As I eluded too above, I’m not sure if any of the IDE platforms really have a strong, user-friendly database browsing base.  They way I imagine the liquibase IDE to work, is to open it up and select your connection.  Then you can browse the tree view of tables, columns, views, etc which you can right click to bring up editing and “refactor” menus.  You should be able to have the normal “edit table definition” type UI where you can see the current definition and make changes such as chanign the name in the textbox or changing a column’s datatype and the UI will know what type of chagneSets to create from that.  There should also be the “refactor” menu for more complex operation such as “introduce lookup table” and easier versions of simple commands like “rename table”.  You also need the ability to view and edit the changelog xml files as XML (or whatever format they are in) from within the tool.  

Most of the above IDEs are primarily code editors with a database tool bolted on.  They either don’t have a database explorer, or it is very limited and somewhat second class (you need to have a java project created and attach database connection information to it).  If one of the IDEs above has a greatly improved database editor since last I looked, that would put it in the front of the pack but I am suspicious that whatever base we choose, we will need to create our own database connection management and browsing support.  It’s not ideal since it takes time away from making the liquibase part of the liquibase IDE, but it is probably what we will need to do if it needs to stand alone as a tool and be user-friendly enough to get users.  

A better option would be to write a plugin to an existing database editing tool.  Unfortunately, I don’t know of any viable candidates.  From the research I did (more should be done, though), they either don’t support plugins, support only one database type, or are not actively supported and maintained.  If you know any good candidates, let me know.  

That is probably a good starting point for discussion, I think.  I have some support thought about in the core liquibase library (affected database objects, change classes knowing what type of database object they can operate on, etc) but I think the base framework needs to be decided on first.

What is your development background and thoughts?

Nathan  

 

Hi,

let me also join to the discussion.

First I’m a java man, but I’m working with liquibase to support some .net projects and one project based on old VS6 codebase (namely VB). Surprisingly this not a coins to prefer the MS-based IDE. By me database-(change)-management is separate to the mainstream code changes. I’m feeling no needs to do database changes inside the VS6 or VS2003. May be due to my conservatism.
Second, I’m agree, IDE is a great facility to propagate using LB by the people, especially for not experienced users, and for reduce learning time/cost for all/other users.
Third, orientation to MS/Windows based IDE imho dramatically reduces area of potential users, excluding Apple and Linux/Unix world. Therefore => Java. My preference -> Eclipse.
Forth, I’m ready to be involved in the development of the IDE.

Please one crazy idea to treating. Go away from .NET and Java and landing to the Mozilla/Gecko? At least i’m running into a plugin for sql-management for sqlite under browser. See https://addons.mozilla.org/de/firefox/addon/5817/. Now I have never thought, what benefits or obstacles could such implementation cause.

Oleg.

The lack of cross-platform support with visual studio is an issue.  Whether it is a show-stopper for that tool, I’m not sure.  Percentage-wise, linux+mac is still small, larger than normal in the liquibase userbase I think, but still small.  While it would be best to have a solution that works for everyone, I would rather have something that works great for 90% of developers, than something that works ok for 100%. 

Has anyone used eclipse on a mac or linux lately?  If I remember correctly, SWT seemed a best fit for windows and mac and linux had performance and bug issues (plus the cocoa vs carbon bindings thing with mac, has that been fully resolved yet?).  Is Eclipse a good framework for Mac and Linux developers? Or would they not really want to use it anyway?

Mozilla/gecko is an interesting idea.  I think the biggest problem is that we need a framework that can call into the java classes nicely so we can call the liquibase.update() etc commands.  I don’t think there is that bridge in a xul based app. 

Some addtional support that a IDE base would bring is version control integration, which would be nice to have.

Nathan

I just happen to start pulling out the codebase from the 1.9.5 tag of the IDE, had a question, saw this thread, and thought to follow along!

I’m currently playing around with mavenizing the current (1.9.5 tag) state of the Eclipse IDE development work, including my recent hard-lessons-learned with the Tycho plugin (which requires maven 3, which is a bit of an issue for me, but work with what ya got).

Things I’ve run into so far:
*There is a ‘common’ in the root of the ide folder, as well as a seperate ‘eclipse/common’ which have some similar classes.  Is one ‘more correct’ than the other?  I was doing to have the root/common be a jar and simply include it in the eclipse/common plugin, but when I saw similar/same-named-and-packaged classes I was a little worried as to the intent of the two.

*The MANIFEST.MF of the eclipse/plugin has a Require-Bundle: org.liquibase.migrator, which I’m not sure where that should be coming from where it is defined in the Manifest of an eclipse plugin.

=======================================

The approach that I was going to take, if this helps:
the idea of having the eclipse-common plugin, then seperate plugin to implement the actual interactions/perspectives actually gave me the inspiration to have DIFFERENT liquibase perspectives to help users slowly migrate.  For example, a ‘beginner’ perspective would have an interaction usecase along the lines of:

*Do you have a liquibase change files somewhere? NO
*Do you want to connect to a database? YES
->user connects to the database through normal DTP methodology
->With liquibase perspective, interrogate the database to see if it is liquibase managed (has CHANGECONTROL table or whatever).
**If managed, go back and ask if they have the liquibase change files somewhere, otherwise…
**If unmanaged, put in big red bracketted [UNMANAGED] around the server.schema.database.
*Right-click option on server.schema.database: Have liquibase create a baseline for managing the database? YES
*Where would you like to save your liquibase change files? XYZ
–>server.schema.database goes from [UNMANAGED] to something like the master change file name, or just [MANAGED].
–>IDE/workspace/project saves these points for later automated retrieval.

…time passes…

Developers make change to the database WITHOUT the ide ( you know it will happen). Get near point where need to deploy changes to a STAGING/TEST environment.

*Open up IDE. See [MANAGED] server.schema.database, with red ‘lq’ or similar marker to identify changes have occured between the current database metadata and the expected changeset metadata (this is where some of the 2.0/extensions may need to come in to help ensure ‘comparable’ and ‘schema diff’ tools can work between the change file, or model, and the connected database).

–>now, at this point, they either make SQL changes and add them to the changeset, or actually use the Refactoring tools. These changesets would turn red ‘lq’ or similar markers into blue ‘lq’ or similar markers until the changes are actually deployed (managed deploy using liquibase) to the database.

Yes, I realize this is not the perfect solution to database change management, but it is what people are looking for.  The 80% scenario of adding a new column to an existing table, brand new views, drop/create of new stored procedures can be managed with this approach until something better flushes out.

==================

Regarding IDE options, I occasionally use Workbench/J, as well as Squirrel (and derivatives), but for first pass I think Eclipse is probably the safer route unless someone from those other projects volunteer. 

Visual Studio/Enterprise Manager options I would hesitate only because the JDBC driver versus ODBC/ADO options would be a bit confusing for those unfamiliar.

Originally posted by: Nathan
Has anyone used eclipse on a mac or linux lately?  If I remember correctly, SWT seemed a best fit for windows and mac and linux had performance and bug issues (plus the cocoa vs carbon bindings thing with mac, has that been fully resolved yet?).  Is Eclipse a good framework for Mac and Linux developers? Or would they not really want to use it anyway?

Some addtional support that a IDE base would bring is version control integration, which would be nice to have.

I daily use Eclipse on Linux, and have one of my coworkers use it on a Mac without issues — and runs better and faster, but this is comparing with Windows 2000 running McAfee (same/similar hardware), which is kind of unfair :-P.  This also coincides with SQuirrel and Workbench/J, where both work great in windows and linux.

http://squirrel-sql.sourceforge.net/
http://www.sql-workbench.net/  (has diff tools and similar data change items, I’ve never used them)

With Eclipse, we would also get ‘commercial offering’ integrations for free…the Websphere/Rational IDE’s, the Redhat/Jboss Studio, etc.

-D

As a warning: I haven not looked at the IDE code for a couple years, and the 1.9 codebase in depth for a year either.

You will run into many eclipse versioning issues, I’m sure, since the IDE is based on a very old version of eclipse and its dependencies.  The idea with the “common” folder is to try to keep a common codebase between the intellij and eclipse and (future) netbeans plugins.  I developed the eclipse plug-in originally, then used it as a base for the intellij plugin.  The common folder was created as part of the intellij implementation with the idea of applying the ideas to the eclipse plugin after, but I never got to that, if I remember correctly.  So the common folder is the newest code examples, but may be disconnected from the eclipse plugin codebase.

I do think you need the ability to modify your changelog xml directly, but like you said, most useage is simply a right-click “refactor” style menu for adding tables and columns.  I’d be concerned about tryign to build too much “diff” functionality into the process, since it tends to not be overly robust.  Even with the changes planned for 2.1, the diff tool will still focus on the common functionality and will miss much of the database-specific and/or advanced data objects.  It works as a final sanity check, but I think it would work best to leave it up to a dev process and focus on adding changesets easy enough that it is what people want to use.

Thanks for the update on mac and linux.  I agree that the jdbc/odbc differences may be confusing, but both are in their own way.  I was hoping that perhaps visual studio would hide some of that better than eclipse may (insert .net dev hand holding joke her), but I’m suspicious it will not.

A summary of my current thoughts/responses:

I think the visual studio option is probably a bit too crazy.  It would require extensive work just to get liquibase running on .net, and I don’t think it is going to bring any more usability/base database tooling than the other options do.  Plus, the fact that it does not run on linux or mac which is a major drawback.

In my mind, it comes down to eclipse vs. intellij community.  Eclipse is probably the correct option because of it’s mindshare, longer time and maturity as a “platform” and integration with commercial versions of eclipse.  I personally prefer intellij over eclipse for development, partly for their UI, but that is probably not enough to sway the answer away from eclipse.

Nathan

I can not help too much with liquibase specifics (I’m still learning on and off as I go), but I can offer to finish off:

*The current migration of the Eclipse liquibase IDE to a maven build system using the Tycho plugin to use as a point-of-reference to start anew.

*Or, create a bootstrap empty project, although this will be harder as the manifest.mf/plugin.xml specifics really do require specific class inclusions.

If this is the direction you would like to move forward with, this is an area I can assist and definitely have interest in completing a UI tool for liquibase.  Let me know how I may assist.

Hi Darren,

I’m very interesting in the participating in the development of the IDE for Liquibase. This is cool that you have experience writing plugins for Eclipse. Sad I have no one because of my freelance kind of work, I can position me as (hopefully) an advanced user of the Eclipse. I want to dive into such kind of development  though.
Please get me answers for a couple of question

  1. You said about Tycho, AFAIK it is intended to develop a RCP. Could it help with development of the Eclipse plugin?
  2. I’m respecting the Sonatype team and those work with maven. I’m with pleasure using the M2Eclipse. But how you could personally estimate the benefits/disadvantages Tycho over ‘native’ plugin/RCP development? Convenience, complexity, robust etc in compare with both approach - Tycho vs ‘native’. May be you could point to some link with such kind of review, I did not found even on the Sonatype site.
  3. What type of the backing Eclipse database machinery you prefer and why? SqlExplorer? DTP? some other? Your personally experience?
  4. What version of Eclipse you’d prefer? What version you using daily? Helios or earlier?

Sorry if I’m asking too much, there are emphasized issues for me this time.

Regards, Oleg

For Oleg,

OT:  Maven/Tycho build versus native RCP/eclipse plugin building

The Tycho plugin can be used to build Eclipse RCP, Eclipse Plugins, Eclipse Update sites/P2 repos (haven’t fully investigated this part yet).

For 1,2, I can answer with one statement - do you want to be able to reliably build your RCP/plugin in an IDE-independent fashion from the command line or from a CI tool (Hudson, Cruisecontrol, continuum, etc)?  If the answer is yes, maven or ant builds are your best option instead of relying on the IDE-native way of ‘assembling’ the plugins, and maven happens to be a better option for most people and the tycho is just a maven plugin.

For 3, I don’t have a preferred Eclipse database option - DTP is generally the default because it’s easier to get and easier to integrate with other plugins (Hibernate Tools for example).  Many of the others I’ve tried do work, but none are overwhelming over another.

For 4, depends on the project I’m working on.  If I start a project with Helios, I try to keep Helios for the lifespan of the project (even if it is several years) unless there is a real pressing need/gain by swapping it on a per-project basis.

Hello,

I posted another comment before seeing this one…

So here it is : we do have an open source database IDE : neXtep designer. We are considering development of a liquibase extension to let it generate Liquibase scripts. As I can see this is a current concern for you at the moment so I think we could collaborate. Our product is not a “Liquibase IDE” but rather an “IDE which could easily generate liquibase artefacts”, so it might differ from what you are willing to build. So I will present you some details here so that you could see if it could fit your needs :

  • It allows to put database developments under version-control using a built-in repository stored in a database : we support checkouts, commits, branches, merges, diffs, activities, users, etc.
  • Every database entity has a lifecycle within this repository through versions
  • You can reverse-engineer an existing database to import it in the repository
  • The IDE provides “disconnected” development : you only work with database structure in the repository.
  • You can synchronize your changes with any database : it generates automatically SQL scripts to update a database
  • When you want to release your developments (and this is where we think we can connect liquibase), you can build what we call a “delivery”. The delivery is generated by comparing version information of the delta to generate. At the moment a delivery is a set of generated SQL files together with a XML descriptor containing information about the required components and versions, expected database structure, etc. This delivery is handled by an installer which can deploy it to a target database. We could easily imagine to make it possible to generate liquibase scripts instead of SQL.
  • It is a RCP product based on eclipse (we’ve just finished migration to the 3.6.1 framework) and has a very extensible plugin-based architecture. We have lots of experience with Eclipse related development and this is our core business.

A rough architecture overview would give :

  • Database capturers (fetching database structure and converting it into our abstract data model)
  • Database model (the objects the IDE is working with)
  • Versioning framework
  • Comparison framework : the ability to compare any element together
  • Database generators (generates scripts from comparison information or from scratch)
  • And of course, all the UI aspects : editors, navigators, menus, version trees, etc.

Integrating liquibase in the product would mean to provide new kind of generators which would generate liquibase scripts instead of SQL. This is not a big deal for us. The thing is that development and refactoring operations that a user can make in the UI will not be restricted to what Liquibase provides. Instead, the “development” part of the product does not depend on generators. So we may need to collaborate tightly to make sure that everything could be generated through Liquibase refactoring options. But this is a benefit from a user point of view: he always make the changes directly on his model and since he is working on our offline repository we can compute what has been done and convert this into Liquibase refactoring actions. 

At the moment we support Oracle, MySql and PostGreSql. DB2 development is on the way. Our main problem with supporting other vendors is the time to implement generators and capturers to generate appropriate SQL for each vendor. Using a bridge to use liquibase would allow us to let liquibase do this job : liquibase would gain a (hopefully) nice IDE and we would benefit from supporting much more database vendors. More, liquibase users could benefit from an IDE (if they want to, as it would be safely decoupled) and neXtep users would benefit from all the good job you did on ant integration and deployment…

Well, I think the best thing would be for you to have a look at the IDE by downloading it from :
http://www.nextep-softwares.com

We made some nice tutorials and getting started guides (and also many articles on database deployment, version control, etc.) in our wiki :
http://www.nextep-softwares.com/wiki

Hope that we could put our forces together to build a great open source solution !
Christophe

Hello Christophe,

Interesting product and certainly a nice GUI. I got interested and downloaded the application to see if we could use it, but the main thing is, the design in NeXtep is done for one particular database vendor or can this be changed somewhere. I did not see it. We use liquibase for Oracle and MySQL but I am thinking about MSSQL and PostgreSQL too. Hw to you see NeXtep to generate generic update scripts which can be applied to different database vendors?

@Nathan; I use NetBeans and Eclipse both on Windows and Mac OS X. I would opt for an IDE which runs on the same platforms where we can use Liquibase.

Chris

Hello Chris,

Actually we implemented a generic “vendor” support for those who may need to deploy a same model to several distinct database vendors (if this is what you need). All you have to do is to create a workspace with the vendor ‘JDBC Generic’. Then you will work in an abstract world (and though will have less features available since not all vendors support sequences, tablespaces, materialized views…).

We think that we could generate liquibase scripts at the end which would mecanically be generic for that kind of “generic” data models. At the moment the product generates one script per vendor for that kind of workspaces. Every vendor-specific script is generated in the vendor-specific dialect. With a liquibase “delivery”, this would become much more generic…

Any idea / comment is welcome…

Christophe.

I was also evaluating neXtep for database change management, but where it is IDE centric it really needs majority database support before it will get broad adoption.

For example, large amount of database changes in my environment are MS SQL, and although everyone is using Enterprise Manager I need to find a better, and hopefully db-agnostic (for my other databases including mysql, postgres, and intersystem), approach to managing change. 

Liquibase can solve this problem by tracking raw SQL/scripted changes (in source control if you choose) through their options/command-line/ant tasks and handle deployment of changes in a variety of ways making it very flexible (and it can handle ANY change, any jdbc database).  But, it also makes it a bit of a learning curve and a little cumbersome (multiple manual steps to follow, writing of raw XML changeset files).

I like what I see in neXtep, and see the potential.  Having neXtep be able to integrate/create liquibase scripts and/or make liquibase api calls would be a great benefit and help people have options in how they want to manage their changes.  However, as an open source community note, it is hard to take a product seriously as an open source product without a clear community page with direction to get the source code, and having downloads require registration also give people pause.  Just my two cents although this is a liquibase forum :-).