Pages

Sunday, October 27, 2013

Managing Dependencies for IntelliJ IDEA Plug-in Development

Probably the most painful part in developing plugins for IntelliJ IDEA is the dependency management of the libraries that your plugin might depend on. There is a multitude of problems that one may encounter:

  • IDEA binaries are not hosted publicly neither in Maven Central, JCenter, or any other repository.
  • Plugins cannot be built with the common tools like Maven or Gradle without setting up your hair on fire.

When your plugin depends on just a few external libraries, and the plugin build itself doesn't require customization, then everything is simple: put the dependencies into lib/ (or whatever) folder in the project and generate the Ant build script through Build -> Generate Ant Build... action. Simple. Well, not really simple: the generated build script will require a few JARs from IDEA by pointing to the IDEA installation directory. So the simplest thing to do in this case is to extract the full IntelliJ IDEA distribution into some directory at the machine where the continuous integration server runs.

Simple, ugly, but works. Why not to make things a bit more kosher?

In my world, any developer in the team should be able to clone the project, open it in the IDE(A), and ideally, launch the project without any additional tuning.

So what could we do in case of IntelliJ IDEA plugins projects? Store the *.iml files in VCS so that when the project is opened by another developer he would automatically have all the dependencies attached to the module? Not kosher.

For me, clearly, the dependencies should be managed by dependency management tool. Options: Maven, Ivy, Gradle. IDEA provides a very good Maven support. But not for its own plugin modules. In fact, for plugin modules I wouldn't even try to use Maven as once IDEA recognizes it as a Maven project, it will erase the information about its plugin origin.

Other options: Ivy and Gradle. Ivy is awesome. It works. There's also a nice plugin for IDEA that will automatically import the dependencies if it locates ivy.xml in the project directory. In that case you could still use the autogenerated Ant build script - just alter it a bit so that it would make a call to ivy task to get the dependencies and incorporate 'em into the build classpath.

Sounds a bit too hardcore to me. There's a high chance that the project import will not be as smooth as you would like it to be. Especially if the project structure isn't very trivial.

Gradle to the rescue!

Gradle is awesome when it comes to non-trivial project structures. Its Ant-like flexibility along with nice Groovy syntax and all the Maven-like goodies is just awesome!

First of all, managing dependencies is very easy. For instance:

  repositories {
    jcenter()
  }

  dependencies {
    compile('org.zeroturnaround:jr-sdk:5.4.1') {
        transitive = false
    }
    compile 'org.ow2.asm:asm-all:4.1'
    compile 'org.slf4j:slf4j-nop:1.6.3'
    compile 'org.apache.commons:commons-compress:1.2'
  }

The little cool part is that JetGradle (in IDEA 12) would also automatically resolve and add the dependencies into the project.

This is cool but it is not enough. First of all, JetGradle cannot import the project as an IDEA plugin module. Secondly, there's no JetGradle in IDEA 13 as Gradle integration is getting a major overhaul. OK, back to the drawing board.

There's an 'idea' plugin for Gradle, how cool is that?

Just run gradle idea and it will generate the project files, incorporating the references to the required dependencies. Good. But the generated module type is a Java module. And I need Plugin module. Have no fear, Gradle's here! We can easily customize the build script to adjust the XML project descriptor to our requirements.
  apply plugin: "idea"

  repositories { ... }
  dependencies { ... }

  idea.project.ipr {
    beforeMerged { project ->
      project.modulePaths.clear()
    }
  }

  idea.module.iml {
    withXml {
      it.node.@type = "PLUGIN_MODULE"
    }
  }

The withXml hook makes the magic here - instead of the auto-generated 'JAVA_MODULE' the final descriptor will contain 'PLUGIN_MODULE' which will make IDEA think that the module is an IntelliJ IDEA plugin.

Only little problem that bothered me: the generated project descriptor is an *.ipr file that is kind of deprecated. It would have been much better if the plugin generated a directory based project metadata. The feature request was filed long ago but still isn't resolved.

After experimenting with the project descriptors for a bit, I actually found a simple workaround for this minor annoyance. The .idea directory structure requires only modules.xml file that is identical to the *.ipr file that idea plugin generates. So I could just use this simple task to create directory based structure:

  task setup {
     dependsOn ideaModule, ideaProject
     doLast {
       copy {
          from '.'
          into '.idea/'
          include '*.ipr'
          rename { "modules.xml" }
       }
       project.delete "${project.name}.ipr"
     }
  }
[hate mode="on"]Just writing delete "*.ipr" would not work. That's annoying.[hate mode="off"]

So now I could just execute gradle setup and it would generate the directory based project structure, with the correct references to the required dependencies and I could import the project into IDEA without any hassle.

This is all good. However, this is not the end of the story! There's more to do and more issues to resolve:

  • Get the required IDEA internal artifacts to a local repository, so that these dependencies could also be downloaded via Gradle dependency manager.
  • Migrate auto-generated Ant build script into Gradle script.
  • Adopt the Gradle script to be able to manage the releases of the plugin.

However, [hate mode="on"]none of this hassle would be needed if IntelliJ IDEA provided a sane way of building the plugins and managing the dependencies.[hate mode="off"]

25 comments:

Anonymous said...

Great post. Thanks for sharing it.

Do you plan to publish some sort of "hello-world-gradle-ified" idea plugin?

Anonymous said...
This comment has been removed by the author.
Hans Dockter said...

Hi Anton, thanks for clarifying your points. BTW: I did never perceive your arguments as bashing.


1.) Gradle definitely has not substituted Ant yet. Far from it. This space has a lot of inertia and for good reasons. At the end of the day it affects all your developers and the products you are shipping to your client. Nonetheless we mad significant inroads considering the dynamic of this space. We have a good idea of the usage of Gradle in the enterprise. I don't think there will ever be a bing bang, just a constant increase in migrations from Ant to Gradle.


2.) The requirements from Android Studio is pushing the Gradle-IntelliJ integration intensely. To make this really good there is still a lot of work to do on our side as well as on the side of JetBrains. But it is for obvious reasons now a very high priority for both sides and the Google Android team is also contributing to this effort. I expect a lot of progress for this over the next months. Let's see. This work will also help to improve the integration with the other IDE's as they all use the Gradle tooling API for this.


3.) Build Usability is an interesting topic. It has multiple aspects in my opinion. It is about executing the build, but also about understanding the build and tailoring the enterprise standard build to the needs of a specific subprojects.


In general I fully agree that improving the IDE integration is important for Gradle. The good thing is that this is work in progress.

arhan said...

Hello Hans! Thanks for your comments.

You know I like Gradle very much and I'm not trying to bash it here. What I'm saying is that its current state is far from perfect

I doubt about enterprise projects because of another survey we have that is filled in only by our customers, which could be characterized as "enterprise" projects. In there, Ant looks really strong! And I do agree that for Ant users Gradle is more intuitive and painless. Mostly because it doesn't constrain them as much as Maven. Imo, Gradle might substitute Ant quite successfully, it is just haven't happened yet. And there's definitely a big share of projects who won't migrate in any case.

IDE support is really basic for Gradle at this current stage. I mostly compare it to Maven's support in IntelliJ, of course. It is very basic in Eclipse and NetBeans, and in IntelliJ 12 it is the most voted issue at YouTrack. My friends are desperately waiting (as well as myself) for better support in IntelliJ 13, but you probably know it yourself - it doesn't look very inspiring in the current EAP at the moment. Ask iNikem (see his comments), how much he wants Gradle support in IntelliJ.

It really think that if IDE support gets more love, then there are higher chances for projects to adopt Gradle. Because, as sad as it is, often developers don't care, how the tool works. No matter, if it is a build tool or is it JRebel. What they care about is _how_ they would use it.

Hans Dockter said...

Hi Anton,

thanks for trying to shed some light on the Java build system space. A couple of comments.

1.) The survey definitely does not reflect the market share in the Jave Enterprise segment. The absolute market leader according to my evidence (surveys, engagements, conferences, ,,,) is still Ant. Nonetheless Gradle is having a very decent and fast growing market share, as is reflected in 1 million downloads over the last 12 months. And if a large Ant build is migrating to a new build system, which most will do over the next years, nowadays I think most of the time they choose Gradle.

2.) Why are you saying: "I really doubt that its share is high in Java enterprise/legacy projects."? Did Jayson told you so :)? Seriously, middle-size and large Enterprise Java based organization are the major drivers for the Gradle adoption. The pain they are having with their current Ant or Maven builds is often tremendous and the ROI for moving to Gradle is very high. I guess it is the same story as for you guys.

3.) I'm surprised by this statement: "But yet again, tooling support for Maven is the main argument why Maven is better." What defines what is a better tool? Just this aspect? It is ironic that you make this comment. Your main argument for enterprises to use JRebel is ROI. The same is true for build tools. The IDE integration is one amongst many parts of this story. For larger teams their current build is often causing them so much pain and is making them so unproductive. If a team is having build times of multiple hours or can't cope with their complex integration testing and build promotion processes, the IDE integration is a completely minor argument. BTW: The Gradle Eclipse tooling is decent and the IntelliJ tooling is much improved in version 13.

jbaruch said...

actually, it's the other way around - maven users are desperate and vocal. Ah, and Intellij IDEA users of Gradle plugin. Those are super-desperate and as vocal as they can be.

arhan said...

Gradle users are desperate and vocal :)

jbaruch said...

Looks good, and I agree, posting the survey in Gradle Users group might affect the objectiveness of the results :)

iNikem said...

Please, send a link to this post to IntelliJ guys :)

Luca Cavagnoli said...

+1

Zaki Mirza said...

Are these talks from the conference available for download?

John said...

doesn't work

NM said...

Thanks for this. my USER_HOME on windows is C:\documents and settings\profileid\ . Unfortunately this has very limited space allowed. How can I change the config, system and plugins path to C:\someother\dir/? I see that idea.configuration file only accepts paths relative to IDEA_HOME. how can I go up couple of directories from IDEA_HOME or USER_HOME?

Thanks.

Mel Cooper said...

-g will instruct the compiler to generate the local variable table AND the line numbers, to generate only the local variable table, use -g:vars

Sergio Ramirez Martinez said...

Awesome features!!, I like the dependency diagram.. thanks for posting this!

Goldin Evgeny said...

>> None of this hassle would be needed if IntelliJ IDEA provided a sane way of building the plugins and managing the dependencies.

Couldn't agree more, always puzzled by that.

Goldin Evgeny said...

Anton, I hope you eventually have something similar to what I did for TeamCity plugins (http://evgeny-goldin.com/wiki/Gradle-teamcity-plugin) - break apart an IDEA distribution, put all the necessary dependencies into publicly available Artifactrory instance and provide a nice and re-usable Gradle plugin to fetch them from there and package an IDEA plugin.

arhan said...

Yup. That's what I want to do.

Pratap Kumar said...

One interesting thing we could spot here is that some of the instructions take a weird operand like #1 or #2, which actually refer to the constant pool of the class. This is now a good point to get more information from the class file.

can you please elaborate the jvm constant pool of the class, i did get this part.

please find some time to explore constant pool information. how source code information translate to constant pool information, where exactly the constant pool being maintain , is it in jvm method area?

above all wonderful article.

Pratap Kumar said...

which javap option you used to view these two.

LocalVariableTable

Exception table

Conficio said...

Too bad the link is gone... Last version I could find --> http://web.archive.org/web/20130928005156/http://evgeny-goldin.com/wiki/Gradle-teamcity-plugin

asit said...

can use
please provide some more information that means it would be more easier to
learn this concepts.

asit said...

thank you for the data which you
have blogged ..hope this will definitely help our students to let them know and
aware all the concepts which is related to this..Amc
Square Reviews

asit said...

..hope this will definitely help our students to let them know and
aware all the concepts which is related to this..Amc
Square Reviews

asit said...

as a fresher who all wanted to go to selenium it would be
really helpful for aware of basics and as well as for interviews

Amc
Square Reviews

Disqus for Code Impossible