Anthony Dahanne's blog

Dynamically proxying annotations to avoid inducing API dependency

anthony.dahanne — Mon, 07 May 2012 13:35:11 +0000

Annotations are API, and as API they need to be on the classpath at compile time. And usually, you do not want your (portable) business code to depend on yet another API.

In this article, I will present you how to allow your library users not to depend on the library API just for the annotations you provide, using java.lang.reflect and its proxy facilities.

My examples will be based on Ehcache source code, since it is for the @IgnoreSizeOf annotation that this whole idea of annotation proxying came up ! (for reference, @IgnoreSizeOf is the annotation used by Ehcache to filter out classes from the SizeOf computation)
Doing this on the Ehcache @IgnoreSizeOf annotation will allow framework developers, who embed Ehcache in their framework (and want to benefit from the sizeOf feature), to not introduce a dependency on Ehcache for their users.

Our use case : you want to use the original annotation but not to depend on it.

Instead of using and depending on Ehcache’s @IgnoreSizeOf (net.sf.ehcache.pool.sizeof.annotations.IgnoreSizeOf) :

package net.sf.ehcache.pool.sizeof.annotations;
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD, ElementType.TYPE, ElementType.PACKAGE })
public @interface IgnoreSizeOf {
    boolean inherited() default false;
}

you want to annotate your classes with your custom com.myproject.ehcache.IgnoreSizeOf annotation , which of course has no relationship (well, it looks like it, acts like it; but it is not the same, see this as duck typing) with the original @IgnoreSizeOf annotation :

package com.myproject.ehcache;
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD, ElementType.TYPE, ElementType.PACKAGE })
public @interface IgnoreSizeOf {
}

You can notice that the inherited method is missing in the custom annotation, more on this in the next chapter.

The big picture : use java.lang.proxy to transform the type of your annotation

Once you have scanned your class or field or package (any AnnotatedElement) for your custom annotation :

Annotation customAnnotation = element.getAnnotation(com.myproject.ehcache.IgnoreSizeOf.class);

It is as simple as writing those 2 lines of code :

import net.sf.ehcache.pool.sizeof.annotations.IgnoreSizeOf;
/* some class and method declarations here */
        InvocationHandler handler = new AnnotationInvocationHandler(customAnnotation);
        IgnoreSizeOf proxiedAnnotation = Proxy.newProxyInstance(IgnoreSize.getClassLoader(), new Class[] {referenceAnnotation}, handler);

Well not exactly… all the interesting code is actually in the InvocationHandler, that will redirect method calls to our custom annotation methods or, if it can’t, to the reference annotation methods.

    private static class AnnotationInvocationHandler implements InvocationHandler {
 
        private final Annotation customAnnotation;
 
        public AnnotationInvocationHandler(Annotation customAnnotation) {
            this.customAnnotation = customAnnotation;
        }
 
        public Object invoke(final Object proxy, final Method method, final Object[] args) throws Throwable {
            //trying to call the method on the custom annotation, if it exists
            Method methodOnCustom = getMatchingMethodOnGivenAnnotation(method);
            if (methodOnCustom != null) {
                return  methodOnCustom.invoke(customAnnotation, args);
            } else {
                    //otherwise getting the default value of the reference annotation method
                    Object defaultValue = method.getDefaultValue();
                    if (defaultValue != null) {
                        return defaultValue;
                }
                throw new UnsupportedOperationException(
                        "The method \""
                         + method.getName()
                         + "\" does not exist in the custom annotation, and there is no default value for"
                         + " it in the reference annotation, please implement this method in your custom annotation.");
            }
        }
 
        private Method getMatchingMethodOnGivenAnnotation(Method method) {
            try {
                Method customMethod = customAnnotation.getClass().getDeclaredMethod(method.getName(), method.getParameterTypes());
                if (customMethod.getReturnType().isAssignableFrom(method.getReturnType())) {
                    return customMethod;
                }
                return null;
            } catch (NoSuchMethodException e) {
                return null;
            }
        }
    }

One method to implement here : the invoke method, called on our proxied annotation every time we call a method on it.
If the method called on the proxy matches a method in our custom annotation, it just delegates the call to it.
But if not (for example remember our custom annotation does not have the inherited() method, but the reference one has), and this is where the JDK annotations are cool, the reference annotation default return value will be returned instead.
What if there is no default value in the reference annotation ? In that case you need to provide this method in your custom annotation (with a default or you can force your consumer to provide a value for this method).

In this example the consumer of Ehcache does not even need to write a complete annotation to mimic the original one ! (since the only method in the original annotation has a default value)

Actually, even in the Oracle JDK they use proxying to resolve annotations, in sun.reflect.annotation.AnnotationParser

/**
 * Returns an annotation of the given type backed by the given
 * member -> value map.
 */
 public static Annotation annotationForMap(Class type, Map memberValues) {
     return (Annotation) Proxy.newProxyInstance(type.getClassLoader(), new Class[] { type }, new AnnotationInvocationHandler(type, memberValues));
 }

using their own sun.reflect.annotation.AnnotationInvocationHandler

Conclusion

Even if this code was written specifically with Ehcache @IgnoreSizeOf annotation in mind, it is generic enough for any framework authors to use (even more since it is Apache 2 licensed)

You can checkout Ehcache source code from http://svn.terracotta.org/svn/ehcache/trunk/ehcache/ehcache-core/ and look for AnnotationSizeOfFilter and AnnotationProxyFactory (or, even better, the test class : FilteredSizeOfTest) to learn more detail on how to provide a mechanism for letting your consumer not depend on your annotations !

Special thanks to Alex Snaps and Chris Dennis for their guidance !

Converting and publishing an Android application (.apk) to run on PlayBook 2.0 (.bar)

anthony.dahanne — Thu, 03 May 2012 02:59:15 +0000

Since its version 2.0 the RIM BlackBerry PlayBook OS acccepts now slightly modified Android application.
Here are the steps to convert your Android application to run on PlayBook OS 2.0 and to publish it on BlackBerry App World.

Convert your Android application to run on PlayBook OS 2.0

Requirements for your Android app :

Your Android application must be targeting version 2.2.3 (API level 10) or earlier to be converted
It must not use the Camera permission (not supported)
The icon (res/drawable/icon.png) has to be converted to 86×86 pixels
There are plenty other requirements, you can read on following this link if you run into trouble

If you can convert your Android app using an eclipse plugin provided by RIM, I preferred to use their online packaging tool, which actually worked great and did not force me to add yet another plugin inside my IDE

So, from the Online Packaging Tool page, follow the instructions :

After accepting the agreement, specify the location of your Android app (.apk) and your Android SDK folder, if everything went OK, you should get this confirmation message :

If you have a warning or error message, it means your app does not meet the requirements, then , have a look at the list of requirements

Request your signing keys to RIM, they’ll then send you (it takes few hours) emails containing 2 certificates : 1 client-RDK-xxx.csj certificate (Rim Development Key) and 1 client-PBDT-xxx.csj certificate (PlayBook Debug Token)
Then configure your computer to actually generate a developer certificate (.p12), still using the online tool
Finally, you can go to the Packaging and Signing section, to sign your application using your developer certificate (.p12)

You finally obtained your BAR archive, ready for submission to BlackBerry App World !

Publish your converted Android application (.bar) to BlackBerry AppWorld

You must register to the vendor portal first (they will ask you to provide an official id such as a passport or driving license), it is free (actually all steps are free), then you’ll receive a notification by email (few days later) telling you your account is approved (read on this post to have some feedback about registration)

Once you are able to login,

you’ll have to create a new product : for that you will need to submit a series of forms about your app (description, screenshots, audience, etc…)

Unfortunately, they impose you with image resolutions.. oh well… gimp is here

Now add a release of your product, this is where you are going to upload and publish the bar version of your Android app (file bundle), target it for Tablet PlayBook 2.0 All

And finally submit it for review : the first time I did that, they got back to me more than a week after – well, it was during the period of the promotional PlayBook giveaway in exchange of a new app submission – to finally deny my submission because there were some Android logos in the screenshots

I finally received a playbook after my app was accepted on the app world; it does not run as smoothly as the original android version, but is still usable.

All in all that was a pretty good experience, I only regret the long time I had to wait between all the validations; it seems that it’s getting better (according to the positive tweets I read on this subject); so you’re an Android developer and want to gain a (even) broader audience, go for it !

Confoo 2012 : notes about the conference

anthony.dahanne — Thu, 08 Mar 2012 06:08:56 +0000

From Wednesday 29th February to Friday 2nd March 2012, took place Confoo 2012, a major conference in Montréal, mainly focused on web technologies; but I could also attend some real cool sessions about Java / JVM, Android, (J)Ruby, TDD, Continuous Integration, software architecture, and Agile/ Scrum / Lean ; talks were given mainly in English and some in French too.

Here are some extensive notes I took while attending those talks :

And below, some quicker notes (really too busy listening !) on some other talks :

I presented a session on Android Continuous Integration
Virtualize your environment by Sean Coates was an interactive session where the speaker would poll the audience about which virtualization environment they are familiar with, on which system (he introduced some tools such as Vagrant to administer Virtual Box VMs with the command line, puppet to centrally configure your system configurations, and OpenVPN for a robust VPN solution, accessible from even poor internet connections)
Propulsez votre architecture grâce aux mocks : Félix Antoine a présenté au public un TDD basé sur les mock objects : comment le fait d’écrire nos classes en partant des couches supérieures (en mockant les couches en dessous) fait émerger un TDD qu’il a qualifié de « London style » , dû au fait que ces techniques ont été documentées en premier par Steve Freeman auteur du livre Growing Object Oriented Dotware, guided by tests

You may now be interested in :

I really enjoyed speaking and attending sessions at Confoo (thanks to my employer who let me get there !) : the organization was really professional and friendly: some sponsor booths would organize some games and give gifts, sessions were on time and cancelled sessions were twitted in real time, 8 10 tracks in parallel, a very interesting keynote, some very good local and international speakers, talks in French and in English (mostly in English since most of the audience was from outside the province of Québec : 600 people from the US/Europe mainly); as a speaker I also could enjoy a very nice dinner with all the other speakers ( I could meet and re connect with developers and trainers I knew from Paris ! and connect with some local software developers) ; and finally this event took place in downtown Montreal, I am so proud I can attend such quality events from here !

I can’t wait for Confoo 2013 !

Confoo 2012 : JRuby

anthony.dahanne — Thu, 08 Mar 2012 05:33:32 +0000

Talk by Charles Nutter, on Friday 2nd March

How to install JRuby :

rvm install jruby and unzip or
download tar/zip file from jruby.org or
you can use a windows installer from jruby.org

What is bad compared to pure Ruby ?

memory footprint, mostly due to the (better) GC; objects are larger, but fewer; BUT : lower memory when using concurrency (for example using rails, each http request will spawn a new Ruby process consuming the same amount of RAM, compared to JRuby spawning some new threads and keeping its amount of RAM consumed stable and keeping barriers between those threads)
c extensions, still experimental

Good

Ruby itself (both 1.8.7 and 1.9.2. modes in the same jruby : start it with jruby –1.9)
sandboxed jruby (for non thread safe programs, runs in isolated mode)

JVM

best GC, support dynamic language
ruby code used with Jruby ends up in native code, optimized by the JVM
stable threading : jvm threads are OS threads;
libraries : tens of thousands of JVM library(Swing demo : Charles built a Swing panel with a button adding statements to jirb )
lots of tools : visualvm (demo stressed gc with old and young objects)
invoke dynamic : ruby is about calling methods, the jvm from 1.7 optimizes those calls

Android

Ruboto : gem install ruboto ; ruboto app; rake : you’re done creating an Android app with JRuby
Ruboto irb
Jruby 1.7 : invoke dynamic, dalvik enhancement, ruby 1.9 by default

Bonus

Charles used the redcar editor, written in jruby + swt
trinidad : tomcat like launcher with ruby wrapper
torquebox : jboss like launcher (messaging, schedulers, etc…)
During his Android demo, Charles used android screencast
warbler for war/jar : to wrap your ruby app in a war
you can deploy your apps to engine yard cloud

Confoo 2012 : Regular Expressions

anthony.dahanne — Thu, 08 Mar 2012 04:57:45 +0000

Talk by Jakon Westhoff (Qafoo) given on Friday 2nd March

Different languages utilize different different regular expression engines : PHP (PCRE), Java, Python, Ruby, etc…

Delimiter with / for example : /foobar/i : i is case insensitive

MetaCharacters :

* : any number of occurrences
+ : at least once
? : once or not at all
{x,y} : occurrences between x and y

Character classes

matches any character EXCEPT new line; but using the s , for example : (The.Point)s it can also match new lines
character classes : [abcdef]+ any character in the bracket would be matched on or several times; ranges : [a--cd-f]+ most of the metacharacters loose their meaning inside brackets; except - for the range
[^abcef]+ : negates a the new line is part of it; to except the new line : [^\n]+
predefined character classes : \d (digit) \s : every whitespace whitespace; the capital letters negate : \D everything but a digit
(something)D : no new line tolerated at the end

Alternatives

Logical OR : Open|Source : matches the first found : Open or Source

Escaping

\ in front of the character (it is supposed to be a literal one, not a special one)
be careful that according to your programming language, \ has also a meaning; so \\n become current…

Anchors

^ beginning of the subject
$ end of the character
(^abcdef$)m enables multiline mode

Sub pattern

((abc)(def)) : to extract part of the string : 1 -> abc and 2 -> def
named sub-pattern : P option

Readability

x : Extend your pattern’s legibility by permitting whitespace and comments.

# is ignored, so you can use it to comment

whitespaces are ignored if they are not escaped

Confoo 2012 : TDD and getting paid

anthony.dahanne — Wed, 29 Feb 2012 22:37:31 +0000

Talk by Rowan Merewood, given on Tuesday the 29th of February

What is TDD ?

Rowan reminded us with what TDD is about :

decide what you want to do,
write the test,
run it and watch it fail,
write minimum piece of code,
re run the test to watch it pass,
refactor,
repeat

Why is it hard ?

do you even know what you / your customer want(s) ?
Rowan took the analogy of a computer scientist with a ninja, repeating kata to master the perfect move: http://codekata.pragprog.com , TDD Kata
Rowan went on with a demo of the design using TDD of a php class (with phpunit checking in the background for modifications to re run the test)

First test : check the instantiation returns the good type ; it fails because the class is not existing, Rowan created the class (no methods no variables) re run, the test passed
Second test : expects that a method called calc returns 0 : it fails since the method add does not exist in the class under test, then Rowan wrote an add method that returns 0, and the test passed
Third test same thing with add 1 : to make it pass, he just returned the param given to the method
Fourth test : add (1,2) should return 3 : he parses the params into an array and array_sums it

So now, we’re ready to apply this at work

you should not try to implement all the tests at once : they won’t be maintanable / understandable and you will miss the deadline
You can try to « force tests » to your client : add a fixed percentage to your estimates, and do not compromise your principles
You can also try to « sell the tests » to your client : use tests to define done, involve the client in creating tests (automated hopefully) you can use fitnesse selenium , behat (can be seen as using cucumber to describe fitnesse tests), and make the tests a deliverable (use CI )
Do not over promise.

The steps to get you started with TDD :

Unit testing and acceptance testing
puppet/chef to enable continuous deployment re creating a fresh environment
continuous integration to make progress visible
issue tracking (it allows reporting on TDD)

Reporting :

Code/branch coverage
Bug origin (tested or untested code )
For a begineer in TDD, a good idea can be to measure the amount of time spent testing vs amount of time spent coding

On a human side : owners vs heroes (a owner can spread the knowledge, a hero just fixes the problems during ight and keeps the knowledge to himself) and ff the team does not care : try games to imply them or… quit (!)
You have to accept not testing the legacy code; but you can wrap the calls to this legacy code and run tests against this wrapping api.

Running late ? drop features or test the happy path / test only the core(reduce the scope)
Broken build ? fix the test or disable/delete the test (some -legacy-tests can be wrong and their impact may be even worse than not having tests at all)

Confoo 2012 : Implanter l’AOP… comment partir du bon pied?

anthony.dahanne — Wed, 29 Feb 2012 22:04:49 +0000

Par Félix-Antoine Bourbonnais, Mardi 29 Février

La programmation orientée aspect permet d’injecter des appels de méthodes (Logger.log() par exemple), avant/après/autour, par exemple des appels aux méthodes commençant par start.

AspectJ peut fonctionner en 2 modes : compile time (il faut utiliser son tisseur/weaver pour compiler) ou au runtime en utilisant un agent (ce qui peut permettre d’appliquer ses aspects sur du code tierce)

Spring AOP par contre est un cadre applicatif AOP pur java.

Parce l’AOP n’a pas décollé ?

Mauvaises raisons et utilisations
Complexité puisqu’on sait difficilement si le comportement de notre code va être modifié par des aspects
Manque de support et d’intégration / manque de connaissance
Trop de promesses

Outillage AspectJ :

on peut construire avec Maven (attention à ne pas ré instrumenter le code avec du code coverage…)
et on a une intégration dans Eclipse avec AJDT.

Il ne faut pas utiliser l’AOP pour cacher des problèmes de design ou pire, créer des problèmes de design.

Bonnes pratiques :

garder les bonnes pratiques orientées objet
ne pas oublier qu’un aspect va coupler du code (attention aux dépendances circulaires et la séparation des responsabilités du code)
les classes sur lesquelles l’aspect va opérer, doivent elles en être « conscientes » (lisibilité, en utilisant des annotations)

Tester des aspects : un aspect n’est pas du code java pur : il est nécessaire de coder 1 « pot de miel » et vérifier que les appels des aspects ont bien été exécutés ET CECI aux bons endroits (pointcut)

Confoo 2012 : JVM Internal for Dummies

anthony.dahanne — Wed, 29 Feb 2012 18:04:01 +0000

This talk was given by Charles Nutter, full time JRuby developer

The slides are are on slideshare .

How the JVM’s JIT work

Monitring theJIT

Finding problems

Dumping assembly (don’t be scared)

So intense (slides with x86 assembly code !) I could not take any notes :-p but really insightful !

Confoo 2012 : Continuous Delivery at speed

anthony.dahanne — Wed, 29 Feb 2012 15:41:53 +0000

Presentation given by Joseph Wilk (co developer of Cucumber, work on Songkick.com) on Tuesday, 29th February

His presentation was about about :

for sure, continuous integration : build small pieces of work, continuously
continuous deployment, putting your process in flow.
Continuous delivery is the addition of those 2 concepts.

As a joke Joseph reminded us how we could do some primal sort of continuous delivery : ssh to the server and edit the files with Vim change is immediate !

To bring back the fun to software engineering practice, Joseph suggests that the delivery process should be faster (today there are too many barriers with CI, QA, etc..)

Tooling ?

Heroku is a tool that deploys your code from your SCM, pushes it to prod, restart the server; it is fun but… dangerous .

Joseph showed us then a tool that, once you save, runs the tests and deploys the code to the live server (why hacking directly with VIM could not work after all ?)

To add safety, he can also deploy to a different web server, on the same live production machine, but running on a different port : the ability to change and experiment, having a live feedback.

What if some code breaks the site ? does it matter if it is the about page that gets 1% of the traffic, in a month is broken ? you can balance the risks with metrics on your website visits for example.

Thinking bigger : when failure is not an option anymore : if you don’t trust your tests, what can you do to be confident on them so that you are also confident about the quality of the code when your tests is green ? smoke tests, acceptance tests, etc..

Some users want a stable experience visiting your website; others volunteer to beta software : those last ones can accept some broken pages.

Joseph is using at his work a Jenkins CI server to run series of tests : unit tests, acceptance tests (using selenium) recording the run of tests on a flash video file that you can replay. (using a python tool)

With Heroku, you can use a « one button deploy », same thing with Jenkins (anyone in the company can deploy ) it triggers a build that will run the tests before deploying

Joseph also uses Zabbix : to monitor CPU, memory, requests : data that gives you immediate feedback.

Zero downtime releases : no sense to deploy if it takes your website down for 10 minutes

canary deploy : use your load balancer to redirect 1% of your traffic to the new code
bug : if you can deploy quicky, then that also means you can fix bugs fast

Joseph asked himself this question from the Poppendieck : how long does it take for a single line of code to be deployed : Joseph was sad to discover that it could be as long as 2 days (he did find 2 days a long time ! !!); a reason for this can be because of all your dependencies : you can try to use libraries (such as rails, will be upgraded every 6 months or so), artifacts (such as ruby gems or debian packages : every machine can get the latest packages from an artifact server) to focus on your business code.

Fast is scary ; but using all this discipline (automated tests running on every commit): fast is fun

Q&A :

can you balance the fun of deploying quickly vs the money you can loose on failure : it depends on your environment, you have to make compromises; tests give you confidence : unit tests, acceptance test, smoke tests; about the infrastructure cost : usually it is not that high; about resources : if you want good developers , you have to help them make their work fun, so continuous deployment can help you hire the best developers
How long did it take for you to put this infrastructure up : about a year, progressively
Do you use code coverage : use it as a guidance : you can prefer more fine grained metrics such as logs, parameters permutation with heckle, etc…

Creating and applying a patch for an Eclipse project with Git, part 2 : using github

anthony.dahanne — Fri, 20 Jan 2012 00:18:11 +0000

In a recent post, I described how to contribute a patch to an eclipse project using git as its version control system.

Patches are OK, but you lack all the social aspect that is enabled with git and its tooling (be it pure git + gerrit or github with the comments on commits and pull requests)

As of today, pull requests are not accepted by Eclipse committers since they lack IP (Intellectual Property) information, and the official git repositories at eclipse are hosted by the foundation not on github (github repositories of Eclipse projects exist for mirroring only).

Recently talking with a committer on the Eclipse Tycho project (Tobias Oberlies not to name him), he suggested to me to share my contributions via github, instead of attaching patches, as long as the git contribution scenario is followed

To comply with this suggestion,

1. I would clone the official repository at git.eclipse.org

git clone http://git.eclipse.org/gitroot/tycho/org.eclipse.tycho.git

2. work on my changes, test them, add them, and commit them for each functional change

git add myfile1 myfile2

git commit -m "BZ xxxxxx : comment"

3. create a new repository on my github account

4. synchronize my new github repo with the official tycho repository

git remote add origin.github git@github.com:anthonydahanne/org.eclipse.tycho.git

« behind proxy » users can use https access instead,

git remote add origin.github  https://anthonydahanne@github.com/anthonydahanne/org.eclipse.tycho.git

5. push my committed changes to my github repository

git push -u origin.github master

6. add a comment to the bugzilla entry describing the fix or new feature I worked on, giving the github commit id

7. accept the IP conditions of the Eclipse foundation

That’s it ! you just contributed some code to a cool Eclipse project !
OK, it seems more complicated than just submitting a patch or a pull request, but once your local repo is set up,
you should have something like :

git remote -v
origin  http://git.eclipse.org/gitroot/tycho/org.eclipse.tycho.git (fetch)
origin  http://git.eclipse.org/gitroot/tycho/org.eclipse.tycho.git (push)
origin.github   https://anthonydahanne@github.com/anthonydahanne/org.eclipse.tycho.git (fetch)
origin.github   https://anthonydahanne@github.com/anthonydahanne/org.eclipse.tycho.git (push)

and it is just a matter of juggling between the original repo, synchronizing the latest changes

git pull origin master

and the github repo (pushing the latest changes)

git push origin.github master

and copying/pasting the commit ids to the bugzilla entry.

I’m still relatively new to Git, so if you think there are better ways to sync the repositories, then please add a comment to this post !

To go deeper on that topic :

Sécuriser son serveur dédié : quelques conseils

anthony.dahanne — Sat, 19 Nov 2011 16:03:35 +0000

Assomption avant de commencer : Bien entendu, vous êtes à jour des derniers correctifs de sécurité de l’ensemble de vos services…

En lisant un article faisant référence à un serveur mal sécurisé ; j’ai été pris de quelques doutes : et si mon serveur dédié était lui aussi mal sécurisé ?
Étant développeur et adepte de l’auto hébergement (oui je ne suis pas un grand fan des services hébergés chez les géants du net avec leurs clauses de confidentialité obscures…) je fais évoluer mon serveur dédié (services qui tournent dessus) au rythme de mes besoins… ce qui fait que la sécurité passe souvent au second plan…
Et rien de tel que de jeter un coup d’œil à la sécurisation de ces services de temps en temps…

Audit rapide et efficace

On commence avec nmap, avec les options qui vont bien :

sudo nmap -A mon.serveurdedie.fr

On peut alors commencer à l’analyse du résultat, et à remédier aux problèmes.

DNS et Bind

Dans le rapport obtenu par nmap, si vous voyez :

| dns-zone-transfer:

et ensuite la liste de vos sous domaines; méfiance, cela veut dire que quiconque peut lister vos sous domaines; pas forcément une bonne idée si certains d’entre eux sont là pour des tests ou pour des services non publics…

Les solutions

Rendez vous dans votre fichier de configuration de bind, /etc/bind/named.conf.options, et ajouter les options suivantes :
 allow-transfer { x.x.x.x; }; --> uniquement donner accès aux informations détaillées des zones à cette IP (votre dns secondaire)
 version "Ma version"; --> pourrait derouter certaines attaques en offuscant la version de Bind; mais comme disent certains, l'obfuscation est rarement une bonne méthode de sécurisation...
 fetch-glue no; --> défendre votre serveur dns aux attaques de type déni de service
 recursion no; --> défendre votre serveur dns aux attaques de type déni de service

Ces informations sont issues d’un article de TechRepublic , et n’hésitez pas à vérifier votre configuration avec zonecheck

HTTP et Apache

Dans le rapport obtenu par nmap, si vous voyez :

80/tcp  open     http         Apache httpd 2.2.9 ((Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny13 with Suhosin-Patch mod_wsgi/2.5 Python/2.5.2)

Et bien cela signifie que votre serveur est très bavard…

La solution

Sur Ubuntu, dirigez vous vers /etc/apache2/conf.d/security et éditez :

ServerTokens Prod

ServerSignature Off

TraceEnable Off

Si vous utilisez PHP, faites aussi un tour du côté de /etc/php5/apache2/php.ini

expose_php = Off

Encore une fois, l’offuscation n’est pas une méthode de sécurisation en soit, mais moins les potentiels attaquants en savent, mieux c’est…
Ces informations sont issues de DebianAdmin

SSH et OpenSSH

Pour l’instant je ne peux que conseiller de lire cet article : http://www.cyberciti.biz/tips/linux-unix-bsd-openssh-server-best-practices.html

à suivre…

Creating and applying a patch for an Eclipse project with Git

anthony.dahanne — Wed, 05 Oct 2011 03:23:35 +0000

Since its move to git, contributing to the Eclipse foundation projects is a bit different.

Nothing terrible actually, let’s take the example of a contribution to the p2 project (part of the equinox umbrella at the Eclipse foundation)

Checkout the project

git clone http://git.eclipse.org/gitroot/equinox/rt.equinox.p2.git

do the changes and then commit them to your local repo (you can perform this step several times)

git commit

create the patch against the master branch

git format-patch origin

You will get a file for each commit you submitted, so one commit = one patch file.

You can attach your patch(es) to a bugzilla entry, for example , make p2 buildable with tycho (click on one of the patches)

what an eclipse committer will do then , is apply your patch to its local repo to test everything is ok

git apply 0001-my-patch.patch

That’s pretty much it ! A bit more tedious than a Github pull request, but it does the job !

Run UI tests on a headless Jenkins / Hudson Continuous Integration server running Ubuntu

anthony.dahanne — Mon, 18 Jul 2011 03:43:59 +0000

If you want to set up a ci build, running ui tests, on a box without any windowing system, you need to pay attention to a few details, here are some of them :

Set up your ubuntu box and Jenkins/Hudson

On your ubuntu box :

# apt-get install vnc4server fluxbox

Vnc4server is a free vnc server, that will act as your X environment, and fluxbox is a tiny window manager; more than enough for UI testing.

In Jenkins / Hudson, Manage Jenkins -> Plugins -> Install new plugins, choose the xvnc plugin

In your job that launches UI tests (in my case it was Eclipse Equinox UI tests launched by a tycho build), tick the box « Run xvnc during build »

Now launch your job, you will run into this problem :

Starting xvnc [workspace] $ vncserver :10 You will require a password to access your desktops. getpassword error: Invalid argument Password:Starting xvnc [workspace]

etc… until vncserver :13

It is because you need to launch vncserver interactively, at least once, to setup a password; connect to your ubuntu box and:

# su jenkins
$ vncserver :10You will require a password to access your desktops.

Password:
Verify:
Password too long - only the first 8 characters will be used
xauth:  creating new authority file /var/lib/jenkins/.Xauthority

New 'host:10 (jenkins)' desktop is host:10

Creating default startup script /var/lib/jenkins/.vnc/xstartup
Starting applications specified in /var/lib/jenkins/.vnc/xstartup
Log file is /var/lib/jenkins/.vnc/host:10.log

Once this is done, re launch your job :

[workspace] $ vncserver :14

New 'host:14 (jenkins)' desktop is host:14 Starting applications specified in /var/lib/jenkins/.vnc/xstartup Log file is /var/lib/jenkins/.vnc/host:14.log

and your build goes on, able to create a UI during the tests !

Have a look at the tested UI while the tests are running

This can be very useful whenever your UI gets stuck (waiting on a dialog to be closed …, in my case I run into the issue of launching an eclipse runtime, and a plugin, Google ADT, was asking to confirm whether or not I wanted to upload usage data– hopefully only the first time) during the tests.

To do so, you first have to know what is the port xvnc is listening to :

# ps aux |grep "vnc"
jenkins  29591  0.0  0.7  33192 14104 ?        S    21:10   0:00 Xvnc4 :14 -desktop host:14 (jenkins) -auth /var/lib/jenkins/.Xauthority -geometry 1024x768 -depth 16 -rfbwait 30000 -rfbauth /var/lib/jenkins/.vnc/passwd -rfbport 5914 -pn -fp /usr/X11R6/lib/X11/fonts/Type1/,/usr/X11R6/lib/X11/fonts/Speedo/,/usr/X11R6/lib/X11/fonts/misc/,/usr/X11R6/lib/X11/fonts/75dpi/,/usr/X11R6/lib/X11/fonts/100dpi/,/usr/share/fonts/X11/misc/,/usr/share/fonts/X11/Type1/,/usr/share/fonts/X11/75dpi/,/usr/share/fonts/X11/100dpi/ -co /etc/X11/rgb

OK, looks like xvnc is listening on port 5914 in this case.
You can then use a vnc viewer (on ubuntu, vinagre is a good choice) of your choice to connect to your ubuntu box on port 5914.
If the ubuntu box has firewall rules impeding you to connect to 5914, then set up a ssh tunnel :

$ ssh -L 5914:localhost:5914 anthony@ubuntubox
then connect your vnc viewer to localhost:5914 , and the magic begins !

A concrete example : m2eclipse-android-integration in Jenkins

I actually set up this configuration for the m2eclipse-android-integration project; this project is an eclipse plugin, that extends the capability of m2eclipse to android projects, here is the link to the Jenkins hosting this job

The author, Ricardo Gladwell, designed exclusively integration tests on the UI, since most of the functionality of this eclipse plugin is UI contribution.

Here are the specific problems I met while setting up this job in Jenkins :

as I previously mentioned, the first time I launched the job against this project, once all the tests passed, the tycho surefire plugin step would not stop… actually the Google ADT plugin in Eclipse, was asking to confirm whether or not I wanted to upload usage data, so I connected to the UI and clicked « ok »; vnc saved the day…
the tests were relying on android projects, requiring to have among the latest Android libraries; to be up to date (so that the tests would not fail), I had to run the command :

$ android update sdk --no-ui --obsolete --force

Overall, that was a good exercise to practice CI on a UI project.

Links :

NAS Synology : pour les sauvegardes et pour diffuser les médias

anthony.dahanne — Mon, 11 Jul 2011 23:58:27 +0000

Séduit par l’idée d’un boîtier pour effectuer mes sauvegardes à faible coût, faible consommation énergétique, faible encombrement, en ligne 24h/24, j’ai récemment fait l’acquisition d’un NAS (Network Attached Storage); et c’est assez naturellement que je me suis tourné vers Synology, un des (sinon le) incontournables du marché (l’utilisation de Linux BusyBox et d’une multitude de logiciels libres n’y est sans doute pas pour rien !); j’ai opté pour le DS211J, qui dispose de 2 baies à disque SATA; pour l’instant je n’y aie inséré qu’un disque de 1.5 To.

Mon objectif premier est de réaliser des sauvegardes robustes, la cerise sur le gâteau (ou sundae) est le fait de pouvoir consulter mes fichiers multimédias sur mon téléviseur (via une vieille XBOX avec XBMC et/ou une PS3)

Architecture de sauvegarde robuste

On réalise des sauvegardes, car on ne veut pas perdre des années de photo, de travail sur documents multimédia (montages vidéos) ses documents personnels (impôts, etc..).

La majorité des utilisateurs sauvegarde leurs fichiers vers un disque dur externe, quand ils y pensent…

Cette approche là est résolument insatisfaisante : il faut automatiser (fréquence) le processus de sauvegarde (donc avoir du matériel toujours disponible pour effectuer la sauvegarde) et aussi diversifier ses lieux de sauvegarde (tout sauvegarder sur des supports qui se trouvent au même endroit que les originaux ouvre la porte à une perte de donnés en cas d’inondation, incendie, cambriolage, etc…)

Je vais décrire ici comment utiliser un NAS Synology pour :

y sauvegarder ses documents (les machines du réseau local, dans notre cas sous Ubuntu, vont sauvegarder vers le NAS)
synchroniser ces sauvegardes vers une machine distante (le NAS va synchroniser les sauvegardes vers une machine dans un data center)
y sauvegarder des sites internet et boîtes courriels hébergés sur un serveur distant (une machine dans un data center va sauvegarder ses données vers le NAS)

Enfin, je décrirai quelques astuces pour profiter de son NAS en serveur multimédia.

Sauvegarde entre les ordinateurs locaux et le NAS

Il existe un excellent article qui introduit ce propos sur le wiki de Synology; ceci dit la technique est la même que celle que j’avais décrit pour des sauvegardes distantes.

Mon objectif est ici de sauvegarder mon home directory (/home/anthony en excluant tous les .*) et rien d’autre
Voici mes remarques et ajouts :

Depuis l’interface graphique de votre NAS, rendez vous dans Panneau de configuration>Utilisateur, sélectionnez votre utilisateur et appuyez et cocher « Accueil Utilisateur » (une traduction sympathique de Home directory :-p) , vous disposez maintenant d’un répertoire personnel dans /volume1/homes/votre_utilisateur
la première étape est la création d’une clef (DSA, c’est important, RSA n’a pas marché chez moi…) sans mot de passe; utilisez la série de commande standard suivantes, dans mon cas :

ssh-keygen -t dsa -b 1024 -f ~/.ssh/diskstation

(pas de mot de passe, il serait difficile de gérer des sauvegardes non interactives avec un mot de passe, vérifiez quand même que votre clef .ssh/diskstation est en 600)

Maintenant envoyer la clef publique à votre NAS :

ssh-copy-id -i .ssh/diskstation.pub anthony@diskstation

Tester la connexion à votre NAS en utilisant la clef : (attention, il est fort possible que votre utilisateur n’aie pas de shell utilisable par défaut; connectez vous alors via SSH en tant que root (même mot de passe que admin) à votre NAS et éditez /etc/passwd, sur la ligne correspondant à votre utilisateur, remplacez /sbin/login par /bin/ash ; l’interpréteur de commandes par défaut)

ssh -i .ssh/diskstation diskstation

vous devriez arriver, sans mot de passe, dans le répertoire /volume1/homes/votre_utilisateur de votre NAS.

rsync pour sauvegarder, la commande est assez simple :

rsync -avz --delete /home/anthony/ --exclude=.* diskstation:/volume1/homes/anthony/sauvegarde

sauf que le vrai défi avec les sauvegardes, c’est qu’il faut qu’elles soient automatiques, et lancées périodiquement ! ce bon vieux cron est encore une fois notre ami :

vim sauvegarde.ssh

on y écrit :

#!/bin/sh
rsync -avz --delete --exclude=.*  -e "ssh -i /home/anthony/.ssh/diskstation"   /home/anthony/  diskstation:/volume1/homes/anthony/x60s

chmod +x

on peut tester :

./sauvegarde.sh

et enfin le rajouter à la crontab :

sudo vim /etc/crontab

et rajouter la ligne (pour lancer la sauvegarde tous les Dimanche à 2h00) :

# m h dom mon dow user  command
0 2    * * 7   anthony      /home/anthony/sauvegarde.sh

Répétez cette opération pour tous les postes du réseau local.

Tout devrait bien se passer quotidiennement, à surveiller que tout est bien en place quand le disque dur de l’hôte commence à faire des bruits étranges ou avant de partir en congés…

Sauvegarde entre le NAS et un serveur distant

Le logiciel d’administration web des NAS Synology, Disk Station Manager (DSM 3.1 au moment où j’écris) dispose d’une efficace interface d’administration, qui comporte un volet « Restaurer et sauvegarder »; nous pourrions très bien faire comme précédemment pour sauvegarder le NAS vers un serveur distant, mais autant profiter de l’interface graphique et des notifications de DSM !

Pour commencer, vous devez préparer votre serveur distant à recevoir les sauvegardes, dans mon cas, nous avons un utilisateur backup, avec son répertoire personnel /home/backup

Dans ce même répertoire personnel, nous allons y créer un fichier rsyncd.conf avec le contenu suivant :

[diskstation]
comment = diskstation
path = /home/backup/diskstation
use chroot = no
read only = no
list = yes
uid = sauvegarde
gid = sauvegarde
strict modes = yes
ignore errors = no
ignore nonreadable = yes
transfer logging = no
timeout = 600
refuse options = checksum dry-run
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

Grâce à ce fichier, lorsque votre NAS fera ses sauvegardes, il trouvera la configuration rsync.

Côté client, donc le NAS, dirigez vous vers l’interface graphique « Restaurer et sauvegarder » et créez une nouvelle sauvegarde

choisissez sauvegarde réseau (compatible rsync)
sélectionnez toutes les options SAUF l’avant dernière (réserver les fichiers; si vous la cochez, lorsque vous supprimerez un fichier/répertoire de la source il sera gardé sur le serveur)
choisissez le nom de module rsync que vous avez configuré sur votre serveur distant (dans notre cas diskstation) et saisissez l’utilisateur (dans notre cas backup) ainsi que son mot de passe
Testez la connexion, paramétrez la fréquence

Et c’est fini !

Sauvegarde entre un serveur distant et le NAS

Dans ce cas ci, vous devez au préalable avoir un moyen d’identifier sur internet votre nas, par exemple, DSM vient avec un client DynDNS.

Dans la suite du paragraphe j’assume que vous avez un nom d’hôte fixe (ou une IP fixe) pour votre NAS, et que vous pouvez y accéder par internet (pour ceci vous devrez configurer aussi les bons port forwarding [seul ssh nous intéresse dans le cas présent] depuis votre routeur ou « box »)

Pour le reste, c’est la même chose que ce que j’ai décrit au chapitre « Sauvegarde entre les ordinateurs locaux et le NAS » …

La particularité que vous pourrez rencontrer, au cas où vous sauvegarder des répertoires qui appartiennent à plusieurs utilisateurs sur le serveur distant à sauvegarder vers le NAS, est de travailler avec l’utilisateur root : c’est le seul utilisateur à pouvoir accéder à tous le système de fichiers.

Donc

côté NAS, créez un utilisateur backup, donnez lui un répertoire personnel et l’accès via SSH;
côté serveur distant à sauvegarder, créez un couple de clefs pour se connecter sans mot de passe au NAS, en tant que utilisateur backup, créez un script sauvegarde.sh, qui vous rentrerez dans la crontab, à lancer par root (il pourra donc travailler sur tout le système de fichiers)

procéder de cette manière vous permet de pouvoir sauvegarder tout le système de fichiers, sans jamais compromettre l’utilisateur root (pas de connexion en tant que root, ni par SSH ni par sudo, etc…)

Astuces pour la performance de votre NAS Synology

Si vous avez beaucoup de photos dans le partage « photo », vous risquez de voir un process appelé « convert » (en fait c’est imagemagick) prendre tout la mémoire vive et une bonne partie du CPU de votre Synology.

Ce process est lancé par le démon suivant :

/usr/syno/etc/rc.d/S77synomkthumbd.sh

donc il vous suffit, une fois connecté par ssh en root (même mot de passe que admin une fois que l’accès ssh est configuré) de lancer :

/usr/syno/etc/rc.d/S77synomkthumbd.sh stop

Si vous voulez définitivement arrêter ce service, éditez le avec vim et rajouter exit à la 2ème ligne (cette astuce est publiée dans le wiki de synology; attention à ne pas comme eux désactiver postgresql, toutes les applications multimédia : photo, son, video, téléchargements nécessitent cette base de donnée); attention quand vous remettrez votre NAS à jour, il faudra ré effectuer l’opération (sigh!)

Diffuser des fichiers vidéos hébergés par le NAS Synology

Le plus simple, si vous disposez d’un lecteur branché à la TV qui peut naviguer sur des partages Samba/CIFS (comme la vieille XBOX avec XBMC), c’est alors de configurer un dossier partagé (par défaut, un dossier partagé est un partage SMB/CIFS) contenant vos vidéos.

Si vous utilisez une PS3, l’approche est différente, il faut utiliser le partage médias par DLNA (car la PS3 ne peut pas monter un partage SMB/CIFS)

Si vous configurez votre NAS comme étant un serveur DLNA, il vous suffit d’avoir vos videos dans le partage video pour que la PS3 puisse les lire…enfin çà va bien pour les formats (container, codes audio et video) que la PS3 reconnaît nativement…

Pour les autres formats de video, comme par exemple container MKV encapsulant du h264, votre PS3 vous dira « format de fichier non compatible ».

Deux options se présentent à vous :

votre PS3 utilise un firmware « non officiel »comme le CFW 3.55 kmeaw, dans ce cas vous pouvez installer le logiciel de lecture Showtime sur votre PS3, qui non seulement lit les partages DLNA, mais est capable aussi d’ouvrir à peu près tous les containeurs vidéo et recense un bon nombre de codecs..(il repose sur libav)
Une autre approche, est de convertir à la volée vos fichiers vidéos dans un format que la PS3 peut ouvrir, le logiciel PS3 Media Player semble pouvoir être installable sur un NAS Synology, ceci dit je n’ai pas essayé…

EclipseCon 2011 blog posts : talks I attended

anthony.dahanne — Wed, 13 Apr 2011 03:09:52 +0000

From the 21st to the 24th of March, 2011, I was a lucky attendee of EclipseCon 2011.

Lucky because it was my first time at EclipseCon, my first major conference in North America (it was located in Santa Clara, in the Californian Silicon Valley) and also because my employer, Compuware, agreed to let me go there, along with 2 of my colleagues.

EclipseCon 2011 was great, because

the main actors of the Eclipse community were there : people I may follow on twitter or ask questions to on several mailing lists; the committers of the projects I use everyday at work; being able to chat with them and ask them questions (even harass them ! no I’m kidding, I did not harass Jeff during 2 days! )
the talks were diversified and most of them were really interesting, you have below links to some of the talks I attended while there.

Attending this conference made me also realize, maybe once more, that meeting with other developers working on the same technologies, is really inspiring, motivating, and it makes you grow technically.

Ok, you got the picture, I enjoyed the conf !

Here are some of the talks I attended:

Under the provisioning (p2) and release engineering themes,

Under the architecture , performance and OSGI themes,

Under the Eclipse tooling theme,

Other really cool talks

If you want my opinion, well, I’d say that Tycho, p2, Virgo, git along with gerrit, and Eclipse Memory Analyzer were predominant topics; you could definitely not ignore them.. and not like them neither !

If you did not, you’ll hear from these great technologies in the next few months !

Keynote on Apache Hadoop at EclipseCon 2011

anthony.dahanne — Fri, 25 Mar 2011 00:33:14 +0000

Todd introduces the audience to the world of huge datasets, what you can do with it, profiling users and customers for example.

False assumptions learned in the last 10 years (that Hadoop has been building with this in mind)

Machines are reliable, Hadoop separates fault tolerance logic from code logic
Machines deserve identities, I put data in a cluster, I don’t care which particular machine hosts the data; Hadoop can swap in and swap out machines across the cluster
Your analysis fits on one machine, Hadoop scales linearly with data size or analysis complexity

A typical Hadoop installation : 5 to 4000 commodity servers (8 cores, 24 GB RAM, 4 to 12 TB hard drive; 2 levels network architecture, 20 to 40 nodes per rack)

The cluster nodes are composed of m

master nodes : 1 NameNode (metadata) and 1 jobtracker
slaves nodes (1 to 4000 each) data nodes and tasktrackers

To access the file system, you would not mount it (even if you could, with fuse), you can use an API, HDFS API (in Java)

Hadoop will write on chunks of 64 MB, which will get replicated across the nodes.

Using HDFS, you will use 2 functions : map() and reduce(); they are run on the node containing the data, so no network overhead to get the info, but HDFS can interpret bytes as key; then reduce is used to aggregate the value.

Hadoop is not only map/reduce, with Hive, you can also use SQL; but there are other tools on top of Hadoop, Pig (DataFlow) or Sqoop (RDBMS compatibilty)

Who uses Hadoop : Yahoo (>82 PB, >40 k machines); FaceBook, 15 TB data/day, 1200 machines; Twitter, etc…

Mozilla uses Hadoop to analyze crash data (FF crashes, you send a report, and they get and analyze the data)

Hadoop Java brings some good tooling (along with integration tools such as Apache, Ivy, etc..) but some bad things such as JVM bugs, JNI libraries to add for non standard features (specific to the OS)

http://www.eclipsecon.org/2011/sessions/?page=sessions&id=2370

Getting Eclipse Preferences Under Control in Teams at EclipseCon 2011

anthony.dahanne — Fri, 25 Mar 2011 00:07:42 +0000

Michael began his talk remainding us that Eclipse configuration lies everywhere :

eclipse.ini
configuration/eclipse.ini
/home/.eclipse
.metadata/
$project/.settings/
runtime option -vmargs …

Michael recommends the audience to try to configure as much as possible preferences on project level.

You can manage team preferences documenting them in a wiki , but it is boring for the user, and also very hard to maintain.

Or you can use Eclipse EPF files (Eclipse Preference File), that you can import manually using the import wizard, but same drawback as wiki documentation, people forget about it.

Better than that, you can manage EPF files with one of those 3 tools :

Eclipse Team etceteras, is a plugin that can do automatic and manual EPF import over HTTP ,set preference between workspace, and tells the user if he has not imported the EPF, suggesting him to download it

Another tool, workspace mechanic, it is a task oriented configuration engine(using Groovy, Java, or other); thing it is file system based, no possibility to transfer preferences through HTTP

Bug 334016 (Common preferences), it is an automatic EPF import over HTTP, without asking the user if he wants it

Which one to choose ?

just importing through HTTP, use ETE,
if EPF are not enough for you, use Workspace Mechanic
need enforcement ? use Common preferences

Then we had a demo of ETE, starting eclipse in a new workspace, a dialog appeared to suggest the user to import the preferences; then changing workspace, there was a dialog to copy settings and preferences.

http://www.eclipsecon.org/2011/sessions/?page=sessions&id=2057

Stop the Architecture Erosion of Eclipse And Open Source Projects at EclipseCon 2011

anthony.dahanne — Thu, 24 Mar 2011 23:49:26 +0000

Bernhard and Frederic began their talk defining software architecture and architectural erosion, which means that your system becomes deprecated and overloaded

Then, Bernhard evoked Findbugs, saying that

back in March 2004, only 4 packages for version 0.7.2, very simple
new features added for annotations for example, few months later, still looking good
in May 2005, a first cyclic dependency appeared, in the svn log, « temporary hack »appears
June 2006, version 1.0.0 many packages, new cyclic dependency,
2009 : many, many tangles

Can you still maintain easily this project ?

He followed his explanation with a tool called Sotoarc to show the dependency between classes and packages; he also mentioned PDE dependency visualization

To check your dependency, you can perform architectural inspection with some architectural tools.

Then Frederic explained to the audience the critical aspect of the architectural quality of an open source project: it is used by many many consumers.

He then enumerated the risks of Erosion in FOSS :

contributors from several organizations (different processes)
lower pressure from management
hazardous funding

What about Eclipse project then ? Bernhard analyzed the architecture of JDT, and found out that the org.eclipse.ant.ui plugin uses JDT, and this dependency could have be avoided.

Also, duplication of code can cause problems of maintenance; he demonstrated this problem with a DialogField class in Eclipse, duplicated since 2.0

Frederic introduced to us the recommendations of the Architecture Council.

Also, some new eclipse projects, such as Orion, the web based ide, introduce new architectures.

To migrate your plugins from Eclipse 3.x to e4 , you have to go through an architectural modernization process which consists in auditing, testing, etc…

To do that, he suggests to use models to represent and manipulate artifacts of existing projects, using MoDisco; so that you can get in EMF the model of a plugin (sources folder, plugin.xml, etc..

Once you have the model represented, you can proceed to the migration of this model to the model of e4 for example.

You need to analyze and proceed to modernization of your tools.

http://www.eclipsecon.org/2011/sessions/?page=sessions&id=2001

Using and Extending Memory Analyzer into Uncharted Waters at EclipseCon 2011

anthony.dahanne — Wed, 23 Mar 2011 22:39:11 +0000

Vladimir first introduced to the audience problems such as:

Tracking memory leaks : hard to debug and leading to OutOfMemory errors
Thread issues : deadlocks, exhausted threads, waiting for external resources, such as DB connection

To solve those problems, you can use Eclipse Memory Analyzer, which is :

a Java Heap Analyzer
it helps find memory leaks and reduce memory consumption
UI and programmatic access to data in the heap dumps
faster then competitors, such as JHat
not that intrusive (in terms of memory overhead)
supporting a lot of JVM (Sun, SAP, 1.4.2 and above)
providing an automatic problem recognition

Then, on to the demo ! Dimitar,

loaded a thread dump (very fast to startup)
showed to us some problems found by the application, such as leak suspects
and some views such as memory consumed by objects, the states of the threads, etc…

Back to the presentation, Vladimir explained how you can extend the tool (you can extend it to save time doing the same thing repeatedly, to share view configurations, etc…)

There are several ways to access a snapshot

through the API : ISnapshot, Iclass, IObject, IArray
extensions points : NameResolver (descriptions of objects) ; Query, for exploring dumps or detailed analysis, in a form of table, tree, histogram ; RequestResolver, to provide the details of what a thread is doing; Vladimir also told us about the presence of 6 other extension points

http://www.eclipsecon.org/2011/sessions/?page=sessions&id=2184

Virgo and RT playing together at EclipseCon 2011

anthony.dahanne — Wed, 23 Mar 2011 19:04:00 +0000

During this talk, Christopher presented the audience Virgo, part of the RT umbrella of the Eclipse Foundation

The virgo packages

Equinox,
then Virgo Kernel,
then Virgo Tomcat Server or Virgo Jetty Server

The Gemini project

blueprint : DI
Web : support for WABs and WARs
JPA : persistence in OSGi
DBAccess : modular JDBC drivers
Management : JMX management (start /stop)
JNDI : JNDI naming in OSGi

Demo :

How to extend virgo to get your own custom server

Download virgo kernel and unzip it (and configure your osgi console port)
starts it, it looks for new applications to install; you can open an osgi console through telnet
add equinox.http to the ext repo, and the same for equinox servlet API bundle (2.5)
Configure http port , in properties, define : org.osgi.service.http.port=5555 for example
add the HttpService to the initial artifacts (a configuration similar to a config.ini osgi.bundles list)
start it, check everything is ok in the osgi console
You can then deploy a jar (osgi bundle) containing a servlet, configured using Spring (osgi:reference on osgi.http service); just copying it to the virgo kernel : it just works

In the works :

P2 integration
integration with Gemini projects
better tooling from libra project

Talking to Christopher and Glynn, I also learned that : (I’m totally a virgo n00b)

Virgo is like having Equinox + Spring OSGi + Jetty; BUT it also provides a unified mechanism system, using logback (and so slf4j logging APIs)
Virgo also solved problems such as Emphatic loading mechanism : take hibernate for example, a bundle using it would normally not be able to see the other bundles providing Hibernate based classes; that’s why they came up with this idea of dynamic bundle creation at runtime, that solves this visibility problem (check Glynn blog entry to know more about this problem)
Not P2 enabled : Virgo just recently joined the Eclipse world, and since uses its own dependency and update mechanism; P2 support is high on the priority list

http://www.eclipsecon.org/2011/sessions/?page=sessions&id=2086