Better, Faster and Simpler Pig Development Using PigUnit

Better, Faster and Simpler Pig Development Using PigUnit

HBase groovy client dependency problem

My groovy HBase client had the following dependencies declared in grab

@Grab(group='org.apache.hbase', module='hbase', version='0.90.4')

@Grab(group='org.apache.hadoop', module='hadoop-core', version='0.20.2')

It threw the following exception

General error during conversion: Error grabbing Grapes -- [unresolved dependency: org.apache.thrift#thrift;0.2.0: not found]

After adding the following grab resolver, it worked!

@GrabResolver(name='Apache Psmith', root='http://people.apache.org/~psmith/hbase/repo')

How to download readonly code from github (using git command)

You might be tempted to do a 'anonymous checkout', but in git you need to clone like this:

git clone https://github.com/tc/cascading-scala-maven-example.git

Grails memory leak on batch inserts and a possible OOM

Recently, I worked on downloading a subset of our huge database into a grails' HSQL. Initially, I thought this was so easy that it almost qualifies as hello world2.

But I hit a roadblock. The job that downloads rows and saves to the HSQL would go out of memory after a 100K rows or so. Specifially, it would throw "java.lang.OutOfMemoryError: Java heap space". I was certain that this was memory failing to get recollect. But without going through the memory analysis, I went after obvious suspects: hibernate session and HSQL transactional memory. I took clues from here and here. But no avail. Finally, this post from Burt Beckwith explained it all.

All I needed to do was set the 'errors' to null after saving domain objects.

To summarize the leak: Grails keeps references to the errors objects that are on the domain objects in the httprequest. So, as long as a request is around, the domain objects will not be garbage collected. Makes sense? :)

Lessons learned : Grails upgrade from 1.2.1 to 1.3.6

Recently (due to the downtime provided by holidays), I upgraded our main project to grails 1.3.6 from 1.2.1. It wasn't a smooth ride. Here are some lessons learned (I'll continue to add):

1. Don't wait too long before grails upgrade. Do an upgrade at your first opportunity. That saves you from being on your own island.

2. If you use groovyws jar, make sure you exclude xmlbeans and jaxen. These are available in jdk1.6. This wasn't a problem in older grails; the classloading has probably changed.

runtime('org.codehaus.groovy.modules:groovyws:0.5.2') {
excludes 'xmlbeans', 'jaxen'
}

3. I am currently trying to find out why the integration tests are not rolling back the data inserts. I'll put the solution here once I know it.

Making grails not ask userid/password for anonymous svn https access

If you happen to create your own repository for grails plugins and host it in your organization's SVN, simply adding the path in BuildConfig.groovy

grails.plugin.repos.discovery.myrepo="https://----plugins repository path----"

will ask your for SVN userId and password for most grails commands. like this:

Authentication for svn repo at local discovery is required. ...
Please enter your SVN username:

Please enter your SVN password:

I avoided this by using "https://anonymous:@----plugins repository path----"

test post.

test post.