Yann’s Blog

I’ve been quiet about my IRM 2 activities recently, and I apologize. This article is not really a status update, but a reflection on mapping programs such as IRM 2 to a traditional RDBMS.

I’ve determined that I just built a database… inside another database. The IRM 2 core currently has a core table, which is quite literally an ID, Column, Value mapping table. The data simply looked liked this:

A100, 1, Name, Computer
A100, 1, Color, Blue
A100, 1, Owner, Bob

Every property simply gets dumped into this table, and requires extensive remapping on the application logic. The RDBMS doesn’t have much insight into schemas, as it subverts its notion of schemas. Searching for explicit property values requires joining huge amounts of data together. Data storage is inefficient, with records scattered throughout the datastore (and in many places on the disk), especially if records were updated after creation.

More and more I’m starting to feel that approaches such as BigTable, SimpleDB, and CouchDB are taking the correct approach to the large data storage problem. The data stored is as a series of documents with arbitrary and per-document schema. Each node is a replicated element, and data storage is not centralized but rather distributed.

The advent of Google’s AppEngine supporting this model of database reaffirms that this is the correct direction to move in for large scale online applications.

Now, what does all of this have to do with IRM 2 and gIRM? Stay tuned - I’m trying to find out myself

I’ve taken the chance now to upgrade some of our APIs, make them more consistent, cleaner, and better documented.

The first primary change is moving from the old /home namespace to the new /api namespace. This allows for some cleaner separation of the logic and not the old mismash of stuff. I’ve also cleaned up the marketstat API, allowing multiple type IDs to be passed in a single query.

The new API has a version header now as well <evec_api version=”2.0″>…</evec_api>. You can distinguish responses, and possibly in the future request specific versions of formatting.

As before, the API is open, not requiring a key to use. I’ve contemplated a key based approach but haven’t deemed in necessary yet. If you suddenly find you can’t connect to the server - you’ve probably been doing something naughty (or it broke).

I have a few IPs blocked for bone-headed use of the system - such as querying every typeID from 1 to 100000000, and in string order at that. Don’t do that

The old APIs based off of /home will continue to work for now. It would be a perfect world if everything would upgrade, but thats not always possible. I’ll monitor usage of both and cut off the old one once its no longer popular.

Many nifty things are afoot for EVE-Central.com. For one, I migrated away from mod_python, which was hanging up the Apache process several times per day for unknown reasons. There was even a monitor script to kill -9, ipcrm, and restart Apache if it wasn’t responding - this horrible restart was happening several times a day. The new EVE-Central.com is running CherryPy as the backend server now - the conversion was quick and painless, plus it performs better (barring any new introduced bugs).

On the new feature front, I’m working on a trade comparison tool for sell orders, trying to find price differences between systems or regions. Its in an alpha prototype stage, so not for public consumption yet. I expect it to launch in a week or two. Here is a link to a sample image.

I’m also mulling over a possible Premium subscription to EVE-Central. Paid in ISK, it would allow access to the aforementioned trade comparison tool, plus other options, such as ad-free view, and the ability to upload “premium” content which would not get pushed to normal users until a small time window had elapsed. Details still sketchy at this point

Just a quick update highlighting what is going on in girm.it, the Git repository for IRM 2 (on Grails).

1. It is now possible to create, view and browse Assets based upon Schemas, which are user defined.
2. You can create Schemas and Fields in a Schema
3. The functionality for linking CRI (Cross Reference Identifiers) is in place - needs more work on the GUI front.

While gIRM isn’t even into alpha stages, its showing a lot of promising functionality already. Next up is a first hash of the Ticketing support system (Tracking mk 2), as well as “personalities”.

Personalities are an abstraction to Schemas which allow you to associate sub-schemas with a master schema. For instance, you can create a “Network Connectivity”personality and associate it with both a printer and a computer. Personalities can also impart custom logic to the application - such as link graphs in the case of a Network Connectivity personality. The mechanism for this isn’t quite designed yet

Feature suggestions are always welcome!

In addition to trying out new Kool-aid with Groovy and Grails, I’ve decided to move from tracking in Subversion to tracking in Git the gIRM code base.

The gitweb is available here: http://stackworks.net/cgi-bin/gitweb.cgi?p=girm.git;a=summary

and cloning is available at

git://stackworks.net/girm.git/

Just a quickie.  A few screenshots presenting IRM on Grails. Developed much faster than my previous solution, and in general making me feel a lot better about its direction

.

Its me again, and this time I bring news of lessons learned with the current development work on IRM 2.

Firstly, while EJB 3.0 is miles better than EJB 2.1, it doesn’t match up to a proper IoC container. By IoC, I am talking both about the Tapestry 5 IoC container, and the Spring IoC container. I find that with writing service methods with EJB, I spend far too much time both massaging domain objects between the EJB layer and the Tapestry layer - the Hibernate/JPA session is no longer valid outside of the EJB layer, so forget about all the “cool” stuff it does, such as lazy fetching, and wallow in pain replicating that functionality.

Second, component based view models are powerful, but not quick to grasp. I grew up (shame shame) on PHP, doing in-line HTML output. I know, this is bad, and I know better. I’ve grown accustomed to other view technologies such as templates (Smarty, Cheetah-Template, Velocity, etc), but still find myself taking a long time developing a workable component based model. This may be a shortcoming in Tapestry, but its taken me far longer to develop views than I would like. I will say that the component model is stronger than JSPs, but sometimes quick & dirty JSP gets you there faster. Some of this may be my experiences with Tapestry 5 talking, which I will discuss further below.

Third, Hibernate is amongst the best ORM wrappers available and a major time saver. Compared to other ORM wrappers out there (like ActiveRecord), it has an unmatched feature set, often backed by very mature Java technologies and libraries.

Java is a good language, and a fantastic environment. Yes, the language isn’t that great - overly verbose, no closure, a little too static. But the Java libraries available, especially on the server-side are un-matched from other web-development languages, including my favorite language, Python.

Groovy is a very promising language. Closures, duck-typing, first class JVM language, and more. Libraries that should have existed ages ago (XmlSlurper comes to mind). If you’re going to do quick development in a Java environment, or looking at a scripting solution in Java, give Groovy a strong look.

Whatever you do, avoid the creation of your own GWF. Also known as the Ghetto Web Framework. We’ve all seem them - you started out with some servlets, and went from there, making half-baked utility functions, some copy and pasting, etc. You get a maintenance nightmare - changes which should simply be a two-liner end up being propagated in odd places in your GWF, requiring exponential time to change. Plus your GWF is so tightly coupled to your application,  you can’t use it for the next application. There are so many web frameworks out there - please use one

Groovy on Grails makes an incredible rapid development environment. Coupled with Groovy, and with idea taken from  Ruby on Rails amongst other systems, you get an incredible Java development environment which is more productive and quicker to develop in than EJB, Tapestry, and maybe even the venerable RoR. Plus, its pure Java nature coupled with libraries such as Hibernate and Spring, you gain all the advantages of scalability and support that you just don’t get in RoR.

Wicket and Tapestry are very well organized frameworks. While I talked somewhat negatively about component based frameworks above, I would like to mention that both of these frameworks are fantastic in their own right. Wicket seems to be gaining a lot of momentum as well. But, on a negative note, Tapestry 5 needs more documentation. You can buy 200 books on Struts (though why you’d want to use it is beyond me), 3 on Grails, 1 (and soon 2) on Wicket, and 0 on Tapestry 5. Tapestry 5 really needs more developer momentum and support to come to fruition. Howard Lewis Ship is doing a fantastic job on being very careful and deliberate in his design and crafting of Tapestry, but he is after all one person, and can not do it all.

So, you may ask,  what does this have to do with IRM 2?

Well, I’ve been once again at the crank. I’ve realized that the code I wanted to avoid writing (simple CRUD), I am writing again, primarily through my mis-choice of EJB3. Partly contributing to this is Tapestry -  I’m writing a fair number of CRUD views - none of which are in any way exciting.

I’ve been converting the existing code from Tapestry/EJB to the Groovy on Grails framework.  Its been amazing easy to accomplish, and the Grails scaffolding support has been an amazing productivity boost.

Look for a developer’s release by the end of the year .

Till then

-Yann

So, long story short. I have a ThinkPad T60P, non-widescreen edition. I run Ubuntu 7.10 Gustry on it (after Windows did a suicide on its self).

It features a Mobility Radeon FireGL V5200 (the Widescreen models may use a V5250 - you’ve been warned) . Out of the box, the last fglrx driver to support this card is 8.35. Yes, the FireGL driver doesn’t support FireGL cards anymore, so says the top-notch driver division at ATI.

The 8.35 driver has some important breakages:

- Can’t suspend or resume (on Kernels with a SLUB allocator, like Ubuntu)

- Can’t hibernate (ditto)

- Weird things happen when you’re dual monitored and start X without a monitor connected.

-

But, perhaps the 8.42.3 driver, hot off the presses, fixes some of that right?  I sought to find out.

The biggest hurdle is 8.42.3 doesn’t support FireGL cards. Ooops. But thanks to these forum posts and mini-patch, it is possible to use the card. For the record, I’m using the provided defaults for CARDIDs.

 http://www.phoronix.com/forums/showthread.php?t=5203

Next up, you have to grab a 8.35 installer packet, extract it (–extract), and then copy common/ati/control over /etc/ati/control, or you’re forever running around with a fantastic watermark in the right corner (whoever thought that was a great idea at ATI deserves to get shot).

So, after mucking with it for an hour, I’ve concluded

- Suspend works, but resume does not, when in multiple monitor mode (wtf)

- Ditto for hibernate

- Compiz runs, but doesn’t really work well at all.

This is in stark contrast to the NVidia-GLX driver, which I use on my desktop, and is a fantastic piece of work, never breaks, and delivers outstanding performance.

Maybe 8.42.4… Or RadeonHD. Or just buy a new laptop with n Nvidia card.

I’ve just published a quicky little program called softwedge which is designed to easily use many serial-port based barcode scanners (including many wireless Bluetooth scanners) with common X11 applications. It takes the serial port data stream and converts the characters into X11 keypress events which are sent via the XTest extension to the active cursor position.

There is no release yet, but you can get to the .git repository here: http://repo.or.cz/w/softwedge.git and check it out. It has been tested with and designed around a Metrologic FocusBT Bluetooth 2d-imager scanner.

Note that you don’t need this software if you have a USB based HID-compatible barcode scanner.

atphttpd is another old software application from the my software archives. Its a very simplistic, memory caching web server, back when it was all the rage to develop web servers, especially for performance It achieved quite some good benchmarks on a dual Pentium 166 running FreeBSD 3.0 - which should give you some idea of the age of the source.

Software license is BSD based. Feel free to mess with it, but word of warning, its not very good (this was a high-school-era programming project - design is not part of the picture).

atphttpd-04btar.gz