Yann's Blog - Software and hardware

I’m pushing out a new update to the site today. Several new features are included and bugs are fixed:

• The API has bug fixes related to the minimum quantity filter being misapplied and overridden in certain cases. The behavior should be consistent now.
• Top bar is now left aligned – looks a bit better?
• The logout button works again (yay!)
• Added a new Trade section, which includes the new Comparison tool I’ve blogged about earlier, in BETA form. There is no cost (ISK wise) to use at this moment (haven’t gotten about getting the subscription interface written yet). You need to be logged in to use this feature though. If you have any feedback and improvements, send me an e-mail or comment here in the blog.

Expect more updates soon

Just a quick one, showing off a new screenshot of the trade compare tool. See it here. Also implemented is a nifty little autocomplete AJAX feature for the system and region names.

I’ll be re-using this YUI component for the market type searching as well, with a fallback to the current page. Note that the autocomplete is not IGB compatible, but the trade comparison tool as a whole is not either (due to large page sizes and lots of presented information), making it a non-issue.

Also, I’m working on a market sell movement application to try to track volumes moved. The data to back this isn’t perfect by any chance, but perhaps will provide an overinflated view of the actual item movement.

I’ve been quiet about my IRM 2 activities recently, and I apologize. This article is not really a status update, but a reflection on mapping programs such as IRM 2 to a traditional RDBMS.

I’ve determined that I just built a database… inside another database. The IRM 2 core currently has a core table, which is quite literally an ID, Column, Value mapping table. The data simply looked liked this:

A100, 1, Name, Computer
A100, 1, Color, Blue
A100, 1, Owner, Bob

Every property simply gets dumped into this table, and requires extensive remapping on the application logic. The RDBMS doesn’t have much insight into schemas, as it subverts its notion of schemas. Searching for explicit property values requires joining huge amounts of data together. Data storage is inefficient, with records scattered throughout the datastore (and in many places on the disk), especially if records were updated after creation.

More and more I’m starting to feel that approaches such as BigTable, SimpleDB, and CouchDB are taking the correct approach to the large data storage problem. The data stored is as a series of documents with arbitrary and per-document schema. Each node is a replicated element, and data storage is not centralized but rather distributed.

The advent of Google’s AppEngine supporting this model of database reaffirms that this is the correct direction to move in for large scale online applications.

Now, what does all of this have to do with IRM 2 and gIRM? Stay tuned – I’m trying to find out myself

I’ve taken the chance now to upgrade some of our APIs, make them more consistent, cleaner, and better documented.

The first primary change is moving from the old /home namespace to the new /api namespace. This allows for some cleaner separation of the logic and not the old mismash of stuff. I’ve also cleaned up the marketstat API, allowing multiple type IDs to be passed in a single query.

The new API has a version header now as well <evec_api version=”2.0″>…</evec_api>. You can distinguish responses, and possibly in the future request specific versions of formatting.

As before, the API is open, not requiring a key to use. I’ve contemplated a key based approach but haven’t deemed in necessary yet. If you suddenly find you can’t connect to the server – you’ve probably been doing something naughty (or it broke).

I have a few IPs blocked for bone-headed use of the system – such as querying every typeID from 1 to 100000000, and in string order at that. Don’t do that

The old APIs based off of /home will continue to work for now. It would be a perfect world if everything would upgrade, but thats not always possible. I’ll monitor usage of both and cut off the old one once its no longer popular.

Many nifty things are afoot for EVE-Central.com. For one, I migrated away from mod_python, which was hanging up the Apache process several times per day for unknown reasons. There was even a monitor script to kill -9, ipcrm, and restart Apache if it wasn’t responding – this horrible restart was happening several times a day. The new EVE-Central.com is running CherryPy as the backend server now – the conversion was quick and painless, plus it performs better (barring any new introduced bugs).

On the new feature front, I’m working on a trade comparison tool for sell orders, trying to find price differences between systems or regions. Its in an alpha prototype stage, so not for public consumption yet. I expect it to launch in a week or two. Here is a link to a sample image.

I’m also mulling over a possible Premium subscription to EVE-Central. Paid in ISK, it would allow access to the aforementioned trade comparison tool, plus other options, such as ad-free view, and the ability to upload “premium” content which would not get pushed to normal users until a small time window had elapsed. Details still sketchy at this point