Reverse engineering the cache files, part 1
So, I’m doing it after all. Reverse engineering the cache files. The format is actually surprisingly simple once you determine the identifiers of all the data fields (and their lengths). No reverse compiling or digging into the EVE runtime was performed, simply educated guesses and a knowledge of CPython internals.
Progress is good. Relevant data identified. More results to be posted, and then an open source code release 
Is CCP going to be ok with making the code open? I thought they didn’t like people sharing that stuff.
Comment by Morgan — August 3, 2009 @ 10:02 am
As far as I’m concerned, the “not open source” is simply conjecture and FUD spread on the forums. From reading the various official CCP responses and the EULA, this library would qualify as receiving information for playing the game, which is generally “ok”. I have never seen an official response regarding the publishing of the cache file format. In all respects, the actual on disk format is very simplistic, and would be trivial to “protect” if CCP were interested in doing so.
Comment by Yann — August 3, 2009 @ 10:23 am
PS: Here is a high level description of the cache file on CCP’s own forum. FWIW, the information here is largely correct. I am working on fully validating all basic data types which this program misses.
http://www.eveonline.com/ingameboard.asp?a=topic&threadID=734561&page=1
Comment by Yann — August 3, 2009 @ 10:33 am
The reason why we over at EVE Metrics and others don’t release source for cache readers is because of the fact that CCP A) Don’t like people releasing source for this, and B) will hold you responsible for any use of your source they don’t like. Basically, if your code sparks the next BACON, you’re getting banned as well as whoever does the modifications. The threat of being banned and the fact that we like to stay on good terms with CCP are reason enough for us
Cachefiles aren’t hard to decode (well, not to ‘decode’. Properly decoding them takes a lot of work, but you can make something ‘good enough’ pretty easily). It’ll be nice to see EC with a cache uploader, though- I’ve been getting worried that EC’s not been seeing much love recently, and it’s nice to see things picking up again
Comment by James Harrison — August 3, 2009 @ 11:06 am
You’d think that CCP wouldn’t send clients locally cached copies of information they don’t want us to have, but I suppose everybody has to cut some data security corners somewhere. If cached files can be used to detect the location of cloakers as they jump through the jumpgates (for example), though, I’d rather know about it right away so I can adjust for the stunts pirates will inevitably pull, instead of only finding out about it after I lose a dozen blockade runners. Then again, something like the POS bug that is still screwing up the prices of T2 components (especially now that nullsec corps like Goonswarm can actually control their production and sale) might also end up being revealed.
Oh well. Is there any chance of getting a “median weighted by volume” statistic added? It would be very useful for sell orders, though I would prefer to also keep the regular median around, just because the weighted median would be nearly useless for buy orders (one order for ten million Anshars at 0.1 isk each would make it return absurd results). The regular median can actually be skewed as well by someone with enough trade-skill alts, posting dozens of single-unit orders, so while it’s more difficult to skew, it would be nice to have an alternative to check when the median starts looking weird.
Comment by Tatterdemalian — August 12, 2009 @ 6:24 am