So the magic 0x2A encoding in the cache files had me stumped for several days. I could see some of the data, but the packing was… odd (and no real price ever materialized out of it, turns out its a 64 bit integer, with lots of 0s 
). The real problem was the zeros, or the lack of them. Turns out the packing algorithm appears to be optimized to simply not encode zero bytes, slightly reducing space. I believe that the first byte (and every “code” byte thereafter), which is split into two nibbles for this algorithm, contains a count of zeros and not zeros (bytes it simply copies to the output stream). This appears to be exclusive – either 0 is inserted or bytes are copied (per nibble), but not both.
I can’t fully verify this as I don’t have any “unpacked” in the same format, but by unpacking I can now locate more relevant data fields. Yay!
Progress is good. The parser doesn’t yet understand type 0x2A yet, but it will soon…
The parser library is a plain-vanilla C++ library, cross platform and embeddable in just about anything. I’m mainly focused on building an AST of the cache file format at this point. We can walk the AST and fetch the needed data later, which is the “easy part.” This is not a “find-magic-byte sequence and use fixed offsets” type parser
You can track progress at gitorious.org/libevecache.
Parse exception Can't identify type 0x2a at position 0x124
Beginning dump...
{Length: 3} <SInt '738197504'> <STuple>
( {Length: 4} <SMarker ID: -120 > <SIdent 'GetOrders'> <SInt '10000016'> <SInt '642'> )
<SDict>
( {Length: 1} <SObject>
( {Length: 2} <SIdent 'objectCaching.CachedMethodCallResult'> <STuple>
( {Length: 2} <SDict>
( {Length: 4} <STuple>
( {Length: 3} <SIdent '1 minute'> <NONE> <NONE> )
<SIdent 'versionCheck'> <SMarker ID: -114 > <SIdent 'sessionInfo'> )
<SSubstream>
( {Length: 3} <SLongLong '2678554370227830784'> <SString 'dbutil.RowList'> <SDict>
( {Length: 4} <SDBHeader>
( {Length: 1} <STuple>
( {Length: 2} <SString 'blue.DBRowDescriptor'> <STuple>
( {Length: 1} <STuple>
( {Length: e} <STuple>
( {Length: 2} <SMarker ID: -117 > <SInt '6'> )
<STuple>
( {Length: 2} <SMarker ID: -95 > <SInt '5'> )
<STuple>
( {Length: 2} <SMarker ID: 74 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -116 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -118 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -96 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -119 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: 116 > <SInt '11'> )
<STuple>
( {Length: 2} <SMarker ID: -125 > <SInt '64'> )
<STuple>
( {Length: 2} <SMarker ID: 126 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -101 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -115 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -106 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: 41 > <SInt '3'> )
)
)
)
)
<SMarker ID: 36 > <STuple>
( {Length: e} <SMarker ID: -117 > <SMarker ID: -95 > <SMarker ID: 74 > <SMarker ID: -116 > <SMarker ID: -118 > <SMarker ID: -96 > <SMarker ID: -119 > <SMarker ID: 116 > <SMarker ID: -125 > <SMarker ID: 126 > <SMarker ID: -101 > <SMarker ID: -115 > <SMarker ID: -106 > <SMarker ID: 41 > )
<SIdent 'columns'> )
)
)
)
)
So, I’m doing it after all. Reverse engineering the cache files. The format is actually surprisingly simple once you determine the identifiers of all the data fields (and their lengths). No reverse compiling or digging into the EVE runtime was performed, simply educated guesses and a knowledge of CPython internals.
Progress is good. Relevant data identified. More results to be posted, and then an open source code release
EVE-Central.com has always sought to provide an in-game browser usable implementation (as usable as the in game browser can be). This has in the past severely limited what can be done in terms of dynamic user interfaces. There are numerous hooks for alternate display methodologies to prevent locking up the IGB browser when loading large pages (say, a global Tritanium listing). Its also responsible for the somewhat “ugly” look of EVE-Central, since maintaining two presentation layers is a fair amount of work.
How many of you are currently actively using the in game browser for browsing EVE-Central? How much would it hurt if you had to switch to using an external browser?
This question comes up every now and then, for the clued in folks who try to determine how the pricing aggregates (median, average) are produced, especially when they notice it vary across the site. This quick guide is designed to settle some of these questions, which undoubtedly will lead to more questions.
The Basics
- All aggregate pricing information has a deadband filter to remove the 0.01 ISK noise. This filter varies by type ID, but generally cuts off at >= 1.0 ISK.
- The average is the weighted average. The weight is simply the amount offered for sale or purchase.
- The median is as advertised. It represents the median order, regardless of the quantity offered.
Exceptions
- The statistics are governed by minimum quantities, which vary according to the type ID and view if not specified by the user.
- Specifically, if using the marketstat API, the minimum quantity for all mineral types (including Morphite) is set to 10,000. This minimum quantity is not used in the quicklook API or when viewing on the web.
- The evemon API uses a minimum quantity of 5,000 (not sure why…) and is the aggregate price for the set of buy and sell orders.
Confused yet?
I’ve been extremely busy lately moving everything from one place to another, and haven’t had any time to push out code changes to EVE-Central. So, in order to make it look like there is some progress (;)), I’ve added a quick “Latest Transactions” section to each item. The filtering is bad (it doesn’t quite get regions yet) and only shows the last 10 entries. There is a lot more backend work, including new aggregate pricing models, which use this data, however the data is brought out to the API or webpages yet.
This transaction data is provided by users (you included!) giving EVE-Central.com Full EVE API access to your account so the system can pull market transaction logs. The more people who contribute, the better it gets.
Just as an FYI, CCP seems to have broken their wallet API recently.
<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
<currentTime>2009-05-17 03:00:34</currentTime>
<error>A general error has occured, please try again later.</error>
<cachedUntil>2009-05-17 04:00:34</cachedUntil>
</eveapi>
Its been down for a day or so. If anyone has some hints on what they broke this time, I’d like to hear it.
Just a quick note for today.
Thank you to everyone who has contributed to the UserVoice feedback page to date. There are some great suggestions mixed in here. I’ve started flagging entries as planned/review/etc. If you head to http://evecentral.uservoice.com/pages/16679-general you can see the activity to date.
More updates soon.
I’ve added a UserVoice feedback link to the front page of EVE-Central. UserVoice is a very handy little feedback suggestion system which allows anonymous users a quick way to add comments to the site and vote on other people’s suggestions. I’ve already collected two good pieces of user feedback from the system (and fixed one bug!).
This is complementary to the other methods I’m (trying) to track, including EVEO forums (please use the official thread if you do comment directly – other threads get lost far too fast) and the Bugzilla list (for developer input bugs).
I’ve only placed the feedback side tab on the front page for now – I may add it other places or provide a less “in your face” version on the new page footers.
First, a small change that others have been asking for. Now, when you search for items in the main interface, blueprints won’t be shown by default. Since there really isn’t a player market for these, there is no reason to show them to you. Uploads and API access will still be accepted for those that want to know.
Edit: You can once again search for Blueprints. If you add the term “Blueprint” to the end of your search, all blueprints which match will be returned. Feature added by popular request.
API transaction pulling is ongoing and the system has been collecting a good amount of data which helps to fine tune algorithms. Work on this feature was a bit slow due to real work and other life activities, but gave me more data to play with, so its not entirely bad. While there is no direct display of data as of the site update tonight, the underlying database contains an entirely new pricing model. I expect the introduce the display of this data in the upcoming week. If you want to help, feel free to register on EVE-Central and input your full API key.
On a side note, I have been playing with producing a market lookup iPhone application, which pulls data from EVE-Central.com. It will take awhile to push it out of the door (and pass through Apple’s approval), but perhaps by the time of the iPhone 3.0 OS launch it will be a reality.
Fly safe!
