Yann's Blog - Software and hardware

Success!

Thanks to some community contributions, libevecache has grown quite a bit this weekend!

New features include:

1. Shared object support (will give an object reference in place of the previous holder type)
2. DBRow decoding (needs more string identifiers for columns in other cache file formats)
3. The dumper utility now learned about --market, which lets it produce market CSV files from a cache file.

   atrus@atp (master) :~/Marketlogs/marketreader$ util/dumper --market tests/Lonetrek-Armageddon-2009.07.30\ 062647.cache > o
   atrus@atp (master) :~/Marketlogs/marketreader$ cat o
   Cache File Dumper
   File: tests/Lonetrek-Armageddon-2009.07.30 062647.cache
   price,volRemaining,typeID,range,orderID,volEntered,minVolume,bid,issued,duration,stationID,regionID,solarSystemID,jumps,
   54000000.0,1,643,32767,1185915256,5,1,False,2009-07-04 09:01:17.000,90,60004012,10000016,30001363,5
   54000000.0,5,643,32767,1187317153,8,1,False,2009-07-06 18:13:31.000,90,60002419,10000016,30001363,5
   55499999.0,1,643,32767,1217266843,1,1,False,2009-07-26 17:58:17.000,90,60000838,10000016,30001367,1
   ...
   52749999.0,1,643,32767,1221169229,1,1,False,2009-07-29 18:25:11.000,30,60003838,10000016,30001377,0
   57999999.88,1,643,32767,1221534269,2,1,False,2009-07-29 23:31:51.000,90,60002326,10000016,30001429,0
   57999999.88,2,643,32767,1221581224,2,1,False,2009-07-30 00:22:49.000,90,60003874,10000016,30001401,0
   45720000.0,5,643,40,1173825903,5,1,True,2009-07-05 16:35:12.000,90,60004027,10000016,30001368,0
   53999.99,1,643,32767,1190937468,1,1,True,2009-07-06 05:58:44.000,90,60004291,10000016,30001368,0
   46956001.5,1,643,3,1212303440,1,1,True,2009-07-22 23:37:54.000,90,60003841,10000016,30001376,0
   53784.31,1,643,32767,1131184395,1,1,True,2009-05-21 14:34:50.000,90,60003889,10000016,30001405,0
   

Update your clones today!

I am going to buckle down and produce an EVE-Central uploader which uses libevecache in the next week. Look for a beta version sometime by Friday.

I’ve done some polish work and integration of various libraries in Yann’s Eagle library repository.

Get it here..

Noteworthy inclusions:

• NAND and SDRAM footprints
• Merging of various MSP430 footprints into the “texas.lbr” library.
• More crystals, including subminature parts (ABM10)
• Cleanup of the Atmel library
• Some TVS components from Tyco and Vishay are included
• Freescale HCS08 footprints for the QB and QK series

libevecache, the open source library for reading EVE Online cache files, has received another update. I have decoded the types for Database types (including much scratching of heads regarding time stamps, I think I have a workable value now with a few hints on the side).

The usual comments came out in the EVE forum thread, but I think everything politically is well squared away now. Its not an “under the radar” project anymore.

Update your clones, and contribute something!

Success!

I give you the first half of a market order cache file, including RLE decompressed database rows. The structure is not 100% correct in regards to nested elements, and there is a bit of unresolved magic occurring at the substream end condition. Also, the naming of the elements is somewhat odd – the first set of orders is actually the Sell orders, and the second half comprises the Buy orders. The exact meaning of the header defining the structure is also not yet determined (suspect its an identifier followed by a numerical “type” code, though either the order differs from the row descriptor or the types have no relation to the serialized types in the cache file…).

The second half of the file can be parsed (except for the odd end condition again…), but the nesting structure is “broken”. This will require some more investigation, but is likely easy to fix.

Another oddity is the dictionary type (or double tuple – the length is half of the number of elements present). It is encoded value followed by key (not key followed by value).

Below is the AST structure dump from the ‘dumper’ utility included. The cache files list Armageddons in Lonetrek as of a couple days ago.

 <SInt '738197504'>
 <STuple>
 (
   <SMarker ID: -120 >
   <SIdent 'GetOrders'>
   <SInt '10000016'>
   <SInt '643'>
 )
 <SDict>
 (
   <SObject>
   (
     <SIdent 'objectCaching.CachedMethodCallResult'>
     <STuple>
     (
       <SDict>
       (
         <STuple>
         (
           <SIdent '1 minute'>
           <NONE>
           <NONE>
         )
         <SIdent 'versionCheck'>
         <SMarker ID: -114 >
         <SIdent 'sessionInfo'>
       )
       <SSubstream>
       (
         <SLongLong '2678554370227830784'>
         <SString 'dbutil.RowList'>
         <SDict>
         (
           <SDBHeader>
           (
             <STuple>
             (
               <SString 'blue.DBRowDescriptor'>
               <STuple>
               (
                 <STuple>
                 (
                   <STuple>
                   (
                     <SMarker ID: -117 >
                     <SInt '6'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -95 >
                     <SInt '5'>
                   )
                   <STuple>
                   (
                     <SMarker ID: 74 >
                     <SInt '2'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -116 >
                     <SInt '2'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -118 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -96 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -119 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: 116 >
                     <SInt '11'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -125 >
                     <SInt '64'>
                   )
                   <STuple>
                   (
                     <SMarker ID: 126 >
                     <SInt '2'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -101 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -115 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -106 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: 41 >
                     <SInt '3'>
                   )
                 )
               )
             )
           )
           <SMarker ID: 36 >
           <STuple>
           (
             <SMarker ID: -117 >
             <SMarker ID: -95 >
             <SMarker ID: 74 >
             <SMarker ID: -116 >
             <SMarker ID: -118 >
             <SMarker ID: -96 >
             <SMarker ID: -119 >
             <SMarker ID: 116 >
             <SMarker ID: -125 >
             <SMarker ID: 126 >
             <SMarker ID: -101 >
             <SMarker ID: -115 >
             <SMarker ID: -106 >
             <SMarker ID: 41 >
           )
           <SIdent 'columns'>
         )
         <DBRow 001882ba7d000000000000000000f03f509bcdfd85fcc90178a1af460500000001000000ac96930390969800d3c8c901050000008302ff7f5a>
         <DBRow 001882ba7d000000000000000000144050a0f97765fec901a105c54608000000010000007390930390969800d3c8c901050000008302ff7f5a>
         <DBRow f0c6933881000000000000000000f03f909da9a71a0eca019b048e480100000001000000468a930390969800d7c8c901010000008302ff7f5a000000>
         <DBRow 004ee77e7d000000000000000000f03f9067651cec00ca013b9b434701000000010000007192930390969800eec8c901070000008302ff7f5a000000>
         <DBRow 003476667b000000000000000000f03f300ef4eb3409ca0173a311480100000001000000c2bf930390969800eec8c901070000008302ff7f5a>
         <DBRow 001882ba7d000000000000000000f03f50b65d2fc40eca0143029f480100000001000000149b930390969800f2c8c901040000008302ff7f5a>
         <DBRow 001882ba7d000000000000000000f03f3024c3b6fc0eca01fe10a6480100000001000000149b930390969800f2c8c901040000008302ff7f5a>
         <DBRow 00e09962820000000000000000001440b0a9347fe3dec901ec889b3c0a00000001000000179b930390969800f3c8c901050000008302ff7f5a>
         <DBRow 00a8b10a870000000000000000000040e0883e7431e2c901039148440b000000010000000490930390969800f9c8c901070000008302ff7f5a>
         <DBRow 9ca7b10a8700000000000000000008404015505adcfdc9012b53f94606000000010000002296930390969800f9c8c901070000008302ff7f5a000000>
         <DBRow 8049016282000000000000000000f03f10c1e8bc0409ca01e9ad0e4801000000010000001996930390969800fbc8c901080000008302ff7f5a000000>
         <DBRow 00f94a8f7c0000000000000000001440c0a05632d10fca01fd20bb4805000000010000003796930390969800fdc8c901060000008302ff7f1e>
         <DBRow 809ddd657b000000000000000000084010aee06a2e10ca010c32c34803000000010000003796930390969800fdc8c901060000008302ff7f1e000000>
         <DBRow 7068e38f7c000000000000000000f03f20ffe8eced0aca012b523b4802000000010000003196930390969800fdc8c901060000008302ff7f5a000000>
         <DBRow 809ddd657b000000000000000000f03f1073931cab0fca01bda1354805000000010000003796930390969800fdc8c901060000008302ff7f1e000000>
         <DBRow 00267c907c000000000000000000f03fb0a3e34d050eca01c4d68a4801000000010000002e9693039096980001c9c901080000008302ff7f5a>
         <DBRow 0070c9b28b000000000000000000f03f50dfb0e8010fca0141d0a64801000000010000009d969303909698000fc9c9010a0000008302ff7f5a>
         <DBRow 8049016282000000000000000000f03f40a6e8b2240bca013434424801000000010000006f9793039096980010c9c901090000008302ff7f5a000000>
         <DBRow f0902e7481000000000000000000f03fb0cfaa9f370fca017790ac480100000001000000639793039096980012c9c9010b0000008302ff7f5a000000>
         <DBRow 00a8b10a87000000000000000000f03fe0277593410fca01bd61ad480200000001000000169093039096980015c9c9010e0000008302ff7f5a>
         <DBRow 00a8b10a87000000000000000000004000cbd3a70010ca0134c5bf480200000001000000169093039096980015c9c9010e0000008302ff7f5a000000>
         <DBRow 001882ba7d000000000000000000f03fb0c2eda85110ca01d44dc5480100000001000000149b930390969800f2c8c901000000008302ff7f5a>
         <DBRow f01373d17a000000000000000000f03fe0ea9be87910ca014d90c9480100000001000000fe95930390969800e1c8c901000000008302ff7f1e000000>
         <DBRow 50a3b10a87000000000000000000f03fb0dae8bfa410ca013d22cf480200000001000000169093039096980015c9c901000000008302ff7f5a000000>
         <DBRow 50a3b10a870000000000000000000040b0d4aadeab10ca01a8d9cf4802000000010000002296930390969800f9c8c901000000008302ff7f5a000000>
         <DBRecords>
         (
           <STuple>
           (
             <STuple>
             (
               <SStreamIdent 2>
             )
             <SDict>
             (
               <SStreamIdent 1>
               <SMarker ID: 36 >
               <STuple>
               (
                 <SMarker ID: -117 >
                 <SMarker ID: -95 >
                 <SMarker ID: 74 >
                 <SMarker ID: -116 >
                 <SMarker ID: -118 >
                 <SMarker ID: -96 >
                 <SMarker ID: -119 >
                 <SMarker ID: 116 >
                 <SMarker ID: -125 >
                 <SMarker ID: 126 >
                 <SMarker ID: -101 >
                 <SMarker ID: -115 >
                 <SMarker ID: -106 >
                 <SMarker ID: 41 >
               )
               <SStreamIdent 3>
             )
           )
         )
         <DBRow 005c3e736a0000000000000000001440d062bb918efdc9016f29f7450500000001000000bb96930390969800d8c8c90100000000830228005a0001>
         <DBRow 9cbe2f2000000000000000000000f03ff0eb1ad2fefdc9017c43fc460100000001000000c397930390969800d8c8c901000000008302ff7f5a000100>
         <DBRow 0427f5536d000000000000000000f03fb0cbb36f250bca015048424801000000010000000196930390969800e0c8c90100000000830203005a000100>
         <DBRow 9cd50e2000000000000000000000f03f7023fa4b21dac9010b816c4301000000010000003196930390969800fdc8c901000000008302ff7f5a000100>
         <DBRow 00a82d336f000000000000000000004030551585dc0fca01f7a5314805000000010000003196930390969800fdc8c9010000000083020a005a0001>
         <DBRow 4068fe3f6e0000000000000000001c4060feb17c410bca01a402a9470a000000010000002e9693039096980001c9c90100000000830214001e000100>
         <DBRow 00ba95b96e000000000000000000f03f30366bc52f0eca01711940480100000001000000919693039096980011c9c90100000000830204005a0001>
         <DBRow e0067fe0680000000000000000001040302ebdaeedf4c901a9d6f0440400000001000000609793039096980012c9c90100000000830205005a000100>
         <DBRow c84cb55b6b000000000000000000f03f408f5a9adb02ca0121208a460100000001000000609793039096980012c9c90100000000830205005a000100>
         <DBRow b085fde068000000000000000000f03ff056816f43fac9016eb19e4601000000010000006c9793039096980014c9c90100000000830205005a000100>
         <DBRow 801bf4266c000000000000000000004090cd97f0cf05ca019d7ead470200000001000000169093039096980015c9c90100000000830203005a000100>
         <DBRow 40fa51b96e000000000000000000004010707d136b0dca01d021a0460300000001000000e69593039096980023c9c90100000000830214005a000100>
         <DBRow 001882ba7d000000000000000000f03f1026ef856e10ca01bb37c8480100000001000000fd93930390969800f3c8c901000000008302ffff5a0001>
         <DBRow 00a446167200000000000000000000408024d1608e10ca01044fcc480200000001000000e293930390969800e0c8c901000000008302ffff070001>
       )
       <SLongLong '128934085416010580'>
     )
   )

So the magic 0x2A encoding in the cache files had me stumped for several days. I could see some of the data, but the packing was… odd (and no real price ever materialized out of it, turns out its a 64 bit integer, with lots of 0s ). The real problem was the zeros, or the lack of them. Turns out the packing algorithm appears to be optimized to simply not encode zero bytes, slightly reducing space. I believe that the first byte (and every “code” byte thereafter), which is split into two nibbles for this algorithm, contains a count of zeros and not zeros (bytes it simply copies to the output stream). This appears to be exclusive – either 0 is inserted or bytes are copied (per nibble), but not both.

I can’t fully verify this as I don’t have any “unpacked” in the same format, but by unpacking I can now locate more relevant data fields. Yay!

Progress is good. The parser doesn’t yet understand type 0x2A yet, but it will soon…

The parser library is a plain-vanilla C++ library, cross platform and embeddable in just about anything. I’m mainly focused on building an AST of the cache file format at this point. We can walk the AST and fetch the needed data later, which is the “easy part.” This is not a “find-magic-byte sequence and use fixed offsets” type parser

You can track progress at gitorious.org/libevecache.


Parse exception Can't identify type 0x2a at position 0x124
Beginning dump...
{Length: 3} <SInt '738197504'> <STuple>
( {Length: 4} <SMarker ID: -120 > <SIdent 'GetOrders'> <SInt '10000016'> <SInt '642'> )
<SDict>
( {Length: 1} <SObject>
( {Length: 2} <SIdent 'objectCaching.CachedMethodCallResult'> <STuple>
( {Length: 2} <SDict>
( {Length: 4} <STuple>
( {Length: 3} <SIdent '1 minute'> <NONE> <NONE> )
<SIdent 'versionCheck'> <SMarker ID: -114 > <SIdent 'sessionInfo'> )
<SSubstream>
( {Length: 3} <SLongLong '2678554370227830784'> <SString 'dbutil.RowList'> <SDict>
( {Length: 4} <SDBHeader>
( {Length: 1} <STuple>
( {Length: 2} <SString 'blue.DBRowDescriptor'> <STuple>
( {Length: 1} <STuple>
( {Length: e} <STuple>
( {Length: 2} <SMarker ID: -117 > <SInt '6'> )
<STuple>
( {Length: 2} <SMarker ID: -95 > <SInt '5'> )
<STuple>
( {Length: 2} <SMarker ID: 74 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -116 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -118 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -96 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -119 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: 116 > <SInt '11'> )
<STuple>
( {Length: 2} <SMarker ID: -125 > <SInt '64'> )
<STuple>
( {Length: 2} <SMarker ID: 126 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -101 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -115 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -106 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: 41 > <SInt '3'> )
)
)
)
)
<SMarker ID: 36 > <STuple>
( {Length: e} <SMarker ID: -117 > <SMarker ID: -95 > <SMarker ID: 74 > <SMarker ID: -116 > <SMarker ID: -118 > <SMarker ID: -96 > <SMarker ID: -119 > <SMarker ID: 116 > <SMarker ID: -125 > <SMarker ID: 126 > <SMarker ID: -101 > <SMarker ID: -115 > <SMarker ID: -106 > <SMarker ID: 41 > )
<SIdent 'columns'> )
)
)
)
)


So, I’m doing it after all. Reverse engineering the cache files. The format is actually surprisingly simple once you determine the identifiers of all the data fields (and their lengths). No reverse compiling or digging into the EVE runtime was performed, simply educated guesses and a knowledge of CPython internals.

Progress is good. Relevant data identified. More results to be posted, and then an open source code release

EVE-Central.com has always sought to provide an in-game browser usable implementation (as usable as the in game browser can be). This has in the past severely limited what can be done in terms of dynamic user interfaces. There are numerous hooks for alternate display methodologies to prevent locking up the IGB browser when loading large pages (say, a global Tritanium listing). Its also responsible for the somewhat “ugly” look of EVE-Central, since maintaining two presentation layers is a fair amount of work.

How many of you are currently actively using the in game browser for browsing EVE-Central? How much would it hurt if you had to switch to using an external browser?

This question comes up every now and then, for the clued in folks who try to determine how the pricing aggregates (median, average) are produced, especially when they notice it vary across the site. This quick guide is designed to settle some of these questions, which undoubtedly will lead to more questions.

The Basics

1. All aggregate pricing information has a deadband filter to remove the 0.01 ISK noise. This filter varies by type ID, but generally cuts off at >= 1.0 ISK.
2. The average is the weighted average. The weight is simply the amount offered for sale or purchase.
3. The median is as advertised. It represents the median order, regardless of the quantity offered.

Exceptions

1. The statistics are governed by minimum quantities, which vary according to the type ID and view if not specified by the user.
2. Specifically, if using the marketstat API, the minimum quantity for all mineral types (including Morphite) is set to 10,000. This minimum quantity is not used in the quicklook API or when viewing on the web.
3. The evemon API uses a minimum quantity of 5,000 (not sure why…) and is the aggregate price for the set of buy and sell orders.

Confused yet?

I’ve been extremely busy lately moving everything from one place to another, and haven’t had any time to push out code changes to EVE-Central. So, in order to make it look like there is some progress (;)), I’ve added a quick “Latest Transactions” section to each item. The filtering is bad (it doesn’t quite get regions yet) and only shows the last 10 entries. There is a lot more backend work, including new aggregate pricing models, which use this data, however the data is brought out to the API or webpages yet.

This transaction data is provided by users (you included!) giving EVE-Central.com Full EVE API access to your account so the system can pull market transaction logs. The more people who contribute, the better it gets.