I have built a new binary of libevecache 0.1.2 for Windows. Included are several important fixes to the CSV export file.
Note that filename globbing (such as using *.cache) is NOT working under cmd.exe in Windows (PowerShell not tested), since Windows does not do filename globbing for commands. As a temporary workaround, use bash 
libevecache just gained two releases this weekend. Version 0.1 was a functional single shot cache file to .CSV file converter capable release, but had huge memory leak and pointer ownership issues, which made Contribtastic! very unstable and generally unusable.
After spending some quality time with Valgrind, all of the remaining issues are now resolved, and version 0.1.1 is available for mass consumption. Look for Contribtastic early betas soon. One new feature of libevecache in version 0.1.1 is the dumper is capable of digesting a whole CachedMethodCalls folder in one shot (just pass it the list of files on the command line). When you run dumper with the --market flag, you can safely redirect stdout to the file of your choice.
Success!
Thanks to some community contributions, libevecache has grown quite a bit this weekend!
New features include:
dumper utility now learned about --market, which lets it produce market CSV files from a cache file.
atrus@atp (master) :~/Marketlogs/marketreader$ util/dumper --market tests/Lonetrek-Armageddon-2009.07.30\ 062647.cache > o atrus@atp (master) :~/Marketlogs/marketreader$ cat o Cache File Dumper File: tests/Lonetrek-Armageddon-2009.07.30 062647.cache price,volRemaining,typeID,range,orderID,volEntered,minVolume,bid,issued,duration,stationID,regionID,solarSystemID,jumps, 54000000.0,1,643,32767,1185915256,5,1,False,2009-07-04 09:01:17.000,90,60004012,10000016,30001363,5 54000000.0,5,643,32767,1187317153,8,1,False,2009-07-06 18:13:31.000,90,60002419,10000016,30001363,5 55499999.0,1,643,32767,1217266843,1,1,False,2009-07-26 17:58:17.000,90,60000838,10000016,30001367,1 ... 52749999.0,1,643,32767,1221169229,1,1,False,2009-07-29 18:25:11.000,30,60003838,10000016,30001377,0 57999999.88,1,643,32767,1221534269,2,1,False,2009-07-29 23:31:51.000,90,60002326,10000016,30001429,0 57999999.88,2,643,32767,1221581224,2,1,False,2009-07-30 00:22:49.000,90,60003874,10000016,30001401,0 45720000.0,5,643,40,1173825903,5,1,True,2009-07-05 16:35:12.000,90,60004027,10000016,30001368,0 53999.99,1,643,32767,1190937468,1,1,True,2009-07-06 05:58:44.000,90,60004291,10000016,30001368,0 46956001.5,1,643,3,1212303440,1,1,True,2009-07-22 23:37:54.000,90,60003841,10000016,30001376,0 53784.31,1,643,32767,1131184395,1,1,True,2009-05-21 14:34:50.000,90,60003889,10000016,30001405,0
Update your clones today!
I am going to buckle down and produce an EVE-Central uploader which uses libevecache in the next week. Look for a beta version sometime by Friday.
I’ve done some polish work and integration of various libraries in Yann’s Eagle library repository.
Noteworthy inclusions:
libevecache, the open source library for reading EVE Online cache files, has received another update. I have decoded the types for Database types (including much scratching of heads regarding time stamps, I think I have a workable value now with a few hints on the side).
The usual comments came out in the EVE forum thread, but I think everything politically is well squared away now. Its not an “under the radar” project anymore.
Update your clones, and contribute something! 
Success!
I give you the first half of a market order cache file, including RLE decompressed database rows. The structure is not 100% correct in regards to nested elements, and there is a bit of unresolved magic occurring at the substream end condition. Also, the naming of the elements is somewhat odd – the first set of orders is actually the Sell orders, and the second half comprises the Buy orders. The exact meaning of the header defining the structure is also not yet determined (suspect its an identifier followed by a numerical “type” code, though either the order differs from the row descriptor or the types have no relation to the serialized types in the cache file…).
The second half of the file can be parsed (except for the odd end condition again…), but the nesting structure is “broken”. This will require some more investigation, but is likely easy to fix.
Another oddity is the dictionary type (or double tuple – the length is half of the number of elements present). It is encoded value followed by key (not key followed by value).
Below is the AST structure dump from the ‘dumper’ utility included. The cache files list Armageddons in Lonetrek as of a couple days ago.
<SInt '738197504'>
<STuple>
(
<SMarker ID: -120 >
<SIdent 'GetOrders'>
<SInt '10000016'>
<SInt '643'>
)
<SDict>
(
<SObject>
(
<SIdent 'objectCaching.CachedMethodCallResult'>
<STuple>
(
<SDict>
(
<STuple>
(
<SIdent '1 minute'>
<NONE>
<NONE>
)
<SIdent 'versionCheck'>
<SMarker ID: -114 >
<SIdent 'sessionInfo'>
)
<SSubstream>
(
<SLongLong '2678554370227830784'>
<SString 'dbutil.RowList'>
<SDict>
(
<SDBHeader>
(
<STuple>
(
<SString 'blue.DBRowDescriptor'>
<STuple>
(
<STuple>
(
<STuple>
(
<SMarker ID: -117 >
<SInt '6'>
)
<STuple>
(
<SMarker ID: -95 >
<SInt '5'>
)
<STuple>
(
<SMarker ID: 74 >
<SInt '2'>
)
<STuple>
(
<SMarker ID: -116 >
<SInt '2'>
)
<STuple>
(
<SMarker ID: -118 >
<SInt '3'>
)
<STuple>
(
<SMarker ID: -96 >
<SInt '3'>
)
<STuple>
(
<SMarker ID: -119 >
<SInt '3'>
)
<STuple>
(
<SMarker ID: 116 >
<SInt '11'>
)
<STuple>
(
<SMarker ID: -125 >
<SInt '64'>
)
<STuple>
(
<SMarker ID: 126 >
<SInt '2'>
)
<STuple>
(
<SMarker ID: -101 >
<SInt '3'>
)
<STuple>
(
<SMarker ID: -115 >
<SInt '3'>
)
<STuple>
(
<SMarker ID: -106 >
<SInt '3'>
)
<STuple>
(
<SMarker ID: 41 >
<SInt '3'>
)
)
)
)
)
<SMarker ID: 36 >
<STuple>
(
<SMarker ID: -117 >
<SMarker ID: -95 >
<SMarker ID: 74 >
<SMarker ID: -116 >
<SMarker ID: -118 >
<SMarker ID: -96 >
<SMarker ID: -119 >
<SMarker ID: 116 >
<SMarker ID: -125 >
<SMarker ID: 126 >
<SMarker ID: -101 >
<SMarker ID: -115 >
<SMarker ID: -106 >
<SMarker ID: 41 >
)
<SIdent 'columns'>
)
<DBRow 001882ba7d000000000000000000f03f509bcdfd85fcc90178a1af460500000001000000ac96930390969800d3c8c901050000008302ff7f5a>
<DBRow 001882ba7d000000000000000000144050a0f97765fec901a105c54608000000010000007390930390969800d3c8c901050000008302ff7f5a>
<DBRow f0c6933881000000000000000000f03f909da9a71a0eca019b048e480100000001000000468a930390969800d7c8c901010000008302ff7f5a000000>
<DBRow 004ee77e7d000000000000000000f03f9067651cec00ca013b9b434701000000010000007192930390969800eec8c901070000008302ff7f5a000000>
<DBRow 003476667b000000000000000000f03f300ef4eb3409ca0173a311480100000001000000c2bf930390969800eec8c901070000008302ff7f5a>
<DBRow 001882ba7d000000000000000000f03f50b65d2fc40eca0143029f480100000001000000149b930390969800f2c8c901040000008302ff7f5a>
<DBRow 001882ba7d000000000000000000f03f3024c3b6fc0eca01fe10a6480100000001000000149b930390969800f2c8c901040000008302ff7f5a>
<DBRow 00e09962820000000000000000001440b0a9347fe3dec901ec889b3c0a00000001000000179b930390969800f3c8c901050000008302ff7f5a>
<DBRow 00a8b10a870000000000000000000040e0883e7431e2c901039148440b000000010000000490930390969800f9c8c901070000008302ff7f5a>
<DBRow 9ca7b10a8700000000000000000008404015505adcfdc9012b53f94606000000010000002296930390969800f9c8c901070000008302ff7f5a000000>
<DBRow 8049016282000000000000000000f03f10c1e8bc0409ca01e9ad0e4801000000010000001996930390969800fbc8c901080000008302ff7f5a000000>
<DBRow 00f94a8f7c0000000000000000001440c0a05632d10fca01fd20bb4805000000010000003796930390969800fdc8c901060000008302ff7f1e>
<DBRow 809ddd657b000000000000000000084010aee06a2e10ca010c32c34803000000010000003796930390969800fdc8c901060000008302ff7f1e000000>
<DBRow 7068e38f7c000000000000000000f03f20ffe8eced0aca012b523b4802000000010000003196930390969800fdc8c901060000008302ff7f5a000000>
<DBRow 809ddd657b000000000000000000f03f1073931cab0fca01bda1354805000000010000003796930390969800fdc8c901060000008302ff7f1e000000>
<DBRow 00267c907c000000000000000000f03fb0a3e34d050eca01c4d68a4801000000010000002e9693039096980001c9c901080000008302ff7f5a>
<DBRow 0070c9b28b000000000000000000f03f50dfb0e8010fca0141d0a64801000000010000009d969303909698000fc9c9010a0000008302ff7f5a>
<DBRow 8049016282000000000000000000f03f40a6e8b2240bca013434424801000000010000006f9793039096980010c9c901090000008302ff7f5a000000>
<DBRow f0902e7481000000000000000000f03fb0cfaa9f370fca017790ac480100000001000000639793039096980012c9c9010b0000008302ff7f5a000000>
<DBRow 00a8b10a87000000000000000000f03fe0277593410fca01bd61ad480200000001000000169093039096980015c9c9010e0000008302ff7f5a>
<DBRow 00a8b10a87000000000000000000004000cbd3a70010ca0134c5bf480200000001000000169093039096980015c9c9010e0000008302ff7f5a000000>
<DBRow 001882ba7d000000000000000000f03fb0c2eda85110ca01d44dc5480100000001000000149b930390969800f2c8c901000000008302ff7f5a>
<DBRow f01373d17a000000000000000000f03fe0ea9be87910ca014d90c9480100000001000000fe95930390969800e1c8c901000000008302ff7f1e000000>
<DBRow 50a3b10a87000000000000000000f03fb0dae8bfa410ca013d22cf480200000001000000169093039096980015c9c901000000008302ff7f5a000000>
<DBRow 50a3b10a870000000000000000000040b0d4aadeab10ca01a8d9cf4802000000010000002296930390969800f9c8c901000000008302ff7f5a000000>
<DBRecords>
(
<STuple>
(
<STuple>
(
<SStreamIdent 2>
)
<SDict>
(
<SStreamIdent 1>
<SMarker ID: 36 >
<STuple>
(
<SMarker ID: -117 >
<SMarker ID: -95 >
<SMarker ID: 74 >
<SMarker ID: -116 >
<SMarker ID: -118 >
<SMarker ID: -96 >
<SMarker ID: -119 >
<SMarker ID: 116 >
<SMarker ID: -125 >
<SMarker ID: 126 >
<SMarker ID: -101 >
<SMarker ID: -115 >
<SMarker ID: -106 >
<SMarker ID: 41 >
)
<SStreamIdent 3>
)
)
)
<DBRow 005c3e736a0000000000000000001440d062bb918efdc9016f29f7450500000001000000bb96930390969800d8c8c90100000000830228005a0001>
<DBRow 9cbe2f2000000000000000000000f03ff0eb1ad2fefdc9017c43fc460100000001000000c397930390969800d8c8c901000000008302ff7f5a000100>
<DBRow 0427f5536d000000000000000000f03fb0cbb36f250bca015048424801000000010000000196930390969800e0c8c90100000000830203005a000100>
<DBRow 9cd50e2000000000000000000000f03f7023fa4b21dac9010b816c4301000000010000003196930390969800fdc8c901000000008302ff7f5a000100>
<DBRow 00a82d336f000000000000000000004030551585dc0fca01f7a5314805000000010000003196930390969800fdc8c9010000000083020a005a0001>
<DBRow 4068fe3f6e0000000000000000001c4060feb17c410bca01a402a9470a000000010000002e9693039096980001c9c90100000000830214001e000100>
<DBRow 00ba95b96e000000000000000000f03f30366bc52f0eca01711940480100000001000000919693039096980011c9c90100000000830204005a0001>
<DBRow e0067fe0680000000000000000001040302ebdaeedf4c901a9d6f0440400000001000000609793039096980012c9c90100000000830205005a000100>
<DBRow c84cb55b6b000000000000000000f03f408f5a9adb02ca0121208a460100000001000000609793039096980012c9c90100000000830205005a000100>
<DBRow b085fde068000000000000000000f03ff056816f43fac9016eb19e4601000000010000006c9793039096980014c9c90100000000830205005a000100>
<DBRow 801bf4266c000000000000000000004090cd97f0cf05ca019d7ead470200000001000000169093039096980015c9c90100000000830203005a000100>
<DBRow 40fa51b96e000000000000000000004010707d136b0dca01d021a0460300000001000000e69593039096980023c9c90100000000830214005a000100>
<DBRow 001882ba7d000000000000000000f03f1026ef856e10ca01bb37c8480100000001000000fd93930390969800f3c8c901000000008302ffff5a0001>
<DBRow 00a446167200000000000000000000408024d1608e10ca01044fcc480200000001000000e293930390969800e0c8c901000000008302ffff070001>
)
<SLongLong '128934085416010580'>
)
)
So the magic 0x2A encoding in the cache files had me stumped for several days. I could see some of the data, but the packing was… odd (and no real price ever materialized out of it, turns out its a 64 bit integer, with lots of 0s ). The real problem was the zeros, or the lack of them. Turns out the packing algorithm appears to be optimized to simply not encode zero bytes, slightly reducing space. I believe that the first byte (and every “code” byte thereafter), which is split into two nibbles for this algorithm, contains a count of zeros and not zeros (bytes it simply copies to the output stream). This appears to be exclusive – either 0 is inserted or bytes are copied (per nibble), but not both.
I can’t fully verify this as I don’t have any “unpacked” in the same format, but by unpacking I can now locate more relevant data fields. Yay!
Progress is good. The parser doesn’t yet understand type 0x2A yet, but it will soon…
The parser library is a plain-vanilla C++ library, cross platform and embeddable in just about anything. I’m mainly focused on building an AST of the cache file format at this point. We can walk the AST and fetch the needed data later, which is the “easy part.” This is not a “find-magic-byte sequence and use fixed offsets” type parser
You can track progress at gitorious.org/libevecache.
Parse exception Can't identify type 0x2a at position 0x124
Beginning dump...
{Length: 3} <SInt '738197504'> <STuple>
( {Length: 4} <SMarker ID: -120 > <SIdent 'GetOrders'> <SInt '10000016'> <SInt '642'> )
<SDict>
( {Length: 1} <SObject>
( {Length: 2} <SIdent 'objectCaching.CachedMethodCallResult'> <STuple>
( {Length: 2} <SDict>
( {Length: 4} <STuple>
( {Length: 3} <SIdent '1 minute'> <NONE> <NONE> )
<SIdent 'versionCheck'> <SMarker ID: -114 > <SIdent 'sessionInfo'> )
<SSubstream>
( {Length: 3} <SLongLong '2678554370227830784'> <SString 'dbutil.RowList'> <SDict>
( {Length: 4} <SDBHeader>
( {Length: 1} <STuple>
( {Length: 2} <SString 'blue.DBRowDescriptor'> <STuple>
( {Length: 1} <STuple>
( {Length: e} <STuple>
( {Length: 2} <SMarker ID: -117 > <SInt '6'> )
<STuple>
( {Length: 2} <SMarker ID: -95 > <SInt '5'> )
<STuple>
( {Length: 2} <SMarker ID: 74 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -116 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -118 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -96 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -119 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: 116 > <SInt '11'> )
<STuple>
( {Length: 2} <SMarker ID: -125 > <SInt '64'> )
<STuple>
( {Length: 2} <SMarker ID: 126 > <SInt '2'> )
<STuple>
( {Length: 2} <SMarker ID: -101 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -115 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: -106 > <SInt '3'> )
<STuple>
( {Length: 2} <SMarker ID: 41 > <SInt '3'> )
)
)
)
)
<SMarker ID: 36 > <STuple>
( {Length: e} <SMarker ID: -117 > <SMarker ID: -95 > <SMarker ID: 74 > <SMarker ID: -116 > <SMarker ID: -118 > <SMarker ID: -96 > <SMarker ID: -119 > <SMarker ID: 116 > <SMarker ID: -125 > <SMarker ID: 126 > <SMarker ID: -101 > <SMarker ID: -115 > <SMarker ID: -106 > <SMarker ID: 41 > )
<SIdent 'columns'> )
)
)
)
)
The FSF has said no thanks to the .Net community promise. Not surprising in the least, and their points are all good. As I’ve said before, the promise is worth only the paper it is written on – it has no legal implications. Microsoft likes sinking their partners (see: Spyglass), and would likely continue to do so.
TwitterPowered by WordPress
