Yann’s Blog

Loyal EVE-Central.com users,

I just migrated the open source beta site to the main site. Along with the improvements comes a new caching framework which should make the API and site much more responsive during peak hours. If you see any anomalous behavior, please leave a comment so I can take a look at it.

Also, as a quick update on Contribtastic (another infamously delayed project), things are moving along. This performance improvement and code migration was required in order to better support the added load expected with the cache reading uploader. Soon you too will have a cache reading uploader. (Here we insert a witty picture along the lines of “Yes, you can’t have a pony”, but Google image search is letting me down).

Have a nice day or evening!

Our open-source beta-site has been getting some updates recently:

1. New alpha trade-finder, which can do route combining and removes some superfluous garabage which can’t be traded.
2. Streamlining some of the layout code to remove IGB dependence (to help us get a better design for the new in game browser
3. New upload methods to work with Contribtastic
4. Small cleanups across the board

There is still work pending on the continuously updated statistics engine (with caching), replacing our very aging market reports system. The exact implementation direction is not yet determined.

If you want to poke at the code, it is also available under the AGPL 3.0 license.

I took some time to learn a bit about computer vision, using OpenCV. The end goal here is to produce a Rock Band playing “robot”, using nothing but a specially modified controller (a AT90USB + bunch of FETs) and a computer with a video capture input.

Below is a video of my first attempt at using OpenCV. It applies several filters to isolate the keys on the racetrack approximately 2 frames ahead of the goal position. The primary heuristic right now is a threshold detect filter after all of the isolation steps. If you watch closely, you can see missed or over-pressed keys, especially when the whole racetrack lights up.

Note that the video output has an overlay color square which announces the software has detected a keypress. The actual gameplay is from another YouTube video – there is no closed loop control yet.

This will take some more tuning to iron out differences, possibly preserving some state from frame to frame. Right now color information is not used, but a separate classifier pipeline and some scoring mechanic might produce better results.

I will publish the git repository with the very-alpha implementation soon.

I have built a new binary of libevecache 0.1.2 for Windows. Included are several important fixes to the CSV export file.

Note that filename globbing (such as using *.cache) is NOT working under cmd.exe in Windows (PowerShell not tested), since Windows does not do filename globbing for commands. As a temporary workaround, use bash

My apologizes for the EVE-Central being down. Minor error on my part – the workers weren’t running under the correct user settings in terms of file handles, so they ran out. Everything should be kosher now.

Contribtastic work is moving in spurts. I’ll get something pushed out soon enough.

libevecache just gained two releases this weekend. Version 0.1 was a functional single shot cache file to .CSV file converter capable release, but had huge memory leak and pointer ownership issues, which made Contribtastic! very unstable and generally unusable.

After spending some quality time with Valgrind, all of the remaining issues are now resolved, and version 0.1.1 is available for mass consumption. Look for Contribtastic early betas soon. One new feature of libevecache in version 0.1.1 is the dumper is capable of digesting a whole CachedMethodCalls folder in one shot (just pass it the list of files on the command line). When you run dumper with the --market flag, you can safely redirect stdout to the file of your choice.

Success!

Thanks to some community contributions, libevecache has grown quite a bit this weekend!

New features include:

1. Shared object support (will give an object reference in place of the previous holder type)
2. DBRow decoding (needs more string identifiers for columns in other cache file formats)
3. The dumper utility now learned about --market, which lets it produce market CSV files from a cache file.

   atrus@atp (master) :~/Marketlogs/marketreader$ util/dumper --market tests/Lonetrek-Armageddon-2009.07.30\ 062647.cache > o
   atrus@atp (master) :~/Marketlogs/marketreader$ cat o
   Cache File Dumper
   File: tests/Lonetrek-Armageddon-2009.07.30 062647.cache
   price,volRemaining,typeID,range,orderID,volEntered,minVolume,bid,issued,duration,stationID,regionID,solarSystemID,jumps,
   54000000.0,1,643,32767,1185915256,5,1,False,2009-07-04 09:01:17.000,90,60004012,10000016,30001363,5
   54000000.0,5,643,32767,1187317153,8,1,False,2009-07-06 18:13:31.000,90,60002419,10000016,30001363,5
   55499999.0,1,643,32767,1217266843,1,1,False,2009-07-26 17:58:17.000,90,60000838,10000016,30001367,1
   ...
   52749999.0,1,643,32767,1221169229,1,1,False,2009-07-29 18:25:11.000,30,60003838,10000016,30001377,0
   57999999.88,1,643,32767,1221534269,2,1,False,2009-07-29 23:31:51.000,90,60002326,10000016,30001429,0
   57999999.88,2,643,32767,1221581224,2,1,False,2009-07-30 00:22:49.000,90,60003874,10000016,30001401,0
   45720000.0,5,643,40,1173825903,5,1,True,2009-07-05 16:35:12.000,90,60004027,10000016,30001368,0
   53999.99,1,643,32767,1190937468,1,1,True,2009-07-06 05:58:44.000,90,60004291,10000016,30001368,0
   46956001.5,1,643,3,1212303440,1,1,True,2009-07-22 23:37:54.000,90,60003841,10000016,30001376,0
   53784.31,1,643,32767,1131184395,1,1,True,2009-05-21 14:34:50.000,90,60003889,10000016,30001405,0
   

Update your clones today!

I am going to buckle down and produce an EVE-Central uploader which uses libevecache in the next week. Look for a beta version sometime by Friday.

I’ve done some polish work and integration of various libraries in Yann’s Eagle library repository.

Get it here..

Noteworthy inclusions:

• NAND and SDRAM footprints
• Merging of various MSP430 footprints into the “texas.lbr” library.
• More crystals, including subminature parts (ABM10)
• Cleanup of the Atmel library
• Some TVS components from Tyco and Vishay are included
• Freescale HCS08 footprints for the QB and QK series

libevecache, the open source library for reading EVE Online cache files, has received another update. I have decoded the types for Database types (including much scratching of heads regarding time stamps, I think I have a workable value now with a few hints on the side).

The usual comments came out in the EVE forum thread, but I think everything politically is well squared away now. Its not an “under the radar” project anymore.

Update your clones, and contribute something!

Success!

I give you the first half of a market order cache file, including RLE decompressed database rows. The structure is not 100% correct in regards to nested elements, and there is a bit of unresolved magic occurring at the substream end condition. Also, the naming of the elements is somewhat odd – the first set of orders is actually the Sell orders, and the second half comprises the Buy orders. The exact meaning of the header defining the structure is also not yet determined (suspect its an identifier followed by a numerical “type” code, though either the order differs from the row descriptor or the types have no relation to the serialized types in the cache file…).

The second half of the file can be parsed (except for the odd end condition again…), but the nesting structure is “broken”. This will require some more investigation, but is likely easy to fix.

Another oddity is the dictionary type (or double tuple – the length is half of the number of elements present). It is encoded value followed by key (not key followed by value).

Below is the AST structure dump from the ‘dumper’ utility included. The cache files list Armageddons in Lonetrek as of a couple days ago.

 <SInt '738197504'>
 <STuple>
 (
   <SMarker ID: -120 >
   <SIdent 'GetOrders'>
   <SInt '10000016'>
   <SInt '643'>
 )
 <SDict>
 (
   <SObject>
   (
     <SIdent 'objectCaching.CachedMethodCallResult'>
     <STuple>
     (
       <SDict>
       (
         <STuple>
         (
           <SIdent '1 minute'>
           <NONE>
           <NONE>
         )
         <SIdent 'versionCheck'>
         <SMarker ID: -114 >
         <SIdent 'sessionInfo'>
       )
       <SSubstream>
       (
         <SLongLong '2678554370227830784'>
         <SString 'dbutil.RowList'>
         <SDict>
         (
           <SDBHeader>
           (
             <STuple>
             (
               <SString 'blue.DBRowDescriptor'>
               <STuple>
               (
                 <STuple>
                 (
                   <STuple>
                   (
                     <SMarker ID: -117 >
                     <SInt '6'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -95 >
                     <SInt '5'>
                   )
                   <STuple>
                   (
                     <SMarker ID: 74 >
                     <SInt '2'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -116 >
                     <SInt '2'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -118 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -96 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -119 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: 116 >
                     <SInt '11'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -125 >
                     <SInt '64'>
                   )
                   <STuple>
                   (
                     <SMarker ID: 126 >
                     <SInt '2'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -101 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -115 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: -106 >
                     <SInt '3'>
                   )
                   <STuple>
                   (
                     <SMarker ID: 41 >
                     <SInt '3'>
                   )
                 )
               )
             )
           )
           <SMarker ID: 36 >
           <STuple>
           (
             <SMarker ID: -117 >
             <SMarker ID: -95 >
             <SMarker ID: 74 >
             <SMarker ID: -116 >
             <SMarker ID: -118 >
             <SMarker ID: -96 >
             <SMarker ID: -119 >
             <SMarker ID: 116 >
             <SMarker ID: -125 >
             <SMarker ID: 126 >
             <SMarker ID: -101 >
             <SMarker ID: -115 >
             <SMarker ID: -106 >
             <SMarker ID: 41 >
           )
           <SIdent 'columns'>
         )
         <DBRow 001882ba7d000000000000000000f03f509bcdfd85fcc90178a1af460500000001000000ac96930390969800d3c8c901050000008302ff7f5a>
         <DBRow 001882ba7d000000000000000000144050a0f97765fec901a105c54608000000010000007390930390969800d3c8c901050000008302ff7f5a>
         <DBRow f0c6933881000000000000000000f03f909da9a71a0eca019b048e480100000001000000468a930390969800d7c8c901010000008302ff7f5a000000>
         <DBRow 004ee77e7d000000000000000000f03f9067651cec00ca013b9b434701000000010000007192930390969800eec8c901070000008302ff7f5a000000>
         <DBRow 003476667b000000000000000000f03f300ef4eb3409ca0173a311480100000001000000c2bf930390969800eec8c901070000008302ff7f5a>
         <DBRow 001882ba7d000000000000000000f03f50b65d2fc40eca0143029f480100000001000000149b930390969800f2c8c901040000008302ff7f5a>
         <DBRow 001882ba7d000000000000000000f03f3024c3b6fc0eca01fe10a6480100000001000000149b930390969800f2c8c901040000008302ff7f5a>
         <DBRow 00e09962820000000000000000001440b0a9347fe3dec901ec889b3c0a00000001000000179b930390969800f3c8c901050000008302ff7f5a>
         <DBRow 00a8b10a870000000000000000000040e0883e7431e2c901039148440b000000010000000490930390969800f9c8c901070000008302ff7f5a>
         <DBRow 9ca7b10a8700000000000000000008404015505adcfdc9012b53f94606000000010000002296930390969800f9c8c901070000008302ff7f5a000000>
         <DBRow 8049016282000000000000000000f03f10c1e8bc0409ca01e9ad0e4801000000010000001996930390969800fbc8c901080000008302ff7f5a000000>
         <DBRow 00f94a8f7c0000000000000000001440c0a05632d10fca01fd20bb4805000000010000003796930390969800fdc8c901060000008302ff7f1e>
         <DBRow 809ddd657b000000000000000000084010aee06a2e10ca010c32c34803000000010000003796930390969800fdc8c901060000008302ff7f1e000000>
         <DBRow 7068e38f7c000000000000000000f03f20ffe8eced0aca012b523b4802000000010000003196930390969800fdc8c901060000008302ff7f5a000000>
         <DBRow 809ddd657b000000000000000000f03f1073931cab0fca01bda1354805000000010000003796930390969800fdc8c901060000008302ff7f1e000000>
         <DBRow 00267c907c000000000000000000f03fb0a3e34d050eca01c4d68a4801000000010000002e9693039096980001c9c901080000008302ff7f5a>
         <DBRow 0070c9b28b000000000000000000f03f50dfb0e8010fca0141d0a64801000000010000009d969303909698000fc9c9010a0000008302ff7f5a>
         <DBRow 8049016282000000000000000000f03f40a6e8b2240bca013434424801000000010000006f9793039096980010c9c901090000008302ff7f5a000000>
         <DBRow f0902e7481000000000000000000f03fb0cfaa9f370fca017790ac480100000001000000639793039096980012c9c9010b0000008302ff7f5a000000>
         <DBRow 00a8b10a87000000000000000000f03fe0277593410fca01bd61ad480200000001000000169093039096980015c9c9010e0000008302ff7f5a>
         <DBRow 00a8b10a87000000000000000000004000cbd3a70010ca0134c5bf480200000001000000169093039096980015c9c9010e0000008302ff7f5a000000>
         <DBRow 001882ba7d000000000000000000f03fb0c2eda85110ca01d44dc5480100000001000000149b930390969800f2c8c901000000008302ff7f5a>
         <DBRow f01373d17a000000000000000000f03fe0ea9be87910ca014d90c9480100000001000000fe95930390969800e1c8c901000000008302ff7f1e000000>
         <DBRow 50a3b10a87000000000000000000f03fb0dae8bfa410ca013d22cf480200000001000000169093039096980015c9c901000000008302ff7f5a000000>
         <DBRow 50a3b10a870000000000000000000040b0d4aadeab10ca01a8d9cf4802000000010000002296930390969800f9c8c901000000008302ff7f5a000000>
         <DBRecords>
         (
           <STuple>
           (
             <STuple>
             (
               <SStreamIdent 2>
             )
             <SDict>
             (
               <SStreamIdent 1>
               <SMarker ID: 36 >
               <STuple>
               (
                 <SMarker ID: -117 >
                 <SMarker ID: -95 >
                 <SMarker ID: 74 >
                 <SMarker ID: -116 >
                 <SMarker ID: -118 >
                 <SMarker ID: -96 >
                 <SMarker ID: -119 >
                 <SMarker ID: 116 >
                 <SMarker ID: -125 >
                 <SMarker ID: 126 >
                 <SMarker ID: -101 >
                 <SMarker ID: -115 >
                 <SMarker ID: -106 >
                 <SMarker ID: 41 >
               )
               <SStreamIdent 3>
             )
           )
         )
         <DBRow 005c3e736a0000000000000000001440d062bb918efdc9016f29f7450500000001000000bb96930390969800d8c8c90100000000830228005a0001>
         <DBRow 9cbe2f2000000000000000000000f03ff0eb1ad2fefdc9017c43fc460100000001000000c397930390969800d8c8c901000000008302ff7f5a000100>
         <DBRow 0427f5536d000000000000000000f03fb0cbb36f250bca015048424801000000010000000196930390969800e0c8c90100000000830203005a000100>
         <DBRow 9cd50e2000000000000000000000f03f7023fa4b21dac9010b816c4301000000010000003196930390969800fdc8c901000000008302ff7f5a000100>
         <DBRow 00a82d336f000000000000000000004030551585dc0fca01f7a5314805000000010000003196930390969800fdc8c9010000000083020a005a0001>
         <DBRow 4068fe3f6e0000000000000000001c4060feb17c410bca01a402a9470a000000010000002e9693039096980001c9c90100000000830214001e000100>
         <DBRow 00ba95b96e000000000000000000f03f30366bc52f0eca01711940480100000001000000919693039096980011c9c90100000000830204005a0001>
         <DBRow e0067fe0680000000000000000001040302ebdaeedf4c901a9d6f0440400000001000000609793039096980012c9c90100000000830205005a000100>
         <DBRow c84cb55b6b000000000000000000f03f408f5a9adb02ca0121208a460100000001000000609793039096980012c9c90100000000830205005a000100>
         <DBRow b085fde068000000000000000000f03ff056816f43fac9016eb19e4601000000010000006c9793039096980014c9c90100000000830205005a000100>
         <DBRow 801bf4266c000000000000000000004090cd97f0cf05ca019d7ead470200000001000000169093039096980015c9c90100000000830203005a000100>
         <DBRow 40fa51b96e000000000000000000004010707d136b0dca01d021a0460300000001000000e69593039096980023c9c90100000000830214005a000100>
         <DBRow 001882ba7d000000000000000000f03f1026ef856e10ca01bb37c8480100000001000000fd93930390969800f3c8c901000000008302ffff5a0001>
         <DBRow 00a446167200000000000000000000408024d1608e10ca01044fcc480200000001000000e293930390969800e0c8c901000000008302ffff070001>
       )
       <SLongLong '128934085416010580'>
     )
   )