Yann's Blog - Software and hardware

We’ve done some quick tuning to the SMTP push service to keep up with the much larger volume of traffic generated by the new uploader. You may notice that the e-mails now contain many types, concatenated together. It turns out that mailman is simply not ideal when distributing 100 e-mails a second

For now, the worst case latency from upload to e-mail is 1 minute, with a fixed upper size per e-mail.

On another note: EVE-Metrics now uses AMQP to distribute uploads. We are reading the uploads from them and incorporating it into the EVE-Central stream and site.

Looks like Yahoo and Hotmail/Live/Microsoft have synced up their e-mail blacklists. The server eve-central.com uses for outbound messages has been blacklisted by both of these entities. I am trying to get this resolved, but in the mean time, if you need a password reset, please e-mail us directly. We’ll use an alternate outbound mail method to get your password to you.

Edit: It looks like the group of IP addresses at my co-location facility has been listed in the SBL. http://www.spamhaus.org/sbl/sbl.lasso?query=SBL80992. Now I am also working with the hosting provider to help clean up this mess. Joy.

Edit 2: Things are in motion to help clean up this mess.

Edit 3: SBL entry closed up, we will see when Yahoo/Hotmail/etc pick this up.

The latest version of Contribtastic is now available. We’re up to alpha3 now.

Fixed in this release:

1. Crash when reading certain NPC standing cache files
2. Possibility of having two upload threads cruising at the same time, leading to duplicate uploads
3. No longer uploads old cache files on first startup or after reseting the configuration

Login functionality is still broken (“by design”).

The download site is still the same: http://dl.stackfoundry.org/contribtastic/. You can also file official bug reports at http://bugs.stackfoundry.org, but comments in the blog or forum are also welcome.

(tl;dr: New uploader with cache reading capability is available as an early alpha release: Download here )

When I was working on libevecache, I was imagining making a quick-turn uploader which used the functionality from the library allowing EVE-Central uploader users to no longer have to hit the magic button.

That was several months ago now. How time flies when everything else is demanding your attention: house stuff, hardware stuff, work stuff, other stuff. I wasn’t even distracted by video games or EVE-Online. My dream of a quick turn uploader clone obviously was just a dream, up until now.

I am releasing Contribtastic 2.0-alpha1, available on the download site as a Windows installer. Contribtastic is a bit different than originally intended (I am reusing the wxPython skeleton from the MarketUploader 1.3, not the pure C++ Qt framework), but this is all to get it out the door quickly: I spent more time downloading and installing the EVE client on my shiny new Windows 7 system than writing code or adapting the new framework to work with libevecache. Go Python go.

There are some known errata:

1. Contribtastic doesn’t always detect the cache folder path correctly
2. The user login functionality is basically totally broken. (Sorry, no credits for uploads yet)
3. Upload suggestions aren’t working
4. 0 item cache files will not be interpreted by the EVE-Central.com parser (a server side issue)
5. Memory usage has not been analyzed
6. Bonking upload now while already scanning for uploads can break things
7. Only tested on Windows 7

This is an alpha version, but please enjoy!

Edit: We should be back up – watch your spam folders though

Yahoo! mail is currently not being accepted from the EVE-Central.com and StackFoundry.com servers (38.113.114.184, 38.113.114.187). We are doing the correct customer support contact magic in order to re-enable delivery with Yahoo!, but cannot give definite timelines.

We would like to remind all EVE-Central.com mail feed users that you should NOT sign up webmail accounts or externally hosted accounts, including rebranded accounts (such as Yahoo or Google for your domain). The mail feed is for live processing on your server, not for batch downloading from another server. The daily CSV dumps are available for this reason.

The old IGB is no longer supported, and we are busily removing all references, hacky workarounds, and alternate styles in the code base which were brought about by its brain dead behavior. However, this means registering on the site will be problematic.

As a workaround, please visit http://eve-central.com/users/ directly to register. We will be implementing the new IGB access methods soon, so you can register from SiSi.

Loyal EVE-Central.com users,

I just migrated the open source beta site to the main site. Along with the improvements comes a new caching framework which should make the API and site much more responsive during peak hours. If you see any anomalous behavior, please leave a comment so I can take a look at it.

Also, as a quick update on Contribtastic (another infamously delayed project), things are moving along. This performance improvement and code migration was required in order to better support the added load expected with the cache reading uploader. Soon you too will have a cache reading uploader. (Here we insert a witty picture along the lines of “Yes, you can’t have a pony”, but Google image search is letting me down).

Have a nice day or evening!

So, I got a WikiReader. The first thing I would like to say is that this device is very very hard to open (and yes I did remove the two screws in the battery compartment ). I haven’t attempted a full disassembly yet as I actually like the main functionality, and didn’t want to completely obliterate the case just yet.

Here is the WikiReader, unpacked. I wasn’t expecting a nice box and manual (even the iPhone doesn’t come with that), but was very pleasantly surprised.

Apologizes for the very poor picture quality – these are simply iPhone snaps – I didn’t take the time to break out the 5D.

The two main complaints I have so far are the scrolling of text and the keyboard. The scrolling makes the text largely unreadable – there are some different display techniques here which could improve the display. The keyboard could also take a few cues from Apple, such as the magnified key presses, which would go a long way to improving usability. There are some other bugs in the contents (such as the lack of full UTF-8 support), but those are minor at this point.

Aside from those complaints, this device is actually quite slick. The performance is EXCELLENT. Random takes no more than a second to load the next article, search automatically filters based on your entry very quickly, and the history feature even remembers where you were in an article. The device is light and portable, and the whimsical asymmetric design is a nice touch and not over-done.

But enough with the quick mini review, what technical fun toys are there for you to play with?

Well, the WikiReader has a built in calculator – while powering up the device, hold the center (History) button.

There is also a factory test mode available – while powering up the device, hold either the Search or Random button:

As you likely (can’t) see in the image, the bulk of the functionality is a set of Forth applications, including the aforementioned calculator. They’re largely there for factory test, but at least one of them lets you draw pretty pictures.

Not drawn using the draw application, but another LCD test app – you don’t want to see my artwork

The WikiReader also contains a thermistor, which appears to be there for maintaining LCD contrast automatically:

Last but not least, there is a Console, which probably is exposed in the hidden programming/debug port available in the battery compartment (peel off the big white rectangular sticker – not the FCC/CE/Regulatory sticker).

Our open-source beta-site has been getting some updates recently:

1. New alpha trade-finder, which can do route combining and removes some superfluous garabage which can’t be traded.
2. Streamlining some of the layout code to remove IGB dependence (to help us get a better design for the new in game browser
3. New upload methods to work with Contribtastic
4. Small cleanups across the board

There is still work pending on the continuously updated statistics engine (with caching), replacing our very aging market reports system. The exact implementation direction is not yet determined.

If you want to poke at the code, it is also available under the AGPL 3.0 license.

I took some time to learn a bit about computer vision, using OpenCV. The end goal here is to produce a Rock Band playing “robot”, using nothing but a specially modified controller (a AT90USB + bunch of FETs) and a computer with a video capture input.

Below is a video of my first attempt at using OpenCV. It applies several filters to isolate the keys on the racetrack approximately 2 frames ahead of the goal position. The primary heuristic right now is a threshold detect filter after all of the isolation steps. If you watch closely, you can see missed or over-pressed keys, especially when the whole racetrack lights up.

Note that the video output has an overlay color square which announces the software has detected a keypress. The actual gameplay is from another YouTube video – there is no closed loop control yet.

This will take some more tuning to iron out differences, possibly preserving some state from frame to frame. Right now color information is not used, but a separate classifier pipeline and some scoring mechanic might produce better results.

I will publish the git repository with the very-alpha implementation soon.