Emergency tuning: Batched e-mail sending
We’ve done some quick tuning to the SMTP push service to keep up with the much larger volume of traffic generated by the new uploader. You may notice that the e-mails now contain many types, concatenated together. It turns out that mailman is simply not ideal when distributing 100 e-mails a second 
For now, the worst case latency from upload to e-mail is 1 minute, with a fixed upper size per e-mail.
On another note: EVE-Metrics now uses AMQP to distribute uploads. We are reading the uploads from them and incorporating it into the EVE-Central stream and site.
Can you update the dev page with the new format info i.e. are the col headers repeated for each type, are types/regions separated by blank lines, etc. Also, what’s the mail format for an empty type i.e. no buy or sell orders in the region? Or does that information not get uploaded/mailed? Do mails exceeding the max size get truncated, or do full uploads just get delayed to the next mail?
Re: eve-metrics. Cool! So if I subscribe to the mail feed, I don’t need to subscribe to eve-metrics too, as you’ll relay the AMQP feed by email?
Any plans to move away from mail, or will the changes allow it to continue to scale reasonably?
Comment by Dal Rath — November 13, 2009 @ 9:27 am
eeek. it is now sending the string “CSV Header” as csv header.
no, no, not the csv header. literaly the string “CSV Header”.
on the pro side, the mails contain full issued timestamp (including the “time” part), whereas the daily batches have the time part clobbered to 00:00:00.000 …
Comment by Anonymous — November 16, 2009 @ 11:31 pm