Yann's Blog - Software and hardware

As you may (or may not) have noticed, the User Settings page on EVE-Cental.com now contains an option to set your full API key. Note that it requires the full API key in order to pull your market transaction logs. I am posting this announcement now as I’d like to collect as many API keys as possible in order to help test the system. If you’re willing to help, please log in to EVE-Central.com and set your API key.

The log data will not currently show up in any online view. I am working on polishing the framework for matching them to existing orders, and other similar fun.

I also have not published the full terms of use for the API logs. The market transaction logs available from the service will however never contain identifying information, such as your character name, IDs, etc. It will be a completely anonymous donation.

The source version of the EVE-Central uploader available from the main page has some issues working correctly on a Macintosh or Linux system do to bad path separators. Below is a link to grab a version of the uploader from the Git repository which contains a small patch to work around the issues. You’ll need an install of Python and wxPython (Unicode) appropriate for your system in order to use it.

EVE-Central uploader with Path fixes

Further up the Git tree is an unfinished UI refactoring effort (that is over a year old). If someone wants to take up the task of trying to complete the multiple-client uploader refactoring, they are more than welcome to.

I’m pleased to announce a new hardware board in development. It isn’t available for sale right now, but prototype testing is under way, and is being double checked for any design problems.

It is a “super” breakout board centered around the Nordic Semiconductor nRF24Z1 Wireless Audio Streamer, a low-power self-contained 2.4GHz digital audio streaming chip. This little chip is capable of sending a 44.1KHz stereo digital signal, straight from a S/PDIF source, to another S/PDIF input, over a 2.4GHz wireless link, with minimal latency. If analog is more your thing, it also features a complete I2S interface (including clocking) which can drive an ADC or DAC (depending if its in transmit or receive mode). Perfect for your DIY projects with outside speakers, wireless headphones, and more.

On this breakout board, you will find:

1. An Atmel ATtiny84 microcontroller, for command and control of the radio link, plus communication between the modules
2. An Octal-Inverter and all of the needed S/PDIF circuitry to correctly use and drive the 1V digital signal
3. A 20 pin breakout header which exposes most of the microcontroller and radio pins.
4. An integrated, inverted-F PCB antenna with superior performance to “chip” antennas, while not hogging large amounts of PCB space

The schematic hardware design is licensed under Creative Commons Attribution Share-Alike 3.0. I’ll post again soon with pictures of the prototypes as well as ordering information.

yyafl, a Python web form library operating on domain objects, now has version 0.2.1 available for download. This release includes numerous fixes for decorators, including the ability to attach multiple decorators to a form field or the wildcard ‘*’ field.

An example is given below, in the context of a CherryPy handler. This example is available in the download as

examples/layout02.py

import cherrypy
import yyafl
import yyafl.layout
from yyafl import fields
from yyafl.widgets import HiddenInput
 
# This example covers the user of custom decorators for required form fields
 
class Form1(yyafl.Form):
    name = fields.CharField(label = "User name", required = True)
    email = fields.CharField(label = "Your e-mail address", required = True)
    optional = fields.CharField(label = "Optional field", required = False)
    hidden = fields.CharField(widget = HiddenInput, default = "123")
    _layout = yyafl.layout.TableLayout()
 
class RequiredDecorator(yyafl.layout.NullDecorator):
    """ Draw a * next to all required fields """
    def extra_markup_widget(self, field, rendered_text):
        if field.required:
            return rendered_text + " * "
        return rendered_text
 
class FormTest(object):
 
    @cherrypy.expose
    def index(self, **kwargs):
        f = Form1(data = kwargs)
        # Add a global decorator
        f.layout().add_decorator('*', RequiredDecorator())
        content = ""
        content += "<html><body>"
 
        if f.is_valid() == False:
            content += "Errors: "
            for error in f.errors:
                content += error + " : " + f.errors[error] + "<br />"
        elif f.is_bound() and f.is_valid():
            content += "Thanks for the data!"
 
 
        content += "<p><form method=\"GET\" action=\"/\">"
 
        # Render using the layout specified in _layout above.
        content += f.render()
        # Or invoke the layout explicitly
        # l = yyafl.layout.TableLayout()
        # l.render(f)
 
        content += "<br /> <input type='submit' value='Submit' /></form>"
        return content
 
 
if __name__ == '__main__':
    cherrypy.quickstart(root = FormTest())

Have you ever wanted to use the UNIX shell utility tee, but instead of simply dumping to a file, invoke a series of subcommands that would run inside the pipeline? If so, I give you ptee, now available in version 1.0.

Edit: As people have pointed out, apparently you can do this in bash already (and this is specific to bash, lighter shells such as dash do not support it), using tee file redirects in subshells (tee >(subshell ) >(subshell ) ). Whoda-thunk. I guess its time for me to turn in my bash scripting license at the door. Either way, its here for your pleasure, if you’d like to use it with other shells beyond the much heavier weight bash

ptee extends the idea of tee by allowing not just filenames, but a whole shell expression (or command). For instance:

... | ptee "grep foo | zcat > outfile" | ...

The expression in quotes is passed to a new instance of the active shell (expressed by $SHELL), and the standard input to ptee is duplicated, both to normal standard output and to this new sub-shell (and any number of commands given to ptee). In effect, the sub-shell is running in parallel with the normal shell pipeline. The simplest case of running ptee is with no arguments, where it acts as an expensive version of the “|” operator. ptee accepts any realistic number of command line arguments, which are all launched and will receive a copy of the pipeline data.

There are of course some restrictions. The sub-shell standard output is thrown away; its simply not generically possible to try re-weaving it with the normal data flow. Also, ptee does not buffer large amounts of data internally, meaning the shell pipeline will run at the speed of the slowest sub-shell (if the slowest command is stalled, the whole pipeline will be stalled). ptee is however tolerant of sub-shells aborting early, and will continue passing data to any remaining outputs. Finally, sub-shell exit codes are ignored and not propagated to the main ptee exit code.

Now, there are likely to be bugs in this 1.0 release, but in some heavy processing tasks I’ve performed it has done admirably. ptee will compile on most POSIX like systems; I have tested both Linux (Ubuntu 8.04) and FreeBSD (7.1).

Feel free to grab the download here (available as a .tar.gz). The very simple C code is made available under an MIT license. It launches very quickly, and is very light weight, perfect for shell scripting.

I’d like to hear ideas you have for using ptee in your normal shell scripting activities. I know I am going to be cleaning up some ugly git post-receive hooks I have created in the past. How about you?

Version 0.2.0 of Yann’s Yet Another Form Library is now available for download. I have been using this code in product for at least a month and felt it is pretty stable. You can grab it from the main site at stackfoundry.com or from the PyPI page.

yyafl is a partial reimplementation and adaptation of the Django ‘newforms’ library allowing it to exist outside of the Django infrastructure. It can be used in other frameworks such as CherryPy and Pylons. YYAFL provides:

• Pluggable and skinnable forms, use either pure Python code or a compatible templating engine (such as Mako). Attach on the fly decorators (such as the ubiquitous red star to required form fields) or even apply whole layouts to forms.
• Default properties can be added to the form object – for example, a default rendering layout can be chosen.
• Comprehensive validation framework for many data types (which is easy to grow, similar in style to newforms)
• Comprehensive widget collection, which is easy to grow, tweak, or extend for your application’s needs. (currently doesn’t support multi-widgets well, this wart will be fixed soon however)
• Soon! Automatic CSRF managed forms – pass it a session store object and it can handle the rest by preventing cross site requests.

YYAFL uses some large chunks of the form-model and metaclass system that newforms used, and as such uses the same license as Django. Work is being done to refactor portions of this code to add some flexibility for multi-widget data fields (such as Date being split amongst several form fields), which were somewhat unclean in Django’s implementation.

I’ve updated the IRM demo at http://www.stackfoundry.com/irmdemo/demo to the latest Git version. Everything should be working, if not, let me know.

Just finished adding in a flexible layout system to yyafl, my reimplementation of Django newforms for other Python web frameworks or WSGI adapters. Now its simple to add default layouts or render a different layout on demand.

For example, to add a simple TableLayout() to a form:

class Form1(yyafl.Form):
    name = fields.CharField(label = "User name", required = True)
    email = fields.CharField(label = "Your e-mail address", required = True)
    hidden = fields.CharField(widget = HiddenInput, default = "123")
    _layout = yyafl.layout.TableLayout()

and to render the form to HTML:

# Render using the layout specified in _layout above.
content += f.render()
# Or invoke the layout explicitly
l = yyafl.layout.TableLayout()
content += l.render(f)

yyafl Layout()s also understand decorators:

decorators = { '*' : MarkRequiredDecorator(),
    'name_field' : HighlightDecorator() }
layout = yyafl.layout.TableLayout(decorators = decorators)
layout.render(f)

Decorators can change attributes and wrap fields in markup blocks. In the above example, the ‘*’ signifies the decorator should apply to all fields, and the HighlightDecorator should only apply to the field called ‘name_field’.

There are still some not yet implemented ideas:

• It doesn’t generated the <form> blocks yet, you need to roll your own.
• Field reordering and grouping isn’t supported by the default layout engines yet, though you could write your own.
• No AJAX form support (though its coming!)
• Automatic CSRF prevention forms.
• Error message printing in the form.

You can find these latest changes in the Git repository linked from the main yyafl web page.

With the popularity of Django or Rails, people have asked before why I use CherryPy. The reason is simple, I love the flexibility. I get to pick what I want to use inside, CherryPy simply worries about how to get the information to a web browser. I found this post to be a good overview of the strengths of CherryPy. Give it a good read. I completely agree with the author’s points, and wish CherryPy would get a bit more attention in the do everything web-framework era.

qsgen, the single-script static web-site generator has just hit the first release, version 0.1.

In short, qsgen is a pure Python functional wrapper around Mako templates and the Pygments library. It lets you build a hierarchical set of .html pages and associated base templates using Mako, which will be transformed into static pages for serving on the web. In addition, it also includes support for Pygments syntax highlighting in-line with the HTML or even with source code sourced from separate files.

I’m using qsgen on both www.stackfoundry.com and tomeapi.com. If you feel making a dynamicaly scripted website just in order to display some simple content is silly, or feel the complexity and headaches of a true CMS system outweigh the benefits, then qsgen is for you.

(qsgen is very similar to a similar script called Maehan which I was using for many years. Consider qsgen a more feature packed and cleaner version of the old Perl Maehan).