Friday 31 December 2010

HTTP 'streaming' from Python generators

One of the more annoying things about HTTP is that it wants to send things in complete chunks: you ask for an object with a particular URL, and some point later you get that object. There isn't a lot you can do (at least from JavaScript) until the complete resource has loaded. For more fine grained control, it's a bit annoying.

Of course web sockets will solve all of this (maybe) once the spec has gone through the mill a few more times. But in the mean-time there is often an impedance mismatch between how we'd like to be able to do things on the server-side and how we are forced to do them because of the way HTTP works.

The following is an example of one way to manage splitting things up. It allows Python generators to be used on the server side and sends an update to the client on every yield, with the client doing long-polling to get the data. This shouldn't be confused with CherryPy's support for yield streaming single responses back to the server (which is discouraged) - the yield functionality is hijacked for other purposes if the method decorator is applied. Also note that this is only of any help for clients which can use AJAX to repeatedly poll the generator.

Example

Let's suppose that we want to generate a large (or infinite) volume of data and send it to a web client. It could be a long text document served line-by-line. But let's use the sequence of prime numbers (because that's good enough for aliens). We want to send it to the client, and have it processed as it arrives. The principle is to use a generator on the server side rather than a basic request function, but wrap that in something which translates the generator into a sequence of responses, each serving one chunk of the response.

Server implementation using CherryPy - note the json_yield decorator.

import cherrypy

class PrimeGen(object):
    @cherrypy.expose
    def index(self):
        return INDEX_HTML # see below

    @cherrypy.expose
    @json_yield   # see below
    def prime(self):
        # this isn't supposed to be efficient.
        probe = 1
        while True:
           for i in range(2, probe):
               if probe % i == 0:
                   break
           else:
               yield probe
           probe += 2

cherrypy.quickstart(PrimeGen)

The thing which turns this generator into something usable with long-polling is the following 'json_yield' decorator.

Because we might want more than one such generator on the server (not to mention generator instances from multiple clients), we need a key - passed in from the client - which associates a particular client with the generator instance. This isn't really handled in this example, see the source file download at the end of the post for that.

The major win is that the client doesn't have to store a 'next-index' or anything else. State is stored implicitly in the Python generator on the server side. Both client and server code should be simpler. Of course this goes against REST principles, where one of the fundamental tenets is that state should be stored only on the Client. But there is a place for everything.

import functools
import json

def json_yield(fn):
    # each application of this decorator has its own id
    json_yield._fn_id += 1

    # put it into the local scope so our internal function
    # can use it properly
    fn_id = json_yield._fn_id

    @functools.wraps(fn)
    def _(self, key, *o, **k):
        """
        key should be unique to a session.
        Multiple overlapping calls with the same 
        key should not happen (will result in
        ValueError: generator already executing)
        """

        # create generator if it hasn't already been
        if (fn_id,key) not in json_yield._gen_dict:
            new_gen = fn(self, *o, **k)
            json_yield._gen_dict[(fn_id,key)] = new_gen

        # get next result from generator
        try:
           # get, assuming there is more.
           gen = json_yield._gen_dict[(fn_id, key)]
           content = gen.next()
           # send it
           return json.dumps({'state': 'ready',
                              'content':content})
        except StopIteration:
           # remove the generator object
           del json_yield._gen_dict[(fn_id,key)]
           # signal we are finished.
           return json.dumps({'state': 'done',
                              'content': None})
    return _
# some function data...
json_yield._gen_dict = {}
json_yield._fn_id = 0

The HTML to go with this is basic long-polling, and separating out the state from the content. Here I'm using jQuery:

INDEX_HTML = """
<html>
<head>
<script src="http://code.jquery.com/jquery-1.4.4.min.js">
</script>
<script>
$(function() {

  function update() {
    $.getJSON('/prime?key=1', {}, function(data) {
      if (data.state != 'done') {
        $('#status').text(data.content);
        //// alternative for appending:
        // $('#status').append($('
'+data.content+'
')); setTimeout(update, 0); } }); } update(); }); </script> </head> <body> <div id="status"></div> </body> </html> """

Uses

The example above is contrived, but there are plenty of possibilities if the json_yield decorator is extended in a number of ways. Long running server side processes can send status information back to the client with minimal hassle. Client-side processing of large text documents can begin before they have finished downloading. One issue is that a chunk of the data should be semantically understandable. Using things like this on binary files or XML (where it is only valid once the root element is closed) won't have sensible results.

There are plenty of possibilities in extending this; the decorator could accumulate the content (rather than the client) and send the entire results up-to-now back, or (given finite memory) some portion of it using a length-limited deque. Additional meta-data (e.g. count of messages so far, or the session key) could be added to the JSON information sent to the client each poll.

Disclaimer: I've not done any research into how others do this, because coding is more fun than research. There are undoubtedly better ways of accomplishing similar goals. In particular there are issues with memory usage and timeouts which aren't handled with json_yield. Also note that the example is obviously silly - it would be much faster to compute the primes on the client.

Download Files

Thursday 16 December 2010

Compiling "Essential Mathematics for Games" sample code on Mac OSX 10.6 Snow Leopard

Before getting into a spot of games programming I thought I would buy a book which covered a few of the relevant topics. Wanting something a bit deeper and not likely to be outdated in the too-near future, I plumped for Van Verth and Bishop's Essential Mathematics for Games and Interactive Applications: A Programmer's Guide[Amazon Associates link]. Seems fairly good so far, mainly because it establishes why a certain approach is most applicable, rather than just telling you 'the' way to do things.

Anyway, the point of this blog post is to share some minor modifications required to the code included on the CD (which seems not available on-line) to get it building on Mac OS X 10.6 (Snow Leopard). From the documentation it seems the code was tested against OS X 10.4 and 10.5. It fails to build out-of-the-CD on 10.6, and I didn't find any updates on the website (www.essentialmath.com), but perhaps in time that will get updated.

Required changes are:

  1. in common/MakefileCommon, change lines 15-17 from:
    ifeq ($(PLATFORM), OSX)
      CFLAGS_EXT = -fvisibility-inlines-hidden
    endif
    
    to
    ifeq ($(PLATFORM), OSX)
      CFLAGS_EXT = -fvisibility-inlines-hidden -ffriend-injection
    endif
    
  2. change the equivalent line 12 CFLAGS_EXT setting in Examples/MakefileExamples in the same way, i.e. adding -ffriend-injection. Presumably this is a requirement of a more recent version of GCC in Snow Leopard.
  3. In common/Graphics/OGL/IvRendererOGL.cpp line 204, remove the GLvoid in the InitGL definition, i.e. change from
    int 
    IvRendererOGL::InitGL(GLvoid)
    
    to
    int 
    IvRendererOGL::InitGL()
    

To build it (this is all in the relevant README files, as is the basic requirement of having the Mac Developer tools - i.e. XCode - installed):

cd <root of directory structure>
chmod -R +w *  # only required once, files/directories copied from CD are read-only
pushd common
make PLATFORM=OSX
popd
pushd Examples
make PLATFORM=OSX
popd
And then the example executables are available under the relevant chapter/section directory as ./Example, e.g. Examples/Ch13-Simulation/Simulation-02-Integration/Example. Note some of the examples seem to segfault if not started from the working directory...

Anyway, hope this helps someone. Now, on to developing some games...!

Thursday 25 November 2010

pylibftdi updated to 0.5

I've done some tidying up of pylibftdi, fixing a few bugs, refactored the pylibftdi package to contain several modules instead of everything dumped in a __init__.py file, and generally made it a bit cleaner. It even has docstrings for most things now, and a test or two!

pylibftdi is a simple interface to libftdi, which in turn allows accessing FTDI's range of USB parallel and serial chips. See here for more details, but very briefly, serial access is provided by a file-like (read/write) interface with a baudrate property, and parallel access is provided by a pair of properties - direction (data direction register) and port (data IO register).

I haven't yet added any more examples, hopefully I'll get round to that in the next week or so. I have been using it as a MIDI interface though, which is fun - I'll get an appropriate example of that out in the next version, together with some diagrams / photos etc.

Tuesday 9 November 2010

BlogPress, iPod, website

So this is the obligatory 'testing' post from BlogPress on my new-ish iPod touch 4th gen.

I wish the battery life was better, but the retina display is awesome for reading tiny text, making ebooks surprisingly practical. It's also far too easy to spend lots of money on the app store. One nice thing is that three fingered touch typing is just about doable in landscape mode - not a skill I anticipated learning!

I've been spending lots of time recently on my website codedstructure.net, and especially on a project for playing with / learning GLSL in the context of WebGL: WebGL Shader Lab, which I'll write more about next time.

If only iOS Safari supported WebGL...

- Posted using BlogPress from my iPod

Monday 4 October 2010

dsPIC33 (and PIC18) programming on my EEEPC at last

I've got an EEEPC 701, and although it is annoying in some ways (small screen, limited SSD space), it's great for portability. One of my goals when I got it was to be able to use it to help developing various microcontroller projects I've got on the go, and I especially like that it doesn't take up lots of space on my (tiny) desk when I'm hacking around. It works great with Arduino, but I prefer PICs to be honest. The issue with this is that Microchip (producers of PIC microcontrollers) want everyone to use MPLAB, which only works on Windows. And even if it worked on my Linux-based EEEPC, it probably wouldn't be too great on a 7 inch screen. Anyway, being born before 1980 (but not by much!) I still prefer command lines to IDEs anyway. My first development environment was debug.com on MS-DOS, and it's never been bettered :-)

At some point I'll get round to changing the OS (Linux Mint looks like the front runner at the mo...), but for now it's still got the Xandros stock install (though I've removed unionfs).

Rather than mess around with getting the (GCC based) Microchip tools compiled on the machine, I'm using wine - both C18 and C30 install and work without any problems.

For actually burning the image onto the controller, I use and the perfectly-working-without-lots-of-hassle pk2cmd and the brilliant PicKit2, (which was so successful Microchip went and broke it).

The following gives the commands I use to build and download the target .hex file - I've only got a single .c file (this one) for input at the moment, and haven't even got a makefile together yet. But getting this working took a couple of hours, so this is as much for reference as anything else...

#!/bin/bash

# exit on errors
set -e

C30_BASE=/home/user/.wine/drive_c/Program\ Files/Microchip/MPLAB\ C30/

echo "Building..."
wine "${C30_BASE}/bin/bin/pic30-coff-gcc.exe" -o dac_music.coff -mcpu=33fj64gp802 -Wl,--script "${C30_BASE}/support/dsPIC33F/gld/p33FJ64GP802.gld" DacMusic.c

echo "bin2hex..."
wine "${C30_BASE}/bin/bin/pic30-coff-bin2hex.exe" dac_music.coff

echo "burning..."
pk2cmd -Pdspic33fj64GP802 -Fdac_music.hex -Q -M -R -T
For reference, I've also got a similar setup for something on the PIC18. I've had this running from fairly soon after I got my EEEPC, and didn't have problems getting it up and running:
#!/bin/bash

# exit on errors
set -e

C18_BASE=/home/user/.wine/drive_c/MCC18

echo "Compiling..."
wine ${C18_BASE}/bin/mcc18-traditional.exe -ml -p=18f252 -k -Oi+ music.c

echo "Linking..."
wine ${C18_BASE}/bin/mplink.exe \\MCC18\\lkr\\18f252i.lkr /l\\MCC18\\lib /aINHX32 music.o

echo "burning..."
pk2cmd -P18f252 -Fa.hex -M -R -T

At some point I'll turn them into makefiles, but for now - I can program tiny microcontrollers with my almost-as-tiny EEEPC. Which is nice and cosy.



Some time soon I'll actually write a blog post or two about the projects I'm using this with - mostly synthesizers and other various things to do with MIDI.

Sunday 26 September 2010

Non-evil SEO?

I've always thought SEO was evil (in the non-theological sense!), but this effort (via @rem) seems well worth supporting. According to my Facebook page my favourite quote is 'It is amazing what you can accomplish if you do not care who gets the credit' - so said Harry Truman. Perhaps amazing is overstating the goal, but maybe it could be paraphrased 'It is amazing what you can make Google do if it is actually helpful and not (directly) self-serving'. I guess we'll find out...

JavaScript JS Documentation: JS Function apply, JavaScript Function apply, JS Function .apply, JavaScript Function .apply

Sunday 19 September 2010

Announcing pylibftdi - a minimal Pythonic wrapper for libftdi

[edit 2013-11-25: note that recent versions of pylibftdi have deprecated and then removed the ability to use 'Driver' in this way; replace Driver and BitBangDriver with Device and BitBangDevice in the code below]

The playing-around I've done with FTDI devices seemed like a good opportunity to actually release something as open source, and so I present 'pylibftdi'. Undoubtedly not the greatest, but right now most likely the latest FTDI-Python bridge in the rather large open source field. There are a few features I know I want to add to it (primarily support for multiple devices), but for a flavour of things:

Toggling an LED or other device from pin D0

from pylibftdi import BitBangDriver
import time

with BitBangDriver() as bb:
    while True:
        bb.port ^= 1
        time.sleep(0.5)

Sending a string to a serial terminal

from pylibftdi import Driver

with Driver() as drv:
    drv.baudrate = 19200
    drv.write('Hello World!')

It's been tested on Linux (Ubuntu 10.10) and Mac OS X (10.6), with libftdi 0.17 and 0.18, but doesn't have any intended platform specific requirements other than having the libftdi library installed. The following goals for this project differentiate it from similar projects:
  • Fully supported on Linux and Mac OS X, using Intra2net's open source libftdi driver.
  • Simple things are very simple, as the example above shows. The basic functionality is made as simple as possible to use, with properties and context managers used where it makes sense. Most other FTDI Python wrappers are 'just' low level bindings of the underlying API.
  • There will be an increasing library of examples showing interaction with various protocols and devices - note this is a goal, not yet an accomplishment, though there is an LCD example there.
pylibftdi is easy_installable (or equivalent) from PyPI, and the code is developed on bitbucket, where you can also raise tickets for problems and feature requests.

For reference, other similar projects include PyUSB, pyftdi, python-ftdi, and ftd2xx. There are probably others...

Monday 30 August 2010

libftdi on Mac OS X

Update 2011/02/02: In trying to port pylibftdi to Python3, I found that the libraries which get built following the instructions below are 64-bit. All well and good, but the Python3.1 installation on Mac OS X 10.6 is 32-bit only (unlike the Python2.6 install).

While it's possible to get both 32-bit and 64-bit universal libraries, I haven't tried that yet. The solution to built 32-bit libraries was to use the following:

CFLAGS="-arch i386" ./configure && make && sudo make install

(Note I omitted libusb and sudo (for make install) in previous instructions - I've updated below). I also found I needed to do:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/

prior to building libusb-compat - I can't remember if this was a problem or not the first time round. Anyway, hopefully I'll have pylibftdi working with Python3 soon.
End of Update

I've not written much about my ftdi projects recently, so here's an update. I've played around with reading values from an encoder and outputting to a RGB LED with PWM, but both of these require a fairly high constant poll rate (or in the case of reading the encoder, ideally interrupt-on-change). The jitter is quite annoying, and decent PWM seems tricky, even when following FTDI's instructions (pdf link) for maxmising the buffer size. There's always going to be a gap when one buffer has finished and the next starts.

On a different note, I'm now mostly using my new Mac mini, so I've spent a few moments getting things to work there. Prerequisites:

These should be installed in the order listed; for each file, download it, untar it (tar -xf filename), then run './configure && make && sudo make install' in the new directory which tar has created.

On the Python side, there were a couple of issues when I tried to run the old code. First was that the library could not be found at all. This was due to the library extension (.so) being hardcoded. For a cross-platform solution, it seems the best way is to use the following:

from ctypes import *
from ctypes.util import find_library

dll = CDLL(find_library('ftdi'))
Rather than specifying 'libftdi.so', we have stripped off the 'lib' prefix and the .so extension. The simple 'ftdi' string would be used if we were linking this library with gcc using the -l flag. Try 'gcc -lftdi' and it should report only that it can't find _main, not that the library is missing (try gcc -lmadeupname and it should complain about being unable to find 'madeupname')

Once this was done, the program would some times run (especially under pdb, which lead me up the garden path of thinking it was a timing issue...), but other times would cause a segfault. This was tracked down to another hard-coded value - the size of the context structure. The following will report the size of the ftdi_context structure:

echo -e '#include \nint main(){return sizeof(struct ftdi_context);}' | gcc -xc - && (./a.out; echo $?)
This is somewhat fragile, as it will fail if the structure ever gets larger than 255 bytes, but this seems unlikely for the time being. On this Mac this gives 112 bytes; on my Linux boxes it gives 84 - though they are running on the previous (0.17) version of libftdi. There is also the issue that the Mac library is x86_64 (i.e. 64-bit), while the Linux libraries are 32-bit.

One solution, though not exactly a purist's, is to allocate far more memory than will be needed. It won't slow anything down, as only a pointer to the start of the block is passed around, and probably won't make a difference to the memory consumption as applications will always use whole numbers of pages (4KB minimum) anyway. So for now, an allocation of 1KB seems a good solution.

The result of this is that the changes needed to the earlier code for compatibility with both Mac OS X and Linux are as follows:

...

from ctypes.util import find_library

def ftdi_start():
    global ctx, fdll  # frown... :P
    fdll = CDLL(find_library('libftdi'))
    # size of ftdi_context varies
    # seen 112 (x86_64, v0.18). 84 (i386, v0.17)
    # over-allocate to 1KB.
    ctx = create_string_buffer(1024)
    fdll.ftdi_init(byref(ctx))
    fdll.ftdi_usb_open(byref(ctx), 0x0403, 0x6001)
    fdll.ftdi_set_bitmode(byref(ctx), 0xFF, 0x01)

...

Me loves os.name == 'posix' :-)

Tuesday 17 August 2010

New website - codedstructure.net

I've not got round to putting much on here for a while, but I have been busy, honest! I've finally got my own website after far too many years of not bothering: codedstructure.net.

I'm probably not the first to use scrolling like that, but I wasn't copying anyone and I think it's quite funky, especially the nice text shadows. Me likes the way web development is going, after years of shunning it...

Oh, and while I'm at it, I've got a twitter account, and hope to be churning out iPhone apps in the near future - though Objective C looks horribly like some terrible collision of Java and Smalltalk - in a bad way. Hope I can grow to like it.

Wednesday 16 June 2010

A Python LCD status display

This is the third in my series of blog posts about doing IO with FTDI's Async BitBang mode on the UM245R / UM232R eval boards, and this time we're actually going to do something useful - have an LCD update with interesting system information. The LCD in question is based on the ubiquitous HD44780 controller, which is interfaced to microcontrollers throughout the world...

Anyway, with only 8 IO lines on the UM2xxR boards, we need to use the 4bit interface mode (two extra IO lines are needed beyond the 'data' path - one to act as a 'data-ready' strobe, and the other to select between 'data' and 'commands'). The wiring up is basically DB0-DB3 on the FTDI device going to D4-D7 on the LCD, with DB6 on the FTDI going to the 'RS' (register select) line on the LCD, and DB7 to the 'E' strobe. If that makes no sense, then hopefully the photo makes things slightly clearer...

I've also got an LED backlight for my display, which makes it look a whole lot cooler :-)

I'll present the code in chunks and try to explain it as I go along. First the initialisation and shutdown code, which are fundamentally unchanged from the previous examples, though they are now in functions to make them just a little tidier. (Note there is still no error checking here...)
"""
Write a string (argv[1] if run from command line) to a HD44780
LCD module connected via a FTDI UM232R/245R module

example usage:

# while true;
>   do python lcd.py $( awk '{print $1}' /proc/loadavg);
>   sleep 5;
> done
"""

from ctypes import *
import time, sys

def ftdi_start():
    global ctx, fdll  # frown... :P
    fdll = CDLL('libftdi.so')
    ctx = create_string_buffer(84)
    fdll.ftdi_init(byref(ctx))
    fdll.ftdi_usb_open(byref(ctx), 0x0403, 0x6001)
    fdll.ftdi_set_bitmode(byref(ctx), 0xFF, 0x01)

def ftdi_end():
    fdll.ftdi_usb_close(byref(ctx))
    fdll.ftdi_deinit(byref(ctx))
The following class is an abstraction of a bus - a collection of one (probably two, technically...) or more electrical lines which should be treated as a single unit. The aim here is to be able to program in a similar style to the embedded programming on a microcontroller, where registers are typically memory mapped and writing to a bus is simply writing into a bitfield. The parameters of this abstraction are the width of the bus (in bits), and the offset from the LSB of the entire port being accessed. It also needs a reference to a driver which allows it to do the reading and writing to the port. By using this as a descriptor, we can define buses within classes representing the various devices we are using; in this case the LCD.
class Bus(object):
    """
    This class is a descriptor for a bus of a given width starting
    at a given offset (0 = LSB).  It needs a driver which does the
    actual reading and writing - see FtdiDriver below
    """
    def __init__(self, driver, offset, width=1):
        self.offset = offset
        self.width = width
        self._mask = ((1<<width)-1)
        self.driver = driver

    def __get__(self):
        val = self.driver.read()
        return (val >> offset) & self._mask

    def __set__(self, obj, value):
        value = value & self._mask
        # in a multi-threaded environment, would
        # want to ensure following was locked, eg
        # by acquiring a driver lock
        val = self.driver.latch()
        val &= ~(self._mask << self.offset)
        val |= value << self.offset
        self.driver.write(val)
The following is the driver which will be used to do the actual data access. Note the use of the latch to store the last value written to the port, which cannot generally be read from the device after having been written (Latch registers for the IO ports was a big advance for the PIC18F series over the earlier 16F series, which needed the application to store this separately in order to do read-modify-write operations properly on the IO ports).
class FtdiDriver(object):
    def __init__(self):
        self._latch = 0

    def read(self):
        z = c_byte()
        fdll.ftdi_read(byref(ctx), byref(z), 1)
        return z.value

    def latch(self):
        return self._latch

    def write(self, val):
        self._latch = val
        z = c_byte(val)
        fdll.ftdi_write_data(byref(ctx), byref(z), 1)
        # the following is a hack specifically to allow
        # me to ignore all the timing constraints of the
        # LCD.  For more advanced LCD usage, this wouldn't
        # be acceptable...
        time.sleep(0.005)
Now we've got a Bus class and a driver to use with it, we can define the LCD module interface. I'm not going to cover the details of the interface, but the wikipedia HD44780 page has some pointers.
# need to instantiate this in global context so LCD
# class can be defined. Could tidy this up...
ftdi_driver = FtdiDriver()

class LCD(object):
    """
    The UM232R/245R is wired to the LCD as follows:
       DB0..3 to LCD D4..D7 (pin 11..pin 14)
       DB6 to LCD 'RS' (pin 4)
       DB7 to LCD 'E' (pin 6)
    """
    data = Bus(ftdi_driver, 0, 4)
    rs = Bus(ftdi_driver, 6)
    e = Bus(ftdi_driver, 7)

    def init_four_bit(self):
        """
        set the LCD's 4 bit mode, since we only have
        8 data lines and need at least 2 to strobe
        data into the module and select between data
        and commands.
        """
        self.rs = 0
        self.data = 3
        self.e = 1; self.e = 0
        self.e = 1; self.e = 0
        self.e = 1; self.e = 0
        self.data = 2
        self.e = 1; self.e = 0

    def _write_raw(self, rs, x):
        # rs determines whether this is a command
        # or a data byte. Write the data as two
        # nibbles. Ahhh... nibbles. QBasic anyone?
        self.rs = rs
        self.data = x >> 4
        self.e = 1; self.e = 0
        self.data = x & 0x0F
        self.e = 1; self.e = 0

    def write_cmd(self, x):
        self._write_raw(0, x)

    def write_data(self, x):
        self._write_raw(1, x)
All that remains is to initialise the FTDI device, initialise the LCD module, and write some data to it.
def display(string):
    ftdi_start()

    lcd = LCD()
    lcd.init_four_bit()

    # 001xxxxx - function set
    lcd.write_cmd(0x20)
    # 00000001 - clear display
    lcd.write_cmd(0x01)
    # 000001xx - entry mode set
    # bit 1: inc(1)/dec(0)
    # bit 0: shift display
    lcd.write_cmd(0x06)
    # 00001xxx - display config
    # bit 2: display on
    # bit 1: display cursor
    # bit 0: blinking cursor
    lcd.write_cmd(0x0C)

    for i in string:
        lcd.write_data(ord(i))

    ftdi_end()


if __name__ == '__main__':
    # note blatant lack of error checking...
    display(sys.argv[1])
and there we have it; a slightly cumbersome but also slightly cool and slightly useful little display utility. In the spirit of UNIX programming, this only does one thing - display the command line argument on the LCD. Obviously it needs major error handling if robustness is required...
while true; do python lcd.py $( awk '{print $1}' /proc/loadavg); sleep 5; done


Tuesday 8 June 2010

Mandelbrot Canvas...

This Mandelbrot Plotter is something I worked on a little while ago. One of the things I want to do in any language I'm learning is know how to get a pixel on a page. A long time ago it was function 0Ch of INT 10h, then memory mapped displays, and now the wonderful ImageData object underlying HTML5's Canvas element. It's so much more fun displaying a picture than 'Hello World' in text...

I'm also fascinated by fractals, and the Mandelbrot set in particular (since I sort-of understand the maths behind it). It makes me wonder whether it is an invention or a discovery, and just why it is what it is - and whether other inscrutable patterns are just waiting to have some new visual representation cast a whole new light on them...

Of course with it being hosted (albeit slightly oddly on Blogger), I can now claim to have written a 'web app', for better or worse. Anyway, it's nice to be able to put random JavaScript up within Blogger...

Thursday 27 May 2010

Let there be LEDs

"In the beginning, [...] God said 'let there be light'".

Some time later, Man discovered Gallium Arsenide, and eye-piercing LEDs have blighted us ever-since.  (Although they used to be quite dim, actually).  Nevertheless, the 'hello world' of the embedded world is to make an LED flash.

In part 1 I 'covered' installation and test of the libftdi module, but the program didn't actually do anything.  The UM232R module is primarily a UART module (unlike the DLP USB245M, which is a FIFO), but as with most recent FTDI devices it supports a 'BitBang' mode.  BitBang is embedded jargon for running a protocol at the digital bit level, specifying in firmware at what point each of the (potentially many) IO lines goes high or low.

To put these devices in to this mode, an API call is needed to the FTDI driver to establish this mode.  The ftdi_set_bitmode function takes three parameters; the FTDI control context (by reference), a direction bitmap, and a mode indicator (which should be 1 for async bitbang mode).  The interesting is the direction bitmap.  For each of the 8 IO lines, the corresponding bit in this mask can be a '1' to indicate an output, or a '0' for an input.  I'm starting with output, so...

if ((ret = ftdi_set_bitmode(&ftdic, 0xFF, 0x01)) < 0) {
        fprintf(stderr, "unable to set bitmode\n");
        return EXIT_FAILURE;
    }
(this fits into the 'DO STUFF HERE' part of the code from last time)

Once this is done, we can start writing things out to the data port.
unsigned char x = 0;
    while (1) {
        ftdi_write_data(&ftdic, &x, 1);
        x ^= 0xFF;      /* toggle all lines */
        usleep(100000)  /* pause for 100ms  */
    }

This is hackish - it never exits for a start - but it should enable an LED to flash!

In terms of wiring things up, the 'GND' pin needs to go to the shorter leg of the LED (the cathode), and a data line (it doesn't matter which in this example, use DB0 to be safe) goes to the longer lead (anode).  A 1K resistor needs to be in either line to limit the current and prevent burning out the LED. As supplied, the jumpers on the UM232R/UM245R module configure it to be self powered. Anyway, it should look something like this, and the LED should be flashing at approx 5Hz.

To do anything more interesting, let's move to Python.  This is where the fun begins - with ctypes. I remember the first time I used ctypes, using puts from the C standard library on a Solaris machine, and the fact that it just worked. Occasionally there is a need to write the equivalent of a function prototype, but most of the time things are incredibly easy.

First up: loading the shared library. Lets guess its name - libftdi.so?

[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-20)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from ctypes import *
>>>
>>> fdll = CDLL('libftdi.so')
>>>
>>> fdll
<CDLL 'libftdi.so', handle 81da028 at b7cfd4ac>
>>>

So far so good :) Next thing is this ftdi_context struct which is referenced by each function in the C code. It starts as an unpopulated structure, and presumably the first functions (init and usb_open) set it up as required.  While the D2XX interface just gives an integer handle, in libftdi we have to pass pointers to this structure around.  So, how do we define this in ctypes?
First we need its size:

#include "ftdi.h"
#include "stdio.h"
int main()
{
    printf ("%d\n", sizeof(struct ftdi_context));
}

Now compile and run...

/home/user/ftdi> gcc get_size.c
/home/user/ftdi> ./a.out
84

So, we need 84 bytes of storage for this context. ctypes will do that for us:

>>> ftdic = create_string_buffer(84) 

Done that. Now we can rattle it off: only two other things: first, where in C you give &var to give the address of a variable (e.g. as a parameter to a function taking *var), in Python with ctypes, it's byref(var). Second, to create a something which has existence in the C world, (e.g. so can have its address taken), there are a whole bunch of c_XXX functions - we'll use c_byte, which is equivalent to an unsigned char.

from ctypes import *

fdll = CDLL('libftdi.so')

ftdic = create_string_buffer(84)

fdll.ftdi_init(byref(ftdic))

fdll.ftdi_usb_open(byref(ftdic), 0x0403, 0x6001)

fdll.ftdi_set_bitmode(byref(ftdic), 0xFF, 0x01)

import time
x = c_byte(0)
try:
    while True:
        fdll.ftdi_write_data(byref(ftdic), byref(x), 1)
        time.sleep(0.1)
        x.value ^= 0xFF # toggle all bits
except KeyboardInterrupt:
    pass

fdll.ftdi_usb_close(byref(ftdic))

fdll.ftdi_deinit(byref(ftdic))

And we're done! It's surprising how easy it is to translate basic C code using libraries into Python using ctypes - normally it just works.

Next time we'll look at interfacing to an LCD module...

Thursday 20 May 2010

Should I teach my children machine code?

I waited till primary school to read this book, but it seems that 'modern' kids can understand these things sooner - or at least Amazon thinks so...

Tuesday 18 May 2010

The outside world...

I'm going to do a series of blog posts on using FTDI devices to access the outside world. There are probably a dozen other similar series out there, so I hope I can introduce enough novelty to make it interesting

I've been interested in low-level programming for as long as I've been programming (anyone else remember this book? - yes, a "children's" book on Machine Code...)

In terms of 'physical computing', things like Arduino are really taking off at the moment, but I'm going to take a step back to the simplicity of simple digital IO based on the BitBang mode of FTDI's latest devices.  There are two reasons for this: firstly it came seem laborious writing two sets of software (for both the host computer and the target micro-controller), and secondly that even if the eventual application is going to be a standalone micro based system, it is still generally quicker to prototype things using only the host computer, avoiding the cross-compile and firmware upload cycle.

Hardware-wise, I'm using a FTDI UM232R (Farnell link) device.  (I've also used one of these, and this either will be usable) This is a DIL module which plugs nicely into a breadboard which can be used to interface with stuff.  If you want to follow along, get a breadboard to plug it into, some LEDs, 1Kohm resistors, and some connecting wire (I like these, but they are waaay over priced).  In a couple of posts I'll be using one of these (N26AZ), too...

On the software side, I'm using Linux on a EEEPC 701 (stock Xandros) with the libftdi drivers, compiled from source found here.  The FTDI supplied drivers are similar and might be a better choice on Windows (not sure if libftdi works on Windows), but some of the function names and interfaces differ somewhat.

First step is getting the simple.c program compiled and working.  The program outline - slightly edited for length - looks something like this.

/* see http://www.intra2net.com/en/developer/libftdi/documentation/ */ 
#include "stdio.h"
#include "ftdi.h"

int main(void)
{
    int ret;
    struct ftdi_context ftdic;
    if (ftdi_init(&ftdic) < 0) {
        return EXIT_FAILURE;
    }

    if ((ret = ftdi_usb_open(&ftdic, 0x0403, 0x6001)) < 0) {
        return EXIT_FAILURE;
    }

    /* DO STUFF HERE */

    if ((ret = ftdi_usb_close(&ftdic)) < 0) {
        return EXIT_FAILURE;
    }

    ftdi_deinit(&ftdic);

    return EXIT_SUCCESS;
}
Compiling this (gcc -o simple simple.c -lftdi) and running the resulting executable should not cause any errors, and should return a successful error code if a FTDI device is attached.

This post is getting long enough for now, so I'll leave it at that. Next time will be configuring bitbang mode, where we can use the device as a configurable 8-bit IO port, blinking lights, and the wonders of Python's ctypes module...

Monday 18 January 2010

The keyboard and the pen

When I was a teenager, I discovered something amazing. A pen that made writing fun, desirable, and (almost) exciting. A Pilot V5. They never lasted long, but maybe that was because I'd actually write stuff with them. Time moved on, and I grew a penchant for nice Parker pens (Frontier rollerball please...), except that I'd get one for Christmas and have lost it by my birthday only a month later, and need another one. Anyways... Moving on more than a few years, I've come to realise that having something which makes writing attractive is more than just a nice incidental thing, but an important consideration.

At least for me, writing is a major way of organising and clarifying my thoughts, and I quite enjoy writing random streams of consciousness on a subject and later reviewing it to see if any interesting things are in there. If writing is fun, I'll do more of it, and having a decent writing instrument does make it fun.

Nowadays, I don't use a pen much (and my handwriting seems to be getting worse from lack of practice), but I still use a writing instrument - my keyboard. And just as there is a great difference between a free biro and a decent pen, so there is with keyboards, if not more so. A decent keyboard makes 'writing' fun, and that is a good thing. My current choice is a small Apple keyboard which is nice and portable, so I can plug it into whichever computer I happen to be using (it's even worth the annoyance of not having a delete key on a PC), and enjoy fine typing, perhaps even leading to some fine thoughts...