The testcases for the tokenizer require the use of simplejson (http://cheeseshop.python.org/pypi/simplejson). Each testcase file can be run directly through a python interpreter (e.g. python test_tokenizer.py).