And now for something completely Pythonic...

Thanks to Thomas Lee, we have now in the Python trunk a built-in way to modify the Abstract Syntax Tree compiled from a piece of Python source, and to compile that to an executable code object.

Quick example:

>>> import _ast
>>> # A dumb source snippet
>>> source = "print 5 + 8"
>>> # Compile using the special PyCF_ONLY_AST flag to get an AST
>>> ast = compile(source, "<string>", "exec", _ast.PyCF_ONLY_AST)
>>> # The toplevel node is always a Module for "exec"
>>> ast
<_ast.Module object at 0xb7b00edc>
>>> # Each AST node has different attributes (see the docs)
>>> ast.body
[<_ast.Print object at 0xb7b020a4>]
>>> # The Print node has a list of values -- expressions to print
>>> ast.body[0].values
[<_ast.BinOp object at 0xb7b02114>]
>>> # This is the addition operator between 5 and 8
>>> ast.body[0].values[0].op
<_ast.Add object at 0xb7c98504>
>>> # Now, replace the operator with subtraction ...
>>> ast.body[0].values[0].op = _ast.Sub()
>>> # ... and compile the resulting AST
>>> code = compile(ast, "<string>", "exec")
>>> # Voila:
>>> exec code
-3

From this, you should be able to work out how to write AST manipulation code that actually makes sense :)

The _ast documentation gives you an idea which classes and attributes are available for individual AST elements.

The de-facto standard UNIX pager less supports an environment variable called LESSOPEN that can be set to the name of an input preprocessor. It is normally used to transparently view compressed files etc., but of course it can also colorize your source files using Pygments!

If you use Gentoo Linux, most of the work needed to set this up has already been done for you -- you just need to set

export LESSOPEN="|lesspipe.sh %s"
export LESSCOLORIZER=pygmentize

and make sure your LESS variable contains -r or -R so that the raw ANSI color codes are passed through by less. Gentoo's lesspipe.sh script will then automatically call Pygments for source code files.

On other platforms, you can set up a lesspipe.sh script yourself; it should look roughly like this:

#!/bin/sh
case "$1" in
   # add all extensions you want to handle here
   *.awk|*.groff|*.java|*.js|*.m4|*.php|*.pl|*.pm|*.pod|*.sh|\
   *.ad[asb]|*.asm|*.inc|*.[ch]|*.[ch]pp|*.[ch]xx|*.cc|*.hh|\
   *.lsp|*.l|*.pas|*.p|*.xml|*.xps|*.xsl|*.axp|*.ppd|*.pov|\
   *.diff|*.patch|*.py|*.rb|*.sql|*.ebuild|*.eclass)
      pygmentize "$1" ;;
   *) exit 0;;
esac

Then export LESSOPEN="|lesspipe.sh %s" and enjoy colored viewing!

Sometimes you get lucky. (Warning: dramatic events ahead.)

I was playing a large video file with mplayer when I made the mistake to delete it because of a command line accident (note to self: relying on the shell history to do what you mean is evil). First I thought "nice, now I can download the whole 500 MB again", but I realized that the file must still be available somewhere on disk because mplayer still needs to access it.

Helpful people from IRC pointed me to /proc/pid/fd/ which is a directory with symlinks to open files, but didn't believe it would do any good because the symlink's target is already gone.

Not one to give up easily, I looked into there, and the listing looked like that:

$ ls -l /proc/pid/fd
total 0
dr-x------ 2 gbr users  0 Mar 25 21:28 .
dr-xr-xr-x 5 gbr users  0 Mar 25 21:27 ..
lrwx------ 1 gbr users 64 Mar 25 21:28 0 -> /dev/pts/7
lrwx------ 1 gbr users 64 Mar 25 21:28 1 -> /dev/pts/7
lrwx------ 1 gbr users 64 Mar 25 21:28 2 -> /dev/pts/7
lr-x------ 1 gbr users 64 Mar 25 21:28 3 -> /home/gbr/video.mov (deleted)
...

Curious. Not the usual "broken symlink" view. And indeed, I could open the file and copy its contents somewhere safe. Isn't that nice?

I've just uploaded a new Sphinx release to PyPI. It should now be compatible with docutils SVN snapshots, as opposed to only docutils 0.4. If you still find problems with some SVN version, let me know!

Also new are some improvements in the doctest and autodoc extensions, see the CHANGES file.

I'm delighted to announce that the Sphinx library, used to build the new Python documentation (for 2.6 and 3.0), is now released for general use.

What is it?

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation for Python projects (or other documents consisting of multiple reStructuredText source files).

Its website is here.

What does it do?

Sphinx is not an API documentation generator like Epydoc. Instead, its focus is on hand-written documentation, such as the Python one.

• Main output formats: HTML (including HTML Help) and LaTeX
• Extensive cross-references: semantic markup and automatic links for functions, classes, glossary terms and similar pieces of information
• Hierarchical structure: easy definition of a document tree, with automatic links to siblings, parents and children
• Automatic indices: general index as well as a module index
• Code handling: automatic highlighting using the Pygments highlighter
• Goodies such as changes overview, and external link checking

What else?

Various extensions are available and in development:

• autodoc: pulls in documentation from docstrings that are written in reST, to avoid having to maintain multiple documentation locations
• doctest: automatically tests snippets in the documentation in doctest fashion
• coverage: documentation coverage checker

I'd like to thank everyone who has given it a try so far and provided me with valuable suggestions and patches: Uli Fouquet, Andre Roberge, Armin Ronacher,  Tim Golden and Mark Summerfield.

Now back to documenting new 2.6 features...

A little new feature in Python 3 that many overviews don't tell you about: extended unpacking. In a way, it complements starred function parameters and is therefore a straightforward extension to the syntax.

In a nutshell:

>>> a, *b = range(5)
>>> a
0
>>> b
[1, 2, 3, 4]

>>> *a, b = range(5)
>>> a
[0, 1, 2, 3]
>>> b
4

It works with any iterable on the right hand side, while the starred target on the rightleft hand side always gets assigned a list. There may be at most one starred target.

There must be at least enough items in the iterable to assign one to all non-starred targets; the starred one will then be assigned an empty list.

More in PEP 3132.

... but you can't say it's ugly. Not anymore. The "unicode" branch of Emacs 23 is now merged with the CVS trunk, and quite stable (I've been using it for months, without apparent problems).

Here are two examples of what a modern Emacs might look like, one editing Python code (with ECB, Org mode agenda and integrated interpreter), one editing reST documentation (with occur-mode).

    

So it's not ugly anymore. It doesn't need hundreds of megabytes of memory either. It's scriptable in Python. It makes you use those 90% of your keyboard you didn't even know exists. So, what's your excuse not to use it? ;-)

Obligatory links: the homepage or, for more and advanced stuff, the comprehensive Wiki.

(Yes, my .emacs is here.)

In the last few days, I've tried to decouple Python's new documentation tool (called "Sphinx") from Python itself, so that other projects can use it too.

I know that the Python world is not short of documentation tools, but it would be a waste of effort not to make this small step and offer another hopefully useful tool to the community.

If you didn't pay attention to the new docs yet (see http://docs.python.org/dev for a daily build), these are some of the main features (implemented as needed for Python core docs):

• Doc source is written in reStructuredText with some custom reST markup
• Output formats are currently HTML, Windows HTML help, LaTeX
• Easy definition of a document hierarchy
  
• Automatic cross-linking of object names, terms and more information items
• Index and module index building
  
• Automated source highlighting

All in all it's a good choice if your documentation consists of reST files that you want to arrange into a consistent tree of docs.

What's different to many other documentation tools is that no generation of docs from docstrings is currently supported (Python never did this, all documentation is maintained separately). Of course, it's not impossible to add that capability, and Armin is looking into that already.

Therefore, I ask those of you who would be interested in using Sphinx for their projects: What features would you need/want/like? (Except for documentation, which is currently restricted to this document and will be extended soon.)

If you didn't notice, last Saturday was a Python bug day. It was quite a success (35 bugs closed) and Christian Heimes and I continued on Sunday, closing 50 more bugs, mainly old ones that didn't apply anymore. A few new developers joined and contributed patches -- hopefully we'll see them again.

Because of that, and in the face of 1700+ open issues (don't worry though, many are "just" feature requests...), we're beginning to tentatively set up the bug day as a monthly institution -- the next will be on February 23.

If you're unfamiliar with the bug day concept: as many core developers as possible meet, discuss bugs and fix them (if possible). Therefore, it's also an opportunity for non-regular contributors to join in, get attention to their patches and fix problems. Why not you too next time?

I've recently given a talk about PyPy at the Munich Python User Group meeting.
The slides (in German) are here.
BTW: There's also a PyPy blog, if you want to stay up-to-date...