Pylons

Entry Date Author Categories RSS

setuptools versioning - wtf?
Nowadays I’m taking Debian’s APT software management system pretty much for granted. It probably represents the most advanced and powerful system. So I’m not instantly turned away if I find other systems that don’t work as well. Observing how Windows users struggle with getting the 50 different installed pieces of software to update that all [...] Jul 19, 2008 18:31 Chris Pylons, Programming Christoph's Blog

Pylons on JVM's (and other VMs)

Phil Jenvey has been making some great progress getting all the components of Pylons running on Jython, and posted a good write-up of the remaining work being done. It’s interesting to note that one of the big issues will affect any web framework on Jython, not just Pylons. That is, the reload time when used in development to restart the server.

While I don’t plan on deploying Pylons apps in WAR files anytime soon, its nice to see Jython emerging as a candidate for deployment.

May 07, 2008 18:40 ben python, pylons Groovie : Category pylons, everything about pylons

You Must Construct Additional Pylons (on other VMs)
Occasionally on the Pylons IRC Channel or mailing list someone will reference the game Starcraft, where "Pylons" are structures that act as a power source. Sometimes the game vocally instructs you to construct additional Pylons.

You could say that last month we've constructed a couple additional Pylons, and by that I mean made Pylons run on different environments:

o Pylons on Jython trunk!
o Pylons on Google App Engine via Ian Bicking's appengine-monkey and mako trunk

With the development versions of both Pylons and its dependencies (and the Mako jython branch) I'm now able to create the flicksearch tutorial app from the Pylons Official Docs on Jython.

The Mako jython branch uses Python 2.5's _ast module, which Jython now supports thanks to Frank Wierzbicki's recent work. Mako's also using Armin Ronacher's handy ast module that'll hopefully find its way into the stdlib. Frank and I are still working on smoothing out a couple of our _ast's rough edges, hence the separate Mako jython branch, but Mako is in good shape with 95% of its 214 tests passing (with most of those failures due to the lack of PEP 263 source code encodings, which we'll get to supporting on Jython soon).

According to the Pylons on Jython buildbot, only a few of the Pylons dependency's tests fully pass -- but most of the failing tests are due to unicode issues (again, lack of PEP 263), CPython dict ordering assumed in doctests, and a few other issues that aren't important enough to cover here. Pylons own test failures are only due to unicode issues and lack of support for the Kid and Cheetah templating engines, which Pylons is only testing for the sake of testing alternative templating engines anyway.

The biggest problem with Pylons on Jython so far is running it in development mode with paster serve --reload, which hot reloads your WSGI app whenever a change to one of its files is made. The paster reloader is working but the reloading process is very slow on Jython. Reloading is done by spawning 2 processes; when a file changes, the child process serving your app exits and the parent creates it anew (last time I looked Django and CherryPy's reloaders also work this way). Since CPython startup time, including loading of the entire WSGI app, is pretty quick, this is the safest and probably easiest way to accomplish a reload.

Unfortunately Jython suffers from a lenghty startup time. It takes about 1.5 seconds best case scenario just to get the >>> prompt in Jython on my 2.33ghz MacBook Pro. It takes almost 10 seconds to get paster serve --reload to fully load a simple Pylons app; that's the parent process startup time plus the child process startup time plus the app load time.

I've been experimenting with using Nailgun to improve startup time, but it actually isn't helping Pylons' startup time as much as I thought it might. Jython must have a startup time bottleneck or two that we need to find here.

Nailgun also poses other problems, such as when System.exit()'ing resources like sockets and files are left open unless they're explicitly cleaned up. Worse yet, threads are left running -- and Pylons' paster serve by default loads 10 listener threads.

If we can't drastically improve the startup time issue in Jython sooner rather than later, we can probably come up with a reloader for Jython for the time being that doesn't require a full restart. These can be tricky and in general are avoided, though.

Here's a couple screenshots of Pylons on Jython, showing off a handy feature of dynamic languages on the JVM; just having the ability to play around with Java in an interactive interpreter:

o Pylons-dev WebError on Jython
o Pylons paster shell on Jython

Pylons is also making progress towards an additional release, 0.9.7, which will utilize the work done by the PyCon sprinters (particularly on WebHelpers), WebOb, WebError (and maybe WebTest) and will also include optional support for a basic SQLAlchemy configuration for new projects. As well as Alpha Jython support. May 05, 2008 00:41 noreply@blogger.com (Philip Jenvey) jython, python, pylons __boss__

Our GSoC project has been accepted - yahoo! :)
This is pretty exciting. I have never bothered about Google’s Summer of Code and have literally been forced (curse you, Kumar) to propose my project I’m working on. And in the end I have a promising student and Google accepted our proposal. Watching #gsoc while everybody was waiting whether they got accepted caused more traffic than [...] Apr 21, 2008 21:40 Chris Pylons, debexpo, Debian Christoph's Blog

MarkMail now indexing Pylons-discuss/devel

I’m thrilled to announce that MarkMail now indexes the Pylons-dicuss and Pylons-devel mail lists. For those looking for a great way to search and browser the Pylons mail lists, the MarkMail interface is top-notch.

For those looking for detailed Pylons docs…. there’s some very exciting developments in the documentation front coming up shortly that will make you rather happy. :)

Apr 09, 2008 07:14 ben python, pylons, thoughts Groovie : Category pylons, everything about pylons

Pylons @PyCon 2008 Wrap-up

Sprints!

Wow, they were great. A lot got done, quite a bit more than I was expecting. I’d like to give a big shout-out to the Pylons sprinters (in random order, sorry if I missed anyone):

Karen Lo

Mike Orr

David Montgomery

Mike Verdone

Wes Devauld

Ian Bicking

Phil Jenvey

and many more TG2 sprinters

WebHelpers saw some significant gains, including the addition of a literal object for safe HTML escaping, and default Mako auto-escaping in Pylons 0.9.7.

There will definitely be more Pylons/TG2 sprints in the future, in multiple locations around the country.

Tutorials

Ouch… setuptools came through, unfortunately the network didn’t, nor did my first built egg of Pylons (unfortunately I got a new laptop recently that was missing some things…). Mark Ramm and I did get the show under way, but the delay in starting up definitely affected how far the tutorial attendee’s made it in the basic Wiki project and proved frustrating for all.

I made it through all the Pylons information I had in the Mastering Pylons/TG2 section, but unfortunately had no time left to go into more detail on various aspects of it in a more hands-on style. I definitely won’t be doing any more tutorials without ensuring lots of USB thumb-drives are handy with virtualenv Pylons/TG2 ready to go.

Sessions and the Conference

They seemed ok, several of them seemed very, “And this person is giving this talk…. why?”. Note to session presenters, honesty is great but I really do wonder why you’re presenting when your opening remarks are, “What I’m about to show you, anyone who’s used this already knows, so there’s nothing to see here.” Show us something with some zing!

The lightning talks, despite a first day of some awkward sponsored ones, were actually the highlight for me. I almost spit water laughing on the “Speech Recognition does war”, followed up by “His base of God would have I done now” (Originally intended to be ‘Speech recognition does work. oh god, what have I done now?’). Unfortunately Ian Bicking failed to give us an update on ZhangoPyloGears, I can only imagine it has fully attained consciousness and freed itself of his clutches.

Coming up…

Pylons 0.9.7 is close at hand, just waiting for some final bits of WebHelpers to drop into place and it should be ready to pop. And one last side-note, I think I finally found the documentation tool of my dreams… Sphinx!

Mar 22, 2008 05:22 ben thoughts, python, pylons Groovie : Category pylons, everything about pylons

setuptools on Jython
A couple weeks ago setuptools added support for Jython trunk. This was made possible with a number of additions and fixes to Jython for setuptools, including distutils and file descriptor support.

A working setuptools is the first big step in getting Pylons working on Jython. Django on Jython has had some recent success; luckily for them Django doesn't rely on/use setuptools at all. So Leo Soto, Jim Baker and the rest of the Django on Jython folks were able to skip directly to kicking around the Django code on Jython.

Now that we have setuptools, the next step is to go through each of Pylons dependencies and ensure their tests pass. Pylons as well as some of its dependencies use the nose test runner. Nose is almost fully working on Jython (after I submit a few upstream patches to nose).

Other dependencies (such as Paste and Beaker) use py.test for their test runners, but there's been some talk of moving some of these over to nose -- which would actually be a good thing right now for Jython. Though technically nose could run through these projects' tests as they already are, they rely on some py.test functionality, such as py.test.raises.

Not to bad mouth py.test -- it has some nice features, like distributed testing -- but I attempted get it working on Jython during the TurboGears 2 sprint a few weeks ago and it wasn't a small task. Part of the problem is the fact that py.test is part of the larger py package. py isn't just a test runner, it also includes other packages that are intertwined with test portion -- all of which need to work before you're able to use just py.test. Feb 09, 2008 07:15 noreply@blogger.com (Philip Jenvey) jython, python, pylons __boss__

Fun and despair with ExtJS
Since I started playing with Javascript about a year ago I became aware of a pretty large number of Javascript libraries that offer additional functions or even widgets for a nicer user interface. I had started with prototype/script.aculo.us because it was included by Pylons. But when I found out that I’m not really bound to [...] Jan 15, 2008 15:01 Chris Open-Source, Work, Pylons, Programming Christoph's Blog

kate isn’t just a NOTEPAD.EXE for KDE
I’ve been a VIM lover for at least a decade. Nothing beats it at doing complex text editing. Surely it’s not intuitive (although I frequently claim that VI stands for “very intuitive”) but no other text editor can so quickly repeat the last action or reformat a paragraph or insert text at a certain column [...] Dec 02, 2007 14:47 Chris Pylons, Find, Programming Christoph's Blog

SQLAlchemy, Declarative layers, and the ORM 'Problem'

There’s been a bit of talk on the Pylons devel list regarding the recommended way to use SQLAlchemy with Pylons mainly regarding how to use SA (SQLAlchemy) in a fashion that is well documented and easy to work with (and maintain!).

Prior to Pylons 0.9.6 and SQLAlchemy 0.4 it was a bit of a mess, with the framework needing to load the config (since thats where your db settings are), then setup globals for SA…. eek. Mike Orr had a good intermediary solution for SA 0.3 called SAContext that handled many of the tricky parts. Unfortunately, this actually caused even more confusion as more ways of doing the same thing came about. SAContext solved some of the global config grabbing issues, but the additional layer of indirection made trouble shooting even harder (despite how small of a library it was).

Less is More

So the fix? Less intermediary layers, less indirection… essentially, KISS. Despite how much Pylons was attempting to help a user to get the db going, the additional layers in the end actually caused more problems then they solved. Of course, I shouldn’t have been too surprised…. Mike Bayer did warn me about many of these things at the beginning. Being overly eager to make things “easier” for new users, I ignored him. :)

This is why Pylons does not recommend Elixir, and with SA 0.4 the recommended usage of SA is to use its plain mapper functionality should you need an ORM layer. Yes, that’s right, despite almost every web framework out there pushing its ORM on you (or someone else’s ORM), there are many times when an app doesn’t even need a full-blown ORM.

Declarative vs Basic SA

For a better look at why one might consider additional layers on SA a bad thing lets compare a fairly basic table setup consisting of users and groups. Each user can be in multiple groups, and lets use proper referential integrity to ensure that groups aren’t deleted when users are still in them.

Compare the following two ways of setting up a basic many to many relation and the tables:
SQLAlchemy 0.4
from sqlalchemy import Column, ForeignKey, MetaData, Table, types from sqlalchemy.orm import mapper, relation metadata = MetaData() person_table = Table('person', metadata, Column('id', types.Integer, primary_key=True), Column('name', types.String, nullable=False), Column('age', types.String) ) group_table = Table('group', metadata, Column('id', types.Integer, primary_key=True), Column('name', types.String, nullable=False) ) persongroups_table = Table('person_groups', metadata, Column('person_id', types.Integer, ForeignKey('person.id', ondelete='CASCADE'), primary_key=True), Column('group_id', types.Integer, ForeignKey('group.id', ondelete='RESTRICT'), primary_key=True), ) class Person(object): pass class Group(object): pass mapper(Person, person_table, properties=dict( groups=relation(Group, backref='people', lazy=False) )) mapper(Group, group_table)
Elixir
from elixir import * class Person(Entity): has_field('name', String) has_and_belongs_to_many('groups', of_kind='Group') class Group(Entity): has_field('name', String) has_and_belongs_to_many('people', of_kind='Person')

On first glance, its pretty obvious that everyone should love Elixir vs the obviously more tedious SA approach of layout out your tables, then mapping them to the class objects. However, look at these two examples, and try to quickly answer the following questions:

How do you add a column to the many to many table to store an additional bit of info for the join?

Do they both enforce referential integrity?

How do you control whether SA is eager loading the relation? Can you restrict it to just one column of the relation?

What are the table names used?

How many tables are in your database?

Where do you change the id column name?

Which one is closer to the Zen of Python?

I think the explicit setup makes many of these questions easier to answer just at a glance. Those with enough Elixir experience can fairly easily answer most of these questions, but consider what that implies. Not only do you need to know SQLAlchemy options and parameters, but you need to know Elixir options and how they map to the SQLAlchemy functions. The desire to reduce the up-front setup of the ORM actually increases the amount of knowledge a user has to have in order to use it, and the most worrisome aspect… how to troubleshoot it.

Setups that Grow with You

With Pylons, a goal has been to provide out of the box recommendations that grow with you. That is, using the set of recommended tools may not be as apparently “easy” as some other frameworks. However, the pay-off is that you don’t hit a wall in 2 months when your application inevitably gets a little more advanced and needs to do something the simple tools either can’t do at all, or it’s incredibly difficult to do even slightly complicated things (eager load 2 columns off a related table, but not all of them). This way, the toolset you learned, you can keep using as you get more advanced and you don’t “outgrow” your tools.

While Elixir definitely appears to be easier at first glance, when you need to get more complicated you can’t exactly turn to the SA docs since Elixir has put a layer between you and SA. This can be very crippling when you eventually hit a wall, and so much ‘magic’ is wrapped up in the declarative layer that you have to troubleshoot additional layers of code when something goes wrong. The result of this is that to effectively use Elixir in complicated setups with SQLAlchemy, you need to really really know both of them which actually requires more work for a user than plain SQLAlchemy.

The SA example clearly requires a little more up-front setup, however, are you really adding tables to your database every day? How often are you going to be actually touching the table and mapper code, vs just adding domain model methods to your Person/Group class? Did the layer make it easier or harder to use multiple databases and/or put more between you and advanced SA functionality you might need later?

Adam Gomaa pointed out some interesting issues with Django’s ORM and Elixir but unfortunately tries to do the same thing Elixir and TurboEntity do…. add more layers. More layers more indirection more to wade through when you need to do something that should be pretty basic with SQLAlchemy (and is probably nicely documented on the SA site, which won’t help with these layers until you dig through the layer to find the basic SA objects the SA site refers to…).

What really makes a lot of this even more trouble-some with SA, is that when setting up complex relationships, the order of declaring table objects becomes important, since relations need to refer to them and the ORM classes. This usually results in some interesting metaclass hackery when you have these Entity’s in multiple modules, importing each other, and doing other fairly common stuff.

SQLAlchemy 0.4

In the end, I’ve been using plain SQLAlchemy 0.4 (at beta5 now, but quite stable) a lot lately, and its really great. Yes, setting up the tables (generally a one-time thing) took me probably 15 mins longer than it would’ve with Elixir. But I’m fairly certain I’ve saved myself significantly more time in the long run since I won’t need to worry about diving into Elixir code to try and find SA objects when I need a complex query, or trying to figure out how to hack Elixir’s connection should I need multiple db connections at once, etc.

So please, new users to SQLAlchemy, use just SQLAlchemy. It definitely seems daunting at first, but the flexibility and comprehensive documentation give you a solution that scales to meet your needs, with no walls in sight.

On a side-note, its interesting to compare my position on this issue to the Django team lack of AJAX helpers. The Django team rightfully claims that Javascript isn’t that hard, so “get over it” and learn a nice Javascript library so you can do powerful things. Also note that by including AJAX helpers, Pylons is encouraging one part that doesn’t scale… as the AJAX helpers will have you hitting a wall sooner or later.

Sep 04, 2007 22:40 ben python, thoughts, pylons Groovie : Category pylons, everything about pylons

Making logging friendly in webapps

Finding out whats going on in an application is always a key point when trying to figure out what’s happening when things don’t go the way you expect. In these kind of annoying situations, following the log of how your request mapped in and what was going on can save a significant amount of time. TurboGears has had great logging throughout it, and with 0.9.6, Pylons add’s the same thorough logging.

But once the entire request is mapped out, it can be a real chore to track through them, toggle modules you want and don’t want logging for, etc. Chainsaw has had excellent graphical support for navigating logging output, and with the help of Phil Jenvey’s XMLLayout package its easy to output logging statements in a format for use with it.

With the recently updated Chainsaw section in the Pylons logging docs, you too can be lumberjacking with Chainsaw.

And of course, a shot of Chainsaw in action:

Aug 21, 2007 02:59 ben python, pylons Groovie : Category pylons, everything about pylons