zope.testbrowser: Automating the Web
Speaking of little known python modules, if you haven't used zope.testbrowser, please raise your hand. Okay, all of you that raised your hand and have ever wanted to automate web forms, check a web site's functionality, or perform screen-scraping related tasks, pull up a chair. This won't take long, because zope.testbrowser's API rocks so hard and is so easy to use.
I first started using zope.testbrowser sometime before consulting for Zenoss. However, when I implemented Zenoss' "Synthetic Transactions" (I believe this was originally part of their open source offering, but has since been moved into the Enterprise Edition), I used it again and profusely. It provided me with a wonderful Python API (a mechanize wrapper and then some), one so intuitive that I could use not only without thinking, but for which I didn't even have to read the docs. This solution proved to be friendly to programmers, but we were really targeting non-programming network administrators and IT managers with the Zenoss plugin, so I completed the plugin using Twill and TestGen4Web (whose python runner I rewrote and to which I added support for zope.testbrowser).
I used it again at Zenoss later when building the community software, enabling users to set their Mailman preferences from their Plone settings. Most recently, I've been using it to publish news articles to various sites from a single file used by a script.
Here are a couple of standard use cases with sample code:
Visit a page and follow some links
>>> from zope.testbrowser.browser import Browser
>>> b = Browser('http://google.com')
>>> b.title
'Google'
>>> b.getLink('About Google')
<Link text='About Google' url='http://www.google.com/intl/en/about.html'>
That gives us the actual link object... but what if we want to follow that link?
>>> b.getLink('About Google').click()
>>> b.title
'About Google'
Simple enough :-)
Check for content
Let's continue on the Google link trail, and check for content:
>>> b.getLink('Investor Relations').click()
>>> b.title
'Google Investor Relations'
>>> b.getLink('Guidelines').click()
>>> 'Corporate Governance' in b.contents
True
Sign into a site
Signing into a site requires filling in form information and submitting that data.
>>> b = Browser('https://launchpad.net/+login')
>>> b.title
'Log in or register with Launchpad'
>>> b.getControl(name='loginpage_email', index=0).value = username
>>> b.getControl(name='loginpage_password').value = passwd
>>> b.getControl(name='loginpage_submit_login').click()
>>> b.title
'Launchpad'
>>> 'Logged in as' in b.contents
True
A couple things to note here:
- you need to look at the HTML so that you know what the form elements are named;
- there are two form elements on the page that are named loginpage_email, and we want the first one; and
- I omitted the part of the code where I pulled my credentials from the file system.
Submit form dataNow that we're logged in, let's change some out-dated information in my profile (my old blog's link has been there for too long):
>>> b.getLink('Duncan McGreggor').click()
>>> b.title
'Duncan McGreggor in Launchpad'
>>> b.getLink('http://www.advogato.org/person/oubiwann/')
<Link text='http://www.advogato.org/person/oubiwann/' url='http://www.advogato.org/person/oubiwann/'>
>>> b.getLink('Change details').click()
>>> b.getControl(name='field.homepage_content').value = 'blog: http://oubiwann.blogspot.com'
>>> b.getControl(name='field.actions.save').click()
Now that we saved it, let's check our results:
>>> b.getLink('http://www.advogato.org/person/oubiwann/')
Traceback (most recent call last):
[snip]
mechanize._mechanize.LinkNotFoundError
>>> b.getLink('http://oubiwann.blogspot.com')
<Link text='http://oubiwann.blogspot.com' url='http://oubiwann.blogspot.com'>
Excellent; just what we expected to see.
These are all really simple examples, but they should be helpful in providing some insight and inspiring you to use it :-) Working with radio buttons is a little more complex in that you have to get the containing object first and use
getControl on that object in order to get the selection you want. However, if you've worked with HTML more than any sane person should have to (I think that exact amount is "any") , this will all be quite expected.
So, who's up for making an async version to work with Twisted? ;-)
A tangential caveat: I made a comment above about the
zope.testbrowser API rocking, and that needs some clarification. As far as I am concerned, there are two major ways in which API's can be graded:
- how easy is the API to use, how well is it documented, how much of the problem domain does it cover? (developer as user)
- how elegantly/efficiently was the solution implemented (developer as contributor)
zope.testbrowser rocks from the perspective of a coder using the API. I've never looked at how it is implemented, because I haven't
needed to. It does everything I require without me having to dig around in its guts. As such, I can make no assertions as to the quality of its implementation. However, it's packages, modules, classes and methods are all named just as I would expect them to be named and do what I would expect them to do. That may not sound like a big deal, but it really is.
Technorati Tags: howto, programming, python, web, zope
Syndicated 2008-02-20 15:57:00 (Updated 2008-02-20 16:02:39) from Duncan McGreggor
