And the winner is ... Twisted

Posted 6 Jun 2003 at 15:09 UTC by Cantanker Share This

If there is an award (I don't read a lot of press), for excellence in open source software, then my nomination is for Twisted, possibly the finest, most thoughtfully designed and useful of active open source projects.

Disclaimer: the author is not affiliated with the Twisted project, nor its lead violinist, Glyph Lefkowitz, who is a nominee for the Python division of the current Active State Active Awards.

What is Twisted

While Twisted is, as many good projects are, many different things to many audiences, the core of Twisted is an event-based programming framework, particularly oriented towards networked applications. But it is also so much more! Event-based programming can be somewhat tedious, and Twisted provides a plethora of packages and tools to make your application not just a cinch to get running, but a pleasure to develop!

Why Twisted is excellent

There are probably pages that could be written detailing

Twisted's virtues, but I'll concentrate on three broad qualities not often seen together in one project:

  • commitment to quality,
  • design excellence, and
  • a very wide application domain.

These qualities are arguably Twisted's greatest strengths.

Commitment to quality

Development on Twisted uses (much of) the eXtreme Programming methodology, which (among other things) dictates that each module must pass an accompanying test before it is released. Twisted goes one better and provides a complete test package, which it, naturally, uses for its own test-driven development.

Twisted's commitment to quality is evinced in the project's comprehensive coding, testing and documentation standards, which are all rigorously adhered to.

Design excellence

Twisted's design is based on the well-researched and established theory of design patterns, and many of its packages are fully documented implementations of some of such patterns.

Twisted's method of implementing patterns is with an object-oriented technique called Aspect-oriented programming, where parts are composed of reusable and adaptable components that declare themselves to implement a given interface. This means all of Twisted's components are completely orthogonal, that is, each component does its one assigned job and does it well, and you only use the parts you need.

Wide application domain

But all this high-falutin abstraction isn't worth a penny if it isn't useful, and that's where Twisted wins big: the patterns and the components and the interfaces all mean that the core parts of Twisted are generalised and can be applied to almost any application domain, network or otherwise.

If Python comes with "batteries included", then Twisted might be considered Python's heavy-duty diesel generator. It ships with fully-functional web (including SOAP and XML-RPC), FTP, mail, news, messaging, DNS and file-sharing servers, a templating engine, a remote object broker, a documentation framework, a complete framework for deploying, managing and configuring daemon processes, and complete integration with the Debian GNU/Linux package management system. And more.

Why you should consider Twisted

Apart from the advantages listed above, there are other reasons you or your organisation should consider using Twisted for your next networked application or managed service.

A complete environment

Twisted comes with everything you need to develop, test, and deploy your application. All you need to write are plugins that contain the business logic.

Powered by Python!

Python is a simple but extremely powerful object-oriented interpreted language. Its simplicity means that any competent programmer can pick it up in a day or less (thanks to Guido van Rossum's excellent tutorial) which, combined with its powerful introspective capabilities make it a perfect environment for rapid development and debugging.

Truly cross-platform

Twisted will run on any platform that supports Python and the select() or poll() system calls (there is support for a variety of different platforms and event loops) and you can even run a Twisted application as a native application on everyone's favourite proprietary operating system.

This means you can use the same code for client and server applications, reducing redundant efforts and development costs.

If you can't use Twisted for both clients and servers, Twisted supports many standard protocols out of the box, and writing a new protocol is greatly simplified with the availability of many abstract base protocol classes to reuse.

Links

If you hadn't already guessed, all things Twisted can be found at www.twistedmatrix.com.


:), posted 7 Jun 2003 at 17:09 UTC by Malx » (Journeyer)

If you whould give an award, then you should at least add it to Best rated Linux software

ps. I am not affiliated with it either :)

If I wanted a press release ..., posted 7 Jun 2003 at 21:05 UTC by dan » (Master)

I'd have read an industry rag.

Spare us the fawning hyperbole and write something with some actual technical content, could you?

Press release, posted 7 Jun 2003 at 21:41 UTC by Cantanker » (Journeyer)

The last press release I read had noticeably less supporting material.

Press release, posted 8 Jun 2003 at 01:44 UTC by djm » (Master)

The last press release that I read was mercifully absent from advogato.

I am so sick of these kinds of complaints, posted 8 Jun 2003 at 06:45 UTC by jbuck » (Master)

It seems that any time anyone posts anything with any content, the trolls come out of the woodwork to say that it doesn't belong: if it's quirky, it should be a diary entry. If it promotes some open source project, it's a press release and doesn't belong. If anyone posted a similar article in the past month, then the new article should have been attached to that thread. And blah blah blah. Even as it is, frequently a week goes by between new articles. Would you rather have it be a month?

dan, djm, give it a rest. I hadn't heard of Twisted, and it sounds interesting; if you guys had your way, I never would have. If you're so confident of your tastes on what an appropriate, interesting Advogato article is, why not write one.

Re: I am so sick of these kinds of complaints, posted 8 Jun 2003 at 07:47 UTC by tk » (Observer)

For once I agree with the trolls. The problem is not that the article promotes a project, but that it promotes without giving useful content. No, Mindless Link Propagation doesn't count. And I wonder what's so important with having at least one article per week -- it's like saying the amount of noise matters more than the S/N ratio. Also, why someone must first write an article before he can say others' articles don't belong. Isn't it the whole point, that if one doesn't have anything interesting enough to post as an article, then don't post anything?

Anyway, I've been hearing about Twisted for quite some time on recent-log already (I wonder why jbuck isn't aware of it). But even this content-free, buzzword-laden, all-praises rave review isn't going to convince me of the advantages of Twisted -- for nothing is ever as good as it sounds.

(First off, why is Twisted a framework, not a library?)

Re: I am so sick of these kinds of complaints, posted 8 Jun 2003 at 09:37 UTC by Cantanker » (Journeyer)

What is the problem?

I am consistently impressed by the quality of the work the Twisted team does, both in terms of the software they produce, and, particularly, the rigour of their approach, and I thought they deserved a little bit of praise outside its own community.

I also thought, given the "advocate" nature of this forum, its mission statement, and the paucity of recent articles (I count no fewer than four SCO articles on the front page, is that technical content?) that such an article would have been welcome on this forum.

My article is structured and uses proper English. It is not a religious or ideological rant. It is not a misplaced complaint about another forum, a rehashing of another SCO article, or a bitch about how stupid American laws are. My claims are not hyperbole; at least not to me. They are quite moderate claims about qualities in a project that are not usually singled out for discussion.

It is, I admit, a light article. It was intended to be, so there is a visible, public and positive summary of the project for evaluative purposes. Evidently my overly positive language has offended the more jaded correspondents.

Can't you people just be nice?

No Seriously, posted 8 Jun 2003 at 10:42 UTC by glyph » (Master)

I wish this had been about another project, so that I could more credibly defend it, but this is actually the variety of article I'd really like to see on advogato. It is, after all, advocating free software.

To the naysayers -

It ships with fully-functional web (including SOAP and XML-RPC), FTP, mail, news, messaging, DNS and file-sharing servers, a templating engine, a remote object broker, a documentation framework, a complete framework for deploying, managing and configuring daemon processes, and complete integration with the Debian GNU/Linux package management system.

This looks like "technical content" to me. Sure, it's light, but it's at least explaining what makes the framework interesting to the author in terms of actual features, not wishy-washy terms like "enterprise-ready" or "web-services enabled". So, in order to round this out, I should probably include some technical content in my comment here -

Twisted is often advertised as providing this huge suite of services in a complete and production-ready way, but this is sadly not the case. While the project is actively developed and substantial improvements are occurring all the time (1.0.5 is VASTLY superior to 1.0.3, for example) the quality level is unfortunately inconsistent

The web server is definitely production ready. It's been running twistedmatrix.com literally for years now with no serious problems. it's very powerful and easy to configure. The FTP server mostly works, but nobody uses it so it has languished somewhat. The remote object broker is very powerful and flexible (absolutely wonderful for prototyping) but has some design issues that make it difficult to implement cross-language portable protocols reliably; a rewrite is in the works. The mail server is a challenge to set up properly. The end-user configuration interface is seriously languishing, to the point where we are considering removing it from the release until it can be re-written.

The documentation framework, Lore, is excellent and should be discussed more often. The test framework, trial, is well-designed and well-integrated with the Emacs environment. The SSH server, conch, is really awesome but has not yet seen enough real-world use - we need applications that exercise it and that expose it to creative white-hat crackers.

That said, I think that the project's status in terms of quality design, testing, and responsiveness to user feedback is pretty good. Most projects could benefit from a similar attention to detail. For example, Brian Warner's excellent BuildBot (More free software advocacy! Hooray!) has a status page for Twisted so that developers working on more common operating systems will know if something breaks on the Microsoft or Apple platform.

However, I think that the real testament to Twisted's success is that we inspire this kind of spontaneous praise from users, even if it's not entirely accurate. I find that a lot of us who, in our youths, were fascinated by computing and excited by its possibilities, are feeling jaded and cynical in the age of the internet. I am no exception to this rule - I found the era of "multimedia" more fun then the current era of "networking". Do you remember when "multimedia" was a buzzword and CD-ROM drives were still cool? We need some fresh enthusiasm to keep coding interesting and fun. Keep the hyperbole coming! :-)

and the winner is ....Twisted, posted 8 Jun 2003 at 16:28 UTC by badvogato » (Master)

there is good parts of twisted in all this.

Back to the topic about "winner", I am reading this book "the upright brush - Yan Zhenquing's calligraphy and song literati politics" . There are some important passages i'd like to share with y'all:

"I explain the notion of characterology as the ancient pseudoscience of assessing a man's character and fitness for government office from examination of his aesthetic effect, both in his physical appearance and comportment and in his practice of the polite arts. Such things as the composing of prose and poetry, the playing of chess or the seven-stringed qin, and the writing of calligraphy result from the gesture of the hand in obedience to the mind and thus were seen as outward manifestations of inner character."
In our time, characterology is really programming. The writing of free/open/propriety software are gestures of science and technology in obedience/rebelion to corporate culture at large. If programming is measured as "outward manifestations of inner character", it is what you do with your pure mathematics besides its aesthetic effect that matters. It is what you know about what mathematics won't do shall ultimately add value to all engineering devices this society is willing to employ. Aesthetic effect and people's will are increasing being marginized. And who are the first to complain about that? Not to my surprise, they are among the best designers of the best tools when chances put them in the other end as users of designers of lower qualities. So my conclusion is, i'd wish people can see our DNAs are upright brush hiding out in twisted codes.

Have you noticed the dedication page of the sensational #1 bestseller "Shibumi" by Trevanian? To the memories of the men who here appear as: Kishikawa, Otake, de Lhandes, Le Cagot. All other characters and organizations in this book lack any basis in reality -- although some of them do not realize that. To differentiate numbers of people and numbers of number are very hard to be of any aesthetic value to far too many high prists of mathematics because sadly enough, not all people are born interesting as equal as they may be.

Re: no seriously, posted 8 Jun 2003 at 17:56 UTC by dan » (Master)

glyph: OK, you've just condensed a seven hundred word article to one paragraph, leaving out approximately nothing that would help me decide whether to use Twisted in a project. And there's still nothing in there about why your DNS system or Web server or FTP server is any better than bind or apache or ProFTPd. Except to Python programmers - who I would suggest probably know about Twisted already - the fact that it's written in Python is not per se a feature. I'm guessing that it implies certain things about extensibility or customizability: an article that made these advantages explicit, with examples of the problems it solves, and without appealing to "it's really high quality, honest, please believe me" would be pretty interesting.

(For that matter, the rest of your comment displayed a much more interesting, informative and balanced view that was far more informative than the original article. More articles like that comment, please ...)

I'm not picking on Twisted here. From what I've read in recentlog and elsewhere on the net it has some pretty neat ideas, so I was actually looking forward to reading something about it. If the article title had been "SCO are in bed with Microsoft and censoring my posts because I work for an open source company" I'd have had much lower expectations - in fact, I'd probably not even have looked at the article.

While I'm ranting, though, weblog-style poorly labelled links aren't useful or stylish, they're just irritating

  1. the reader should not be expected to follow a link without at least some indication of what they might find there - it takes much less time for him to read six additional words than to pick up the mouse and hover it over the link text to guess what the subject is

  2. the visual effect of highlighing random words makes you sound like some mid-1990s net.kook.

Re: no seriously, posted 8 Jun 2003 at 19:24 UTC by moshez » (Master)

dan:

so I was actually looking forward to reading something about it

That strikes me as highly hypocritical. If you really wanted to read in-depth papers about Twisted, why did you wait until Cantanker posted his article, went to his article, waited until glyph posted his comment, and then complained?

You could have, for example, gotten more papers about Twisted than you have time to read had you simply bothered to check the Twisted CVS. Instead, you chose to be abusive to someone who contibuted a fun-to-read article to the community. Good going!

Re: no seriously, posted 8 Jun 2003 at 22:39 UTC by dan » (Master)

"Hypocritical", I believe, is applying different standards of behaviour to others than one does to oneself. I don't see anything in either of my comments that suggests that a similar article would be more interesting if I'd written it. So, hypocritical how exactly?

Did I even mention "in-depth papers"? As you identify, I - and most of the free software community, I'd expect - have limited time. When I saw the article title on the front of advogato, I was hoping it would be a short introduction to Twisted that would tell me if it would be (a) interesting/useful to learn, (b) worth borrowing from. It wasn't and didn't.

I'm happy to hear that you have more papers about Twisted than I have time to read (and thank you for the link, I'll add it to my list), but if the article we're talking about here had had a little more information and a little less hype, I would be far better placed to decide whether it's worth making the time to read some of them. I assume the purpose of posting to Advogato about Twisted is for it to reach a wider audience? A more technical article could have done that far more effectively.

Content, posted 9 Jun 2003 at 00:42 UTC by djm » (Master)

jbuck: I wouldn't say that the posted article had any content, at least none of the form that I'd expect for Advogato. It is advertising, pure and simple and characterising those who disagree with you as trolls is a pretty cheap shot.

If the poster has included some *technical* content (beyond buzzwords) and turned the hype down, then this could have been an interesting article. Perhaps an example of some real-life applications (e.g. the neat little ssh implementation), a critical analysis of its benefits and flaws, or an introduction to its event model. There is no shortage of interesting things to talk about...

On PR, Twisted, posted 9 Jun 2003 at 01:30 UTC by ianb » (Journeyer)

While I hate to become involved in meta-commentary, I do think there is an issue surrounding Twisted and the way it is marketed. It is offered as the best choice for too many things, and with too much enthusiasm (at least inside the Python community). This was the way Zope was advocated at one time, I believe to the significant detriment of the Python community. I think people need to step back and be a bit more objective about these things (and it's nice to see Glyph has done so in a comment here). I'm not saying this as a right-tool-for-the-job kind of guy -- I really do believe that the right-tool-for-every-job exists, even if it hasn't been implemented yet. But I believe the argument about what the right tool is needs to be made thoughtfully, and including alternate perspectives. Some of Twisted's choices -- choices that you may not like -- are not well presented in this sort of marketing.

Now, to switch out of meta-mode, and comment on Twisted itself. I don't believe an asynchronous framework can achieve much popularity (and I don't think it's a good idea either). For the unfamiliar, asynchronous code is code where all long-running tasks are done asynchronously. That is, if you have to do something long-running (a database fetch, a network send, etc), you run a function that has a callback, and when the task is completed that callback is run.

As a result you don't need threads or multiple processes. You don't have to worry about a lot of concurrency issues, because there isn't any concurrency -- you are running one thing at a time, full speed, and if you don't have anything to do (or you're waiting on something) you let someone else run.

There's lots of arguments for this, but to me it boils down to: asynchronous is weird and forces you to think in a certain way. It's like highly functional programming -- something which has never caught on, because it's too hard for people to become comfortable with.

The other problem, and the one which makes me worry about Twisted being marketed too hard, is that asynchronous APIs look weird and have strong requirements. If you use something in your asynchronous program, and it takes a long time to return, your entire server process is stalled on that. A significant portion of otherwise reusable Python code has some condition in which it can block, by error or intention. You have to guard every use of such code in Twisted, to protect yourself against possible blocking. This reminds me of using Python code with Zope, which has security requirements that make code reuse very difficult.

From the outside, Twisted code is strange to use, because asynchronous APIs are funny-feeling. I expect functions with useful return values, instead I get functions with no return value, that return before they are finished, and where I have to use a callback to find the result of the function. You can get around both sides of this problem (certainly more easily than in Zope), but the result is a cultural divide. It's that cultural divide that really worries me if Twisted draws lots of unsuspecting people in.

That said, I think Twisted may serve as a useful networking foundation, where asynchronous programming perhaps makes more sense. I hope instead of trying to make Twisted fill server needs at all levels, that instead Twisted will develop mature ways to bridge from the low level to more conventional high level applications.

Two-way forum, posted 9 Jun 2003 at 02:42 UTC by Cantanker » (Journeyer)

These comments seem to me the perfect place for the more technical discussions. Remember there are many people with a wide range of abilities who use this forum, and a lighter article may mean more readers will find it accessible. This comments mechanism allows for points to be clarified and technical details to be fleshed out: exactly what has happened here. There is simply no need to abuse an author because his article does not conform to someone's idea of what is "interesting".

Are not quality and usefulness interesting to software engineers involved with higher-level design and project direction? Or are they just too fuzzy for the likes of dan and tk to appreciate?

My exact crime was evidently to target a wider audience, in a lull in interesting articles (although my faith in the community to produce an interesting discussion has not been eroded!)

Still, your author humbly accepts and notes the criticisms.

We are the programmers who say NIH, posted 9 Jun 2003 at 03:44 UTC by itamar » (Master)

dan: the benefit? Partially it's for flexibility (having all these servers in Python means doing random wacky stuff is easy), partly it's for fun (that's why we have a SSH server), partially it's for the eventual goal of having secure network programs (look ma, no buffer overflows).

The real reason though is NIH syndrome. As ianb pointed out, integrating random Python code with Twisted is hard, since it might block. This forces us to rewrite every potential piece of code we might need, from scratch, to fit our needs.

This doesn't explain why we have a documentation system that can generate prosper and magicpoint slides from HTML and LaTeX docs from HTML and HTML from HTML. Or the mouseman mouse and GPS receiver wire protocols using our serial support (I'm not kidding), and oooh look it's some sort of horrible database and a web templating system and why is this package called "twisted.spread.jelly". Oh, and there's the object serialization system that outputs python source code (just to make old school Lispers wince with horror). And then there's the time washort started writing a mini-lisp which would be used to output C code (why? macros!). Not in CVS anymore, but it's no suprise one of our developers is responsible for reality distorting code, which is of course quite evil.

I'm not even going to mention the business plans based on Pokey.

We may be crazy, but we're cool as hell (does your open source project have videos on the web of Tekken DDR on giant screens?)

Democracy == everyone listens to _me_!!!, posted 9 Jun 2003 at 04:54 UTC by tk » (Observer)

I am consistently impressed by the quality of the work the Twisted team does, both in terms of the software they produce, and, particularly, the rigour of their approach, and I thought they deserved a little bit of praise outside its own community.

Mr. Cantanker... have you actually used Twisted? Or are you just basing your rave review on what other rave reviewers have written about Twisted?

This comments mechanism allows for points to be clarified and technical details to be fleshed out: exactly what has happened here.

Sounds like a handy excuse for an article writer to make lots of blunders and then wait for other people to correct his mistakes......

There is simply no need to abuse an author because his article does not conform to someone's idea of what is "interesting".

...and this sounds like an excuse that someone might use if he/she just wants to write random garbage.

Are not quality and usefulness interesting to software engineers involved with higher-level design and project direction? Or are they just too fuzzy for the likes of dan and tk to appreciate?

Precisely because quality and usefulness aren't fuzzy terms, that we absolutely insist that there be some technical content.

These comments seem to me the perfect place for the more technical discussions. Remember there are many people with a wide range of abilities who use this forum, and a lighter article may mean more readers will find it accessible.

Looks like the article's meant for people with lower "abilities" who nevertheless are "involved with higher-level design and project direction". How many such people are on Advogato? Hands up please......

(I count no fewer than four SCO articles on the front page, is that technical content?)

Well, at least one is -- it deals with legal technicalities. It doesn't just scream "SCO sux0rs, Linux r0x0rs" etc. etc. etc.

my two cents, posted 9 Jun 2003 at 05:34 UTC by dalke » (Journeyer)

I'm someone who's heard about Twisted for a while, this year started looking into it, and started using it over the last month or so.

I have also found it to be overhyped. There's quite a few useful chunks in it, but there most of the code I've looked at needs polishing, for lack of a better work. I've pointed out problems with the logging system (which went into place during an import of Twisted), the web server can fail to report anything when there's an error (lack of proper error-handling) and there's a simple denial-of-service attack possible against the server which I just reported this evening, and last week I found an easily triggered bug in its MS Windows support when connecting to a remote machine.

I agree that all the bugs I've reported are in the edge cases, that is, they don't occur if everything is used correctly (no one attacks the server, people don't mistype host names, etc). I also agree that they can be fixed rather easily. That's why I called it polish. But that's also why I consider it to be overhyped.

I also found the wxWindows support lacking despite claims made in the documenation. (That's been addressed with help from the mailing list.)

Regarding the various coments so far. "complete test package". When I tried one of the regression tests, it failed. I since reported it and it was fixed, but it means that people aren't always running all the regression tests. (Admittedly, it wasn't in the standard unit tests, but it was a failing test in released code.)

Twisted does not come with SOAP built-in. It depends on the SOAPpy external package. Others, like the FTP server "need a rewrite". Announcements, like the one which started this thread, make no comment about the various limitations to Twisted. Those limitations do exist, so the announcements get interpreted as overhype.

tk: First off, why is Twisted a framework, not a library?
Because Twisted wants to take over the world and doesn't easily acknowledge that other circumstances exist. Eg, I wanted to use Twisted as part of an Excel add-in with wxWindows as a GUI, which means Twisted cannot be in charge of the app. (Luckily, with a bit of tweaking it can be made to work as library.)

glyph: It's been running twistedmatrix.com literally for years now with no serious problems
And how many serious attempts at trying to break it? I showed one way to give a DoS against it. What other ones exist? Those might not be serious for you, because you can fix it easily, but suppose I sell a product based on Twisted to one of my non-programmer customers. Trying to diagnose and fix that problem remotely would be serious.

dan: And there's still nothing in there about why your ... Web server ... is any better than ... apache I'm using Twisted because my web architecture uses a semi-ReST style architecture, and Twisted makes it easier to manipulate how the server responds to different URIs. I also want to add dynamically generated WebDAV views of the URI/objects, and that would be hard under Apache. I think Apache is still more secure, but as I'm using twisted on an intranet it's okay. I also like its support for persistent state even across server restarts, but I would rather use ZODB for that, and not loading/dumping to pickle files.

moshez: You could have, for example, gotten more papers about Twisted than you have time to read
Follow that link, drill down to any of the CVS leaves, and try the "download" or "as test" links. It'll generate messages like "/users/jh.twistd/viewcvs/cgi/viewcvs.cgi/doc/historic/2003/pycon/pb/pb.html: unknown location". Other links do work, but those are the two links I always use, so I didn't try them until after a few failures. There's also no link from the main twistedmatrix.com site to those pages, at least that I can find. (There might be, but definitely not on the /documents/ page.)

ianb: I agree that async programming is a strange viewpoint. However, it isn't that different than CGI programming which is also async. And I've also been doing too much SAX programming :). BTW, Twisted does support threaded programming using the same framework, which means you can spin off long running tasks and not worry about blocking the rest of the system. This is what Twisted Enterprised does when doing a SQL query.

Andrew Dalke kicks ass, posted 9 Jun 2003 at 05:58 UTC by itamar » (Master)

He writes wonderful bug reports.

Two comments - persistent app state (TAPs) are for *configuration*, and in my opinion simple configuration only. You are quite right that they shouldn't be used for application data, and they aren't intended for that.

We do run all the tests, and automatically on checkin - just not on Windows. Most of us (including me) use Debian as main platform.

Oddly enough, Windows support *is* a good reason to use Twisted. We've found and worked around most of the oddities in the way it does select() and error handling for failed connections and what not by this point (I'm down to only two or three real bugs reported by the test suite on windows, hopefully none by next release).

Next time anyone writes their own async network support in Python "because it's easy", I will simply start counting the ways their code will fail on Windows (1. you know how UNPv1 says exception list in select() is for OOB data? on windows that's how you detect connect errors. 2. reading from socket will return EWOULDBLOCK even if connection failed 3. SO_REUSEADDR means you can bind to same TCP port multiple times, and there may be a 4 or even a 5 I forget about).

state machines, posted 9 Jun 2003 at 13:04 UTC by lkcl » (Master)

twisted requires that you write code as a state machine.

writing code as a state machine is pretty much standard practice. if you don't understand how to do a state machine (e.g. as pointed out already: cgi scripts have to do state machines) then you really shouldn't be doing any programming at all.

it's the _extent_ to which you do the state machine that's the issue (with twisted it's mandatory), yes?

but then again, _any_ client/server-based system is a state machine! apache modules; the SMB protocol; NFS.

about the only client/server systems that really don't "look" like they are state machines are RPC systems such as DCE/RPC - and even then there are two "modes" (context-based programming and stateless connections).

Windows 3.0/1/FWG forced people to write code as a state machine with message passing - no multitasking.

the "next level" of evolution for twisted is of course to have at the back-end a multi-threaded approach where several "states" can be entertained at once.

this would be the simplest work-around to the problems of blocking.

Testing...?, posted 9 Jun 2003 at 13:54 UTC by tk » (Observer)

I agree that all the bugs I've reported are in the edge cases, that is, they don't occur if everything is used correctly (no one attacks the server, people don't mistype host names, etc).

This is strange, considering that Twisted is supposed to have a test suite. During software engineering courses, I was taught that a good test suite is one that breaks a program. That is, a test suite checks not only a program's response to proper input, but also its response to improper input (robustness).

Re: state machines, posted 9 Jun 2003 at 14:09 UTC by mpr » (Journeyer)

... don't have to look that way. Check out the State Threads library: it does away with the synchronization issues usually associated with threads, without making your code any less straightforward.

thread-states and tests, posted 9 Jun 2003 at 15:03 UTC by itamar » (Master)

Sound similar to other systems I've seen, e.g. a couple built using continuations for Python, which turn the traditional event dispatching system inside out.

Another alterantive is CSP-style threads (I'm not sure I have an accurate definition of this, so I'm not going to expand, but Google should help).

In Python at least you can't really do either efficiently and portably on a plain Python system (you need Stackless). You can fake the first way though, up to a point, using generators, and people have done so on top of Twisted.

Also take a look at Mozart Oz.

tk: tests never catch everything, there's always more bugs, duh. And so we never have enough and always need to write more tests, again, duh. I'll be a lot happier when Twisted has 5 times as many tests as it has now. But one of the bugs Andrew Dalke mentioned was caught, just never fixed since it only happened on Windows and we never got around to it (fixed in CVS now). Have you ever written a test suite in your life, or is it something you feel you are expert on since you learned about it in school?

Re: thread-states and tests, posted 9 Jun 2003 at 17:14 UTC by tk » (Observer)

itamar: Well, I've written test cases both for academic projects and a (now defunct) personal project of my own. Of course, this doesn't mean that I'm a great guru and the Twisted developers are morons, but when I hear dalke give general remarks like "most of the code I've looked at needs polishing", I can't help wondering......

Dalke is quite right, posted 9 Jun 2003 at 17:48 UTC by itamar » (Master)

Twisted is indeed lacking polish. Zope 3, as another similar-scale Python project, has about 3 times as many tests as Twisted, maybe more (measured by tests/lineofcode.) It's a pretty new project, after all.

The reliability varies by parts of the code and platform. TCP networking on Linux is *good*. In next release it will likewise be quite good on Windows, and it probably is on most Unixish platforms (but there are always small edge cases, e.g. a specific errno I had to handle for FreeBSD to correctly detect reason for connection failure). HTTP server is pretty good if what you want is a Python HTTP server, but it certainly needs work in certain areas. And so on.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page