Older blog entries for oubiwann (starting at number 186)

21 May 2008 »

darkstat, For the Win

This is a quick self-response to my tweet to the lazyweb (is it still a tweet when it's Pownce and not Twitter?) today. I couldn't remember the name of a really handy network monitoring tool I used to use. It was similar to ntop but used a fraction of the resources and had a very limited yet perfectly satisfactory feature set. I've been having some crazy network utilization weirdness at the office lately, and I've wanted to peek at some trends without setting up NetFlow for my router or messing with ntop.

The answer was darkstat. It was my own memory that eventually came to the rescue, not Keyword Roulette on Google. Version 3.x is out and available for Mac OS X 10.5 via the latest MacPorts version (1.6).

This is all I needed to get it running:

sudo port selfupdate
sudo port install darkstat
sudo /opt/local/sbin/darkstat --debug -i en0 -l 192.168.4.0/255.255.255.0

Then, I just had to hit localhost:667...

I don't know what's up with the Google Juice for this guy's page, but it took me forever to find! I was searching for all the keywords like "ntop" (which was mentioned on his site at one point, I think), "network", "dark", "lightweight", "monitoring", etc. You get the picture. Hopefully this blog post will help when others are looking for it, too.

Syndicated 2008-05-20 22:15:00 (Updated 2008-05-20 22:22:44) from Duncan McGreggor

20 May 2008 (updated 12 Aug 2008 at 22:07 UTC) »

Required Reading: Ultra Large-Scale Systems

At Divmod, we're always talking about the future of computing, software, and the network. This usually focuses on our work with Twisted or the Divmod platform. But we have also spent considerable time assessing research in the area of what is called "ultra large-scale systems." Our primary business interest with this revolves around development, deployment, and management. However, there is a great deal of work that needs to be done to make ULS systems a reality.

I have a series of blog posts planned to discuss ULS systems in the areas where I have a vested interest. Despite the fact that a popular study on the matter was funded by the U.S. Department of Defense (conducted by the Carnegie Mellon Software Engineering Institute), I will not be discussing this technology in the context of war efforts nor national defense. Instead, I will be engaging in this discussion within the context of the medical/health services field, per the example given by Richard Gabriel in his presentation (PDF) to the Chinese University of Hong Kong.

This is especially pertinent today, as Google prepares for its Google Health announcement. Even though there are by some estimates (optimistic ones, in my view) 25 years of research ahead of us, it is generally agreed that ULS systems will consist aggregate sub-systems, built incrementally over time. I don't believe anyone is under the illusion that Google does not want to produce the first ULS system in history, and they are making rapid progress towards this goal. Google Health brings this point home very clearly, in the context of the research that has been done in this area.

The world is rapidly changing. The most important issues in technology and business are not who is going to create the next catchy "Web 2.0" application, or what mega corp games are being played in the Great Silicon Valley Soap Opera. The important issues are how the systems of the next 100-200 years are being built now, who they are being built by, and who as access to that technology.

We are a powerful and creative community. We are concerned about distributing power to the people, education, privacy, and freedom -- open source is built upon these principles. If we want those who are building ULS systems to build them fairly and with our concerns in mind, then we must get involved now. We must start building the necessary tools.

As for the reading, a couple of these were linked to in this blog post. Here is a list that should get your brain revved up and ready to roll (these are all PDF files):

I will have more specifics about the sorts of things we're exploring in future blog posts, but until then -- don't get left behind!

Update: Allen Short just pointed out some additional material that predates the Carnegie Mellon research: the Agoric Papers, co-authored by Mark Miller who is a co-creator of the E programming language.

Update 2: Glyph Lefkowitz just sent me a link to the Big Ball of Mud that discusses similar concepts.

Technorati Tags: technology, business, community , open source, uls

Syndicated 2008-05-19 19:58:00 (Updated 2008-08-12 17:53:56) from Duncan McGreggor

18 May 2008 (updated 1 Aug 2008 at 22:03 UTC) »

The Twisted Think Tank

I'm pleased to announce that we've got a new Divmod site up! We're still making tweaks, but it's ready for public viewing, and open for business.

This change currently doesn't affect our subscriber services... but it will, very shortly :-) JP's working on that now.

Anyone who knows us, knows that we know Twisted. We really know it. And how could we not, with Twisted superheroes like Glyph and JP? We've been solving very interesting problems for the past couple years, and other companies have availed themselves of this expertise. We're no longer trying to hide from our destiny as "the Twisted company."

We've found that providing specialized consulting services has not detracted from our core competency as software developers, but has rather done quite the reverse: provided a great deal of insight and clarity. The two activities have established a complementary feedback mechanism for growth and invention.

Syndicated 2008-05-16 18:32:00 (Updated 2008-08-01 16:37:47) from Duncan McGreggor

18 May 2008 (updated 1 Aug 2008 at 22:03 UTC) »

App Engine Haiku

I've been playing with Google App Engine a bit more tonight, and I've come to some additional conclusions:

For the best results, you gotta drink the whole pitcher of koolaide; and
It's really quite tasty koolaide.

One of the things I really wanted to test was imaging. I use Flickr far more than Picasa, but Flickr's authentication API is kind of a pain in the ass (especially when used with App Engine). Google auth is (for obvious reasons) easier to work with. This, of course, led me to explore Picasa. To my surprise, it was delightful to use. I chopped up gdata for only the parts needed to support Picasa and was uploading images within mere minutes. And not only was the effort minimal, but the performance was outstanding.

One of the things that makes application development a more efficient process is having a unified platform on which to work. This is why working with the Divmod platform is so nice when I do extensive Twisted application development: all the infrastructure is already built and ready for me to use. Today, I got to really taste that same experience with the Google platform. And I liked it.

I did notice one interesting psychological side effect of working with the file count limitation: I inadvertently treated it like a game. Not unlike the rules in poetry, it provided a structure and bounds within which I was forced to operate, adjust for, and test my creativity against. I rather enjoyed this microgame, and that was a surprise :-)

Syndicated 2008-05-16 06:19:00 (Updated 2008-08-01 16:36:41) from Duncan McGreggor

14 May 2008 (updated 1 Aug 2008 at 22:03 UTC) »

Founding Sponsor Opportunity Closing

Last month we announced that there were only 30 days left to become a Founding Sponsor of the Twisted Software Foundation. There are only two days left, so get your donate on!

We've had some really amazing sponsor conversations during this process, and we're excited about all the new Twisted stories that can be told. We've got enough material for 3 years of Twisted Shows, so I expect you'll be hearing more about how folks are using Twisted in the coming months.

In particular, I want to give a shout out to two different organizations.

Zenoss recently published a press release about their sponsorship, exposing Twisted to an "enterprise" audience that may not have heard of it yet. From the release:

"Twisted provides the foundation for Zenoss' scalable, agent-less collectors. It allows Zenoss daemons to talk to hundreds of enterprise systems simultaneously, with low overheads, using standard protocols like SNMP. In addition, Twisted provides the internal communications between distributed collectors."

United Business Media (formerly known as CMP) has also donated to Twisted, becoming a recent Founding Sponsor.

This was a rather unexpected bit of good fortune... but it gets better: they hadn't heard of Twisted before, and after reading up on it, they were so impressed that they immediately agreed to have three of their research and publication organizations become founding sponsors. They were amazed at the sophistication and power that the Twisted framework provides to a community of developers who are creating the future trends in software. I expect we'll be hearing more from them in the future ;-)

Update: The three UBM companies which sponsored the TSF are Contentinople, Internet Evolution, and Light Reading.

Syndicated 2008-05-13 20:29:00 (Updated 2008-08-01 16:35:38) from Duncan McGreggor

2 May 2008 (updated 1 Aug 2008 at 22:03 UTC) »

Twisting the Planet

As Steve blogged the other day, we've been jamming on some Twisted lately. But it's not the kind of thing you usually hear from us. We're not doing something esoteric and mind-blowing. We're doing something much harder: working out how to bring Twisted to the masses.

The motivation for this is philanthropic: we believe in Twisted's goodness :-) As Allen Short paraphrased on IRC the other day after listening to MIT entrepreneur Raffi Krikorian "it sounds like he's saying Twisted makes you smarter." Humor aside, Allen is right. Twisted does make you smarter: with increased familiarity and experience, you start thinking outside the box. Twisted helps you become a more creative problem solver.

In particular, we're reviewing the "Teach Me Twisted" open space session we had at PyCon. A bunch of you showed up for it, and the energy in that room was just phenomenal. 30 minutes after the session, people were still talking excitedly about what they were learning or how they were using Twisted or just sharing their love for the code :-)

For those of you that missed it, Steve Holden was the headliner while Alex Martelli played impromptu co-star. The humor and enthusiasm from these two was just incredible. Glyph, Itamar, and Chris played educators while JP, Zooko and I handled one-on-one questions in the audience. There were more players, but you get the point: it was a highly dynamic, lively and fun experience. Folks were so jazzed that conversations that night lasted long into the wee hours of the morning.

After almost two months' worth of post-PyCon follow-up, we're finally getting around to comparing notes. My biggest concern is for the absolute new-comer and the lack of intuitive and useful metaphors that would help aspiring Twisted users grasp the event-driven concepts of our code quickly. Steve and I are both interested in establishing a Proper and Good motivation for using Twisted. My girlfriend, who has a Masters in anthropology, was also there. Thanks to her insight and background, she has a completely different perspective of the community (and the new-comer dynamic at the session that night) and has some completely unique ideas for crafting a new generation of tutorial materials.

We're just getting started, but it's quite exciting. We expect to have more thoughts to share on the matter... in the form of materials as well as potential news items.

One last parting thought: despite the rumors and well-earned reputation to the contrary, Twisted coders are not exclusionists: everyone's invited to the party. We're just trying to make it easier to get there :-)

Syndicated 2008-05-01 23:42:00 (Updated 2008-08-01 16:35:10) from Duncan McGreggor

24 Apr 2008 (updated 1 Aug 2008 at 22:03 UTC) »

Synthesis Studios Twisted Interview

This is a re-blog, in case some of you aren't subscribed to http://labs.twistedmatrix.com/ -- we've got a new episode of the Twisted Show up, and it was a excellent interview with Synthesis Studios. Go check it out!

Syndicated 2008-04-24 05:15:00 (Updated 2008-08-01 16:40:00) from Duncan McGreggor

15 Apr 2008 (updated 16 Apr 2008 at 07:05 UTC) »

TSF Founding Sponsor Round

Earlier today, I posted to the Twisted mail list and to our Labs blog about the deadline for the Founding Sponsors (individuals and companies/organizations) for the Twisted Software Foundation, so I thought now would be a good time to give a quick status report :-)

Things are going well. We're still working with 4 major donors (some of whom are surprising!), with an additional 3 outliers who may or may not donate. We're hoping to have all that confirmed and settled within the next two weeks. For the full list of current sponsors, please see the Founding Sponsors page.

So what does all this mean? It means that Twisted has a continually-increasing chance of meeting your needs and exceeding your expectations :-)

Sponsors will have access to a private email list that is managed by the TSF, and this will be the primary forum where desired features and issues to be addressed will be discussed. The donations help us address resource issues and the collective voice of the sponsors will help provide a focus on important topics.

Not only do the current sponsors have their logos+links on the front page, but Founding Sponsors also get this on a dedicated page on the Twisted site in perpetuity. We've got some amazing Google Juice (search), so this works out well for all. We've also been approached by sponsors who are using it as a means of recruiting Python and Twisted talent in their shops. There are all sorts of creative ways that this can be of benefit :-)

If you haven't stopped by the TSF pages, give them a look and see if it's something you or your organization could be interested in.

Update: Thanks to feedback from Grig Gheorghiu, we now have two domains that direct to the TSF page:

Be aware that the DNS for those may still be propagating when you click on them :-)

Technorati Tags: tsf, twisted

Syndicated 2008-04-15 21:42:00 (Updated 2008-04-16 06:21:21) from Duncan McGreggor

10 Apr 2008 (updated 10 Apr 2008 at 05:05 UTC) »

Twisted on Nexenta/OpenSolaris

A few hours ago, I had the chance to install Nexenta/OpenSolaris in Parallels. The install was pretty straight-forward and quick. Gnome isn't on the .iso, but this was easily addressed with a/etc/apt/sources.list update, and an apt-get (a ~450MB dependency download and install). With Gnome was up and running, I was amazed at its responsiveness: Gnome on Nexenta seems to be much snappier than Ubuntu 7.10. This is the first time I've seen something I could use instead of Ubuntu, and that's saying a lot.

Next up was getting the Twisted and Divmod code installed. This required the following additional package installs:

sudo apt-get install python2.4-zopeinterface
sudo apt-get install python2.4-profiler
sudo apt-get install python2.4-pyopenssl
sudo apt-get install python2.4-crypto

With that done, I ran trial twisted, and watched the tests zip by. The end result? Only 1 failure and 2 errors; that's a pretty significant improvement over Twisted in Solaris 10. The failure was actually a little bit weird: the test can't find /dev/tty, however the device does exist (and I can open it from the python prompt). The two errors came from the UDP "multi listen" test, and were are result of the test timing out.

If I can get those two tests sorted out, I'll start testing the Divmod code. If all goes well, this could very well end up being my new development platform.

Technorati Tags: divmod, gnome, python, solaris, twisted

Syndicated 2008-04-10 02:48:00 (Updated 2008-04-10 04:30:30) from Duncan McGreggor

8 Apr 2008 (updated 10 Apr 2008 at 05:05 UTC) »

The Problem with and Solution to Google's App Engine

I know everyone is all aglow with the new web development offering from Google, but let me do the unpopular thing and put some things into perspective: there are limitations.

In fact, the limitations that exist will prevent me from using App Engine with all of my projects, save one (that one being a very simple web site). First, the limitations that prevent me from using App Engine (from one of their FAQs):

Sockets are disabled with Google App Engine
The system does not allow you to invoke subprocesses, as a result some os module methods are disabled
Threading is not available

This means that I can't write a deferred wrapper for their data layer, I can't use Twisted for such things as XML-RPC or AMP-based communications, and I can't use an async templating system (like Nevow). I'm stuck with CGI and blocking code. And for all but the simplest projects, that's a big "No Thank You" from me.

This doesn't mean that I won't use it -- I will. I have one project that this will be perfect for... but it's for someone else, not me.

However, these limitations are actually good news :-) Here's the silver lining:

As Glyph as alluded to in his recent blog post (and in our tweets), we've recently completed a massive week-long BizDev Divmod sprint in Boston. One of the results of this is based on community feedback we've had over the last year, and which culminated at PyCon 2008 in Chicago with multiple requests for particular services from The Twisted Company. That result is a set of tools, features, and management options folks will be able to use with our software (app server, smart object db, network services, etc.). People really want to start using our stuff in cloud/grid computing environments. They need support for multiple and diverse network services, inter-store communications, massive deployments, etc. Two months before PyCon, we started working on tickets to support this, and we're making excellent progress toward providing the requested features.

We're still unclear as to which parts of this will be open source, as that will be driven by a combination of business and community demand. Regardless, Google's lack of support for this stuff has (for now) left the field wide open for us. And that, folks, is a big "Thank You Google!" :-)

Technorati Tags: business, community, divmod, google, grid, mantissa, networking, nevow, cloud, pycon, python, services, software

Syndicated 2008-04-08 19:56:00 (Updated 2008-04-10 04:36:41) from Duncan McGreggor

177 older entries...

Jun	JUL	Aug
	01
2016	2017	2018