Older blog entries for Nafai77 (starting at number 24)

Handy script to search for and install Debian packages

I often find myself searching for a package with apt-cache search and then installing interesting matches with apt-get install. Whenever I find myself doing something over and over again, I want to automate it, to make it easier, faster, repeatable. To this end, I have come up with search_and_install.sh.

Some may say that I should just use synaptic or a similar tool. I find doing this is much quicker and easier to use for this use case.

To Use

To use, just run search_and_install.sh or search_and_install.sh <search terms>. If you leave off the search terms, a dialog box will prompt you for the search terms. If there are matches, a dialog box will come up listing the matching packages with their name and the short description returned by apt-cache search. Each will have a checkbox beside it. Check the packages you want to install and then hit OK. You will then be prompted for your password (or your root password, if you are on a non-Ubuntu system), and synaptic will run, downloading and installing the selected packages.

Caveat

Currently, packages you already have installed are shown in the list. In future versions, I'd like to filter those out.

Dependencies

bash, zenity, apt, synaptic, gksu

A standard Ubuntu install should have these, and these are easily installable on any other Debian-derived distro.

Download search_and_install.sh.

This and other scripts will be linked from my Random Code page. Hopefully something there will be useful.

Syndicated 2007-07-27 00:46:00 from Travis B. Hartwell / Software Craftsman

How to redirect stderr and stdout to a file plus display at the same time

The biggest search term that has brought people to my blog this month has been "redirecting stderr". I realize that I probably should expound more on Redirecting stderr and stdout to a file plus displaying them so that it makes more sense.

The original example script:

  #!/bin/bash

OUTPUT_LOG=output.log
OUTPUT_PIPE=output.pipe

if [ ! -e $OUTPUT_PIPE ]; then
    mkfifo $OUTPUT_PIPE
fi

if [ -e $OUTPUT_LOG ]; then
    rm $OUTPUT_LOG
fi

exec 3>&1 4>&2
tee $OUTPUT_LOG < $OUTPUT_PIPE >&3 &
tpid=$!
exec > $OUTPUT_PIPE 2>&1

echo "This is on standard out"
echo "This is on standard err" >&2

exec 1>&3 3>&- 2>&4 4>&-
wait $tpid

rm $OUTPUT_PIPE

The Explanation

First,

  if [ ! -e $OUTPUT_PIPE ]; then
    mkfifo $OUTPUT_PIPE
fi

I needed to make a named pipe -- otherwise known as a FIFO (First In First Out) -- to provide means for the shell and tee to share the output. You're probably familiar with pipes, such as ls -l | grep myfile. Named pipes are merely a way for you to do the same thing via a "file" in the filesystem. That same command can actually be done this way:

  mkfifo tmpfifo && (ls -l > tmpfifo &) && grep myfile < tmpfifo

Yeah, it's a little messy for that simple command. But it can be used in cases where a simple pipe cannot be used.

Next,

  exec 3>&1 4>&2

This saves the file descriptors for stdout (file descriptor 1) and stderr (file descriptor 2) to file descriptors 3 and 4.

Then,

  tee $OUTPUT_LOG < $OUTPUT_PIPE >&3 &

tee will, to quote from the manpage tee(1): "read from standard input and write to standard output and files". So, let's break up this line into parts to describe what's going on:

  • tee $OUTPUT_LOG: tee will take from stdin and write to the file named in the variable $OUTPUT_LOG.
  • < $OUTPUT_PIPE: tee's stdin will be redirected from the named pipe named in the variable $OUTPUT_PIPE
  • >&3: redirect tee's stdout to file descriptor 3, which is the stdout of the shell script
  • &: start tee as a background process of the shell

Next,

  tpid=$!

This line saves the process id of the tee process to the variable tpid.

Then,

  exec > $OUTPUT_PIPE 2>&1

In parts:

  • exec > $OUTPUT_PIPE: redirects the stdout of the script to the named pipe named in the variable $OUTPUT_PIPE.
  • 2>&1: redirects stderr to stdout so both stderr and stdout are redirected to the named pipe.

Then comes whatever you need your script to do. I included a few echo statements to show that both stdout and stderr are shown both in the terminal and in the log file.

After the meat of the script:

  exec 1>&3 3>&- 2>&4 4>&-

  • 1>&3 and 2>&4 restore the original file descriptors for stdout and stderr from file descriptors 3 and 4.
  • 3>&~ 4>&~: now that they are not needed, close file descriptors 3 and 4

Then,

  wait $tpid

This "pauses" the script until the tee process (whose process id is saved in the variable $tpid) exits. It will exit because its stdin (the output from the named pipe) has been closed. The named pipe closed the stdin because its input (file descriptors 1) was closed when the file descriptor stored in descriptor 3 was restored back to the original stdin.

Finally,

  rm $OUTPUT_PIPE

remove the named pipe, because it is not needed any more.

Addtional Information

For additional information, Chapter 19. I/O Redirection and Appendix E. A Detailed Introduction to I/O and I/O Redirection of the Advanced Bash-Scripting Guide are very useful in understanding these concepts. Also, a good introduction to named pipes can be found in this Linux Journal article.

I hope this helped make things a little bit clearer.

Syndicated 2007-07-27 00:46:00 from Travis B. Hartwell / Software Craftsman

How to host Pyblosxom as a WSGI application with Twisted

I recently switched to PyBlosxom for my blog software. I have been using Twisted to host my website since I had my own server and wished to continue to do so if possible.

Dependencies

  • Twisted Web 2: Twisted Web2 has built-in support for WSGI, which is used to connect to PyBlosxom. The 0.2.0 release should work fine, though I'm running off of Twisted svn trunk.

    Note: The stated requirement to download twisted_wsgi.py found in wsgi_app.py in the Pyblosxom distribution only applies to Twisted Web 1, not Twisted Web 2, as Twisted Web 2 has WSGI support built in.

  • Twisted: If you chose to download the release tarball of Twisted Web2, you'll also have to download the 2.4.0 release. If, however, you have checked it out from subversion, you'll have both Twisted Web2 and Twisted. Refer to the Twisted site for more details.

  • PyBlosxom: The latest release should be fine, but note that I'm also running off of PyBlosxom SVN Trunk.

General Setup

  1. Get all of the above modules set up. The easiest way is to refer to the INSTALL file in each of the downloaded or checked out distributions.
  2. Create a directory for all of your configuration and run-time files. For example, I use $HOME/bin.
  3. Copy web/wsgi_app.py from the PyBlosxom distribution in the directory.
  4. Copy the config.py PyBlosxom configuration file into the directory as well, making any customizations that are needed.

Twisted Configuration

I use a .tac file for my Twisted server configuration. It is simply a set of Python code required to set up the server and its parameters. Here is a minimum working configuration, with no Twisted logging enabled:

   from twisted.application import service, strports
 from twisted.web2 import server, channel, wsgi
 from twisted.web2.channel import http
 from wsgi_app import application as pyblosxom_application

 pyBlosxomResource = wsgi.WSGIResource(pyblosxom_application)

 site = server.Site(pyBlosxomResource)
 application = service.Application('web')
 s = strports.service('tcp:80', channel.HTTPFactory(site))
 s.setServiceParent(application)

I could probably make it a bit cleaner, but this is a pared down version of the more complex working version that hosts my sites. You can download and save blog.tac in the directory created above. Be sure to change the port number from tcp:80 to tcp:<custom port number> if you want the server to listen on a different port.

Startup Script

I also wrote a quick script to make it easy to start the blog server. You'll have to modify the pathnames appropriately to your particular setup. Here is the script:

   #!/bin/sh

 cd ~/bin/
 export PYTHONPATH=/home/nafai/code/Twisted:/home/nafai/code/pyblosxom-new/pyblosxom:$PYTHONPATH
 python2.4 /home/nafai/code/Twisted/bin/twistd -y /home/nafai/bin/blog.tac --pidfile=blog.pid --logfile=blog.log

Download and save startblog.sh in the directory created above. Make it executable by doing chmod +x startblog.sh.

Run it!

Now try running the blog service (for example, if the configuration files were saved in ~/bin:

   ~/bin/startblog.sh

You should be able to see your blog if you browse to http://localhost/ with your web browser. If you get a 503 or other error, check blog.log in the ~/bin directory for any tracebacks or errors. Also check the pyblosxom specific log if you have one configured in your config.py.

Caveat

You may run into a bug that I've seen, but haven't taken the time to track down where it is coming from. I am getting this traceback fairly often in my blog.log file, though everything else seems to run normal:

   2006/08/24 22:22 MDT [HTTPChannel,39,127.0.0.1] Traceback (most recent call last):
 2006/08/24 22:22 MDT [HTTPChannel,39,127.0.0.1]   File "logging/__init__.py", line 737, in emit
 2006/08/24 22:22 MDT [HTTPChannel,39,127.0.0.1] ValueError: I/O operation on closed file

I should track this down sometime soon.

Congratulations!

Happy TwistedPyBlogging!

Syndicated 2007-07-27 00:46:00 from Travis B. Hartwell / Software Craftsman

New Languages

I've recently been feeling the desire to learn a new programming language. The last language I learned on my own (not directly for school or work) was Python, and that was 7 years ago! In that time, I've learned Common Lisp for my Artificial Intelligence class, Torque Script -- the scripting language for the Torque Game Engine -- for my last job, and Java for my latest job. I do feel some desire to get back to Lisp, but I'm also drawn to other languages.

Looking back, I think there were a few factors that lead me to learn and stick with Python:

  • Between a well-designed language and a great standard library, I was able to produce something that was useful within my first afternoon of learning Python. It's an entry for another day, but suffice to say, in that first afternoon I wrote the first iteration of a small utility that I used every day for 3 or 4 years.

  • A great community. I had some questions and I found #python (which was on irc.efnet.org at the time) and there were a lot of very helpful people there that helped me with my silly questions.

From that, some of my main interests in learning a language or program ultimately boil down to these two things:

  1. Is there sufficient power and ability to do something that is ultimately useful to me?
  2. Is there an active community behind and supporting the language or program?

That said, it sometimes goes against something Alan Perlis (from Perlisisms) said:

"A language that doesn't affect the way you think about programming, is not worth knowing."

In congruence with this, Eric Raymond, in How To Become A Hacker, said this:

LISP is worth learning for a different reason -- the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot.

I do love learning new things that make me look at problems differently. But, unfortunately, my laziness as a programmer has made me ultimately follow what Eric Raymond has said -- I learned Lisp, I got it, I loved it -- but I never use it.

So I'm conflicted. On one hand, I have projects that I want to do -- useful things I want to write for my use. On the other hand, I want to learn something new, something elegant. So there is that conflict between utility and beauty.

That said, I'm trying to start to learn something new. I'm looking at two languages now, each with an active, helpful, and intelligent community behind it: Haskell and Factor. Each has an active channel on the IRC network irc.freenode.net (#haskell and #concatenative respectively). I've started reading the tutorials and writing a little bit of code. Haskell is a pure functional language. Factor is stack-based. Neither paradigm is something that I've done anything extensively with, so there is a bit of a learning curve.

Now all I have to do is come up with something useful to write so I can meet my first criteria above. I'll probably be writing more about what I learn of each of these languages.

Syndicated 2007-07-27 00:46:00 from Travis B. Hartwell / Software Craftsman

New Laptop

A couple of weeks ago my wife Torie called me at work, telling me that the computer had locked up and it wouldn't even boot when she tried to reboot. I took it apart when I got home and discovered that the CPU fan had stopped and the CPU had fried. The computer was a freebie that a coworker of my father-in-law had given us, so it wasn't exactly new. Instead of trying to replace the CPU or CPU, motherboard, memory, power supply, and fan — as would have been required — Torie and I decided it was time to upgrade. I do a bit of contract work, Torie needs a computer for school work, and a million other reasons justify why we couldn't be without a computer.

We got in the car and went to Best Buy (a bit impulsive, in retrospect, but we needed a computer, and, besides, it was the only thing still open at 9 PM). After talking with a surprisingly knowledgeable and helpful staff, we picked out the Toshiba Satellite U205-S5034 laptop. Light, fast (Intel Core 2 Duo Processor), Intel chipset for video and wireless so it would work with Linux, great keyboard — it had all that we needed. I upgraded the RAM to 2 gig to make it a little more useful.

So we bought it. I downloaded the Ubuntu Edgy install disk as soon as we got home. I was shocked. Everything worked out of the box. Sound. Video. Accelerated 3-D Video. Wireless. The only thing left hardware-wise for me to tweak is the set up of the dual monitors. In Windows XP, you can use the external monitor port and use both the laptop LCD and an external LCD. Awesome. I just need to take the time to figure out how to do that in Ubuntu.

The only downside is that the old desktop system had SCSI for its main drive. I only have an IDE USB enclosure. I didn't have backups set up yet. So I haven't yet been able to get at our old data. Luckily, a coworker is going to dig out an old system of his for me to borrow. I just have to throw in the PCI SCSI controller and the drive and copy the data elsewhere. But it sure is a pain. One of my next projects is going to be getting a decent backup system working for us. Gotta do that soon.

Syndicated 2007-07-27 00:46:00 from Travis B. Hartwell / Software Craftsman

My Link Blog

One of my good friends and I are always pasting links back and forth to each other, "Have you seen this?" or "Have you read about this"? A quick search of my chat logs from last few months shows that I shared anywhere from 25 links to 65 links a month with him. It's probably safe to say that most of them come from things I've read via Google Reader.

I'm currently subscribed to 220 feeds. Not quite the 622 feeds Robert Scoble reads, but blogging isn't my day job. :)

I got the idea from Robert Scoble. He uses Google Reader's Feed Sharing feature for his link blog. Since I also am using Google Reader, I thought I would also share mine. Perhaps the things I find interesting will be of interest to others, especially those who don't take the time to read as much as I do.

Again, here's the links:

Syndicated 2007-07-27 00:46:00 from Travis B. Hartwell / Software Craftsman

My Link Blog

One of my good friends and I are always pasting links back and forth to each other, "Have you seen this?" or "Have you read about this"? A quick search of my chat logs from last few months shows that I shared anywhere from 25 links to 65 links a month with him. It's probably safe to say that most of them come from things I've read via Google Reader.

I'm currently subscribed to 220 feeds. Not quite the 622 feeds Robert Scoble reads, but blogging isn't my day job. :)

I got the idea from Robert Scoble. He uses Google Reader's Feed Sharing feature for his link blog. Since I also am using Google Reader, I thought I would also share mine. Perhaps the things I find interesting will be of interest to others, especially those who don't take the time to read as much as I do.

Again, here's the links:

Syndicated 2007-06-04 03:48:00 from Travis B. Hartwell / Software Craftsman

New Laptop

A couple of weeks ago my wife Torie called me at work, telling me that the computer had locked up and it wouldn't even boot when she tried to reboot. I took it apart when I got home and discovered that the CPU fan had stopped and the CPU had fried. The computer was a freebie that a coworker of my father-in-law had given us, so it wasn't exactly new. Instead of trying to replace the CPU or CPU, motherboard, memory, power supply, and fan — as would have been required — Torie and I decided it was time to upgrade. I do a bit of contract work, Torie needs a computer for school work, and a million other reasons justify why we couldn't be without a computer.

We got in the car and went to Best Buy (a bit impulsive, in retrospect, but we needed a computer, and, besides, it was the only thing still open at 9 PM). After talking with a surprisingly knowledgeable and helpful staff, we picked out the Toshiba Satellite U205-S5034 laptop. Light, fast (Intel Core 2 Duo Processor), Intel chipset for video and wireless so it would work with Linux, great keyboard — it had all that we needed. I upgraded the RAM to 2 gig to make it a little more useful.

So we bought it. I downloaded the Ubuntu Edgy install disk as soon as we got home. I was shocked. Everything worked out of the box. Sound. Video. Accelerated 3-D Video. Wireless. The only thing left hardware-wise for me to tweak is the set up of the dual monitors. In Windows XP, you can use the external monitor port and use both the laptop LCD and an external LCD. Awesome. I just need to take the time to figure out how to do that in Ubuntu.

The only downside is that the old desktop system had SCSI for its main drive. I only have an IDE USB enclosure. I didn't have backups set up yet. So I haven't yet been able to get at our old data. Luckily, a coworker is going to dig out an old system of his for me to borrow. I just have to throw in the PCI SCSI controller and the drive and copy the data elsewhere. But it sure is a pain. One of my next projects is going to be getting a decent backup system working for us. Gotta do that soon.

Syndicated 2007-01-20 05:30:00 from Travis B. Hartwell / Software Craftsman

New Languages

I've recently been feeling the desire to learn a new programming language. The last language I learned on my own (not directly for school or work) was Python, and that was 7 years ago! In that time, I've learned Common Lisp for my Artificial Intelligence class, Torque Script -- the scripting language for the Torque Game Engine -- for my last job, and Java for my latest job. I do feel some desire to get back to Lisp, but I'm also drawn to other languages.

Looking back, I think there were a few factors that lead me to learn and stick with Python:

  • Between a well-designed language and a great standard library, I was able to produce something that was useful within my first afternoon of learning Python. It's an entry for another day, but suffice to say, in that first afternoon I wrote the first iteration of a small utility that I used every day for 3 or 4 years.

  • A great community. I had some questions and I found #python (which was on irc.efnet.org at the time) and there were a lot of very helpful people there that helped me with my silly questions.

From that, some of my main interests in learning a language or program ultimately boil down to these two things:

  1. Is there sufficient power and ability to do something that is ultimately useful to me?
  2. Is there an active community behind and supporting the language or program?

That said, it sometimes goes against something Alan Perlis (from Perlisisms) said:

"A language that doesn't affect the way you think about programming, is not worth knowing."

In congruence with this, Eric Raymond, in How To Become A Hacker, said this:

LISP is worth learning for a different reason -- the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot.

I do love learning new things that make me look at problems differently. But, unfortunately, my laziness as a programmer has made me ultimately follow what Eric Raymond has said -- I learned Lisp, I got it, I loved it -- but I never use it.

So I'm conflicted. On one hand, I have projects that I want to do -- useful things I want to write for my use. On the other hand, I want to learn something new, something elegant. So there is that conflict between utility and beauty.

That said, I'm trying to start to learn something new. I'm looking at two languages now, each with an active, helpful, and intelligent community behind it: Haskell and Factor. Each has an active channel on the IRC network irc.freenode.net (#haskell and #concatenative respectively). I've started reading the tutorials and writing a little bit of code. Haskell is a pure functional language. Factor is stack-based. Neither paradigm is something that I've done anything extensively with, so there is a bit of a learning curve.

Now all I have to do is come up with something useful to write so I can meet my first criteria above. I'll probably be writing more about what I learn of each of these languages.

Syndicated 2006-12-02 18:44:00 from Travis B. Hartwell / Software Craftsman

How to host Pyblosxom as a WSGI application with Twisted

I recently switched to PyBlosxom for my blog software. I have been using Twisted to host my website since I had my own server and wished to continue to do so if possible.

Dependencies

  • Twisted Web 2: Twisted Web2 has built-in support for WSGI, which is used to connect to PyBlosxom. The 0.2.0 release should work fine, though I'm running off of Twisted svn trunk.

    Note: The stated requirement to download twisted_wsgi.py found in wsgi_app.py in the Pyblosxom distribution only applies to Twisted Web 1, not Twisted Web 2, as Twisted Web 2 has WSGI support built in.

  • Twisted: If you chose to download the release tarball of Twisted Web2, you'll also have to download the 2.4.0 release. If, however, you have checked it out from subversion, you'll have both Twisted Web2 and Twisted. Refer to the Twisted site for more details.

  • PyBlosxom: The latest release should be fine, but note that I'm also running off of PyBlosxom SVN Trunk.

General Setup

  1. Get all of the above modules set up. The easiest way is to refer to the INSTALL file in each of the downloaded or checked out distributions.
  2. Create a directory for all of your configuration and run-time files. For example, I use $HOME/bin.
  3. Copy web/wsgi_app.py from the PyBlosxom distribution in the directory.
  4. Copy the config.py PyBlosxom configuration file into the directory as well, making any customizations that are needed.

Twisted Configuration

I use a .tac file for my Twisted server configuration. It is simply a set of Python code required to set up the server and its parameters. Here is a minimum working configuration, with no Twisted logging enabled:

   from twisted.application import service, strports
 from twisted.web2 import server, channel, wsgi
 from twisted.web2.channel import http
 from wsgi_app import application as pyblosxom_application

 pyBlosxomResource = wsgi.WSGIResource(pyblosxom_application)

 site = server.Site(pyBlosxomResource)
 application = service.Application('web')
 s = strports.service('tcp:80', channel.HTTPFactory(site))
 s.setServiceParent(application)

I could probably make it a bit cleaner, but this is a pared down version of the more complex working version that hosts my sites. You can download and save blog.tac in the directory created above. Be sure to change the port number from tcp:80 to tcp:<custom port number> if you want the server to listen on a different port.

Startup Script

I also wrote a quick script to make it easy to start the blog server. You'll have to modify the pathnames appropriately to your particular setup. Here is the script:

   #!/bin/sh

 cd ~/bin/
 export PYTHONPATH=/home/nafai/code/Twisted:/home/nafai/code/pyblosxom-new/pyblosxom:$PYTHONPATH
 python2.4 /home/nafai/code/Twisted/bin/twistd -y /home/nafai/bin/blog.tac --pidfile=blog.pid --logfile=blog.log

Download and save startblog.sh in the directory created above. Make it executable by doing chmod +x startblog.sh.

Run it!

Now try running the blog service (for example, if the configuration files were saved in ~/bin:

   ~/bin/startblog.sh

You should be able to see your blog if you browse to http://localhost/ with your web browser. If you get a 503 or other error, check blog.log in the ~/bin directory for any tracebacks or errors. Also check the pyblosxom specific log if you have one configured in your config.py.

Caveat

You may run into a bug that I've seen, but haven't taken the time to track down where it is coming from. I am getting this traceback fairly often in my blog.log file, though everything else seems to run normal:

   2006/08/24 22:22 MDT [HTTPChannel,39,127.0.0.1] Traceback (most recent call last):
 2006/08/24 22:22 MDT [HTTPChannel,39,127.0.0.1]   File "logging/__init__.py", line 737, in emit
 2006/08/24 22:22 MDT [HTTPChannel,39,127.0.0.1] ValueError: I/O operation on closed file

I should track this down sometime soon.

Congratulations!

Happy TwistedPyBlogging!

Syndicated 2006-08-25 05:43:00 from Travis B. Hartwell / Software Craftsman

15 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!