Older blog entries for timj (starting at number 10)

Using mod_disk_cache with MediaWiki


MediaWiki is a pretty fast piece of software out of the box. It’s written in PHP and covers a lot of features, so it can’t serve pages in 0 time, but it’s reasonably well written and allows use of PHP accelerators or caches in most cases. Since it’s primarily developed for Wikipedia, it’s optimized for high performance deployments, caching support is available for Squid, Varnish and plain files.

For small scale use cases like private or intranet hosts, running MediaWiki uncached will work fine. But once it’s exposed to the Internet, regularly crawled and might receive links from other popular sites, serving only a handful of pages per second is quickly not enough. A very simple but effective measure to take in this scenario is the enabling of Apache’s mod_disk_cache.

Here’s a sample benchmark for the unoptimized case:

$ ab -kt3 http://testbit.eu/Sandbox
Time taken for tests:   3.33173 seconds
Total transferred:      301743 bytes
Requests per second:    6.26 [#/sec] (mean)
Time per request:       159.641 [ms] (mean)
Transfer rate:          96.93 [Kbytes/sec] received

Now we configure mod_disk_cache in apache2.conf:

CacheEnable   disk /
CacheRoot     /var/cache/apache2/mod_disk_cache/
And enable it in Apache:
$ a2enmod disk_cache
Enabling module disk_cache.
Run '/etc/init.d/apache2 restart' to activate new configuration!

This in itself is not enough to enable caching of MediaWiki pages however, this is due to some bits in the HTTP header information it’s sending:

$ wget -S --delete-after -nd http://testbit.eu/Sandbox
--2011-02-09 00:48:21--  http://testbit.eu/Sandbox
  HTTP/1.1 200 OK
  Date: Tue, 08 Feb 2011 23:48:21 GMT
  Vary: Accept-Encoding,Cookie
  Expires: Thu, 01 Jan 1970 00:00:00 GMT
  Cache-Control: private, must-revalidate, max-age=0
  Last-Modified: Tue, 08 Feb 2011 03:24:32 GMT
2011-02-09 00:48:21 (145 KB/s) - `Sandbox' saved [14984/14984]

The Expires: and Cache-Control: headers both prevent mod_disk_cache from caching the contents.

A small patch against MediaWiki-1.16 fixes that by removing Expires: and adding s-maxage to Cache-Control:, which allows caches to serve “stale” page versions which are only mildly outdated (a few seconds).

With the patch, the headers changed as follows:
$ wget -S --delete-after -nd http://testbit.eu/Sandbox
--01:03:03--  http://testbit.eu/Sandbox
 HTTP/1.1 200 OK
 Date: Wed, 09 Feb 2011 00:03:03 GMT
 Vary: Accept-Encoding,Cookie
 Cache-Control: s-maxage=3, must-revalidate, max-age=0
 Last-Modified: Tue, 08 Feb 2011 03:24:32 GMT
01:03:03 (386.21 MB/s) - `Sandbox' saved [14984/14984]

Upon inspection, there’s no Expires: header now and Cache-Control: adapted as described. Let’s now rerun the benchmark:

$ ab -kt3 http://testbit.eu/Sandbox
Time taken for tests:   3.5511 seconds
Total transferred:      38621189 bytes
Requests per second:    831.14 [#/sec] (mean)
Time per request:       1.203 [ms] (mean)
Transfer rate:          12548.95 [Kbytes/sec] received

That looks good, 831 requests instead of 6!

Utilizing mod_disk_cache with MediaWiki can easily speed up the number of possible requests per-second by more than a factor of one hundred for anonymous accesses. The caching behavior in the above patch can also be enabled for logged-in users with adding this setting to MediaWiki’s LocalSettings.php:

$wgCacheLoggedInUsers = true;

I hope this helps people out there to speed up your MediaWiki installation as well, happy tuning! ;)

flattr this!

Syndicated 2011-02-09 01:03:45 from Tim Janik

New Beast Website

Last week the Beast project went live with a new website that has been in the making since December:

The old website was several years old, adding or changing content was very cumbersome and bottlenecked on Stefan or me. All edits had to go into the source code repository, adding content pages meant editing web/Makefile.am and changing a menu entry required the entire site to be rebuilt and re-synced. Also beast.gtk.org went offline for several weeks due to hosting problems at UC Berkeley.
So in the last few weeks the Beast website has been gradually moved from beast.gtk.org to beast.testbit.eu and a different hosting service that has more resources available. In the last few years, I’ve gained experiences with Plone, Drupal, DokuWiki, Confluence, a beast-specific markup parser, Joomla, WordPress, etc. They all have their up and down sides, and while I prefer WordPress for my own blog, I’ve settled on MediaWiki for the new Beast website.
Running the new site entirely as a wiki makes the contents easily accessible for everyone willing to contribute and MediaWiki’s markup is what most people already know or are likely to learn in the future. MediaWiki must be the hardest tested collaborative editing tool available, turns out to be impressively feature rich compared to other Wiki engines, has a rich set of extensions, scripting facilities and due to Wikipedia weight a reliable maintenance future.
Much of the previously hand crafted code used for site generation and operation becomes obsolete with the migration, like the screenshot gallery PHP snippets. The entire build-from-source process can be eliminated, and running a dedicated Beast theme on MediaWiki allows editing of the menu structure in a wiki page.
Also MediaWiki allows running multiple front ends under different domains and with different themes on the same Wiki database, which allowed me to merge the Beast site and testbit.eu to reduce maintenance.
A small set of patches/extensions were used to tune MediaWiki for the site’s needs:
  • Enhancing the builtin search logic, so it automatically resorts to partial matches for empty result lists.
  • Adjusting Expires/Cache-Control headers to utilize mod_disk_cache – this increases the number of possible requests per-second by more than a factor of one hundred.
  • Adding support for [[local:/path/to/download/area]] URLs, to refer to downloadable files from within wiki pages.
It took a while to migrate contents gradually into MediaWiki format, as some files had to be migrated from a very old ErfurtWiki installation, some came from the source code repository and some were available in HTML only. Big Kudos to David Iberri, his online html2wiki converter (html2wiki on CPAN) has been a huge help in the process.
I hope the new site is well received, have fun with it!

flattr this!

Syndicated 2011-02-05 05:05:23 from Tim Janik

Got The Time?

flattr this!

Syndicated 2010-12-18 20:55:06 from Tim Janik

10.09.2010 BEAST/BSE release 0.7.2 available

BEAST/BSE and BSE-ALSA version 0.7.2 are available for download:


BEAST is a music composition and modular synthesis application. The “Bedevilled” portion of the names has no religious background, please refer to the About page for more details.

    Homepage:           http://beast.gtk.org/
    Downloads:          http://beast.gtk.org/beast-ftp/
    Feedback:           http://mail.gnome.org/mailman/listinfo/beast/

The 0.7.2 release provides new plugins and instruments, and a long list of bug fixes, improvements and translation updates.

TRANSLATORS: Please help us to improve the BEAST translations at Transifex.

Overview of Changes in BEAST/BSE 0.7.2:

  • Moved Beast/BSE to GNU LGPL, use AS-IS license for examples
  • Module changes and additions: ArtsCompressor – Relicensed to LGPL with permission from Matthias Kretz BseContribSampleAndHold – Relicensed to LGPL with permission from Artem Popov DavXTalStrings – Use deterministic random numbers for unit tests BseNoise – Improved random number generator
  • Switched to autogenerated ChangeLogs
  • Error bell can be muted in beast preferences dialog
  • Added multisample creation/editing command line tool: bsewavetool
  • Support adjustable volume, pitching and drum envelopes in .bsewave files
  • Added Retro Acoustic drum kit [Tim, Stefan]
  • New loadable Instruments/Effects:
    - BQS Bass Drum E8012 [Tim, Stefan]
    - BQS Slow Hum [Stefan, William DeVore]
    - FSM Fresh Water Bass instrument [Krzysztof Foltman]
    - FSM Growl Bass instrument [Krzysztof Foltman]
    - FSM Synth String Sweep [Krzysztof Foltman]
  • Added support for loading 32bit and 24bit PCM-format WAV files
  • Added support for gcc-4.4 and automake-1.10
  • Added support for guile-1.8, guile-1.6 remains as minimum requirement
  • Various fixes, improvements and much improved test coverage.
  • Bug fixes: #452604, #468229, #344388, #451086, #450724, #454121, #491552, #450490, #441936, #336766, #433431, #474332, #474244, #456879, #456408, #424897 [Tim Janik, Stefan Westerfeld]
  • Migrated translation support to use awk, sed and po/Makefile.am.
  • Updated German translation [Mario Blättermann]
  • Updated Italien translation [Michele Petrecca]
  • Updated Occitan translation [Yannig Marchegay]
  • Updated Brazilian Portugues translation [Leonardo Ferreira Fontenelle]
  • Updated British English translation [David Lodge]
  • Updated Spanish translation [Jorge Gonzalez]
  • Updated Slovenian translation [Andrej Znidarsic]
  • Updated Danish translation [Joe Hansen]
  • Updated French translation [Bruno Brouard]
  • Added Norwegian bokmal translation [Kjartan Maraas]
  • Added Ukrainian translation [Maxim V. Dziumanenko]

Overview of Changes in BSE-ALSA 0.7.2:

  • Fixes for automake-1.10 builds
  • Moved Beast/BSE to GNU LGPL

UPDATES: Updated translator instructions and uploaded a new release tarball with build fixes.

flattr this!

Syndicated 2010-09-10 01:28:56 from Tim Janik

09.09.2010 Request to support voting in GNOME Bugzilla

These days I often have a hard time to keep up with the tasks on my TODO lists, but I do manage to sneak in a spare hour here or there to look into code I authored sometime ago and that’s waiting for maintenance attention. For projects like Gtk+/GLib it’s incredibly hard to figure a good start and order for bug processing if time is sparse and the number of bugs is flooding you.

Other projects on the net use issue trackers that support user voting of individual requests, here are two examples:

Of course I do realize that we have priority and severity fields in GNOME Bugzilla, but those are for a different purpose than polling the public opinion on which bugs should be fixed best/next, or which bugs have the largest pain impact on our user base.

At least for me, a publicly open voting system for GNOME Bugs would be immensely useful to judge where it’s best to concentrate my development efforts.

flattr this!

Syndicated 2010-09-09 12:53:11 from Tim Janik

30.08.2010 Rapicorn release 10.08.0 available

Rapicorn 10.08.0 is available for download:


Rapicorn is an Experimental UI Toolkit. Most things in a toolkit implementation will benefit from proper application of modern technologies (e.g. pthreads, XCB, Cairo, compositing, IDL, XML notation, path evaluation, DSLs, unit tests, SVG). Rapicorn is developed on this base with the aim to significantly improve developer efficiency and user experience.

This release contains a multitude of new features like Python bindings, an automated test suite, myriads of bug fixes and small improvements.

    Homepage:           http://rapicorn.org/
    Downloads:          http://rapicorn.org/dist/rapicorn/
    Feedback:           http://rapicorn.org/mailman/listinfo/rapicorn-list

Updates in Rapicorn 10.08.0:

  • Introduced Cairo dependency.
  • Introduced C++ TR1 dependency (for shared_ptr, etc).
  • Added $RAPICORN environment variable to control logging and more.
  • Added PLIC, a pluggable IDL compiler.
  • Added Remote Object Programming Extension.
  • Added window and widget addressing functionality.
  • Allow signal disconnections by Id.
  • Added simple expression parser to evaluate property value assignments.
  • Added sample expression evaluator as: ui/tests/sinfextest –shell
  • Added XML UI file error reporting.
  • Changed packing properties into regular Item properties.
  • Merged h/v-scale and -align into Item pack properties.
  • Replaced hfill/vfill in Table by h/v-scale and -align properties.
  • Replaced *_attach in table by Item’s h/v-position and -span properties.
  • Got rid of the packer class entirely.
  • Added testing framework with macros and reference files.
  • Added ‘-x’ to rapidrun to auto-exit.
  • Added ‘–list’ to rapidrun to list gadgets from a GUI file.
  • Provide regular expression parser API in rapicorn-core.
  • Introduced a basic typed value system and model APIs.
  • Started list area/widget development.
  • Optimized resizing performance.
  • Cosmetic fixups in colors and gradients.
  • Implemented horizontal and vertical size groups.
  • Renamed string_to_cescape() and fixed string_to_cquote() quoting.
  • Generate internal C++ bindings (to IDL specifications).
  • Provide a main loop implementation in Python.
  • Generate external Python bindings.
  • Majorly extended test suite.
  • Optimized remote invocations and reference counting.
  • Started migrating interfaces from headers to IDL.
  • Hand optimized threading primitives and main loop handling.
  • Applied fixes for AMD64 compilation. [Stefan Westerfeld]
  • Updates to compile with GCC-4.4.
  • Linker optimizations to improve ELF relocations.
  • Loads of other cleanups and bug fixes.

flattr this!

Syndicated 2010-08-20 21:57:50 from Tim Janik

29.07.2010 Lanedo at GUADEC

Like every year, the entire Lanedo Crowd is currently attending Guadec. If you’re around as well, we can strongly recommend attending one of the talks we’re giving on:

Feel free to approach us for a chat or for handing over your CV. ;-)

flattr this!

Syndicated 2010-07-29 07:31:09 from Tim Janik

25.06.2010 Using distcheck with ccache + colorgcc

Waiting for unit compilations to finish during development, particularly with G++ and heavy optimizations can be quite time consuming. A run of make distcheck escalates the problem, because all files during a dist need to be rebuilt. Small errors triggered late in the dist build might require several time consuming restarts of the build process.

With ccache, a tool exists that can majorly speed up this process. I’ve been using it successfully for several years now. A related development aid is colorgcc which is a colorization wrapper for gcc. This also is under heavy use for my development. It turns out that some tweaks are needed to get it working together with ccache though.

Here’s how to use the tools in combination to speed up regular project builds and also distcheck:

  • Get and install ccache.
    Wrapping compiler invocations with ccache will cache recompilations of the same units. ccache needs a directory to store the cache contents; this defaults to ~/.ccache which might not be the best choice for network homes or encrypted homes. Using /tmp is also suboptimal since it is usually cleaned upon reboots. So I’d recommend to put this into your .bashrc:
    export CCACHE_DIR="/var/tmp/ccache-$USER"

    This will retain the compiler cache across reboots in /var/tmp.

  • Get and install colorgcc.
    Then some patching is in order, especially if you use multiple versions of gcc and need ccache integration. Based on colorgcc from Ubuntu-9.10, I’ve prepared a patch for this (it has been sent to Jamie Moyers some while ago) to be found here: http://testbit.eu/~timj/blogstuff/xyv-colorgcc.diff

    To briefly summarize its effects:

    • This allows colorgcc to be used through a bunch of links, e.g. by creating:
      ln -s colorgcc colorg++
      ln -s colorgcc colorgcc-4.4
      ln -s colorgcc colorg++-4.4

      Invoking the script through those links will cause it to strip the ‘^color’ prefix and invoke the respective compiler.

    • Using this patch, /usr/bin/ccache will be used to wrap the compiler invocation if it exists.
    • The patch also contains some UTF-8 quotation fixes needed for gcc-4.4 support.
    • Finally, the patch enables colorgcc to correctly deal with g++ error message that span multiple lines (e.g. template error messages).
  • Use colorgcc as a compiler replacement for project builds:
    rm -f config.cache
    ./autogen.sh CC=colorgcc-4.4 CXX=colorg++-4.4
    nice make all -j4
  • Use colorgcc as a compiler replacement for distcheck builds:
    CC=colorgcc CXX=colorg++ nice make all distcheck

Finally, you might want to tweak the colors of colorgcc’s output which can be adjusted in /etc/colorgcc/colorgccrc. FYI, here’s the color setup I prefer to use in a black terminal window:

  srcColor:             bold white
  introColor:           bold magenta
  warningFileNameColor: cyan
  warningNumberColor:   bold cyan
  warningMessageColor:  yellow
  errorFileNameColor:   cyan
  errorNumberColor:     bold cyan
  errorMessageColor:    bold yellow

UPDATE: Fixed ccache default dir thanks to anonymous.

flattr this!

Syndicated 2010-06-25 17:17:16 from Tim Janik

23.04.2010 Doxer release 10.04.0 available

Doxer 10.04.0 is available for download:


Doxer is a documentation tool initiated originally for generating source code documentation. This is the first public release of Doxer as a separate package (it has been included in other packages previously).

This release contains a wiki markup parser, a corresponding HTML generator, and a Drupal input format module. The markup syntax is designed to cover the feature set required to write source code documentation and general purpose documents. The parser and HTML generator have a strong focus on robustness to support the full range of user sophistication found on general purpose websites. An extensive test suite accompanies the development.

    Homepage:           http://rapicorn.org/doxer.html
    Download:           http://rapicorn.org/dist/doxer/
    Wiki Markup:        http://rapicorn.org/DoxerMarkup.html
    Feedback:           http://rapicorn.org/mailman/listinfo/rapicorn-list

User visible features in Doxer 10.04.0:

  • Added Doxer wiki markup parser.
  • Added docs/DoxerMarkup.txt to describe the markup.
  • Added HTML 4.01 Transitional generator for wiki markup.
  • Added extensive Doxer wiki markup test suite.
  • Wrapped the wiki markup parser as Drupal 6 input format module.
  • Rewrote documentation extractor (Doxyscan).
  • Started new Doxyscan Html documentation generator.

flattr this!

Syndicated 2010-04-23 00:31:18 from Tim Janik

14.09.2009 OSiM 2009

Together with Martyn Russell, Carlos Garnacho and Kristian Rietveld, I’m attending OSiM this week. None of us has been here before, so we’re quite curious about the conference and will keep our eyes open.
My schedule still has some holes, so if you would like a chat at the conference, drop me a line and we can arrange a meeting.

PS: Yes, I’ve seen Alex recent work and will take a look once I’m back from the conference.

flattr this!

Syndicated 2009-09-15 07:50:10 from Tim Janik

1 older entry...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!