Older blog entries for caolan (starting at number 186)

libexttextcat: text guessing feature

LibreOffice inherited a text language guesser, based on textcat from wise-guys.nl and extended by Jocelyn Merand to basically handle UTF-8 text. This is the thing that makes the suggestions as to what language your text might really be in when you right click on some misspelled text and chose set language.

We’ve now spun this off as a standalone libexttextcat and fixed up some conversion problems from the original selection of 8bit encodings and generated new language fingerprints in other cases, which should give better results for various languages, and allow us to enable checking for some languages which was disabled until now.

The current list of languages it attempts to detect can be seen here

Here’s a plausible process to add your favourite language to it, given git clone git://anongit.freedesktop.org/libreoffice/libexttextcat and bootstrapping from the insanely-translated UDHR using Abkhaz as an example.


cd libexttextcat/langclass/ShortTexts/
wget http://unicode.org/udhr/d/udhr_abk.txt
#skip english header, name result using BCP-47
tail -n+7 udhr_abk.txt > ab.txt
cd ../LM
../../src/createfp < ../ShortTexts/ab.txt > ab.lm
echo ab.lm ab--utf8 >> ../fpdb.conf

Then update the check target in src/Makefile.am to confirm the detection of ShortTexts/ab.txt as ab works using make check

I’ll remove the necessity of a configuration file in a later version, and convert the result to a BCP-47 tag. For the moment it remains a drop in replacement for the original solution which necessitates retaining the slightly odd language tag syntax.

Syndicated 2011-09-28 14:10:11 from Caolan McNamara

git, really nifty after all

Maybe there’s something to the cult-of-git after all :-) . vcl/unx/source/fontmanager/fontcache.cxx had some code which painstakingly constructed a string, only to do nothing with it. Clearly at some time in the past it was used, so when did its use go away. This is a file which has been moved around over the years from place to place, hmm, potentially tricky to scratch the itch of knowing when it happened ?, not at all…

git log --follow --oneline -S'suspiciously missing variable' /path/to/file.cxx

and 2 seconds later I have a list of 5 commits, there it is at the top of the list. Back in 2005, a rework of the font cache where the stat on a file was optimized out, while the constructed path to the file remained. No undetected nightmare merge bug then, just a missed micro optimization opportunity.

Syndicated 2011-09-19 23:57:25 from Caolan McNamara

all text rendered with cairo

So, as of today all LibreOffice (3.5 onwards) text rendering under X goes through cairo. This was already the case in practice for horizontal text for quite a while, the additional change is that its true for vertical text as well now.

before

after

Yes, I know it’s still rather sub-optimal. The current implementation is basically intended to be bug-for-bug compatible for now, though I couldn’t resist improving the positioning of 0x30FC.

Test-case at http://cgit.freedesktop.org/libreoffice/core/tree/qadevOOo/testdocs/vertical-testcase.odt

Syndicated 2011-08-19 11:56:02 from Caolan McNamara

sgv, StarDraw 2.0 examples with text ?

I wonder if anyone has any sgv documents left around, not svg, but sgv, the StarDraw 2.0 format. Looking for .sgv documents that contain text, and ideally text outside of the ASCII range. A few umlaut’s would probably suffice.

Syndicated 2011-08-08 11:12:50 from Caolan McNamara

unused code, libreoffice style

The return of callcatcher derived lists of unused code list in LibreOffice. I tweaked callcatcher to understand the additional gcc command line options used by the new gbuild module so it can be dropped in as a gcc replacement in that environment.

There’s now a findunusedcode target in the top level Makefile and a cached list of easy to remove methods in the tree as unusedcode.easy. These are non-virtual C++ methods which are not called directly, nor have their address taken by any code in a stock debug level Linux build.

What distinguishes unusedcode.easy from not-easy is simply that the easy list is restricted to C++ name-mangled class-level symbols and so omits any non-mangled C-style symbols which might be dlsymed from some not easy to find entry point.

At a count of 5176 easy unused methods there’s enough there to be getting on with for the moment, and can revisit the C-style symbols with a whitelist of known dlsym names on completion of those.

Syndicated 2011-07-11 12:34:48 from Caolan McNamara

regression testing libreoffice filters

For regression testing LibreOffice filters I’ve now arranged things so that each import filter’s cppunit test comprises of three data dirs, a pass dir, a fail dir and an indeterminate dir. Files in pass must parse without error, those in fail are expected to fail, but fail gracefully by returning an error or throwing an exception, i.e. a crash is a fail on a “fail” test, while “can’t parse” is the expected pass state.

The pass/fail dirs are typically pre-filled in the tree with a small sample of tricky documents which get tested at every build time to ensure they remain working.

indeterminate dirs on the other hand are expected to be empty in the tree, and the cppunit tests don’t care if their contents can be parsed or not, only that they don’t crash. This is really convenient for searching for crashes in a large document collection (horde), given that its an order of magnitude faster than using the full application to load and layout the results.

So I/we can just take a large document horde of e.g. docs and throw them in sw/qa/core/data/ww8/indeterminate and run make -sr in sw and sit back and wait to see if anything in there is a crasher at the parser level. For extra goodness export VALGRIND=memcheck to run the whole lot under valgrind.

FWIW, today anyway

  1. All 3721 attachments of (alleged) mime-type application/msword in openoffice.org’s bugzilla pass without crash when placed into ww8/indeterminate. To be re-run under valgrind later
  2. And all (ok, only 128) attachments of (alleged) mime-type application/msword in freedesktop.org’s bugzilla pass under VALGRIND=memcheck when placed into indeterminate.

I’ve got doc, rtf, qpro, wmf, emf, hwp, lwp and sxw organized and pre-seeded with a sample handful files so far. Plenty more filters than that of course, but .doc is my current focus as the richest vein of available had-bugs-reported documents.

Syndicated 2011-07-11 11:58:27 from Caolan McNamara

Naoise

I’ve been somewhat remiss, our new baby Naoise being held by his big sister.

Syndicated 2011-06-06 20:56:24 from Caolan McNamara

when cut and paste goes wrong

So, what’s wrong with this picture…

void AnimationNode::acquire( ) throw ()
{
OWeakObject::acquire();
}
...
void AnimationNode::release( ) throw ()
{
OWeakObject::acquire();
}

Don’t need to know anything about the code in question to have grave doubts about a “release” method that has the same body as its “acquire” method :-) .

This big old leak in presentation animation nodes is fixed now

Syndicated 2011-05-26 08:25:24 from Caolan McNamara

fixing annoying stuff 2

So Sébastien (Le Ray) strikes again. I like autocorrect, I know a lot hate it, but I generally like it, but it drives me insane when it does the wrong thing, e.g. you type http:// at the start of a sentence, and it autocorrects to Http://. So now it simply doesn’t autocorrect protocols when you type that :. http://, ftp://, etc. now stays as http://, ftp://

Syndicated 2011-02-21 21:36:44 from Caolan McNamara

order styles by natural sort

So, one of those little annoying things. Headings sorted awkwardly.

and now after Sébastien Le Ray’s mods:


ta-da.

Syndicated 2011-02-13 21:44:26 from Caolan McNamara

177 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!