Recent blog entries for oubiwann

Scientific Computing and the Joy of Language Interop

The scientific computing platform for Erlang/LFE has just been announced on the LFE blog. Though written in the Erlang Lisp syntax of LFE, it's fully usable from pure Erlang. It wraps the new py library for Erlang/LFE, as well as the ErlPort project. More importantly, though, it wraps Python 3 libs (e.g., math, cmath, statistics, and more to come) and the ever-eminent NumPy and SciPy projects (those are in-progress, with matplotlib and others to follow).

(That LFE blog post is actually a tutorial on how to use lsci for performing polynomial curve-fitting and linear regression, adapted from the previous post on Hy doing the same.)

With the release of lsci, one can now start to easily and efficiently perform computationally intensive calculations in Erlang/LFE (and any other Erlang Core-compatible language, e.g., Elixir, Joxa, etc.) That's super-cool, but it's not quite the point ...

While working on lsci, I found myself experiencing a great deal of joy. It wasn't just the fact that supervision trees in a programming language are insanely great. Nor just the fact that scientific computing in Python is one of the best in any language. It wasn't only being able to use two syntaxes that I love (LFE and Python) cohesively, in the same project. And it wasn't the sum of these either ;-) You probably see where I'm going with this ;-) The joy of these and many other great aspects of inter-operation between multiple powerful computing systems is truly greater than the sum of its parts.

I've done a bunch of Julia lately and am a huge fan of this language as well. One of the things that Julia provides is explicit interop with Python. Julia is targeted at the world of scientific computing, aiming to be a compelling alternative to Fortran (hurray!), so their recognition of the enormous contribution the Python scientific computing community has made to the industry is quite wonderful to see.

A year or so ago I did some work with Clojure and LFE using Erlang's JInterface. Around the same time I was using LFE on top of  Erjang, calling directly into Java without JInterface. This is the same sort of Joy that users of Jython have, and these are more examples of languages and tools working to take advantage of the massive resources available in the computing community.

Obviously, language inter-op is not new. Various FFIs have existed for quite some time (I'm a big fan of the Common Lisp CFFI), but what is new (relatively, that is ... as I age, anything in the past 10 years is new) is that we are seeing this not just for programs reaching down into C/C++, but reaching across, to other higher-level languages, taking advantage of their great achievements -- without having to reinvent so many wheels.

When this level of cooperation, credit, etc., is done in the spirit of openness, peer-review, code-reuse, and standing on the shoulders of giants (or enough people to make giants!), we get joy. Beautiful, wonderful coding joy.

And it's so much greater than the sum of the parts :-)


Syndicated 2015-01-01 20:56:00 (Updated 2015-01-01 21:01:38) from Duncan McGreggor

Improved Python Support in Erlang/LFE

The previous post on Python support in Erlang/LFE made Hacker News this week, climbing in fits and starts to #19 on the front page. That resulted in the biggest spike this blog has seen in several months.

It's a shame, in a way, since it came a few days too early: there's a new library out for the Erlang VM (written in LFE) which makes it much easier to use Python from Erlang (the language from Sweden that's famous for impressing both your mum and your cats).

The library is simply called py. It's a wrapper for ErlPort, providing improved usability for Python-specific code as well as an Erlang process supervision tree for the ErlPort Python server. It has an extensive README that not only does the usual examples with LFE, but gives a full accounting of usage in the more common Prolog-inspired syntax Erlang. The LFE Blog has a new post with code examples as well as a demonstration of the py supervision tree (e.g., killing Python server processes and having them restart automatically) which hasn't actually made it into the README yet -- so get it while it's hot!

The most exciting bits are yet to come: there are open tickets for:
  • work on multiple Python server processes
  • scheduling code execution to these, and
  • full Python distribution infrastructure with parallel execution.
This could drastically change the picture for compute-intensive tasks in Erlang, Elixir, LFE, and Joxa. The Erlang VM was never intended to excel at the sort of problems that Python has traditionally focused on... yet it provides the sort of infrastructure that the Python community has been agonizing over for more than a decade. For Pythonistas, this may not be a very big deal ... but for the Erlang and functional programming communities, the LFE py project could be a life-saver for any number of projects which need easy-access to the strengths of Python.


Syndicated 2014-12-28 01:05:00 (Updated 2014-12-28 01:09:34) from Duncan McGreggor

28 Nov 2014 (updated 28 Nov 2014 at 20:13 UTC) »

Scientific Computing with Hy and IPython

This blog post is a bit different than other technical posts I've done in the past in that the majority of the content is not on the blog in or gists; instead, it is in an IPython notebook. Having adored Mathematica back in the 90s, you can imagine how much I love the IPython Notebook app. I'll have more to say on that at a future date.

I've been doing a great deal of NumPy and matplotlib again lately, every day for hours a day. In conjunction with the new features in Python 3, this has been quite a lot of fun -- the most fun I've had with Python in years (thanks Guido, et al!). As you might have guessed, I'm also using it with Erlang (specifically, LFE), but that too is for a post yet to come.

With all this matplotlib and numpy work in standard Python, I've been going through Lisp withdrawals and needed to work with it from a fresh perspective. Needless to say, I had an enormous amount of fun doing this. Naturally, I decided to share with folks how one can do the latest and greatest with the tools of Python scientific computing, but in the syntax of the Python community's best kept secret: Clojure-Flavoured Python (Github, Twitter, Wikipedia).

Spoiler: observed data and
polynomial curve fitting
Looking about for ideas, I decided to see what Clojure's Incanter project had for tutorials, and immediately found what I was looking for: Linear regression with higher-order terms, a 2009 post by David Edgar Liebke.

Nearly every cell in the tutorial notebook is in Hy, and for that we owe a huge thanks to yardsale8 for his Hy IPython magics code. For those that love Python and Lisp equally, who are familiar with the ecosystems' tools, Hy offers a wonderful option for being highly productive with a language supporting Lisp- and Clojure-style macros. You can get your work done, have a great time doing it, and let that inner code artist out!

(In fact, I've started writing a macro for one of the examples in the tutorial, offering a more Lisp-like syntax for creating class methods. We'll see what Paul Tagliamonte has to say about it when it's done ... !)

If you want to check out the notebook code and run it locally, just do the following:

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!