gccgo and Autotools

As part of a personal project, I wrote a small Go program recently to marshal some data from a MySQL database as JSON over HTTP. I hadn't written a lot of (read: any) Go before this, and I found the process relatively painless and the implementation much more concise than the alternatives in Java, PHP, or Python. However, when I went to integrate my program with the rest of the Autotools build for my project, I ran into some obstacles, mostly related to the incomplete / broken support for Go in the current stable version of Autoconf (2.69). There are apparently fixes for the most obvious bugs coming in Autoconf 2.70, but that release has been in a development for years; and even once released, to the best of my knowledge, it won't include important features like tests for available Go libraries. So I spent a few days working on a better Autotools Go integration, which I'm attaching below.

A few notes to make what follows a bit clearer:

First, Go already has a build system, more or less - it's called go. If you've got a library or a program called "mypackage/myproject/myprog," and you've put the source files in the right directory, you can run...

go build mypackage/myproject/myprog

...and wind up with a working, statically-linked executable. What is the right directory? It's the "src" directory under one of the directories in $GOPATH. The Go project has a good amount of documentation on the topic of code organization, but in brief, the layout of a directory that forms a component of your GOPATH should be:

• pkg/[COMPILER_]$GOOS_$GOARCH/: Compiled library files go here
• bin/: Compiled executable files go here
• src/: Source code goes here, grouped by package

The go command is a front-end for the native go compilers (6g, 6l, 8g, 8l, etc.) as well as for gccgo (via the -compiler flag). It figures out where all the external dependencies are in your $GOPATH and passes flags and libraries to the compilers and linkers. If you run gccgo directly - that is, without using the go front-end - you have to assemble these arguments and paths yourself.

`go build' is the mainstream, nicely literate way of executing a build for .go files, and it's how most people familiar with the language will go about it. However, Autotools' existing support for Go unsurprisingly depends on direct interaction with gccgo, and I wouldn't expect that to change in any near-term releases. `go build' is convenient for fast, iterative builds; I find Autotools-based builds useful for packaging a source distribution for delivery to environments that need to be interrogated to locate dependencies. I wanted my project's build to work both for people doing `./configure && make' as well as for people running `go build'.

The files below provide:

• A behind-the-scenes patch for the broken `AC_PROG_GO' in Autoconf 2.69
• A new macro implementation - AC_CHECK_GOLIB - that finds and tests dependencies for Go programs and libraries, and which behaves similarly to pkg-config's `PKG_CHECK_MODULES'.
• A working example of an Autotools build for a small project that uses Autotools for its build and depends on some common Go web service and database libraries.

m4/golib.m4

Provides the patch and macro implementation. Bundle this file with your project to apply the changes locally, or put it in `/usr/local/share/aclocal' to make it available system-wide.

# Undefine the broken _AC_LANG_IO_PROGRAM from autoconf/go.m4...

m4_ifdef([_AC_LANG_IO_PROGRAM(Go)], m4_undefine([_AC_LANG_IO_PROGRAM(Go)]))

# ...and redefine it to use a snippet of Go code that compiles properly.

m4_define([_AC_LANG_IO_PROGRAM(Go)],
  [AC_LANG_PROGRAM([import ( "fmt"; "os" )],
  [f, err := os.OpenFile("conftest.out", os.O_CREATE | os.O_WRONLY, 0777)
   if err != nil {
     fmt.Println(err)
     os.Exit(1)
   }
   if err = f.Close(); err != nil {
     fmt.Println(err)
     os.Exit(1)
   }
   os.Exit(0)
  ])])

#
# Support macro to check that a program that uses LIB can be linked.
#
# _AC_LINK_GOLIB(VARIABLE, LIB, [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND])

AC_DEFUN([_AC_LINK_GOLIB],[

  # This little "embedded" shell function outputs a list of dependencies for a
  # specified library beyond the set of standard imports.

  AS_REQUIRE_SHELL_FN([ac_check_golib_nonstd_deps],
    [AS_FUNCTION_DESCRIBE([ac_check_golib_nonstd_deps], [LINENO],
      [Find the non-standard dependencies of the target library.])],
    [for i in `$ac_check_golib_go list -f {{.Deps}} $[]1 | tr -d [[]]`; do
       $ac_check_golib_go list -f {{.Standard}} $i | grep -q false
       if [[ $? = 0 ]]; then
         echo $i
       fi
     done])

  ac_check_golib_paths=`$ac_check_golib_go list -compiler gccgo \
    -f {{.Target}} $2`
  ac_check_golib_seeds=`ac_check_golib_nonstd_deps $2`
  ac_check_golib_oldseeds=""

  # Compute the transitive closure of non-standard imports.

  for ac_check_golib_seed in $ac_check_golib_seeds; do
    ac_check_golib_oldseeds="$ac_check_golib_oldseeds $ac_check_golib_seed"
    ac_check_golib_newseeds=`ac_check_golib_nonstd_deps $ac_check_golib_seed`

    for ac_check_golib_newseed in $ac_check_golib_newseeds; do
      if ! echo "$ac_check_golib_oldseeds" | grep -q "$ac_check_golib_newseed"
      then
        ac_check_golib_oldseeds="\
	  $ac_check_golib_oldseeds $ac_check_golib_newseed"
      fi
    done

    ac_check_golib_seeds="$ac_check_golib_seeds $ac_check_golib_newseeds"
    ac_check_golib_path=`$ac_check_golib_go list -compiler gccgo \
      -f {{.Target}} $ac_check_golib_seed`
    ac_check_golib_paths="$ac_check_golib_paths $ac_check_golib_path"
  done

  ac_check_golib_save_LIBS="$LIBS"
  LIBS="-Wl,-( $LIBS $ac_check_golib_paths -Wl,-)"

  AC_LINK_IFELSE([],
    [$1[]_GOFLAGS="-I $ac_check_golib_root"
     $1[]_LIBS="$ac_check_golib_paths"
     AC_MSG_RESULT([yes])
     $3
     LIBS="$ac_check_golib_save_LIBS"
     break],
    [AC_MSG_RESULT([no])
     m4_default([$4], AC_MSG_ERROR([Go library ($2) not found.]))
     LIBS="$ac_check_golib_save_LIBS"])
])

#
# Attempts to locate a Go library LIB somewhere under $GOPATH that can be used
# to compile and link a program that uses it, optionally referencing SYMBOL.
# Calls ACTION-IF-FOUND if a usable library is found, in addition to setting
# VARIABLE_GOFLAGS and VARIABLE_LIBS to the requisite compiler and linker flags.
#
# AC_CHECK_GOLIB(VARIABLE, LIB, [SYMBOL], [ACTION-IF-FOUND],
# [ACTION-IF-NOT-FOUND], [GO])

AC_DEFUN([AC_CHECK_GOLIB],[
  AC_ARG_VAR([$1][_GOFLAGS], [Go compiler flags for $2])
  AC_ARG_VAR([$1][_LIBS], [linker flags for $2])

  AC_MSG_CHECKING([for Go library $2])

  ac_check_golib_go="$6"
  if test -z "$ac_check_golib_go"; then
    ac_check_golib_go="go"
  fi

  # The gccgo compiler needs the `pkg/gccgo_ARCH` part of the GOPATH entry that
  # contains the target library, so use the `go' command to compute the full
  # target install directory and then subtract out the library-specific suffix.
  # E.g., /home/user/gocode/pkg/gccgo_linux_amd64/foo/bar/libbaz.a ->
  # /home/user/gocode/pkg/gccgo_linux_amd64

  ac_check_golib_root=`$ac_check_golib_go list -compiler gccgo \
    -f {{.Target}} $2`
  ac_check_golib_root=`dirname $ac_check_golib_root`  
  ac_check_golib_path=`dirname $2`

  ac_check_golib_root="${ac_check_golib_root%$ac_check_golib_path}"

  # Save the original GOFLAGS and add the computed root as an include path.

  ac_check_golib_save_GOFLAGS=$GOFLAGS
  GOFLAGS="$GOFLAGS -I $ac_check_golib_root"

  AS_IF([test -n "$3"],
    [AC_COMPILE_IFELSE([AC_LANG_PROGRAM([import ("os"; "$2")],[
        if $3 == nil {
          os.Exit(1)
        } else {
          os.Exit(0)
        }])],

      # Did it compile? Then try to link it.

      [_AC_LINK_GOLIB([$1], [$2], [$4], [$5])],

      # Otherwise report an error.

      [AC_MSG_RESULT([no])
       m4_default([$5], AC_MSG_ERROR([Go library ($2) not found.]))])],

    # If there was no SYMBOL argument provided to this macro, take that to mean
    # this library needs to be imported but won't be referenced, so craft a test
    # that exercises that kind of import clause (i.e., one with the `_'
    # modifier).

    [AC_COMPILE_IFELSE([AC_LANG_PROGRAM([import ("os"; _ "$2")],
        [os.Exit(0)])],
      [_AC_LINK_GOLIB([$1], [$2], [$4], [$5])],
      [AC_MSG_RESULT([no])
       m4_default([$5], AC_MSG_ERROR([Go library ($2) not found.]))])])

  # Restore the original GOFLAGS.

  GOFLAGS="$ac_check_golib_save_GOFLAGS"  
])

configure.ac

Food for Autoconf. Note the call to `AC_CONFIG_MACRO_DIR' to make golib.m4 visible.


AC_INIT([My Cool Go Program], [0.1], [me@example.com], [myprog], [])
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_SRCDIR([src/mypackage/myproject/myprog.go])

AM_INIT_AUTOMAKE(1.6)

AC_LANG_GO
AC_PROG_GO

AC_CHECK_GOLIB([MARTINI], [github.com/go-martini/martini], [martini.Classic])
AC_CHECK_GOLIB([GORM], [github.com/jinzhu/gorm], [gorm.Open])
AC_CHECK_GOLIB([RENDER], [github.com/martini-contrib/render], [render.Renderer])
AC_CHECK_GOLIB([MYSQL], [github.com/go-sql-driver/mysql])

AC_CONFIG_FILES([Makefile])
AC_OUTPUT


Makefile.am

Food for Automake. From what I can tell, Automake has no explicit support for building Go programs, but it does include general support for defining build steps for arbitrary source languages. Note the use of the ".go.o" suffix declaration to specify the compilation steps for .go source files and the "LINK" variable definition to specify a custom link step. The `FOO_GOFLAGS' and `FOO_LIBS' variables are created by the expansion of `AC_CHECK_GOLIB([FOO]...)' in configure.ac.

bin_PROGRAMS = myprog

myprog_SOURCES = src/mypackage/myproject/myprog.go
myprog_userd_GOFLAGS = $(GOFLAGS) @MARTINI_GOFLAGS@ @RENDER_GOFLAGS@ \
        @GORM_GOFLAGS@ @MYSQL_GOFLAGS@

myprog_DEPENDENCIES = builddir
myprog_LDADD = @MARTINI_LIBS@ @RENDER_LIBS@ @GORM_LIBS@ @MYSQL_LIBS@
myprog_LINK = $(GOC) $(GOFLAGS) -o bin/$(@F)

builddir:
        if [ ! -d bin ]; then mkdir bin; fi

.go.o: 
        $(GOC) -c $(myprog_GOFLAGS) -o $@ $<

CLEANFILES = bin/myprog

Here's a snippet from the output of configure when the integration is working:


checking for gccgo... gccgo
checking whether the Go compiler works... yes
checking for Go compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking for go... /usr/bin/go
checking for Go library github.com/go-martini/martini... yes
checking for Go library github.com/jinzhu/gorm... yes
checking for Go library github.com/martini-contrib/render... yes
checking for Go library github.com/go-sql-driver/mysql... yes


To apply this code to your own project, copy / adapt the snippets above along with your own Go code into the following directory layout.


configure.ac
Makefile.am
m4/golib.m4
src/[package]/[project]/*.go
bin/


If you add this project root to your GOPATH, you should be able to run `go build package/project' in addition to `./configure && make'.

Problems? Let me know.