Emacs

I'm one of the few people who converted from vi to Emacs (FAQ/Emacswiki). I've been using vim for years, but after a couple of days with Emacs, I don't think I will use vim  for anything other than simple one line changes to config files.


The reason I like Emacs so much is, because it really is a geek toy. Sure, you can also do some serious text editing with it, but mostly I just love to tweak it and make it do silly things. Which is easy as it neatly integrates into the operating system - as long as it's a *nix system. I went through some pains to get even basic stuff like spell checking working on Windows. Also I think I didn't Tramp setup correctly, which I expect to be a breeze on *nix. Mainly because all those "standard" tools ain't standard on Win. Anyway it's still fun.


A couple of features I like about Emacs

• Buffers
  Buffers work like tabs in other editors. However, they can contain more than only files. They can also hold scratchpad areas, shells, output from running processes, and anything else Emacs can access. It is extremely convenient to have all output (e.g. from a compiler) saved in a buffer. Or how many times did you have to redirect shell output to a file? You can run a shell in Emacs instead of a terminal. And the best thing is that most shells understand Emacs' navigation keys.
• Kill Ring
  Works like the clip board. Well actually, it works like a cyclical buffer of 30 clip boards. So you can copy multiple words, lines, regions and still paste them 15 minutes later. How many times did you wish for a second clip board? Now you have 30.
• Mark Ring
  Every time you take diversions (e.g. by searching, or pressing M-< or M->), Emacs uses the mark to save your previous position, kind of like sticking your finger behind one page of a book while you go to glance at another page. More information about the mark ring.
• Tramp
  Transparent file access through Tramp. Tramp can open local or remote files through various protocols like ftp or ssh and it can also open shells. Although I still need some time to get used to the syntax - seems like I always forget an semicolon or slash etc. More information about Tramp.
• Version Control
  Transparent version control (be it CVS, SVN, Git etc) through the VC package. The commands are all the same regardless of what version control system. 99 out of a 100 times you'll either want to update, commit or add files - Emacs is more than capable to do that. More information about VC.

C++ Template Metaprogramming

Template Metaprogramming

In a recent post I admired the functional programming features (among many other features) that the BOOST libraries add to the C++ runtime system. Well guess what there is a pure, lazy functional programming language looming in C++'s template system! That means all the features you know and love from Haskell.


I've been reading "C++ Template Metaprogramming: Concepts, Tools, and Techniques from Boost and Beyond". It outlines how can make use of metaprogramming features in your programs. 


A Simple Example
This is an introductory example from the book. Let's say you want to conveniently express binary numbers in your code. You could write a conversion function like this:


/*
 * general case
 */
template<unsigned long N>
struct binary

{
    static unsigned const value =
    binary::value<N/10> * 2 + N%10;
}


/*
 * special case, invoked to terminate recursion
 */
template<>
struct binary<0>
{
    static unsigned const value = 0;
}

In your code you could then write something like the following


/*
 * the binary expression is converted to 42 at compile time
 */
int fortytwo = binary<101010>::value;


If you're familiar with functional programming this should look familiar to you. And even if you don't know FP, the example shouldn't be too complicated to grasp. When a binary expression is compiled, the general binary struct is invoked recursively until the remainder is zero, at which point the so called template specialization is called which terminates the recursion.


Even from this simple example I think you can see potential uses Metaprogramming, especially when it comes to crafting domain specific languages for your application. The book dedicates a whole chapter to the design of domain specific embedded languages.


BOOST Support
BOOST also provides the metaprogramming library (mpl), which is a template metaprogramming framework of compile-time algorithms, sequences and metafunctions. mpl is also extensively used in the book. Another objective of the mpl is reduce the syntactic noise the templates produce. Template syntax is becoming really messy really quickly if expressions grow just little complex.

Programming Collective Intelligence

Programming Collective Intelligence is a new book from O'Reilly written by Toby Segaran.

Even though the subtitle read "building smart web 2.0 applications", it's about more than that. I'm less interested in building websites and all the more data mining of existing sources. And I was not let down.
 
Toby really manages to explain the underlying concepts in an accessible way. Which important because much of it is based on methods of statistical analysis. He covers discovering groups, searching, ranking, optimization, document filtering, decision trees, price models or genetic algorithms. The book explains how to implement Simulated Annealing, k-Nearest Neighbors, Bayesian Classifier and many more.

Throughout the book, Toby explores various other AI techniques, and explains clearly, how to implement them in Python. One nice touch at the end of the book, was to include a reference to modules used in the book, and to include small usage examples. I was quite pleased by Collective Intelligence, and I would recommend the book to any intermediate to advanced programmer who wants to learn more about AI, and also web specific applications of AI theory. Take a look at the table of contents (it does not list all the algorithms, tho).

All code in the book is written exclusively in Python. Don't worry if you're not familiar with Python. In the introduction there is a short description of the language and the code is usually well explained as well as commented. I use very little Python, but I was still able to follow the code listings.

Learning C++

C++
Recently I joined a new project at work. It's some application running on an embedded Linux platform. So it was decided that we would write it in C++. C++ is great for these kind of jobs, where you have limited space and processing power.

The only problem was, I had never worked with C++ before. Why was I even assigned to the team you ask. Well, I had previously been working on a related project for the same client. So I had a lot of domain specific knowledge that the other team members didn't have. Also I'm the resident UNIX guy so I would help them when their machines started acting up.

So there I was learning C++ as I went. Reading a lot of books by Scott Meyers and a ton of BOOST documentations. And after a couple of months I'm still a C++ rookie. Frankly, I didn't think it was a Language worth learning. I though it was outmoded and inferior to modern languages.

BOOST

The BOOST library is an amazing piece of software. I can't imagine where we would be without it. From Wikipedia:

"The BOOST libraries are aimed at a wide range of C++ users and application domains. They range from general-purpose libraries like the smart_ptrlibrary, to OS abstractions like Boost FileSystem, to libraries primarily aimed at other library developers and advanced C++ users, like the template metaprogramming (MPL) and DSL creation (Proto).

In order to ensure efficiency and flexibility, Boost makes extensive use of templates. Boost has been a source of extensive work and research into generic programming and metaprogramming in C++."

Most of all I was pleasantly surprised to see how many functional programming features is provides. I mean I use boost::bind, boost::function or boost::signals every single day. boost::bind allows you to create a callable object and bind to an arbitrary member function. It also allows partial function application and the like. There is also the BOOST lambda which allows you to use lambda expression just the way you're used from FP languages.

Valgrind
I've also installed Valgrind on our development machines. We generally use it to detect memory leaks and other memory management issues. Apart from suchmemory checks, it also features several profilers. But the coolest feature IMO is Helgrind, which can detect race conditions in multi-threaded applications and/or improper locking of resources. 

To Infinity and Beyond

Sigfpe made the effort to collect many of his great blog posts he wrote over the years into a single, structured post. Basically writing an index. He covers Monads, Comonads, Fold, Unfold, the general Category Theory. I've been reading and rereading for weeks and there's still more to discover. Check it out here.

Update: Lucid Lynx on the HP Mini 210

Bye Bye Koala, Hello Lynx

Some time ago, I installed Koala on my HP Mini 210. Shortly after the next Ubuntu version "Lucid Lynx" was released. So I upgraded in hopes that some the hardware would be better supported. So I figured I should write a quick update about how things are.

Improvements

Bluetooth

Bluetooth works consistently now even after many suspend/resume cycles. This is definitely an improvement for people who need Bluetooth enabled peripherals. 

Remaining Problems

No Multitouch

Unfortunately there still doesn't seem to be touch pad drivers which support multi-touch properly. Normally I wouldn't care much. But the way the Mini's touch pad is designed makes this a bit of an annoyance. The left and right mouse buttons are embedded in the touch pad. This means that the cursor jumps around the screen every time I touch the pad with a second finger. Unfortunately this happens all the time when I point and click. 

Short Battery Life

Sadly, the one improvement I was hoping for hasn't happened. Even relying on powerttop to reduce the power consumption, the battery life is still only about two thirds of what it should be. I also figured out that shutting down the wireless controller before suspending, reduces power consumption while the netbook is suspended.

Conclusion

So that's that. Lucid's hardware support has improved compared to Karmic. That and the new look and additional features speak for an upgrade.

Mixed Locale in Ubuntu

If you're not a native English speaker the odds are that you most likely want to change the default locale. Well, the right thing in that case would be to select the correct language during installation. Or perhaps you come from a multilingual country and a single locale just won't cut it. I'm from Switzerland where we have four official languages (English not included - yet). I like to have my OS talking to me in English, like the makers intended, but I also want the currency, date & time, etc. formatted in the familiar way.

Installing necessary locales

Before you can start to tweak the locale, you must first install all locales you are planning to use. 

The easiest way to install support for additional languages in Ubuntu is through the aptly named 'Language Support' tool. It is located in System -> Administration -> Language Support. This tool will also allow you to specify the language of your user environment as well as the language for the login screen.  I keep both settings at US English. At the bottom you will see a button called "Add/Remove Languages". When you press it, another menu will pop up where you can check all languages that you require. In my case, I added German. When you apply the changes support for all selected languages will be installed.

If you're working on the command line. You can install language-pack packages through your favorite CLI package management tool, like apt or aptitude or whatever you like. There are a couple of language-packs for different languages. 

When I additionally picked support for German, the "Language Support" tool installed the following packages on my system:

• language-pack-de
• language-pack-gnome-de
• language-pack-de-base
• language-pack-gnome-de-base
• language-support-writing-de

There are also language-pack-kde-de and language-pack-kde-de-base, respectively for KDE. But you probably won't need them unless you are on Kubuntu.

Mixing locales

As I mentioned in the previous paragraph. You can select your locale in System -> Administration -> Language Support. But there you can only select one locale for everything. For example, I like English GUI & CLI a lot, but I don't like their date & time formats so much - I'm just not used to that. Not to mention the American system of measurements :)

Luckily UNIX/Linux actually allows more fine grained control over the locale settings than the Ubuntu GUI wants to make you believe. The language you select on the GUI will act as the default, but you can also tweak it to your liking.

First create a new file called .custom_locale in your home directory. This will contain the custom locale settings for you alone:

# Custom locale settings:
# use Swissgerman date, time, currency, numbering etc.
export LANG=de_CH.utf8
# keep the CLI/GUI in US english
export LC_MESSAGES=en_US.utf8
# sort in alphabet the Swiss way
export LC_COLLATE=de_CH.utf8
# change currency formatting behavior of strfmon(3)
export LC_MONETARY=de_CH.utf8
# change time formatting behavior of strftime(3)
export LC_TIME=de_CH.utf8
# change number formatting behavior of printf(3) and scanf(3)
export LC_NUMERIC=de_CH.utf8

The LC_* variables are described in more detail in man locale(7). The most important settings are the first two. This file alone won't do much. You have to include it to your environment.

In ~/.profile, add the following lines:

# Include ustom locale settings if they exist
# (gdm parses .profile so these settings will 
# also be active within the Gnome environment).

if [ -f "$HOME/.custom_locale"]; then
   . "$HOME/.custom_locale"
fi

Additionally, you will also have to add the same snippet to your ~/.bashrc or ~/.zprofile (or rc file of whatever shell you use) because unlike gdm, the shells usually don't read .profile. Once you're done, reboot.

After the login, you can verify the locale settings e.g. from the shell with the command locale:

~$ locale
LANG=de_CH.utf8
LANGUAGE=en_US.UTF-8
LC_CTYPE="de_CH.utf8"
LC_NUMERIC=de_CH.utf8
LC_TIME=de_CH.utf8
LC_COLLATE=de_CH.utf8
LC_MONETARY=de_CH.utf8
LC_MESSAGES=en_US.utf8
LC_PAPER="de_CH.utf8"
LC_NAME="de_CH.utf8"
LC_ADDRESS="de_CH.utf8"
LC_TELEPHONE="de_CH.utf8"
LC_MEASUREMENT="de_CH.utf8"
LC_IDENTIFICATION="de_CH.utf8"
LC_ALL=

Systemwide Changes

The above tweak will only work for individual users. If you want to make a system wide change, you should change move the custom_locale file to /etc and then modify /etc/profile rather than ~/.profile. And likewise /etc/bash.bashrc & /etc/zsh/zprofile

Ubuntu on HP Mini 210 - Taming the fierce Koala

Bad Koala! Nasty Koala!

I got an HP mini 210 last weekend and of course the first thing I had to do, was getting rid of Windows 7 Starter. Initially I wanted to give Easy Peasy a try, but it didn't boot properly. So I switched to the next best thing, which is Ubuntu 9.10 "Karmic Koala" Netbook Remix.

Before kissing windows 7 good-bye for good (HP doesn't provide neither Windows7 nor any driver CDs), I wanted to make sure that Ubuntu would support the hardware. So I created a bootable USB stick with Unetbootin. In the live CD environment almost everything worked nicely. Most importantly wifi ran out of the box with the proprietary Broadcom STA driver. Suspend & resume also worked. The only thing that I noticed was that the multitouch pad didn't work. So I went along and installed Karmic. To my surprise the wireless driver failed to load! Just great. Fortunately for me, I knew that the driver had to be somewhere after all it ran in the live image. After a little searching I found it in /media/usb/pool/restricted/b/bcmwl/.

You can install it directly from Nautilus or in the shell. Whatever you prefer. After that, wireless should work.

The Good - Things that work Out of the Box™

Just like its namesake the Karmic Koala is pretty tame already and many things worked:

• Wireless
• Ethernet
• Suspend/Resume
• Sound
• Webcam
• Microphone
• Bluetooth
• SD/MMC/MS/xD card reader
• USB
• Touchpad (single touch and no right click)

Moreover, the Mini 210 has SIM card slot for mobile Broadband. I haven't tested that because I'm not using such a service.

The Bad - Stuff I could fix

Touchpad

Solution #1. Initially, presumed that fixing the Touchpad was just a matter of installing the Synaptics driver. But that didn't help. After some googling, I found a quick and dirty fix here. Essentially it suggests adding "options psmouse proto=exps" to psmouse.modprobe like so:

$ sudo echo options psmouse proto=exps > /etc/modprobe.d/psmouse.modprobe

I call this quick and dirty because it will essentially make the OS treat the touchpad like a PS mouse. It also breaks the edge-scrolling. A feature I can't live without.

Solution #2. After some more searching I discovered a touchpad driver patch on the Ubuntu forums. You can download and apply the patch and then hand roll your own driver. This will fix left/right clicks and edge-scrolling. But real multitouch doesn't seem to be currently supported.

The Ugly - Issues I couldn't fix

Bluetooth is gone after suspend

I also noticed that the Bluetooth device is only available after a fresh reboot. This is not really a problem for me since I don't normally use any Bluetooth peripherals. I didn't find any solutions for the problem. As a matter of fact, I'm not even sure what the root cause is...

Short Battery Life

I saved the worst part for last. I noticed that the battery life is not nearly a good as it is supposed to be. I have a 6 cell lithium ion battery which should last around 8+ hours. Effectively, I only get about half that.

I installed a tool called powerttop. It combines various sources of information from the kernel into one convenient screen so that you can see how well your system is doing at saving power, and which components are the biggest problem. Running powertop on my system revealed that the kernel spends half the time (i.e. half the battery) scheduling!

After further digging I discovered that this was a well known regression bug that affects various Atom based netbooks by different manufacturers. Apparently this was no problem in Jaunty. Hopefully the bug will be fixed with the next Ubuntu release which is due April 30th. If it isn't, I likely will revert to Jaunty for better or worse.

Conclusion
All in all, Karmic runs ok on the HP Mini. Although there a couple of hurdles that are very difficult to overcome by the average user.

Porting Cabal packages to FreeBSD - Part 2

Back in part 1, I tried to give a basic overview of FreeBSD porting. In this part, I'll show you how I ported hs-HTTP. Actually another maintainer (Jacula Modyun) beat me to it! I think he's the most active Haskell maintainer. Anyway, I'm going to walk you through all steps required to port the Cabal package to a FreeBSD port. Let's start with the sources. The source tarball of the HTTP package is available on Hackage.

Next install the tools:

#cd /usr/ports/ports-mgmt/portlint && make install clean

#cd /usr/ports/ports-mgmt/porttools && make install clean

#cd /usr/ports/ports-mgmt/genplist && make install clean

This will install a binary called port (part of the porttools). When you call it the first time it generates the configuration file .porttools in your home directory. Try it.

#port

===>; Generating /root/.porttools configuration file

FreeBSD Port Tools 0.99

Usage:    port []

Available commands:
commit  - commit a port into the FreeBSD Ports CVS Repository
create  - create new port from template using newfile(1)
diff    - generate diff against original port version
fetch   - fetch distfile(s) of new port version
getpr   - get patch/shar from a PR
help    - display this command summary
install - install a port
submit  - submit Problem Report with new port, or port change/update
test    - test port (build, install, package, deinstall)
upgrade - upgrade a port

You should now edit the newly created .porttools file. It contains settings that will be used to create template files for your port. You should at least put in your correct email address. What settings are supported can be found in the manpage (man 5 porttools).

# FreeBSD Port Tools configuration file - see porttools(5)

# vim: ft=sh

EMAIL="joeyjojo@mail.com"

FULLNAME="Joeyjojo Shabadoo"

ORGANIZATION=""

BUILDROOT="/tmp"

ARCHIVE_DIR=""

DIFF_MODE="CVS"

DIFF_VIEWER="more"

PORTLINT_FLAGS="abct"

Creating the templates

I prefer to create new ports in /tmp. To create a new directory for the port HTTP do the following:

#cd /tmp

#port create HTTP

porttools will also create some template files for us:

#ls /tmp/HTTP

Makefile    pkg-descr    pkg-plist

We can now go and set the variables CATEGORIES, COMMENT, PORTVERSION, MASTER_SITES and PKGNAMEPREFIX.

#cd /tmp/hs-HTTP

#cat Makefile

# Whom:                 Joeyjojo Shabadoo
#
# $FreeBSD$
#

PORTNAME=       HTTP
PORTVERSION=    4000.0.9
#PORTREVISION=  0
#PORTEPOCH=     0
CATEGORIES=     www haskell
MASTER_SITES=   http://hackage.haskell.org/packages/archive/${PORTNAME}/${PORTVERSION}/
#MASTER_SITE_SUBDIR=
PKGNAMEPREFIX=  hs-
#PKGNAMESUFFIX=
#DISTNAME=
#EXTRACT_SUFX=
#DISTFILES=
#DIST_SUBDIR=   ${PORTNAME}
#EXTRACT_ONLY=

MAINTAINER=     joeyjojo@mail.com
COMMENT=        A library for client-side HTTP

.include <bsd.port.pre.mk>

.include <bsd.port.post.mk>

The PORTVERSION and the COMMENT were directly copied from the Hackage website. I chose CATEGORIES www and haskell. Note that the category haskell is only virtual and can therefore never be declared at the first position. www on the other hand is a real category which fits very nicely for our HTTP-client library. Refer the Porter's Handbook for more information on proper categorization and a list of existing categories.

The PKGNAMEPREFIX for all Haskell ports is 'hs-'. So you should always set that as well.

And finally MASTER_SITES contains a list of URLs where the source tarball is located. It is recommended to specify a list rather than a single location for redundancy reasons. Unfortunately I know only the one URL specified on Hackage.

A word about formatting

There are a couple of formatting rules you have to follow. If you don't, portlint will tell you about it.

• always use tabs instead of spaces for indentation. Also, indent all variable assignments.

• you mustn't put consecutive empty lines into the Makefile. 

• avoid spaces or tabs at the end of a line 

First test run - fetching the source

At this point the Makefile has enough information to download the tarball and calculate the checksums. So lets try that to verify that the URL is correct.

#port fetch
=> HTTP-4000.0.9.tar.gz doesn't seem to exist in /usr/ports/distfiles/.

=> Attempting to fetch from http://hackage.haskell.org/packages/archive/HTTP/4000.0.9/.

HTTP-4000.0.9.tar.gz                          100% of   58 kB   40 kBps

#cat distinfo

MD5 (HTTP-4000.0.9.tar.gz) = bbd005935537ed8883bfefb624e8bf3c

SHA256 (HTTP-4000.0.9.tar.gz) = 1e2b4a8b782ad1417c8755bb0d248851bc142b351366ed460e07f2945a5e95ba

SIZE (HTTP-4000.0.9.tar.gz) = 59528

Documentation

So far I haven't talked about how to install any documentation. The variable NOPORTDOCS controls whether or not documentation should be installed. Most Haskell modules contain documented source files. This documentation can be extracted and converted to browsable HTML files using Haddock.

Haddock

Haddock can be either built into ghc or be installed as separate package. This can be checked through make -V PORT_HADDOCK. It will return a two digit number indicating whether or not haddock was installed as separate port.

PORT_HADDOCK=  (cd  ${PORTSDIR}/lang/ghc && ${MAKE} -V PORT_HADDOCK)

You can try it on the shell prompt to see what it does:

#cd /usr/ports/lang/ghc && make -V PORT_HADDOCK

#10

If Haddock hasn't been integrated into ghc, we have to declare it as additional build dependency so that it will be installed if necessary. PORT_HADDOCK:M?0 will match either "10" or "00". In case you wonder, the meaning of these variable values is defined inside ghc's Makefile (/usr/ports/lang/ghc) file. It is much more involved than the average Haskell port but still worth a look.

.if !empty(${PORT_HADDOCK:M?0})

BUILD_DEPENDS+= haddock:${PORTSDIR}/devel/hs-haddock

.endif

BTW, all options, commands, conditionals, variable assignments and pattern matching used in the Makefile are documented in make's manpage (man 1 make).

In addition to extracting documentation, Haddock can also convert, format and highlight the source code itself using the port hs-hscolour. So that means another build dependency.

BUILD_DEPENDS+= HsColour:${PORTSDIR}/print/hs-hscolour

Finally, we also have to set the variable PORTDOCS. It is intended to hold a list of all conditionally installed documentation. Which will also be included into the final packing list. The Handbook specifies:

If a directory is listed in PORTDOCS or matched by a glob pattern from this variable, the entire subtree of contained files and directories will be registered in the final packing list. If NOPORTDOCS is defined then files and directories listed in PORTDOCS would not be installed and neither would be added to port packing list.

To give you an idea, the whole block looks like this:

.ifndef(NOPORTDOCS)

CHECK_HADDOCK=  (cd  ${PORTSDIR}/lang/ghc && ${MAKE} -V PORT_HADDOCK)

.if !empty(${CHECK_HADDOCK:M?0})

BUILD_DEPENDS+= haddock:${PORTSDIR}/devel/hs-haddock

.endif

BUILD_DEPENDS+= HsColour:${PORTSDIR}/print/hs-hscolour

HSCOLOUR_VERSION=       1.15

HSCOLOUR_DATADIR=       ${PREFIX}/share/hscolour-${HSCOLOUR_VERSION}

PORTDOCS=       *

.endif

The block is identical for virtually every Haskell port. But I think it is still important to understand what all this stuff means.

Other Documentation

There some more variables, that influence pkg-plist, which are related to PORTDOCS that you should know about.

DOCSDIR: by default this variable will get expanded to PREFIX/share/doc/PORTNAME. But we'll redefine it to DOCSDIR=${PREFIX}/share/doc/${DISTNAME}. Where DISTNAME is just PORTNAME plus PORTVERSION. This way we can install different docs for different versions of the same port.

PLIST_SUB: If you need to make other substitutions, you can set this variable with a list of VAR=VALUE pairs and instances of %%VAR%% will be substituted with VALUE in the pkg-plist.

If your port installs files conditionally on the options set in the port, the usual way of handling it is prefixing the pkg-plist lines with a %%TAG%% and adding that TAG to the PLIST_SUB variable inside the Makefile with a special value of @comment, which makes package tools ignore the line. So in our case we do:

.ifdef(NOPORTDOCS)

PLIST_SUB+=     NOPORTDOCS=""

.else

PLIST_SUB+=     NOPORTDOCS="@comment "

.endif

If the port was installed without additional documentation, it will still put a LICENSE file in its doc folder - /usr/local/share/doc/HTTP-4000.0.9 in our case. So that will be accounted for in the pkg-plist.

Defining the Make targets

In order to configure, build and install the port, we have to define the corresponding targets. I will also define the two variables SETUP_CMD = ./setup and GHC_CMD = ${LOCALBASE}/bin/ghc (LOCALBASE normally points to /usr/local/).

Which reminds me. I need ghc to be present on the system to build as well as to run the HTTP library. So I have to declare BUILD_DEPENDS and RUN_DEPENDS. If it isn't, it will be automatically installed as a dependency.

BUILD_DEPENDS+= ghc:${PORTSDIR}/lang/ghc

RUN_DEPENDS+=   ghc:${PORTSDIR}/lang/ghc

We can then copy the default targets which I've also shown in the first part. They too are identical for most ports. The only important thing you might have to adjust is the Setup source. Some packages contain Setup.lhs while others use Setup.hs, you have to pick the correct one in the do-configure target.

do-configure:

        cd ${WRKSRC} && ${GHC_CMD} --make Setup.lhs \
                                   -o setup -package Cabal \

                     && ${SETUP_CMD} configure \
                      --haddock-options=-w --prefix=${PREFIX}

do-build:

        cd ${WRKSRC} && ${SETUP_CMD} build \

                     && ${SETUP_CMD} register --gen-script

.if !defined(NOPORTDOCS)

        cd ${WRKSRC} && ${SETUP_CMD} haddock \

                 --hyperlink-source \

                 --hscolour-css=${HSCOLOUR_DATADIR}/hscolour.css

.endif

do-install:

        cd ${WRKSRC} && ${SETUP_CMD} install \
                     && ${INSTALL_SCRIPT} register.sh ${PREFIX}/${HTTP_LIBDIR_REL}/register.sh

post-install:

        ${RM} -f ${PREFIX}/lib/ghc-${GHC_VERSION}/package.conf.old

Note: The command setup register --gen-script in the do-build target registers this package with the compiler, i.e. makes the modules it contains available to programs. The setup install command actually incorporates this action. The main use of this separate command is in the post-installation step for a binary package.

The whole Makefile now looks like this:

# New ports collection makefile for:    HTTP
# Date created:         2010-01-28
# Whom:                 your name
#
# $FreeBSD$
#

PORTNAME=       HTTP
PORTVERSION=    4000.0.9
CATEGORIES=     www haskell
MASTER_SITES=   http://hackage.haskell.org/packages/archive/${PORTNAME}/${PORTVERSION}/
PKGNAMEPREFIX=  hs-

MAINTAINER=     your.name@mail.com
COMMENT=        A Haskell library for client-side HTTP

BUILD_DEPENDS=  ghc:${PORTSDIR}/lang/ghc
RUN_DEPENDS+=   ghc:${PORTSDIR}/lang/ghc

GHC_VERSION=    6.10.4
HTTP_VERSION=   ${PORTVERSION}

GHC_CMD=        ${LOCALBASE}/bin/ghc
SETUP_CMD=      ./setup

DOCSDIR=                ${PREFIX}/share/doc/${DISTNAME}
HTTP_LIBDIR_REL=        lib/${DISTNAME}

PLIST_SUB=      GHC_VERSION=${GHC_VERSION} \
                HTTP_VERSION=${HTTP_VERSION} \
                HTTP_LIBDIR_REL=${HTTP_LIBDIR_REL}

.ifdef(NOPORTDOCS)
PLIST_SUB+=     NOPORTDOCS=""
.else
PLIST_SUB+=     NOPORTDOCS="@comment "
.endif

.ifndef(NOPORTDOCS)

PORT_HADDOCK=   (cd  ${PORTSDIR}/lang/ghc && ${MAKE} -V PORT_HADDOCK)
.if !empty(PORT_HADDOCK:M?0)
BUILD_DEPENDS+= haddock:${PORTSDIR}/devel/hs-haddock
.endif
BUILD_DEPENDS+= HsColour:${PORTSDIR}/print/hs-hscolour

HSCOLOUR_VERSION=       1.15
HSCOLOUR_DATADIR=       /usr/local/share/hscolour-${HSCOLOUR_VERSION}

PORTDOCS=       *
.endif

.SILENT:

do-configure:

        cd ${WRKSRC} && ${GHC_CMD} --make Setup.lhs \
                                   -o setup -package Cabal \

                     && ${SETUP_CMD} configure \
                      --haddock-options=-w --prefix=${PREFIX}

do-build:

        cd ${WRKSRC} && ${SETUP_CMD} build \

                     && ${SETUP_CMD} register --gen-script

.if !defined(NOPORTDOCS)

        cd ${WRKSRC} && ${SETUP_CMD} haddock \

                 --hyperlink-source \

                 --hscolour-css=${HSCOLOUR_DATADIR}/hscolour.css

.endif

do-install:

        cd ${WRKSRC} && ${SETUP_CMD} install \
                     && ${INSTALL_SCRIPT} register.sh ${PREFIX}/${HTTP_LIBDIR_REL}/register.sh

post-install:

        ${RM} -f ${PREFIX}/lib/ghc-${GHC_VERSION}/package.conf.old

Port Description

We need to put a description into pkg-descr. I usually just copy whatever description is already part of the source. So that's straight forward.

Packing List

The last missing file is pkg-plist. It contains the listing of all files and directories created by our port. The packing list is very important, without it ports could not be cleanly deinstalled.

Handling the documentation

As we've seen in the paragraph about Documentation, you can specify PORTDOCS and PLIST_SUB dynamically within the Makefile instead of explicitly listing every file in pkg-plist. So the whole documentation installed in /usr/local/share/doc/YourPort/ is already taken care of. That's very convenient because sometimes it is difficult to tell beforehand what files will be installed as part of the documentation.

Handling the no-documentation

Remember, I've defined a substitution similar to the one above for NOPORTDOCS in case no documentation should be installed. In that case the LICENSE file will be put into the port's doc-folder nontheless. That, we have to deal with. So I'm going to add the following two lines to pkg-plist:

%%NOPORTDOCS%%%%DOCSDIR%%/LICENSE
%%NOPORTDOCS%%@dirrmtry %%DOCSDIR%%

Ok, so what does this do? Remember what we've defined in the Makefile:

.ifdef(NOPORTDOCS)

PLIST_SUB+=     NOPORTDOCS=""

.else

PLIST_SUB+=     NOPORTDOCS="@comment "

.endif

That means if NOPORTDOCS is defined, %%NOPORTDOCS%% will be substituted with and empty string. So the final packing list will contain the two lines:

%%DOCSDIR%%/LICENSE
@dirrmtry %%DOCSDIR%%

The @dirrmtry is an instruction to remove this port's doc-folder when it is being deinstalled, but only if that directory is empty. Otherwise it won't do anything. There is also the related @dirrm instruction. It will fail if the directory to remove in not empty. There is no such thing as @filerm to remove files. All listed files will be removed automatically during deinstallation.

On the other hand, if  NOPORTDOCS is not defined, %%NOPORTDOCS%% will be replaced with "@comment " and we end up with:

@comment %%DOCSDIR%%/LICENSE
@comment @dirrmtry %%DOCSDIR%%

All lines beginning with @comment will be ignored in the final packing list. So both cases are covered.

 

Handling pre-built packages

Until now I always explained things from the vantage point of what happens when you compile the port from source. But FreeBSD also supports the installation of pre-built packages (through pkg_add). We have to consider this as well. Think about it. We can easily compile a library and put it into a package. But when installing such a package, you also have to register the library. In the case of Haskell libraries, we have to register them with ghc. This is also done in pkg-plist with another two directives, @exec and @unexec respectively:

@exec /bin/sh %D/%%HTTP_LIBDIR_REL%%/register.sh
@exec /bin/rm -f %D/lib/ghc-%%GHC_VERSION%%/package.conf.old

This will run the register-script we've built in the do-build target. That script will be part of the package.

@unexec %D/bin/ghc-pkg unregister HTTP
@unexec /bin/rm -f %D/lib/ghc-%%GHC_VERSION%%/package.conf.old

Unexec is the reverse operation that unregisters the Haskell module as part of the deinstallation process of the port. Be careful to unregister the correct one.

Handling the rest of the files

I promised to show you how to automatically generate pkg-plist. However, all of the above has to be managed manually. So what's left you ask? All the files that are compiled from the Haskell source. I'll use genplist to track those. The path /tmp/plists is the working directory of genplist. genplist's man page suggests using /tmp but that would cause a conflict since our  port is located there.

#mkdir /tmp/plists && genplist create /tmp/plists

 PORTNAME = HTTP
PREFIX = /tmp/plist/HTTP
===>  Deinstalling for www/hs-HTTP
===>   hs-HTTP not installed, skipping
===>   hs-HTTP-4000.0.9 depends on executable: ghc - found
===>   hs-HTTP-4000.0.9 depends on executable: haddock - found
===>   hs-HTTP-4000.0.9 depends on executable: HsColour - found
===>   hs-HTTP-4000.0.9 depends on executable: ghc - found
./bin missing (created)
./etc missing (created)
./etc/pam.d missing (created)
./etc/rc.d missing (created)
./include missing (created)
./include/X11 missing (created)
./info missing (created)
./lib missing (created)
./lib/X11 missing (created)
./lib/X11/app-defaults missing (created)
./lib/X11/fonts missing (created)

...

...

ar: warning: creating dist/build/libHSHTTP-4000.0.9.a
Writing registration script: register.sh for HTTP-4000.0.9...
Preprocessing library HTTP-4000.0.9...
Running hscolour for HTTP-4000.0.9...
setup: /tmp/plist/HTTP/share/hscolour-1.15/hscolour.css: copyFile: does not exist (No such file or directory)
*** Error code 1

Stop in /usr/ports/www/hs-HTTP.
*** Error code 1

Stop in /usr/ports/www/hs-HTTP.
       21.26 real        15.46 user         1.20 sys

What happened? genplist is working in its own sandbox in /tmp/plists. In our Makefile we specified the path to the hscolour css-file as ${PREFIX}/share/hscolour-1.15. However PREFIX has been set to /tmp/plists. I haven't really found a good solution to this. As a workaround, I will temporarily replace that path in our Makefile with the absolute path /usr/local/share/hscolour-1.15. I also have to delete the contents of /tmp/plists directory before I can start over.

#rm -r /tmp/plists/* && genplist create /tmp/plists

...

ocumentation created: dist/doc/html/HTTP/index.html
===>  Installing for hs-HTTP-4000.0.9
===>   hs-HTTP-4000.0.9 depends on executable: ghc - found
===>   Generating temporary packing list
===>  Checking if www/hs-HTTP already installed
Installing library in /tmp/plists/HTTP/lib/HTTP-4000.0.9/ghc-6.10.4
Registering HTTP-4000.0.9...
Reading package info from "dist/installed-pkg-config" ... done.
Writing new package config file... done.
===>   Registering installation for hs-HTTP-4000.0.9
       28.10 real        18.10 user         2.57 sys

This time it worked. A new file pkg-plist.new has been created. It contains all the files created by the port.

#vim pkg-plist.new

lib/HTTP-4000.0.9/ghc-6.10.4/HSHTTP-4000.0.9.o
lib/HTTP-4000.0.9/ghc-6.10.4/Network/Browser.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/BufferType.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/Auth.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/Base.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/Base64.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/Cookie.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/HandleStream.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/Headers.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/MD5.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/MD5Aux.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/Proxy.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/Stream.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP/Utils.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/Stream.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/StreamDebugger.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/StreamSocket.hi
lib/HTTP-4000.0.9/ghc-6.10.4/Network/TCP.hi
lib/HTTP-4000.0.9/ghc-6.10.4/libHSHTTP-4000.0.9.a
lib/HTTP-4000.0.9/register.sh
%%PORTDOCS%%%%DOCSDIR%%-4000.0.9/LICENSE
...

... more %%PORTDOCS%% stuff we can ignore

...

%%PORTDOCS%%%%DOCSDIR%%-4000.0.9/html/src/hscolour.css
%%PORTDOCS%%@dirrm %%DOCSDIR%%-4000.0.9/html/src
%%PORTDOCS%%@dirrm %%DOCSDIR%%-4000.0.9/html
%%PORTDOCS%%@dirrm %%DOCSDIR%%-4000.0.9
@dirrm lib/HTTP-4000.0.9/ghc-6.10.4/Network/HTTP
@dirrm lib/HTTP-4000.0.9/ghc-6.10.4/Network
@dirrm lib/HTTP-4000.0.9/ghc-6.10.4
@dirrm lib/HTTP-4000.0.9

We've already dealt with the PORTDOCS, so we can safely remove all those lines. We could keep the remaining lines - everything in lib/. However, imagine that the GHC version is updated to 6.12.1 in the near future. Then we would have to not only have to adjust the Makefile, but the pkg-plist as well. That would be much more complicated than necessary. So we also define plist substitutions for the LIBDIR_REL and GHC_VERSION in the Makefile.

PLIST_SUB+= LIBDIREL=lib/HTTP-4000.0.9

PLIST_SUB+= GHC_VERSION=6.10.4

Then we can rewrite the lines. For example the line:

lib/HTTP-4000.0.9/ghc-6.10.4/Network/Browser.hi

can be written as:

%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/Browser.hi

This way we can leave the pkg-plist untouched if the version of GHC or HTTP were to change. That is much more convenient. The complete pkg-plist looks like this:

%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/HSHTTP-%%HTTP_VERSION%%.o
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/Browser.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/BufferType.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/Auth.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/Base.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/Base64.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/Cookie.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/HandleStream.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/Headers.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/MD5.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/MD5Aux.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/Proxy.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/Stream.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP/Utils.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/Stream.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/StreamDebugger.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/StreamSocket.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/TCP.hi
%%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/libHSHTTP-%%HTTP_VERSION%%.a
%%HTTP_LIBDIR_REL%%/register.sh
%%NOPORTDOCS%%%%DOCSDIR%%/LICENSE
%%NOPORTDOCS%%@dirrmtry %%DOCSDIR%%
@dirrm %%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network/HTTP
@dirrm %%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%/Network
@dirrm %%HTTP_LIBDIR_REL%%/ghc-%%GHC_VERSION%%
@dirrm %%HTTP_LIBDIR_REL%%
@exec /bin/sh %D/%%HTTP_LIBDIR_REL%%/register.sh
@exec /bin/rm -f %D/lib/ghc-%%GHC_VERSION%%/package.conf.old
@unexec %D/bin/ghc-pkg unregister HTTP
@unexec /bin/rm -f %D/lib/ghc-%%GHC_VERSION%%/package.conf.old

To copy the pkg-plist.new to pkg-plist do commit.

#genplist commit

To test whether your pkg-plist really covers everything you can test it with.

#genplist test

If everything checks out, you can clean up.

#genplist clean

Testing the Port

At this point our port is complete. Time to test it.

#port test

===> Validating port with portlint
WARN: Makefile: [70]: possible direct use of command "install" found. use ${INSTALL_foobaa} instead.
WARN: Makefile: possible use of absolute pathname "/usr/local/share/hsc...".
WARN: Makefile: possible direct use of "/usr/local" found. if so, use ${PREFIX} or ${LOCALBASE}, as appropriate.
WARN: Makefile: only one MASTER_SITE configured.  Consider adding additional mirrors.0 fatal errors and 4 warnings found.
===> flags: PREFIX=/tmp/hs-HTTP-4000.0.9 NO_DEPENDS=yes PKG_DBDIR=/tmp/pkg_db.AkQQPcPV --
===> Cleaning workspace before port test
===>  Cleaning for hs-HTTP-4000.0.9
===>  Vulnerability check disabled, database not found
===>  Extracting for hs-HTTP-4000.0.9
=> MD5 Checksum OK for HTTP-4000.0.9.tar.gz.
=> SHA256 Checksum OK for HTTP-4000.0.9.tar.gz.
===>  Patching for hs-HTTP-4000.0.9
===>  Configuring for hs-HTTP-4000.0.9
[1 of 1] Compiling Main             ( Setup.lhs, Setup.o )
Linking setup ...
Configuring HTTP-4000.0.9...
...

...
===>  Installing for hs-HTTP-4000.0.9
===>   Generating temporary packing list
===>  Checking if www/HTTP already installed
Installing library in /tmp/hs-HTTP-4000.0.9/lib/HTTP-4000.0.9/ghc-6.10.4
Registering HTTP-4000.0.9...
Reading package info from "dist/installed-pkg-config" ... done.
Writing new package config file... done.
===>   Registering installation for hs-HTTP-4000.0.9
===>  Building package for hs-HTTP-4000.0.9
Creating package /usr/ports/packages/All/hs-HTTP-4000.0.9.tbz
Registering depends:.
Creating bzip'd tar ball in '/usr/ports/packages/All/hs-HTTP-4000.0.9.tbz'
===> Checking pkg_info
hs-HTTP-4000.0.9    A library for client-side HTTP
===> Checking shared library dependencies
===>  Deinstalling for www/HTTP
===>   Deinstalling hs-HTTP-4000.0.9
/tmp/hs-HTTP-4000.0.9/bin/ghc-pkg: not found
pkg_delete: unexec command for '/tmp/hs-HTTP-4000.0.9/bin/ghc-pkg unregister HTTP' failed
pkg_delete: couldn't entirely delete package (perhaps the packing list is
incorrectly specified?)
===> Extra files and directories check
===> Cleaning up after port test
===>  Cleaning for hs-HTTP-4000.0.9
===>  Removing existing /tmp/hs-HTTP-4000.0.9 dir
===> Done.

I've highlighted the important messages. The first three warnings come from portlint who is reminding us that we used an explicit path to hscolor.css for testing. Now is probably a good time to revert that change.

The other warning was generated because ghc-pkg was not found in the the /tmp/hs-HTTP directory. That's not very surprising either. For a "real" port unregistering will work.

Portlint also suggested specifying more than a one source in MASTER_SITE, which is good idea, but there doesn't seem to exist another one.

Submitting the Port

If all tests passed without errors, the last step is to submit the port. Before you do that make sure you read DOs and DON'Ts from the porters handbook to safe yourself and a potential committer a lot of headaches.

#port submit

This will fire up a problem report (PR) editor. The most important values have already been set and our new port has already been attached as shar-archive. When you close the editor, it asks you what to do with the report. Hit 's' to submit it.

s)end, e)dit or a)bort? s

After some time the report will be listed in GNATS and added to the portstree by one of the FreeBSD committers. If there are problems, you will be contacted. That's all there is to it : )