peper's blog

After a fair time of using the hook described in the previous post I realized that it noticeably slows down the uninstalls of packages with lots of files like sys-kernel/gentoo-sources and hence I decided to rewrite it as a Python hook to see how it performs:

UNINSTALL_PROTECT=["/lib64/modules/"]

def hook_run_unmerger_unlink_file_override(env, hook_env):
    for path in UNINSTALL_PROTECT:
        if hook_env["UNLINK_TARGET"].startswith(path):
            return "skip"
        else:
            return ""

I was expecting a pretty big difference as for Python hooks an embedded interpreter is used and .hook hooks need a separate bash interpreter each time they are run. Nonetheless I was still surprised to see how big the difference really was for sys-kernel/gentoo-sources-2.6.23-rX uninstallation:

.hook hook ~ 8m
python hook ~ 1m 15s
no hook ~ 1m

Python hook is not much slower than no hook at all and hence I didn't care to try and write it as a .so hook, which probably would be even faster.

Today, after a lot of polishing, I have finally committed Python Hooks to trunk.

If you are not familiar with Paludis Hooks, please read Hooks guide first.

A .py hook is much like .hook hook, but written in Python and with full access to the current Paludis environment through Python bindings. For each hook it can handle it must, at minimum, define a function named hook_run_$HOOK which accepts exactly two positional arguments: the current Environment used by Paludis, and the additional Hook environment variables represented by a Python dictionary. It may also define the hook_depend_$HOOK and hook_after_$HOOK functions which must take exactly one argument, the Hook environment, and return a list of strings. For example:

def hook_run_install_all_post(env, hook_env):
    from paludis import *

    print "* Checking for monkeys..."

    if list(env.package_database.query(Query.Package("nice/monkey"))):
        print "Found a monkey!"
    else:
        print "No monkeys found"

def hook_depend_install_all_post(hook_env):
    # we need to run after the Paludis standard eselect_env_update hook
    return ["eselect_env_update"]

def hook_after_install_all_post(hook_env):
    # if checking for rabbits or squirrels, do those first
    return ["check_for_rabbits", "check_for_squirrels"]

To play with Python Hooks, you need to install paludis-scm with python use-flag enabled and also have >=boost-1.34.0 installed.

Google Summer of Code is over, but, hopefully, the projects are not - at least I am not going to stop working on mine. I must say that SoC was very successful for me, I have learned plenty of new things and my dream of contributing to the Paludis itself came true. I must also thank all of the folks at #paludis, especially Ciaran McCreesh, my mentor Saleem Abdulrasool and rest of the Gentoo SoC staff for all the help.

By the way, it seems like a good time to start thinking about a project for SoC 2008 :)

As Python 2.5 is now unmasked in Gentoo I decided to try it with Boost.Python, more specifically with Python bindings for Paludis. Everything seems to work, but, of course, boost needs to be rebuilt first.

For quite some time I wanted to add docstrings to enums exposed with Boost.Python as my project's API docs seemed incomplete. Let's see how I found the solution eventually:

Enum we want to expose:

// Nice enum!
enum Foo
{
    blah
};

While exposing the enum itself couldn't be any simpler:

BOOST_PYTHON_MODULE(enum_test)
{
    bp::enum_<Foo> enum_foo("Foo");
    enum_foo.value("blah", blah);
    ...

... adding __doc__ is completly another story.

First (silly) attempt:

    ...
    PyObject_SetAttrString(enum_foo.ptr(), "__doc__", PyString_FromString("Nice enum!"));
};

"It must work" I thought, but what I got on import then was far from satisfying:

TypeError: attribute '__doc__' of 'type' objects is not writable


After digging in Boost.Python, Python C API docs and harassing people at #python:

    ...
    PyTypeObject * pto = reinterpret_cast<PyTypeObject *>(enum_foo.ptr());
    pto->tp_doc = "Nice enum!";
};

Thils looked really promising... and what? Nothing at all! Changing tp_doc had no effect whatsoever, and only after some more digging, it turned out it was too late to change it and the right way is:

    ...
    PyTypeObject * pto = reinterpret_cast<PyTypeObject *>(enum_foo.ptr());
    PyDict_SetItemString(pto->tp_dict, "__doc__", PyString_FromString("Nice enum!"));
};

Boost.Python and Python C API are fun ;)

Python bindings are now available in the paludis-scm.ebuild from the Paludis overlay. Just set the python use-flag and reinstall. Comments are welcome, but keep in mind it's far from the final version :]

Additional links: API docs, repository

I was supposed to give some updates about my SoC project, so here is the much delayed first one.

I have been working on the bindings since the moment I thought about the project and had some code ready even for the initial project proposal. After it had been accepted I was continuing the work with bigger and smaller breaks. Meanwhile I was doing some contributions to Paludis in other areas, learning its internals and getting better at C++.
I believe the first big date for my project was 5 April 2007 when bindings code was initially imported to the Paludis repository (r2881). The second big date is still coming, which will be the release of Paludis 0.26 with the Python bindings included.
The exact status of the bindings is best seen in the repository or in the API docs, which I update every now and then.

Currently I am working on bringing *DepSpec up to date and preparing for yet another Paludis API change. The bigger plan is to finish bindings for all the core classes sometime soon...

I need to make a follow-up again as the hack described in my previous post is now obsolote as Paludis' trunk has now _override hooks for merger and unmerger actions. They are a little different than all the hooks till now as their effect is determined from their output, more specifically unmerger_unlink_*_override has two options:

• "skip" - skips the action
• "force" - force it (without the usual tests like type or mtime check)

And merger_install_*_override has only the "skip" option. Also no output means the default.

Let me show an example:

/etc/paludis/hooks/unmerger_unlink_file_override/uninstall_protect.hook:

#!/bin/bash

UNINSTALL_PROTECT="/lib64/modules/"

hook_run_unmerger_unlink_file_override() {
    for PROTECT in ${UNINSTALL_PROTECT}; do
        if [[ "${UNLINK_TARGET}" == "${PROTECT}"* ]]; then
            echo "skip"
        fi
    done
}

As easily you can make INSTALL_MASK hook and others...

This and more in forthcoming 0.26.

This is somehow related to my previous posts about the moduledb as the idea also arose when playing with kernel modules.

When installing modules for a new kernel I noticed that Paludis treated them as any other package and removed the old installs leaving my old kernel without the modules. Nothing really surprising here, just different than the portage behaviour. Wanting to avoid this in the future my first thought was just to add /lib64/modules to the CONFIG_PROTECT, but after a second I realised it wasn't the best idea as I didn't want to review modules' changes when upgrading, I just wanted them to be not removed when uninstalling. Hence that it wasn't long until making an Unmerger Hook came to my mind.

/etc/paludis/hooks/uninstall_protect.hook:

#!/bin/bash

UNINSTALL_PROTECT="/lib64/modules/"

hook_run_unmerger_unlink_file_pre() {
    for PROTECT in ${UNINSTALL_PROTECT}; do
        if [[ "${UNLINK_TARGET}" == "${PROTECT}"* ]]; then
            mv "${UNLINK_TARGET}" "${UNLINK_TARGET}.protect"
        fi
    done
}

hook_run_unmerger_unlink_file_post() {
    if [[ -e "${UNLINK_TARGET}.protect" ]]; then
        mv "${UNLINK_TARGET}.protect" "${UNLINK_TARGET}"
        echo "protected '${UNLINK_TARGET}'"
    fi
}

And two symlinks pointing at it:
/etc/paludis/hooks/unmerger_unlink_file_pre/uninstall_protect.hook -> ../uninstall_protect.hook
/etc/paludis/hooks/unmerger_unlink_file_post/uninstall_protect.hook -> ../uninstall_protect.hook

This is only a short follow-up to my previous post.
Support of dynamic configuration files has been implemented a few days ago and for a start I have made two simple dynamic sets:

kernel-modules-ver.bash (set of kernel-modules with exact versions):

#!/bin/bash

sed -e 's/.*:/* =/' /var/lib/module-rebuild/moduledb

kernel-modules.bash (set of kernel-modules - unversioned):

#!/bin/bash

shopt -s extglob

while read PKG; do
    PKG=${PKG##*:}
    PKG=${PKG%%-scm*([[:digit:]])}
    PKG=${PKG%%-[[:digit:]]*([^-]|-[^[:digit:]])}
    echo "* ${PKG}"
done < /var/lib/module-rebuild/moduledb

And that's really only the beginning...

With portage I was quite often using module-rebuild, more or less after every kernel update and thus I thought it would be nice to make something similar for Paludis, especially as it seemed really simple with Paludis' Hooks and Sets.

So I made two simple hooks:

• ${paludis_confdir}/hooks/ebuild_postinst_post/modules_set_add.bash
• ${paludis_confdir}/hooks/ebuild_postrm_post/modules_set_rm.bash

And a simple one-liner to import entries from the moduledb:
sed -e 's/.*:/* =/' /var/lib/module-rebuild/moduledb > ${PALUDIS_CONFDIR}/sets/modules.conf

But then I realized that it's not so perfect:

• Won't work when using Environment != Paludis ( ${PALUDIS_CONFDIR} won't be reliable then )
• To actually rebuild the modules one has to use: paludis --dl-reinstall always --dl-deps-default discard -i modules and hope that the deps are satisfied :]
• Doubles the work of the linux-mod.eclass

So I decided to talk with Ciaran about the above issues. Yet while we were talking he implemented a new paludis option (--dl-reinstall-targets auto|always|never), which affects the way paludis treats targets. Up till now the default "auto" was always used(="never" for sets and "always" for ordinary packages). When it is released the module rebuild will be simply done with:
paludis --dl-reinstall-targets always -i modules and there won't be any risk of missing deps.
Going further ciaran thought about allowing bash scripts in sets/, in short paludis runs a foo.bash script from sets/ and its output is interpreted as foo set. The moment it is implemented (probably today) and released we can forget about the two hooks above and make a modules.bash similar to the import one-liner.

So don't forget to update Paludis when it's out ;]

Welcome to my brand new blog!

Thanks to seredipity setting it up was quite easy :] I've been thinking about making one for quite some time and today I have eventually decided to do so. Why today? Well, GSoC! My proposal was accepted as one of the Gentoo projects. In short, my task is to make python bindings for Paludis, the Other Package Mangler for Gentoo. If you want more details, check out my application and, if you are even more desperate, code, which I have already written. Also ciaran was nice enough to write a few words about it, but don't forget to ignore the rubbish about Python :]
So... I am planing to blog about progress of my project, some Gentoo stuff, and... time will tell.

P.S. 1) My name is Piotr Jaroszyński (hence the domain), I live in Warsaw and am first year Computer Science student at Warsaw University.
P.S. 2) I hope my pathetic 256 kbit/s upload will manage ;]