Le blog des salariés

Aller au contenu | Aller au menu | Aller à la recherche

jeudi, avril 10 2014

Backporter un package Debian testing vers Debian stable

On souhaite backporter un package de testing vers la stable. En l'occurrence, pour notre exemple on prendra le package python-numpy pour illustrer les manipulations.


Forger un paquet debian, peut parfois prendre des allures de champs de bataille, surtout lorsque le paquet en question a un certain nombre de dépendance. Pour cloisonner le travail, je vous propose d'utiliser deboostrap.

On créer un environnement ::

apt-get install debootstrap debootstrap --arch amd64 wheezy ~/dbs-builddeb http://ftp.fr.debian.org/debian/

On se chroot dans le debootstrap ::

chroot dbs-builddeb

Nous allons avoir besoin de quelques outils de développement, que nous installons ::

apt-get install devscripts build-essential dh-buildinfo echo "export LANG=C" >> ~/.bashrc

Howto par l'exemple

On configure apt dans /etc/apt/source.list, tel que ::


## Wheezy deb http://ftp.fr.debian.org/debian wheezy main deb-src http://ftp.fr.debian.org/debian wheezy main # wheezy-backports deb http://ftp.fr.debian.org/debian wheezy-backports main contrib non-free ## Jessie #deb http://ftp.fr.debian.org/debian jessie main deb-src http://ftp.fr.debian.org/debian jessie main

On update le tout

apt-get update

On récupère les sources

apt-get source python-numpy

On récupère les dépendances, que l'on installe

apt-get build-dep python-numpy

On compile le code source

cd python-numpy-1.8.1 dch -i

python-numpy (1:1.8.1-1~etalabbpo70+1) unstable; urgency=low * Non-maintainer upload. * Backport to wheezy. -- Felix Defrance <felix.defrance@data.gouv.fr> Thu, 10 Apr 2014 14:22:32 +0000

dpkg-buildpackage -tc

C'est terminé ! On peut voir le package forgé dans le répertoire parent.

python-numpy_1.8.1-1~etalabbpo70+1.debian.tar.gz python-numpy_1.8.1-1~etalabbpo70+1_amd64.deb python-numpy_1.8.1-1~etalabbpo70+1.dsc python-numpy_1.8.1-1~etalabbpo70+1_amd64.changes

Installation du package

Pour une utilisation personnelle un dpkg -i suffira, sinon on ajoutera le package à un depot spécifiquement établi pour l'occasion par exemple..

mercredi, décembre 11 2013

Active/Backup iptables tracking connexions between two gateway

This setup is interesting when you want to avoid SPOF on your firewall/gateway which are on top of your network architecture.

This article is about how to improve high availability on stateful firewalls using netfilter's conntrack synchronization. In a later article we will discuss on how to automatically remove static routes when a gateway is down (Gateway Fail Over Wan)

Need stateful mode

Stateful based firewalling is now used on most part of firewalling architectures. The stateful mode is based on keeping track of the network connections to make sysadmin's life better ;)

To view active conntrack and deal with it, you could install conntrack package. it will provide this kind of commands :

conntrack -S (Show statistics)


conntrack -L (List conntrack)

Stateful Syncing between nodes

In our use case, we need to synchronize network connections tracking on two firewalls nodes. This is ensured by a daemon called conntrackd

apt-get install conntrackd

Conntrackd, has three replication approaches, “no track”, “ft-fw” and “alarm”.

  • no-track: use the best effort syncing tables and no control was made when tables are replicate.
  • ft-fw: use reliable protocol to perform message tracking. So that sync error or corruption are permitted.
  • alarm: Which allow to set syncing tables interval. This option require a lot of bandwhitch.

More information: http://conntrack-tools.netfilter.org/manual.html#sync

We choose ft-fw mode because it's ready for production environnement, more stable and it works well.

To use ft-fw, you could reuse example as your configuration and make some little changes, as your network addresses.

zcat /usr/share/doc/conntrackd/examples/sync/ftfw/conntrackd.conf.gz > /etc/conntrackd/conntrackd.conf

Conntrackd, should start as daemon at boot starting, so we define this by init scripts and /etc/default/conntrackd in Debian.

Iptables Rules

As you drop all undesired traffic, we need to add some rules to allow traffic came from conntrackd on both nodes:

# ------------------------- Conntrack
iptables -A INPUT -p udp -i $IFCONN -d --dport 3780 -j ACCEPT
iptables -A INPUT -p udp -i $IFCONN -s $IPCONN  --dport 694 -j ACCEPT

Check your synchronisation

As your configuration should work without any problem, now we could play with the daemons.

Conntrackd, provide commands that they works like a client/server. So we can ask conntrackd by cli commands to know cache / statistics /etc..

Here are some examples :

To show tables which are synchronised , we could use this commands. See external cache (cache which is on gw02 was synchronised to gw01):

root@gw02:~# conntrackd -e 

See internal cache :

root@gw02:~# conntrackd -i

You can compare results and counting them :

root@gw02:~# conntrackd -e | wc -l
root@gw02:~# 325
root@gw01:~# conntrackd -i | wc -l
root@gw02:~# 328

And show more statistics :

conntrackd -s

As you can see, ft-fw is asynchronous. Our setup is “Active-Backup”. You can sync mannually for fun:

root@gw02:~# conntrackd -n

Conntrackd, provide Active-Active setup but it's still in asymmetric mode. For more information you can read the manual : http://conntrack-tools.netfilter.org/manual.html#sync-aa

mardi, novembre 5 2013

Handling misencoded HTTP request in Python WSGI applications

At Easter-eggs we use Python and WSGI for web applications development.

The last few months some of our applications crashed periodically. Thanks to WebError ErrorMiddleware, we receive an email each time an internal server error occurs.

For example someone tried to retrieve all of our french territories data with the API.

The problem is simple: when the request headers contains non UTF-8 characters, the WebOb Request object throws an UnicodeDecodeError exception because it expects the headers to be encoded in UTF-8.

End-user tools like web browsers generate valid UTF-8 requests with no effort, but non UTF-8 requests can be generated by some odd softwares or by hand from a ipython shell.

Let's dive into the problem in ipython :

In [1]: url = u'http://www.easter-eggs.com/é'

In [2]: url
Out[2]: u'http://www.easter-eggs.com/\xe9'

In [3]: url.encode('utf-8')
Out[3]: 'http://www.easter-eggs.com/\xc3\xa9'

In [4]: latin1_url = url.encode('latin1')
Out[4]: 'http://www.easter-eggs.com/\xe9'

In [5]: latin1_url.decode('utf-8')
[... skipped ...]
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 27: unexpected end of data

This shows that U+00E9 is the Unicode codepoint for the 'é' character (see Wikipedia), that its UTF-8 encoding are the 2 bytes '\xc3\xa9', and that decoding in UTF-8 a latin1 byte throws an error.

The stack trace attached to the error e-mails helped us to find that the UnicodeDecodeError exception occurs when calling one of these Request methods: path_info, script_name and params.

So we wrote a new WSGI middleware to reject mis-encoded requests, returning a bad request HTTP error code to the client.

from webob.dec import wsgify
import webob.exc

def reject_misencoded_requests(req, app, exception_class=None):
    """WSGI middleware that returns an HTTP error (bad request by default) if the request attributes
    are not encoded in UTF-8.
    if exception_class is None:
        exception_class = webob.exc.HTTPBadRequest
    except UnicodeDecodeError:
        return exception_class(u'The request URL and its parameters must be encoded in UTF-8.')
    return req.get_response(app)

The source code of this middleware is published on Gitorious: reject-misencoded-requests

We could have guessed the encoding, and set the Request.encoding attribute, but it would have fixed only the read of PATH_INFO and SCRIPT_NAME, and not the POST and GET parameters which are expected to be encoded only in UTF-8.

That's why we simply return a 400 bad request HTTP code to our users. This is simpler and does the work.

vendredi, septembre 27 2013

When CEPH comes to rescue


Here @Easter-eggs[1], like others, we start playing with the awesome CEPH[2] distributed object storage. Our current use of it is the hosting of virtual machines disks.

Our first cluster was just installed this week on tuesday. Some non production virtual machines where installed on it and the whole cluster added to our monitoring systems.


On thursday evening, one of the cluster nodes went down due to cpu overhead (to be investigated, looks like a fan problem).

Monitoring systems send us alerts as usual, and we discovered that CEPH just did the job :) :

  • the server lost was detected by other nodes
  • CEPH started to replicate pgs between other nodes to maintain our replication level (this introduced a bit of load on virtual machines during the sync)
  • virtual machines that were running on the dead node were not alive anymore, but we just add to manually start them on another node (pacemaker is going to be setup on this cluster to manage this automagically)

On friday morning, we repaired the dead server, and boot it again:

  • the server automatically joined the CEPH cluster again
  • osd on this server were added automatically in the cluster
  • replication started to get an optimal replication state

Incident closed!

What to say else?

  • thanks to CEPH and the principle of server redundancy to let us sleep in our home instead of working a night in the datacenter
  • thanks to CEPH for being so magical
  • let's start the next step: configure pacemaker for automatic virtual machines failover on cluster nodes


[1] http://www.easter-eggs.com/

[2] http://ceph.com/

mardi, septembre 24 2013

[Libvirt] Migrating from on disk raw images to RBD storage


As we just configured our first CEPH[1] cluster, we needed to move our current virtual machines (using raw images stored on standard filesystem) so they use the RBD block device provided by CEPH.

We use Libvirt[2] and Kvm[3] to manage our virtual machines.


Migration with virtual machine downtime

This step can be done offline:

  • stop the virtual machine
 virsh shutdown vmfoo
  • convert the image to rbd
 qemu-img convert -O rbd /var/lib/libvirt/images/vmfoo.img rbd:libvirt-pool/vmfoo
  • update the VM configuration file
 virsh edit vmfoo
 <disk type='file' device='disk'>
   <driver name='qemu' type='raw' cache='none'/>
   <source file='/var/lib/libvirt/images/vmfoo.img'/>
   <target dev='vda' bus='virtio'/>
   <alias name='virtio-disk0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>


 <disk type='network' device='disk'>
   <driver name='qemu'/>
   <auth username='libvirt'>
     <secret type='ceph' uuid='sec-ret-uu-id'/>
   <source protocol='rbd' name='libvirt-pool/vmfoo'>
     <host name='' port='6789'/>
     <host name='' port='6789'/>
   <target dev='vda' bus='virtio'/>
   <alias name='virtio-disk0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
  • restart the virtual machine
virsh start vmfoo

Migration without downtime

The trick here is to use migration support in libvirt/kvm and the ability to provide a different xml definition for the target virtual machine:

  • get the current vm disk informations
 qemu-img info /var/lib/libvirt/images/vmfoo.img
  • create an empty rbd of the same size
 qemu-img create -f rbd rbd:libvirt-pool/vmfoo XXG
  • get the current vm configuration
 virsh dumpxml vmfoo > vmfoo.xml
  • edit this configuration to replace the on disk image by the rbd one
 <disk type='file' device='disk'>
   <driver name='qemu' type='raw' cache='none'/>
   <source file='/var/lib/libvirt/images/vmfoo.img'/>
   <target dev='vda' bus='virtio'/>
   <alias name='virtio-disk0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>


 <disk type='network' device='disk'>
   <driver name='qemu'/>
   <auth username='libvirt'>
     <secret type='ceph' uuid='sec-ret-uu-id'/>
   <source protocol='rbd' name='libvirt-pool/vmfoo'>
     <host name='' port='6789'/>
     <host name='' port='6789'/>
   <target dev='vda' bus='virtio'/>
   <alias name='virtio-disk0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
  • start the migration process
virsh migrate --live --persistent --copy-storage-all --verbose --xml vmfoo.xml vmfoo qemu+ssh://target_node/system
  • wait until the process finished. The time to wait depends of your cluster performances and your VM size, but there is no interruption of the virtual machine!
  • you're done, your virtual machine is now running over rbd and once checked you can safelly archive or destroy your old disk image.


  • of course you have to use libvirt/kvm with rbd support on target node
  • you have to use a recent version of kvm, we had memory exhaustion problems on the hypervisor during the migration process with debian wheezy version


[1] http://ceph.com/

[2] http://libvirt.org/

[3] http://www.linux-kvm.org/

Convert an unsparse vm image to sparse vm image

Convert an unsparse vm image to sparse vm image

Few weeks ago, I needed to convert qcow2 image to raw image. So, I executed this command:

qemu-img convert -f qcow2 -O raw vm-foo.qcow2 vm-foo.raw

After that, I had an unsparse image because qemu-img don't output sparse file. I saw this by running this command:

qemu-img info vm-foo.img


ls -lksh vm-foo.img

So now, I want to convert this new vm-image to a sparse file, because I want to free space in my file system. As you could know, in sparse file, zero-data, don't takes place in your file system instead of unsparse file.

Moreover, when files are deleted, their data stay in place on the disk (just indexes was deleted).

In my case, i want to optimize my future sparse file vm-image, and I decide to force zero-data in my vm-image.

So, on my started guest, I wrote zero-data as far as possible, using this command:

root@foo# dd if=/dev/zero of=/tmp/zerotxt bs=1M
root@foo# sync
root@foo# rm /tmp/zerotxt

Now, I shutdown the guest, and convert unsparse file to sparsed file by using cp command:

cp --sparse=always vm-foo.raw vm-foo.raw-sparse

Well done, I got a clean sparse file!

  qemu-img info vm-foo.raw-sparse
image: vm-foo.raw-sparse
file format: raw
virtual size: 40G (42949672960 bytes)
disk size: 6.3G
This article on my blog.

vendredi, septembre 13 2013

How to self published your code with Git over http


Today i want to publish my scripts. Few days ago, I decided to use Git to release them. But it's only visible by me on my servers. So i decided to use Viewgit, a web interface in php. It's cool! Now, i can see my scripts with my browser! But in fact, I'm unhappy because nobody is able to use git mechanism like “git clone”. So, I want to use “git over http”, with git-http-backend.

For this environment, I use Nginx web server over Debian to serve files.


The installation of viewgit is pretty easy, just download, untar and play. You must drop your git projects, in “projects” directory, like me :


And declare your projects in /var/www/viewgit/inc/localconfig.php

Your nginx config looks like this at this time :

vi /etc/nginx/sites-available/viewgit
server {
root /var/www/viewgit;
index index.php;
server_name git.d2france.fr;
location ~ \.php$ {
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

Git over http

Before using git over http, you need to know two fundamentals. First, you want to allow people to download your projects, and second, you want to allow people to make modifications on your projects.

To play around git clone, fetch and pull requests, git uses http.uploadpack service.

To play around git push, git uses http.receivepack service.

To provide those services, your need to use GIT-HTTP-BACKEND as a backend cgi script for your web server and nginx cgi server (fcgiwrap) to run it.

apt-get install git-http-backend fcgiwrap

With Nginx, the configuration could be like this :

server {
root /var/www/viewgit;
index index.php;
server_name git.d2france.fr;
location ~ \.php$ {
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
location ~ ^projects/.*/(HEAD|info/refs|objects/info/.*|git-upload-pack)$ {
root /var/www/viewgit/projects;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME   /usr/lib/git-core/git-http-backend;
fastcgi_param PATH_INFO         $uri;
fastcgi_param GIT_PROJECT_ROOT  /var/www/viewgit/projects;
fastcgi_param GIT_HTTP_EXPORT_ALL "";
fastcgi_pass unix:/var/run/fcgiwrap.socket;

Here, i just want to share my scripts. So, I only allow git-upload-pack requests

It works!

Now you can clone your git repositories with this command:

 git clone http://server/projects/foobar

As you can see, on each project in viewgit you can't add any information like the url of your git. A friend made a plugin for that. You should find his work at viewgit-projectinfos-plugin.

This article on my blog.

vendredi, septembre 25 2009

I've lost my files !!!


À qui cela n'est-il donc jamais arrivé ???

Avec le système de fichiers ext2 il 'y avait la possibilité de retrouver des fichiers effacés à l'aide de Midnight Commander mais avec Ext3 que néni.

Après quelques recherches, j'ai découvert ext3undel. Celui-ci peut être utiliser directement, avec ses options, ou bien propose deux autres exécutables, gabi et ralf qui sont plus simples d'utilisations mais plus globaux.

Ext3undel s'appuie, entre autre, sur foremost, disponible en paquets Debian (Lenny).

Foremost permet de restaurer des fichiers effacés du système en spécifiant des types de fichier, la partition ou image dd concernée, un répertoire de destination des fichiers restaurés ainsi que quelques autres options... Il a l'avantage de pouvoir travailler dur une partition montée ce qui n'est pas forcément le mieux pour éviter l'écrasement d'inodes mais permet de travailler sur des filsystem que l'on ne peut pas démonter ou monter en readonly.

Dans le cas présent, après avoir créer le répertoire de destination, j'ai utilisé la commande :

foremost -s 512 -o /root/photo-finds/ -t jpg /dev/md1

Le résultat est surprenant, pour un répertoire effacé d'une taille d'environ 5 Go, il a récupéré 199 Go de fichiers correspondant au type et sur la partition que j'avais spécifié. Foremost a donc restauré des fichiers effacés depuis longtemps :)