[EN] FreeBSD QOS with FreeBSD DUMMYNET

The latest released FreeBSD version 6.2  comes with 3 firewall systems:

- pf (ported from OpenBSD)
- ipfw
- ipfilter

All these three are stable as of this release. I've recently needed to run a QOS system, and chosen
ipfw "DUMMYNET" for this purpose.

My network is about couple hundreds computers and I needed limit bandwidth for each user.

I chose DUMMYNET because of it's simplicity, and "dynamic pipe" creation. I know "pf" (especially HFSC alghoritm) is a good solution too, but natively it supports up to 64 queues
per interface, and you need to patch the source in order to enable more. It's safe to use hundreds of queues, but I simply don't like patching.. (at least by now).

DUMMYNET provides FreeBSD users with dynamic pipe creation.

Suppose you want to create a new 256Kbit pipe for incoming, and 128Kbit pipe for outgoing traffic.

# ipfw pipe 1 config bw 256Kbit/s mask dst-addr 0xffffffff
# ipfw pipe 1 config bw 128Kbit/s mask src-addr 0xffffffff

Now, you need to put the traffic into these pipes.

# ipfw add pipe 1 ip from any to 192.168.0.0/24, 192.168.1.0/24
# ipfw add pipe 2 ip from 192.168.0.0/24, 192.168.1.0/24 to any

By these 4 lines of code you will get hunderds of pipes for each ip in the network for
both incoming and outgoing traffic.

You could write it this way as well:

# ipfw pipe 1 config bw 256Kbit/s
# ipfw pipe 2 config bw 256Kbit/s
...
# ipfw pipe 345 config bw 256Kbit/s

and appropriate rules:
# ipfw add pipe 1 ip from any to 192.168.0.2
# ipfw add pipe 2 ip from any to 192.168.0.3
....
# ipfw add pipe 345 ip from any t0 192.168.1.23

I think the first solution is better!

Now that we stand with another problem - some users have more bandwidth enabled, some less.

You could write pipes with mask-src (dynamic), and put hosts into these.. which would cause lots of rules written.

ipfw (ipfw2 for FreeBSD 4.X) comes with lookup tables, which are extremely fast!

There are 2 values stored in each record of the table:
ip address or network, integer

To create a new table and put the record you would:

# ipfw table 1 add 192.168.0.2 23
# ipfw table 1 add 192.168.0.3,345

These 2nd values lets filter by these in the rules.

# ipfw add allow ip from table\(1\,23) to any # ( = ipfw add allow ip from 192.168.0.2 to any)

Now, in order to create more complex ruleset for pipe assigning look at the example below:

# incoming traffic
# ipfw pipe 1 config bw 1024Kbit/s mask dst-addr 0xffffffff
# ipfw pipe 2 config bw 512Kbit/s mask dst-addr 0xffffffff
# ipfw pipe 3 config bw 256Kbit/s mask dst-addr 0xffffffff

# outgoing traffic
# ipfw pipe 11 config bw 1024Kbit/s mask src-addr 0xffffffff
# ipfw pipe 12 config bw 512Kbit/s mask src-addr 0xffffffff
# ipfw pipe 13 config bw 256Kbit/s mask src-addr 0xffffffff

# Create 2 lookup tables, one with values (IP,INCOMING_BANDWIDTH), second with (IP, OUTGOING_BANDWIDTH)
# TABLE 1 (incoming)
# ipfw table 1 add 192.168.0.2 1024
# ipfw table 1 add 192.168.0.3 1024
# ipfw table 1 add 192.168.1.0/24 512 # each host in network 192.168.1.0/24 gets 512 Kbit/s)
....
# ipfw table 1 add 192.168.2.23 256

# TABLE 2 (outgoing)
# ipfw table 2 add 192.168.0.2 512
# ipfw table 2 add 192.168.0.3 512
# ipfw table 2 add 192.168.1.0/24 256 # yeah, network 192.168.1.0 got too much traffic last month, now they will get less

# RULES
# rule for incoming pipe 1024kbit (pipe #1)
# ipfw add pipe 1 ip from any to table\(1,1024\)
# rule for incoming pipe 512 (pipe #2)
# ipfw add pipe 2 ip from any to table\(1,512\)
# rule for incoming pipe 256 (pipe #3)
# ipfw add pipe 3 ip from any to table\(1,256\)

# outgoing traffic
# fule for outgoing pipe 1024Kbit (pipe #11)
# ipw add pipe 11 ip from table\(2,1024\) to any
# rule for outgoing pipe 512Kbit (pipe #12)
# ipfw add pipe 12 ip from table\(2,512\) to any
# rule for outgoing pipe 256Kbit (pipe #13)
# ipfw add pipe 13 ip from table\(2,256\) to any

That's all! No more hundreds of static pipes, hundreds of rules.

Keep in mind, that this example is a strcitly pipe related.
Depending on the NAT you are using, make sure these rules are evaluated
before being nated (if using "natd") for outgoing traffic, and after being nated (if using "natd")
for incoming traffic.

You can get this by using rules like:

# ipfw add 100 divert natd ip from any to any in recv $OUTGOING_INTERFACE

Put all the pipe rules here

# ipfw add 4000 divert natd ip from any to any out xmit $OUTGOING_INTERFACE

NOTE:
1. Remember to set sysctl variable "net.inet.ip.fw.one_pass" to 0, which prevents packets from
not being reinjected into the firewall.

2. You may also want to put a "skipto" rule after ip to pipe assigning (if you are using lots of such rules).

3. Personally I'm using ipnat+dummynet, which works fine for bigger networks. natd is not
fast, as it works in userland. ipnat is entirely kernel attached, so it's faster.

4. If using FreeBSD 4.x make sure you apply the ipfw/ipfilter order patch (lookup for "ipfw ipnat order patch" in google)

5. Assign appropriate "queue" values for each pipe depending on the pipe bw parameter.

6. In FreeBSD 5.5? 6.X you can even simplify the ruleset, read about "tablearg" parameter for ipfw!

7. Tune all "net.inet.ip.dummynet.*" sysctls if there are lots of hosts in your network, and there's a danger of creating lots of dynamic pipes.

8. Some guys use pf+dummynet for FreeBSD 6.X nat. I think it's tricky too, as pf is a great firewall too!

Some useful links:
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-ipfw.html
http://info.iet.unipi.it/~luigi/ip_dummynet/

[EN] FreeBSD multiple Jails with "ezjail"

Linux has a great support for VPS systems, so that you can easily create
multiple virtual environments using xen f.e.

As my favourite OS is FreeBSD I've decided to look for a similar solution.
I've been aware of great "jail" feature, but managing multiple jails
is not that handy as it is in linux.

Recently, I found a great solution , "ezjail". ( /usr/ports/sysutils/ezjail )

"ezjail" is a shell script which lets you easily:

- create new jails
- running/shutting down jails
- update existing jails

You can create jail environment as:
- file system in your existing os
- file system in either plain, gdbe or geli encrypted image

Before you begin, make sure you are familiar with jails by reading:

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html

A very interesting note is that "ezjail" doesn't create a completely new base system for new jails.
Instead it makes use of mount_nullfs and create "ro" mounts of the main base system
dirs like "/bin", "/sbin" etc.

When you create a new jail, it creates the "basejail" which in my case was 114MB size.
Any other jails (in case you don't do any other customizations) is 2MB large!.


Let's get started with "ezjail" !

First, create the base jail system.

# ezjail-admin install

This will fetch files needed for base system according to your current os version (`uname -r`)

# ezjail-admin create testjail 192.168.3.1
where IP is the IP you assing to your jail. Make sure it's already existing and set up.

Let's list existing jails now.

# ezjail-admin list
STA JID IP Hostname Root Directory
--- ----- --------------- ---------------------------- -------------------------
DS N/A 192.168.3.21 testjail /usr/jails/testjail
#

"DS" flags mean that it's a Directory Based jail and it's Stopped.

To create a geli encrypted image you would:

# ezjail-admin create -s 200M -c eli gelibasedjail 192.168.3.22
200+0 records in
200+0 records out
209715200 bytes transferred in 12.010512 secs (17460971 bytes/sec)
Initialising crypto device. Enter a new passphrase twice...
Enter new passphrase:
Reenter new passphrase:

Your jails is ready!
You can easily attach your jail filesystem:

# ezjail-admin config -i attach gelibasedjail
Attaching eli device for image jail ...
Enter passphrase:

Let's check the system mounts
# mount
...
/dev/md0.eli on /usr/jails/gelibasedjail (ufs, local)
...

and go into your jail file system:

# cd /usr/jails/gelibasedjail

# ls
.snap bin etc media rescue sys var
COPYRIGHT boot lib mnt root tmp
basejail dev libexec proc sbin usr


To start your jails on reboot, add ezjail_enable="YES" entry into your /etc/rc.conf file.

After reboot all your jails (which do not require password) will start.

You can also start/shutdown particular jails with:

# /usr/local/etc/rc.d/ezjail.sh start|stop gelibasedjail

You would ask "What if I want some customizations when creating new jails?".

"ezjail" comes with "flavours" functionality.

Let's say you want to create a new bunch of jails for HTTP web serving.
Your customers will put the Web data in /web/sites, and you want to avoid creating
this directories anytime you create a new jail.

With ezjail it's simple!

# cd /usr/jails/flavours/
# mkdir httpd
# cd httpd
# mkdir -p /web/sites

Now when creating new jail let "ezjail" know about the flavour:
# ezjail-admin create -f httpd newjail JAILIP

This will create a new jail with /web/sites directory created within your new jail!
The directory tree will recursively be copied to your new jail.

You can also perform other desired actions like running script when initiating a new jail install.
Create users, fetch packages.. do whatever you want.

Put the "ezjail-flavour" script in your new flavour jail definition (/usr/jails/httpd/ezjail-flavour) in my example.

Hope I convinced you to try "ezjail".

Some notes:
- it does create/delete /etc/fstab.jailname entries, so you don't have to worry about filesystem mountpoints after reboot. (you can avoid running specified jail at startup by `ezjail-admin config -r run|norun jailname` command)
- it's about creating/deleting/running/shutting down jails at the moment. It does not allow you
to manage jail f.e. resource usage. Look at: http://wiki.freebsd.org/JailResourceLimits for more information.


Look at the URL below, to find out more about the functionality of "ezjail".
Also, consider donation :)

http://erdgeist.org/arts/software/ezjail

[EN] Django extending User, custom User methods

Django doesn't allow to subclass the User model yet.

Each Django model lets you easily create model methods.
Behind the scene, each django model has its own methods, f.e. save()
which saves the model data into the database.

Django provides the lightweight User framework in "django/contrib/auth/",
where "User" model is defined with fields:

username, password, is_staff, first_name, last_name, email, date_joined, last_login,
is_superuser, is_active

What if we need the user model with additional data like: age, location ?

The answer is: "extending the User model" by creating a new model
and linking it with "User" model via ForeignKey.

Example:

from django.contrib.auth.models import User

class Userprofile(models.Model):
        user = models.ForeignKey(User, unique=True)
        age = models.IntegerField()
        location = models.CharField(maxlength="100", verbose_name"Country")

To create a new user, django comes with a special "create_user" method.

user = User.objects.create_user('robert', 'rsome@email', 'somepassword')
user.save()

You could do this also using the standard model create like:

user = User(username='someusername', is_active=True, email='some@email', first_name='john', last_name='doe')
user.save()
user.set_password('somepassword')
user.save()

As we have a new User record, to create a new "linked" UserProfile you would:

user = User.objects.get(name='someusername')
user.userprofile_set.create(age=22, location='Poland')

Django will automagically create the new UserProfile with "user" ForeignKey pointing
to the right User instance. Cool !

Question! How do I work on two separated models? Do I need to grab the UserProfile
instance for each request.user I operate in my views ?

Answer: Normally, you woule have to use the code below:

userprofile = Userprofile.objects.get(user=request.user)

Django comes with a special settings.py variable "AUTH_PROFILE_MODULE' which should point to your Userprofile model.

Example:
AUTH_PROFILE_MODULE = 'yourapp.Userprofile'

which lets you easily get into the "UserProfile" by using the "get_profile()" method.

Example: (in your either views.py or template):

location = request.user.get_profile().location
{{ user.get_profile.location }}

Note:
user_profile() does not only allows you to get the model data.
It allows you to get the model methods as well.

Example:

class Userprofile(models.Model):
        user = models.ForeignKey(User, unique=True)
        age = models.IntegerField()
        location = models.CharField(maxlength="100")

        def __str__(self):
                return self.location

        def is_of_age(self):
                if self.age < 18:
                        return False
                else:
                        return True

This lets you do some conditioning in your views:

if request.user.get_profile().is_of_age():
        print "Yes, I will sell you this beer"

Delicious!

Another +1 for Django !

Links for reading:
http://www.djangoproject.com/documentation/authentication/

[EN] Django model managers, Do Not Repeat Yourself !

I've recently come up with a great django feature "model Managers".

Each django model has it's own manager class, which can be overriden by your custom class.

Let's say there's a model containing fields:

class Person(models.Model):
        name = models.CharField(maxlength="100", blank=False, verbose_name="Your name")
        age = models.IntegerField(verbose_name="Your age")

        def __str__(self):
                return self.name

You need to use this model in many places, and you need to select people by age and mark
them as child, teen, older.

Each time you select them you would need to crate a queryset like:

children = Person.objects.filter(age__lt=13)
teen = Person.objects.filter(age__gte=13).filter(age__lt=18)
older = Person.objects.filter(age__gte=18)

Anytime you use this function, you would need to write the filter statements.

Thanks to model managers, you can do it once, and use it in your code anywhere.

class PersonManager(models.Manager):
        def get_children(self):
                return self.filter(age__lt=13)
        def get_teens(self):
                return self.filter(age__ge=13).filter(age_lt=18)
        def get_olders(self):
                return self.filter(age__gte=18)

and put:
objects = PersonManger() in your model class so the class looks like:

from django.db import models

class Person(models.Model):
        name = models.CharField(maxlength="100", blank=False, verbose_name="Your name")
        age = models.IntegerField(verbose_name="Your age")

        objects = PersonManger()

                def __str__(self):
                        return self.name

To select teens, you can use: Person.objects.get_teens().
You can use any standard queryset options like .filter, .exclude etc, for example:

To get teens, with name not starting with "s".
Person.objects.get_teens.exclude(name__startswith: "s")

Also, you can change parameters for model managers functions.

Let's say you want to select teens, olders, children with name starting with some letter.

Modify your model manager functions, so they look like (I will modify just the one here).

def get_teens(self, name_filter_letter=None):
        if name_filter_letter is None:
                return self.filter(age__ge=13, age_lt=18)
        else:
                return self.filter(age__ge=13).filter(age__lt=18).filter(name__startswith=name_filter_letter)

And use it like:
Person.objects.get_teens("s")

Custom managers can do many other things, they are a standard python methods.

Note that you don't need to use the name:
objects = PersonManager()

You can use your own name as well:
persons = PersonManager()

which results in:
Person.persons.get_teens()

One more thing:
You can go deeply and override the models.Manager get_query_set() method, for example:

You want to grab teens from the model:

Create model subclass like:

class PersonTeensManager(models.Manager):
        def get_query_set(self):
                return super(PersonTeensManager, self).get_query_set().filter(age__lt=18).filter(age__gte=3)

and put:

teens = PersonTeensManager() in your model class.

In your views, use the queryset:

teens = Person.teens.all()

Isn't that simple?!

For more documentation take a look here:
http://www.djangoproject.com/documentation/models/custom_managers/
http://www.djangoproject.com/documentation/models/get_object_or_404/