Selection of most fitting partition/backend upon account creation

[Julien Coloos]

Hi,

In latest cyrus versions (beginning with 2.3.x branch), cyrus is
capable of selecting the most fitting partition when creating a new
account. To do so, the default partition has to be left unspecified,
and in this case cyrus selects the partition with the most free space.
Similarly, when creating an account from a murder proxy, it is
possible to let cyrus select the backend with the most free space.

Due to some of our clients needs, we had to rework those features to
handle other modes of selection (see below). We currently have code
for 2.3.16 version.
If you are interested, we could create a patch out of it, adapted for
2.4.5 version, and add an entry for it on bugzilla.


Here are some details on the changes we made:

Partitions/backend are managed using a new structure which contains
all valuable data. Those data are retrieved the first time it is
needed, and then cached (configuration allows to refresh data).

New configuration options manage partition/backend selection:
   - partition_mode: how most fitting partition is selected; value is one of
      - random: (pseudo-)random selection
      - freespace-most: partition with the most free space (KiB)
         -> same as current cyrus behaviour
      - freespace-percent-most: partition with the most free space (%)
      - freespace-percent-weighted: each partition is weighted
according to its free space (%)
         -> the more free space the partition has, the more chances it
has to be selected
      - freespace-percent-weighted-delta: each partition is weighted
according to its difference of free space (%) compared to the most
used partition
         -> the more the partition is lagging behind the most used
partition, the more chances it has to be selected
         -> actually the code is made so that even the most used
partition has a few chances to be selected, and those chances increase
when other partitions get closer
   - partition_mode_exclude: list of partitions to exclude from selection mode
   - partition_mode_weighted_usage_limit: limit of partition usage (%)
      -> if a partition is over that limit, it is automatically
excluded from selection mode
      -> if all partitions are over that limit, this feature is not used anymore
   - partition_mode_usage_reinit: for a given session, number of
"operations" (e.g. partition selection) for which partitions usage
data are cached
      -> useful for clients that massively create new mailboxes using
the same session

   - serverlist_mode: same as partition_mode, but used on proxy for
selecting most fitting backend
      - random: (pseudo-)random
      - freespace-most: backend with the most (total) free space (KiB)
         -> same as current cyrus behaviour
      - freespace-percent-most: backend whose partition has the most
free space (%)
         -> and not the backend with the most (total) free space (%);
the goal is to compare the most fitting partition of each backend
      - freespace-percent-weighted: same as for partition selection,
comparing the free space (%) of the least used partition of each
backend
         -> again the goal is to compare the most fitting partition of
each backend
      - freespace-percent-weighted-delta: same as for partition
selection, comparing the free space (%) of the least used partition of
each backend
         -> again the goal is to compare the most fitting partition of
each backend
   - serverlist_mode_weighted_usage_limit: same as
partition_mode_weighted_usage_limit
   - serverlist_mode_usage_reinit: same as partition_mode_usage_reinit

Necessary data are fetched from backend through new annotations
(vendor name was replaced by yours):
   - "/vendor/cmu/cyrus-imapd/freespace/total": returns the (sum of)
available and total space of configured partitions, in the form
"available;total"
   - "/vendor/cmu/cyrus-imapd/freespace/percent/most": returns the
available and total space of the partition with the most free space
(%), in the form "available;total"

For the proxy, the function find_free_server was modified to select
the most fitting backend according to the configured mode, thanks to
the two new annotations.

For the backend, the function find_free_partition was replaced to
select the most fitting partition according to the configured mode.


We refactored our code recently, and if necessary the options can be
renamed (as well as the different modes or annotations): better use a
consistent naming scheme if you decide to integrate it in cyrus :)
So, let us know if you are interested in that code.

Regards