From brong at fastmail.fm  Mon Sep  1 22:17:55 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 02 Sep 2008 12:17:55 +1000
Subject: [PATCH] Set lock pointer at the start of foreach (Re: 2.3.12
	transaction problem within skiplist DB->foreach())
In-Reply-To: <8275B86A-F955-43BC-AFCD-CF551194504D@umich.edu>
References: <20080828221744.GA71671@exuma.irbs.com>
	<20080829063127.GA16173@brong.net>
	<8275B86A-F955-43BC-AFCD-CF551194504D@umich.edu>
Message-ID: <1220321875.31180.1271642283@webmail.messagingengine.com>


On Fri, 29 Aug 2008 09:36:40 -0400, "Wesley Craig" <wes at umich.edu> said:
> On 29 Aug 2008, at 02:31, Bron Gondwana wrote:
> > Please find attached a tested patch that fixes the problem and  
> > refactors
> > the bloody mess that was locking code into a couple of nice, neat
> > functions.
> 
> A quick rewiew looks good to me.  I'll review it closely this  
> afternoon and deploy it on a test cluster.  Presuming I find no  
> issues I'll commit it thereafter.  Any chance of submitting it to  
> bugzilla?

Sure.  Have done.  Bug 3075.

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm


From brong at fastmail.fm  Tue Sep  2 02:29:33 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 2 Sep 2008 16:29:33 +1000
Subject: On the topic of nested skiplist transactions
Message-ID: <20080902062933.GB26500@brong.net>

I've been thinking a bit about transactions and skiplists these past few
days, since sending the patch that refactors the skiplist locking code.

Basically the issue is this:  any piece of code that MIGHT be used
within a transaction currently needs a way to pass that transaction 
down through all the layers.  This messes up the API quite considerably
and means it needs to keep track of whether it was called within a
transaction so it knows if it's supposed to commit or not.

The alternative is nested transactions, so that you have:

int a()
{
  struct txn *t;
  DB->fetch(db, key, keylen, &val, &vallen, &t);
  // ...
  b();
  DB->store(db, key, keylen, val, vallen, &t);
  DB->commit(db, t); // bloody transaction pointer indirection level
                     // changes in the API
  return 0;
}

int b()
{
  DB->fetch(db, key2, key2len, &val2, &val2len, NULL);
  return 0;
}

That currently works now thanks to the "hacky" transaction promotion
magic.

The problem is when you want to do:

int b()
{
  struct txn *t2;
  DB->fetch(db, key2, key2len, &val2, &val2len, &t2);
  // ...
  DB->store(db, key2, key2len, &val2, &val2len, &t2);
  DB->commit(db, t2);
  return 0;
}

Suddenly you have transaction nesting.

NOW - this _could_ work.

How it would work.

"struct db" would keep a linked list of transactions rather than just a
current_txn pointer.

GLOSSARY:
  struct txn **tidptr;

if (tidptr == NULL) {
  CASE: transactions on the list 
    - read needs no lock.
    - write needs a new transaction on the list that will autocommit at
      the end of the function
  CASE: no transactions on the list
    - read creates a readlock only
    - write needs a new transaction on the list that will autocommit at
      the end of the function
}
else if (*tidptr == NULL) {
  - add a new transaction to the list
  - do all operations inside the new transaction
  - *tidptr = newtxn;
}
else {
  CASE: transactions on the list
	  - assert(*tidptr == list_top->tid);
	CASE: no transactions
	  - abort.
}

OK - now what happens with locks?

newtxn:
  CASE: transactions on the list
    - don't do any more locking on create
  CASE: no transactions
	  - write lock on create

commit/abort:
  CASE: this is the last item on the list
	  - unlock after commit
  CASE: there are further transactions
	  - don't unlock


WHAT ABOUT ROLLBACK?

Each transaction has a base pointer, that points to the end of the
skiplist before this transaction started.  When you rollback, you start
from this point and read each record that was appended.  When you hit
the LAST record you reverse its actions, then truncate to that point and
loop until there's nothing left on the end of the file.

With nested transactions, you rollback to the start of THIS transaction
only.  Let me draw a picture:

t1:
  STORE 'abc' '123'
	STORE 'foo' '3456'
	DELETE 'abc'
	t2:
	  DELETE 'foo'
		STORE 'foo' '789'
    COMMIT
	STORE 'bar' 'beer, woot!'
	t3:
	  DELETE 'bar'
		STORE 'bar' 'girly drink'
		ROLLBACK
  COMMIT


DB CONTENT IS:
foo: 789
bar: beer, woot!

Actual on disk data is:

STORE 'abc' '123'
STORE 'foo' '3456'
DELETE 'abc'
DELETE 'foo'
STORE 'foo' '789'
STORE 'bar' 'beer, woot!'
COMMIT

Notice, we don't actually generate a COMMIT record to disk for commits
where there are further transactions, all we do is update the "end of
transaction pointer" on the parent transaction when we commit, rather
than running the rollback logic to remove the unwanted records.  This
means that a crash will never commit the partial transactions.


Sooo... why do we want this (those of you who have followed along this
far?)

Basically - it means you can do a self-contained unit of work without
caring whether it's being called in an outer transaction or not.  Just
start your own "transaction" and call commit at the end.  It still won't
be "committed" and accessible to other users until the outer transaction
makes its commit, but it means you don't have to pass around transaction
pointers and worry about return codes and mapping the abort all the way
out.  Your part of the code can appear to have never happened without
needing to roll back the outer transaction and start over.


It's not actually complex implementation-wise, and it shouldn't break
anything currently in existance because trying this now gets you an
assertion failure.

It will also "unhacky" the way that transient non-transactional reads
are rolled into the wrapping transaction.

Thanks for reading!  Sorry it got so long...

Bron.

From brong at fastmail.fm  Tue Sep  2 03:11:51 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 02 Sep 2008 17:11:51 +1000
Subject: On the topic of nested skiplist transactions
In-Reply-To: <20080902062933.GB26500@brong.net>
References: <20080902062933.GB26500@brong.net>
Message-ID: <1220339511.16303.1271671017@webmail.messagingengine.com>


On Tue, 2 Sep 2008 16:29:33 +1000, "Bron Gondwana" <brong at fastmail.fm> said:
> I've been thinking a bit about transactions and skiplists these past few
> days, since sending the patch that refactors the skiplist locking code.

For reference, here's what BDB does:

http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/nested.html

My proposal is different in that it doesn't support multiple nested concurrent
transactions.  If you open up a new transaction, you must complete it (one way
or the other) before anything can happen outside that transaction.

You can, however, open yet another sub-nested transaction like so:

t1 (
  STORE 'a' 'x'
  t2 ( 
    t3 (
      DELETE 'a'
      t4 (
        STORE 'a' 'z'
        ROLLBACK
      )
      COMMIT
    )
    COMMIT
  )
  COMMIT
)

But basically every transaction can be considered equivalent to a single operation
in the parent transaction.

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm


From Rudy.Gevaert at UGent.be  Tue Sep  2 10:26:19 2008
From: Rudy.Gevaert at UGent.be (Rudy Gevaert)
Date: Tue, 02 Sep 2008 16:26:19 +0200
Subject: [PATCH] Set lock pointer at the start of foreach (Re: 2.3.12
	transaction problem within skiplist DB->foreach())
In-Reply-To: <1220321875.31180.1271642283@webmail.messagingengine.com>
References: <20080828221744.GA71671@exuma.irbs.com>	<20080829063127.GA16173@brong.net>	<8275B86A-F955-43BC-AFCD-CF551194504D@umich.edu>
	<1220321875.31180.1271642283@webmail.messagingengine.com>
Message-ID: <48BD4D0B.3050700@UGent.be>

Bron Gondwana wrote:
> On Fri, 29 Aug 2008 09:36:40 -0400, "Wesley Craig" <wes at umich.edu> said:
>> On 29 Aug 2008, at 02:31, Bron Gondwana wrote:
>>> Please find attached a tested patch that fixes the problem and  
>>> refactors
>>> the bloody mess that was locking code into a couple of nice, neat
>>> functions.
>> A quick rewiew looks good to me.  I'll review it closely this  
>> afternoon and deploy it on a test cluster.  Presuming I find no  
>> issues I'll commit it thereafter.  Any chance of submitting it to  
>> bugzilla?
> 
> Sure.  Have done.  Bug 3075.
> 
> Bron.


I just hit the a similar problem (see the thread skiplist errors on 
Cyrus 2.3.12).

Bron, I applied your patch on my test system and the problem doesn't 
happen to me any more.

I was facing the UNLOCKED too when a I deleted a top level mailbox of a 
user.



-- 
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Rudy Gevaert          Rudy.Gevaert at UGent.be          tel:+32 9 264 4734
Directie ICT, afd. Infrastructuur ICT Department, Infrastructure office
Groep Systemen                    Systems group
Universiteit Gent                 Ghent University
Krijgslaan 281, gebouw S9, 9000 Gent, Belgie               www.UGent.be
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

From brong at fastmail.fm  Tue Sep  2 23:25:23 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Wed, 03 Sep 2008 13:25:23 +1000
Subject: [PATCH] Set lock pointer at the start of foreach (Re: 2.3.12
	transaction problem within skiplist DB->foreach())
In-Reply-To: <48BD4D0B.3050700@UGent.be>
References: <20080828221744.GA71671@exuma.irbs.com>
	<20080829063127.GA16173@brong.net>
	<8275B86A-F955-43BC-AFCD-CF551194504D@umich.edu>
	<1220321875.31180.1271642283@webmail.messagingengine.com>
	<48BD4D0B.3050700@UGent.be>
Message-ID: <1220412323.32013.1271863947@webmail.messagingengine.com>


On Tue, 02 Sep 2008 16:26:19 +0200, "Rudy Gevaert" <Rudy.Gevaert at UGent.be> said:
> Bron Gondwana wrote:
> > On Fri, 29 Aug 2008 09:36:40 -0400, "Wesley Craig" <wes at umich.edu> said:
> >> On 29 Aug 2008, at 02:31, Bron Gondwana wrote:
> >>> Please find attached a tested patch that fixes the problem and  
> >>> refactors
> >>> the bloody mess that was locking code into a couple of nice, neat
> >>> functions.
> >> A quick rewiew looks good to me.  I'll review it closely this  
> >> afternoon and deploy it on a test cluster.  Presuming I find no  
> >> issues I'll commit it thereafter.  Any chance of submitting it to  
> >> bugzilla?
> > 
> > Sure.  Have done.  Bug 3075.
> > 
> > Bron.
> 
> 
> I just hit the a similar problem (see the thread skiplist errors on 
> Cyrus 2.3.12).
> 
> Bron, I applied your patch on my test system and the problem doesn't 
> happen to me any more.
> 
> I was facing the UNLOCKED too when a I deleted a top level mailbox of a 
> user.

Thanks Rudy :)

Just wait until I add nested transactions on top of this!  In particular
it will be nice for that issue someone had a while back of startup taking
FOREVER on an external filesystem with slow locking.  We could easily
take a write lock on the database in a wrapper transaction while we
check through the mailboxes.db if we don't have to worry about passing
that transaction down all the layers when we clean up the partially
created mailboxes.

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm


From brong at fastmail.fm  Wed Sep  3 00:32:05 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Wed, 03 Sep 2008 14:32:05 +1000
Subject: On the topic of nested skiplist transactions
In-Reply-To: <20080902062933.GB26500@brong.net>
References: <20080902062933.GB26500@brong.net>
Message-ID: <1220416325.6064.1271870195@webmail.messagingengine.com>


On Tue, 2 Sep 2008 16:29:33 +1000, "Bron Gondwana" <brong at fastmail.fm> said:
> I've been thinking a bit about transactions and skiplists these past few
> days, since sending the patch that refactors the skiplist locking code.

Wow, this was scarily simple to write.  Really the code changes are
so easy!

Basically:

* add 'struct txn *next_txn' to struct txn
* in newtxn, check if there's a current txn and copy values from it if
  so, including put it into next_txn of the new transaction
* don't assert that current_txn is NULL when starting a new transaction
* in commit and abort, set current_txn to tid->next_txn rather than NULL
* if current_txn is now NOT NULL, then don't unlock.
* also in commit, if current_txn is NOT NULL, then don't write the COMMIT
  record to the DB or fsync.


It really was that simple... and it compiled first time, which has to be
some sort of a sign.  Ho hum.  OK, so it broke my skiplist tuning patch
and I had to re-build it - just because I changed the logic of the
checkpoint checker - but that's what quilt is so nice for :)


It passes the hammer tests and a bunch of imap rename tests, which isn't
a big surprise, because no code actually uses the "another transaction
still exists" codepaths yet :)

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm

-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrus-skiplist-nested-transactions-2.3.12.diff
Type: text/x-diff
Size: 4957 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080903/aae94883/attachment.bin 

From hdembkowski at alcatel-lucent.com  Mon Sep  8 10:07:41 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Mon, 8 Sep 2008 16:07:41 +0200
Subject: Core dump from cyrusdb_skiplist.c
Message-ID: <B576524D0905194499F90736CAA2758A01D8E3BC@DEEXC1U03.de.lucent.com>

Hi,  


We are using cyrus-imapd-2.3.7 to store our messages.
And we got core from imap - file cyrusdb_skiplist.c


#0  myforeach (db=0x814b618, prefix=0xbfffb8c0 "user.6165602930", prefixlen=16, goodp=0x8087dc0 <find_p>,
    cb=0x80871d0 <find_cb>, rock=0xbfffb8b0, tid=0xbfffc91c) at cyrusdb_skiplist.c:1019
1019                    ptr = db->map_base + FORWARD(ptr, 0);
(gdb) t 1
[Switching to thread 1 (process 14237)]#0  myforeach (db=0x814b618, prefix=0xbfffb8c0 "user.6165602930", prefixlen=16,
    goodp=0x8087dc0 <find_p>, cb=0x80871d0 <find_cb>, rock=0xbfffb8b0, tid=0xbfffc91c) at cyrusdb_skiplist.c:1019
1019                    ptr = db->map_base + FORWARD(ptr, 0);
(gdb) where
#0  myforeach (db=0x814b618, prefix=0xbfffb8c0 "user.6165602930", prefixlen=16, goodp=0x8087dc0 <find_p>,
    cb=0x80871d0 <find_cb>, rock=0xbfffb8b0, tid=0xbfffc91c) at cyrusdb_skiplist.c:1019
#1  0x08087ecf in annotatemore_findall (mailbox=0x814b618 "@?\024\b\a", entry=0x80b4834 "*", proc=0x4effb8, rock=0x4effb8,
    tid=0xbfffc91c) at annotate.c:391
#2  0x0808801b in annotatemore_rename (oldmboxname=0xbfffd730 "user.6165602930",
    newmboxname=0x4effb8 <Address 0x4effb8 out of bounds>, olduserid=0x4effb8 <Address 0x4effb8 out of bounds>,
    newuserid=0x4effb8 <Address 0x4effb8 out of bounds>) at annotate.c:2093
#3  0x08055e39 in cmd_rename (tag=0x8151a28 "1", oldname=0x8151b08 "user.6165602930", newname=0x81c6010 "user.6163078011",
    partition=0x0) at imapd.c:5408
#4  0x0805e3ac in cmdloop () at imapd.c:1532
#5  0x08060a57 in service_main (argc=1, argv=0x8149008, envp=0xbffffd64) at imapd.c:789
#6  0x0804c3a8 in main (argc=3, argv=0xbffffd54, envp=0xbffffd64) at service.c:532


It refers to function myforeach() and code


    980             /* save KEY, KEYLEN */
    981             if (KEYLEN(ptr) > savebuflen) {
    982                 savebuflen = KEYLEN(ptr) + 1024;
    983                 savebuf = xrealloc(savebuf, savebuflen);
    984             }
    985             memcpy(savebuf, KEY(ptr), KEYLEN(ptr));
    986             savebufsize = KEYLEN(ptr);
    987
    988             /* make callback */
    989             cb_r = cb(rock, KEY(ptr), KEYLEN(ptr), DATA(ptr), DATALEN(ptr));
    990             if (cb_r) break;
    991
    992             if (!tid) {
    993                 /* grab a r lock */
    994                 if ((r = read_lock(db)) < 0) {
    995                     return r;
    996                 }
    997             } else {
    998                 /* make sure we're up to date */
    999                 update_lock(db, tp);
   1000             }
   1001
   1002             /* reposition */
   1003             if (!(ino == db->map_ino && sz == db->map_size)) {
   1004                 /* something changed in the file; reseek */
   1005                 ptr = find_node(db, savebuf, savebufsize, 0);
   1006
   1007                 /* 'ptr' might not equal 'savebuf'.  if it's different,
   1008                    we want to stay where we are.  if it's the same, we
   1009                    should move on to the next one */
   1010                 if (savebufsize == KEYLEN(ptr) &&
   1011                     !memcmp(savebuf, KEY(ptr), savebufsize)) {
   1012                     ptr = db->map_base + FORWARD(ptr, 0);
   1013                 } else {
   1014                     /* 'savebuf' got deleted, so we're now pointing at the
   1015                        right thing */
   1016                 }
   1017             } else {
   1018                 /* move to the next one */
   1019                 ptr = db->map_base + FORWARD(ptr, 0);
   1020             }
   1021         } else {
   1022             /* we didn't make the callback; keep going */
   1023             ptr = db->map_base + FORWARD(ptr, 0);
   1024         }
   1025     }



Do you know something about it?



Kind Regards,
Henryk

From brong at fastmail.fm  Mon Sep  8 21:11:36 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 09 Sep 2008 11:11:36 +1000
Subject: Core dump from cyrusdb_skiplist.c
In-Reply-To: <B576524D0905194499F90736CAA2758A01D8E3BC@DEEXC1U03.de.lucent.com>
References: <B576524D0905194499F90736CAA2758A01D8E3BC@DEEXC1U03.de.lucent.com>
Message-ID: <1220922696.16736.1272906357@webmail.messagingengine.com>

Yeah, you've got a corrupted skiplist file.  There are a variety
of reasons why this could be the case, but most of all, the
skiplist library in 2.3.7 is pretty buggy.  Well, no more buggy
than it had been for ages, but I got a LOT of fixes added since
then.

Probably the best skiplist library in current use is 2.3.11.  2.3.12
is theoretically better, but it actually exposes a bug that just
silently worked in 2.3.11.

Hopefully, if Wes and Ken are happy that the locking refactor is
tested enough, there will be a 2.3.13 soonish with a "perfect"
skiplist library.

Meanwhile - that doesn't fix your problem!  How attached are you
to the data in the skiplist file?  It's your annotations DB.  Do you have
a recent backup before it got corrupted?  It's sometimes hard to tell.

I can probably repair it for you, but I haven't got a good working
automated tool for it.  I'd be going in and fiddling the bits by hand!

Bron.


On Mon, 8 Sep 2008 16:07:41 +0200, "DEMBKOWSKI, Henryk (Henryk)" <hdembkowski at alcatel-lucent.com> said:
> Hi,  
> 
> 
> We are using cyrus-imapd-2.3.7 to store our messages.
> And we got core from imap - file cyrusdb_skiplist.c
> 
> 
> #0  myforeach (db=0x814b618, prefix=0xbfffb8c0 "user.6165602930",
> prefixlen=16, goodp=0x8087dc0 <find_p>,
>     cb=0x80871d0 <find_cb>, rock=0xbfffb8b0, tid=0xbfffc91c) at
>     cyrusdb_skiplist.c:1019
> 1019                    ptr = db->map_base + FORWARD(ptr, 0);
> (gdb) t 1
> [Switching to thread 1 (process 14237)]#0  myforeach (db=0x814b618,
> prefix=0xbfffb8c0 "user.6165602930", prefixlen=16,
>     goodp=0x8087dc0 <find_p>, cb=0x80871d0 <find_cb>, rock=0xbfffb8b0,
>     tid=0xbfffc91c) at cyrusdb_skiplist.c:1019
> 1019                    ptr = db->map_base + FORWARD(ptr, 0);
> (gdb) where
> #0  myforeach (db=0x814b618, prefix=0xbfffb8c0 "user.6165602930",
> prefixlen=16, goodp=0x8087dc0 <find_p>,
>     cb=0x80871d0 <find_cb>, rock=0xbfffb8b0, tid=0xbfffc91c) at
>     cyrusdb_skiplist.c:1019
> #1  0x08087ecf in annotatemore_findall (mailbox=0x814b618 "@&#351;\024\b\a",
> entry=0x80b4834 "*", proc=0x4effb8, rock=0x4effb8,
>     tid=0xbfffc91c) at annotate.c:391
> #2  0x0808801b in annotatemore_rename (oldmboxname=0xbfffd730
> "user.6165602930",
>     newmboxname=0x4effb8 <Address 0x4effb8 out of bounds>,
>     olduserid=0x4effb8 <Address 0x4effb8 out of bounds>,
>     newuserid=0x4effb8 <Address 0x4effb8 out of bounds>) at
>     annotate.c:2093
> #3  0x08055e39 in cmd_rename (tag=0x8151a28 "1", oldname=0x8151b08
> "user.6165602930", newname=0x81c6010 "user.6163078011",
>     partition=0x0) at imapd.c:5408
> #4  0x0805e3ac in cmdloop () at imapd.c:1532
> #5  0x08060a57 in service_main (argc=1, argv=0x8149008, envp=0xbffffd64)
> at imapd.c:789
> #6  0x0804c3a8 in main (argc=3, argv=0xbffffd54, envp=0xbffffd64) at
> service.c:532
> 
> 
> It refers to function myforeach() and code
> 
> 
>     980             /* save KEY, KEYLEN */
>     981             if (KEYLEN(ptr) > savebuflen) {
>     982                 savebuflen = KEYLEN(ptr) + 1024;
>     983                 savebuf = xrealloc(savebuf, savebuflen);
>     984             }
>     985             memcpy(savebuf, KEY(ptr), KEYLEN(ptr));
>     986             savebufsize = KEYLEN(ptr);
>     987
>     988             /* make callback */
>     989             cb_r = cb(rock, KEY(ptr), KEYLEN(ptr), DATA(ptr),
>     DATALEN(ptr));
>     990             if (cb_r) break;
>     991
>     992             if (!tid) {
>     993                 /* grab a r lock */
>     994                 if ((r = read_lock(db)) < 0) {
>     995                     return r;
>     996                 }
>     997             } else {
>     998                 /* make sure we're up to date */
>     999                 update_lock(db, tp);
>    1000             }
>    1001
>    1002             /* reposition */
>    1003             if (!(ino == db->map_ino && sz == db->map_size)) {
>    1004                 /* something changed in the file; reseek */
>    1005                 ptr = find_node(db, savebuf, savebufsize, 0);
>    1006
>    1007                 /* 'ptr' might not equal 'savebuf'.  if it's
>    different,
>    1008                    we want to stay where we are.  if it's the
>    same, we
>    1009                    should move on to the next one */
>    1010                 if (savebufsize == KEYLEN(ptr) &&
>    1011                     !memcmp(savebuf, KEY(ptr), savebufsize)) {
>    1012                     ptr = db->map_base + FORWARD(ptr, 0);
>    1013                 } else {
>    1014                     /* 'savebuf' got deleted, so we're now
>    pointing at the
>    1015                        right thing */
>    1016                 }
>    1017             } else {
>    1018                 /* move to the next one */
>    1019                 ptr = db->map_base + FORWARD(ptr, 0);
>    1020             }
>    1021         } else {
>    1022             /* we didn't make the callback; keep going */
>    1023             ptr = db->map_base + FORWARD(ptr, 0);
>    1024         }
>    1025     }
> 
> 
> 
> Do you know something about it?
> 
> 
> 
> Kind Regards,
> Henryk
-- 
  Bron Gondwana
  brong at fastmail.fm


From hdembkowski at alcatel-lucent.com  Tue Sep  9 04:20:13 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Tue, 9 Sep 2008 10:20:13 +0200
Subject: Core dump from cyrusdb_skiplist.c
In-Reply-To: <1220922696.16736.1272906357@webmail.messagingengine.com>
References: <B576524D0905194499F90736CAA2758A01D8E3BC@DEEXC1U03.de.lucent.com>
	<1220922696.16736.1272906357@webmail.messagingengine.com>
Message-ID: <B576524D0905194499F90736CAA2758A01D8E516@DEEXC1U03.de.lucent.com>

Bron,

Thanks for help. 

Yes, we have backup and we successfully recovered our annotations
database. 

However it could be good if we could have fix for cyrusdb_skiplist.c and
core will 
not be seen anymore.

Kind Regards,
Henryk


-----Original Message-----
From: Bron Gondwana [mailto:brong at fastmail.fm] 
Sent: Tuesday, September 09, 2008 3:12 AM
To: DEMBKOWSKI, Henryk (Henryk); Info Cyrus; Cyrus Devel
Cc: DEMBEK, Adam (Adam)
Subject: Re: Core dump from cyrusdb_skiplist.c

Yeah, you've got a corrupted skiplist file.  There are a variety of
reasons why this could be the case, but most of all, the skiplist
library in 2.3.7 is pretty buggy.  Well, no more buggy than it had been
for ages, but I got a LOT of fixes added since then.

Probably the best skiplist library in current use is 2.3.11.  2.3.12 is
theoretically better, but it actually exposes a bug that just silently
worked in 2.3.11.

Hopefully, if Wes and Ken are happy that the locking refactor is tested
enough, there will be a 2.3.13 soonish with a "perfect"
skiplist library.

Meanwhile - that doesn't fix your problem!  How attached are you to the
data in the skiplist file?  It's your annotations DB.  Do you have a
recent backup before it got corrupted?  It's sometimes hard to tell.

I can probably repair it for you, but I haven't got a good working
automated tool for it.  I'd be going in and fiddling the bits by hand!

Bron.


On Mon, 8 Sep 2008 16:07:41 +0200, "DEMBKOWSKI, Henryk (Henryk)"
<hdembkowski at alcatel-lucent.com> said:
> Hi,
> 
> 
> We are using cyrus-imapd-2.3.7 to store our messages.
> And we got core from imap - file cyrusdb_skiplist.c
> 
> 
> #0  myforeach (db=0x814b618, prefix=0xbfffb8c0 "user.6165602930", 
> prefixlen=16, goodp=0x8087dc0 <find_p>,
>     cb=0x80871d0 <find_cb>, rock=0xbfffb8b0, tid=0xbfffc91c) at
>     cyrusdb_skiplist.c:1019
> 1019                    ptr = db->map_base + FORWARD(ptr, 0);
> (gdb) t 1
> [Switching to thread 1 (process 14237)]#0  myforeach (db=0x814b618, 
> prefix=0xbfffb8c0 "user.6165602930", prefixlen=16,
>     goodp=0x8087dc0 <find_p>, cb=0x80871d0 <find_cb>, rock=0xbfffb8b0,
>     tid=0xbfffc91c) at cyrusdb_skiplist.c:1019
> 1019                    ptr = db->map_base + FORWARD(ptr, 0);
> (gdb) where
> #0  myforeach (db=0x814b618, prefix=0xbfffb8c0 "user.6165602930", 
> prefixlen=16, goodp=0x8087dc0 <find_p>,
>     cb=0x80871d0 <find_cb>, rock=0xbfffb8b0, tid=0xbfffc91c) at
>     cyrusdb_skiplist.c:1019
> #1  0x08087ecf in annotatemore_findall (mailbox=0x814b618 
> "@&#351;\024\b\a",
> entry=0x80b4834 "*", proc=0x4effb8, rock=0x4effb8,
>     tid=0xbfffc91c) at annotate.c:391
> #2  0x0808801b in annotatemore_rename (oldmboxname=0xbfffd730 
> "user.6165602930",
>     newmboxname=0x4effb8 <Address 0x4effb8 out of bounds>,
>     olduserid=0x4effb8 <Address 0x4effb8 out of bounds>,
>     newuserid=0x4effb8 <Address 0x4effb8 out of bounds>) at
>     annotate.c:2093
> #3  0x08055e39 in cmd_rename (tag=0x8151a28 "1", oldname=0x8151b08 
> "user.6165602930", newname=0x81c6010 "user.6163078011",
>     partition=0x0) at imapd.c:5408
> #4  0x0805e3ac in cmdloop () at imapd.c:1532
> #5  0x08060a57 in service_main (argc=1, argv=0x8149008, 
> envp=0xbffffd64) at imapd.c:789
> #6  0x0804c3a8 in main (argc=3, argv=0xbffffd54, envp=0xbffffd64) at
> service.c:532
> 
> 
> It refers to function myforeach() and code
> 
> 
>     980             /* save KEY, KEYLEN */
>     981             if (KEYLEN(ptr) > savebuflen) {
>     982                 savebuflen = KEYLEN(ptr) + 1024;
>     983                 savebuf = xrealloc(savebuf, savebuflen);
>     984             }
>     985             memcpy(savebuf, KEY(ptr), KEYLEN(ptr));
>     986             savebufsize = KEYLEN(ptr);
>     987
>     988             /* make callback */
>     989             cb_r = cb(rock, KEY(ptr), KEYLEN(ptr), DATA(ptr),
>     DATALEN(ptr));
>     990             if (cb_r) break;
>     991
>     992             if (!tid) {
>     993                 /* grab a r lock */
>     994                 if ((r = read_lock(db)) < 0) {
>     995                     return r;
>     996                 }
>     997             } else {
>     998                 /* make sure we're up to date */
>     999                 update_lock(db, tp);
>    1000             }
>    1001
>    1002             /* reposition */
>    1003             if (!(ino == db->map_ino && sz == db->map_size)) {
>    1004                 /* something changed in the file; reseek */
>    1005                 ptr = find_node(db, savebuf, savebufsize, 0);
>    1006
>    1007                 /* 'ptr' might not equal 'savebuf'.  if it's
>    different,
>    1008                    we want to stay where we are.  if it's the
>    same, we
>    1009                    should move on to the next one */
>    1010                 if (savebufsize == KEYLEN(ptr) &&
>    1011                     !memcmp(savebuf, KEY(ptr), savebufsize)) {
>    1012                     ptr = db->map_base + FORWARD(ptr, 0);
>    1013                 } else {
>    1014                     /* 'savebuf' got deleted, so we're now
>    pointing at the
>    1015                        right thing */
>    1016                 }
>    1017             } else {
>    1018                 /* move to the next one */
>    1019                 ptr = db->map_base + FORWARD(ptr, 0);
>    1020             }
>    1021         } else {
>    1022             /* we didn't make the callback; keep going */
>    1023             ptr = db->map_base + FORWARD(ptr, 0);
>    1024         }
>    1025     }
> 
> 
> 
> Do you know something about it?
> 
> 
> 
> Kind Regards,
> Henryk
--
  Bron Gondwana
  brong at fastmail.fm


From brong at fastmail.fm  Wed Sep 10 02:43:26 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Wed, 10 Sep 2008 16:43:26 +1000
Subject: Time for 2.3.13?
Message-ID: <20080910064326.GC15156@brong.net>

I just noticed that 2.3.12 was released back in April, and
there are a few fixes changes kicking around.  I've just
compared CVS HEAD to 2.3.12 using trusty old 'diff' and
it looks like the big changes are:

* sql database support
* "can not" => "cannot"
* capabilities tidy up
* cyr_virusscan


I'd also like to push the following patches upstream:

* http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-locking-rework-2.3.12.diff
  - we've discussed this one in detail in other threads,
    and people who've tested it report it working well.

* http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-safeunlock-2.3.12.diff
  - pretty trivial fix to make consistency checks safe
    against failures.  Probably pointless, but it makes
    the code nicer.

* http://cyrus.brong.fastmail.fm/patches/cyrus-always-unlink-2.3.12.diff
  - absolutely trivial patch in the style of other fixes.
    Put an "unlink" call before an open call on a file that's
    not supposed to be there in the stage area.  Just in case[tm].

* http://cyrus.brong.fastmail.fm/patches/cyrus-longwords-2.3.4.diff
  - seriously, who is doing email with a 10Mb memory machine these
    days?  We've had a query just recently on the mailing list where
    a site hit line length issues.  Let the VM handle it.

I don't think I'd push nested transactions upstream at this stage -
it works fine for me, but it's pretty experimental still!  I'm more
interested in having a robust skiplist implementation that handles
the 'foreach used first' bug.  Then if anyone asks about skiplist
issues I can just say "use 2.3.13" rather than "use 2.3.12 plus
these patches" :)

Bron.

From wes at umich.edu  Wed Sep 10 09:45:07 2008
From: wes at umich.edu (Wesley Craig)
Date: Wed, 10 Sep 2008 09:45:07 -0400
Subject: Time for 2.3.13?
In-Reply-To: <20080910064326.GC15156@brong.net>
References: <20080910064326.GC15156@brong.net>
Message-ID: <1F1680B6-DB1F-4FF8-BB87-E797728239FC@umich.edu>

I see 12 items in the bugzilla (one of which is the skiplist foreach  
bug) which I think can be closed.  So, yes, I think we should get  
these various fixes committed and begin rolling 2.3.13.  Can you make  
sure bugzilla has the versions of these various patches that you  
think should be applied to HEAD?

:wes

On 10 Sep 2008, at 02:43, Bron Gondwana wrote:
> I just noticed that 2.3.12 was released back in April, and
> there are a few fixes changes kicking around.  I've just
> compared CVS HEAD to 2.3.12 using trusty old 'diff' and
> it looks like the big changes are:
>
> * sql database support
> * "can not" => "cannot"
> * capabilities tidy up
> * cyr_virusscan
>
>
> I'd also like to push the following patches upstream:
>
> * http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-locking- 
> rework-2.3.12.diff
>   - we've discussed this one in detail in other threads,
>     and people who've tested it report it working well.
>
> * http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist- 
> safeunlock-2.3.12.diff
>   - pretty trivial fix to make consistency checks safe
>     against failures.  Probably pointless, but it makes
>     the code nicer.
>
> * http://cyrus.brong.fastmail.fm/patches/cyrus-always- 
> unlink-2.3.12.diff
>   - absolutely trivial patch in the style of other fixes.
>     Put an "unlink" call before an open call on a file that's
>     not supposed to be there in the stage area.  Just in case[tm].
>
> * http://cyrus.brong.fastmail.fm/patches/cyrus-longwords-2.3.4.diff
>   - seriously, who is doing email with a 10Mb memory machine these
>     days?  We've had a query just recently on the mailing list where
>     a site hit line length issues.  Let the VM handle it.
>
> I don't think I'd push nested transactions upstream at this stage -
> it works fine for me, but it's pretty experimental still!  I'm more
> interested in having a robust skiplist implementation that handles
> the 'foreach used first' bug.  Then if anyone asks about skiplist
> issues I can just say "use 2.3.13" rather than "use 2.3.12 plus
> these patches" :)
>
> Bron.

From thomas.jarosch at intra2net.com  Wed Sep 10 10:00:07 2008
From: thomas.jarosch at intra2net.com (Thomas Jarosch)
Date: Wed, 10 Sep 2008 16:00:07 +0200
Subject: Time for 2.3.13?
In-Reply-To: <20080910064326.GC15156@brong.net>
References: <20080910064326.GC15156@brong.net>
Message-ID: <200809101600.07571.thomas.jarosch@intra2net.com>

Hello Bron and Wes,

I've experienced another bug and I'm not sure if the skiplist/locking code is 
at fault or not. I was deleting huge amounts of mail in many subfolders
on a slow box. The tool that started the request got a timeout after some 
minutes and this left cyrus-imapd in a broken state:

The mailboxes were still listed in mailboxes.db and the mails were still on 
disc. Every cyrus.* file in the mail directories were already deleted and so 
cyrus was unable to recover from this.

Is this somehow related or is this another issue that needs tracing?

Thomas


From brong at fastmail.fm  Wed Sep 10 19:11:41 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 11 Sep 2008 09:11:41 +1000
Subject: Time for 2.3.13?
In-Reply-To: <1F1680B6-DB1F-4FF8-BB87-E797728239FC@umich.edu>
References: <20080910064326.GC15156@brong.net>
	<1F1680B6-DB1F-4FF8-BB87-E797728239FC@umich.edu>
Message-ID: <1221088301.2763.1273321133@webmail.messagingengine.com>

On Wed, 10 Sep 2008 09:45:07 -0400, "Wesley Craig" <wes at umich.edu> said:
> I see 12 items in the bugzilla (one of which is the skiplist foreach  
> bug) which I think can be closed.  So, yes, I think we should get  
> these various fixes committed and begin rolling 2.3.13.  Can you make  
> sure bugzilla has the versions of these various patches that you  
> think should be applied to HEAD?

Will do :)  Feel free to glance through any of the other patches on
cyrus.brong.fastmail.fm as well, but at this stage I think it's better
to get the nested transaction stuff done and rebuild some of those
patches on top of it.  It turns out you have to do ugly complex things
with transaction passing without nested transactions, and some of those
patches are too complex for their own good!
-- 
  Bron Gondwana
  brong at fastmail.fm


From brong at fastmail.fm  Wed Sep 10 21:33:15 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 11 Sep 2008 11:33:15 +1000
Subject: Time for 2.3.13?
In-Reply-To: <200809101600.07571.thomas.jarosch@intra2net.com>
References: <20080910064326.GC15156@brong.net>
	<200809101600.07571.thomas.jarosch@intra2net.com>
Message-ID: <1221096795.14593.1273338561@webmail.messagingengine.com>


On Wed, 10 Sep 2008 16:00:07 +0200, "Thomas Jarosch" <thomas.jarosch at intra2net.com> said:
> Hello Bron and Wes,
> 
> I've experienced another bug and I'm not sure if the skiplist/locking
> code is 
> at fault or not. I was deleting huge amounts of mail in many subfolders
> on a slow box. The tool that started the request got a timeout after some 
> minutes and this left cyrus-imapd in a broken state:
> 
> The mailboxes were still listed in mailboxes.db and the mails were still
> on 
> disc. Every cyrus.* file in the mail directories were already deleted and
> so 
> cyrus was unable to recover from this.
> 
> Is this somehow related or is this another issue that needs tracing?

Can you post the logic that your script was using?  That might help.

Doesn't look like a skiplist issue directly to me.  Possibly it's a
"was in a transaction and rolled back" issue, but that's separate,
and it doesn't explain the mails still being on disk!

Also - do you have split meta?  I.e. are the mail files in a different
location than the cyrus.* files?

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm


From hdembkowski at alcatel-lucent.com  Thu Sep 11 06:30:04 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Thu, 11 Sep 2008 12:30:04 +0200
Subject: Problem with skiplist
Message-ID: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>

Hi,

We are using skiplist for our annotations.db.  And we have big problems
with it. 

We are using tool validate-skiplist
(http://kolab.org/cgi-bin/viewcvs-kolab.cgi/utils/admin/)
to check if annotations.db is fine. Some time ago we got the following
error:

validate-skiplist: Bad node: expected ADD or DELETE at 0x001CFDB4!
(found0x78742F70)

Unfortunately we don't know exact scenario which could cause this
problem. We just 
run validate-skiplist in cron (every one hour) and from time to time we
see such error.

So we found patches to fix some issues (our problems are the same as
described at
http://www.intevation.de/roundup/kolab/issue840) - source can be found
here

http://permalink.gmane.org/gmane.mail.imap.cyrus/28033

However it doesn't help. We still have error 

validate-skiplist: Bad node: expected ADD or DELETE at 0x002A855C!
(found 0x38373033)


We noticed that this error/problem is self repaired. I mean after few
hours everything is fine. 
For example, we had errors  

validate-skiplist -v annotations.db
Skiplist file Version 1.2
Max level: 20
Current level: 17
26549 active items
Log start at 0x002A855C
Last recovery on Tue Aug 26 22:18:33 2008

Checking log part...
validate-skiplist: Bad node: expected ADD or DELETE at 0x002A855C!
(found 0x38373033)


But after few hours 


validate-skiplist -v annotations.db
Skiplist file Version 1.2
Max level: 20
Current level: 17
34415 active items
Log start at 0x003C9ED8
Last recovery on Wed Sep 10 01:10:40 2008

Checking log part...
Found 3830 ADD, 5336 DELETE and 5431 COMMIT nodes in log.

It has been repaired without any actions from our side. Interesting
thing is 
that "Last recovery" has been changed. Do you know what is it? Who
changed it?
Why it is done? When it is done? "Last recovery" suggest that something 
recovered annotations.db. 



Kind Regards,
Henryk Dembkowski

From thomas.jarosch at intra2net.com  Thu Sep 11 08:05:43 2008
From: thomas.jarosch at intra2net.com (Thomas Jarosch)
Date: Thu, 11 Sep 2008 14:05:43 +0200
Subject: Problem with skiplist
In-Reply-To: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
References: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
Message-ID: <200809111405.44029.thomas.jarosch@intra2net.com>

Hello Henryk,

which version of cyrus-imapd are you using?

Which operation system are you using?
If it's Linux, what kernel version do you use?

In the past there was a bug in the kernel (mmap)
that could cause skiplist corruption.

Thomas


From hdembkowski at alcatel-lucent.com  Thu Sep 11 08:40:03 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Thu, 11 Sep 2008 14:40:03 +0200
Subject: Problem with skiplist
In-Reply-To: <200809111405.44029.thomas.jarosch@intra2net.com>
References: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
	<200809111405.44029.thomas.jarosch@intra2net.com>
Message-ID: <B576524D0905194499F90736CAA2758A01D8EC71@DEEXC1U03.de.lucent.com>

Thomas,

We are using cyrus-imapd 2.3.7. We had problems so we applied fixes 
from http://permalink.gmane.org/gmane.mail.imap.cyrus/28033
which are now in 2.3.11 and higher. Do you know if there are additional 
changes in skiplist code in 2.3.11 beside above fixes? BTW, we are
planning 
to use new version 2.3.12 or 2.3.13..

We are using Linux and 

uname -a
Linux ms01 2.6.5-7.276.PTF.205323.0-bigsmp #1 SMP Mon Jul 24 10:45:31
UTC 2006 i686

At http://www.intevation.de/roundup/kolab/issue840 we saw that someone
recomended
a linux 2.6.19.2  http://www.intevation.de/roundup/kolab/msg12474

	My hypothesis is that using a linux 2.6.19.2 or later 
	_with_ skiplist should also develop no problems.

Do you think that we have to move to higher linux? And even we have
fixes
for skiplist we will still get errors due to linux?


Thanks a lot!

Kind Regards,
Henryk


-----Original Message-----
From: Thomas Jarosch [mailto:thomas.jarosch at intra2net.com] 
Sent: Thursday, September 11, 2008 2:06 PM
To: cyrus-devel at lists.andrew.cmu.edu
Cc: DEMBKOWSKI, Henryk (Henryk)
Subject: Re: Problem with skiplist

Hello Henryk,

which version of cyrus-imapd are you using?

Which operation system are you using?
If it's Linux, what kernel version do you use?

In the past there was a bug in the kernel (mmap) that could cause
skiplist corruption.

Thomas


From hdembkowski at alcatel-lucent.com  Thu Sep 11 09:48:25 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Thu, 11 Sep 2008 15:48:25 +0200
Subject: Problem with skiplist
In-Reply-To: <20080911130229.GA28674@brong.net>
References: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
	<20080911130229.GA28674@brong.net>
Message-ID: <B576524D0905194499F90736CAA2758A01D8ECE2@DEEXC1U03.de.lucent.com>

Bron,

Thank you for help.

Do you think that we should also change linux kernel to  2.6.19.2 or
higher?
As Thomas Jarosch suggested. Or new version of skip list will be enough?

Kind Regards,
Henryk



-----Original Message-----
From: Bron Gondwana [mailto:brong at fastmail.fm] 
Sent: Thursday, September 11, 2008 3:02 PM
To: DEMBKOWSKI, Henryk (Henryk)
Cc: info-cyrus at lists.andrew.cmu.edu; cyrus-devel at lists.andrew.cmu.edu;
ZAKRZEWSKI, Marcin (Marcin); ILNICKI, Waldek (Waldek)
Subject: Re: Problem with skiplist

On Thu, Sep 11, 2008 at 12:30:04PM +0200, DEMBKOWSKI, Henryk (Henryk)
wrote:
> Hi,
> 
> We are using skiplist for our annotations.db.  And we have big 
> problems with it.
> 
> We are using tool validate-skiplist
> (http://kolab.org/cgi-bin/viewcvs-kolab.cgi/utils/admin/)
> to check if annotations.db is fine. Some time ago we got the following
> error:
> 
> validate-skiplist: Bad node: expected ADD or DELETE at 0x001CFDB4!
> (found0x78742F70)
> 
> Unfortunately we don't know exact scenario which could cause this 
> problem. We just run validate-skiplist in cron (every one hour) and 
> from time to time we see such error.
> 
> So we found patches to fix some issues (our problems are the same as 
> described at
> http://www.intevation.de/roundup/kolab/issue840) - source can be found

> here
> 
> http://permalink.gmane.org/gmane.mail.imap.cyrus/28033
> 
> However it doesn't help. We still have error
> 
> validate-skiplist: Bad node: expected ADD or DELETE at 0x002A855C!
> (found 0x38373033)
> 
> 
> We noticed that this error/problem is self repaired. I mean after few 
> hours everything is fine.
> For example, we had errors
> 
> validate-skiplist -v annotations.db
> Skiplist file Version 1.2
> Max level: 20
> Current level: 17
> 26549 active items
> Log start at 0x002A855C
> Last recovery on Tue Aug 26 22:18:33 2008
> 
> Checking log part...
> validate-skiplist: Bad node: expected ADD or DELETE at 0x002A855C!
> (found 0x38373033)
> 
> 
> But after few hours
> 
> 
> validate-skiplist -v annotations.db
> Skiplist file Version 1.2
> Max level: 20
> Current level: 17
> 34415 active items
> Log start at 0x003C9ED8
> Last recovery on Wed Sep 10 01:10:40 2008
> 
> Checking log part...
> Found 3830 ADD, 5336 DELETE and 5431 COMMIT nodes in log.
> 
> It has been repaired without any actions from our side. Interesting 
> thing is that "Last recovery" has been changed. Do you know what is 
> it? Who changed it?
> Why it is done? When it is done? "Last recovery" suggest that 
> something recovered annotations.db.

You really, really want a new version of cyrusdb_skiplist.c!  I
recommend the attached one.  It's 2.3.12 plus the patches that will go
into 2.3.13.

Bron ( there won't be any compatibility issues, just drop it in your
       cyrus-imapd-2.3.12/lib/ directory and rebuild )

From thomas.jarosch at intra2net.com  Thu Sep 11 10:02:29 2008
From: thomas.jarosch at intra2net.com (Thomas Jarosch)
Date: Thu, 11 Sep 2008 16:02:29 +0200
Subject: Problem with skiplist
In-Reply-To: <B576524D0905194499F90736CAA2758A01D8EC71@DEEXC1U03.de.lucent.com>
References: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
	<200809111405.44029.thomas.jarosch@intra2net.com>
	<B576524D0905194499F90736CAA2758A01D8EC71@DEEXC1U03.de.lucent.com>
Message-ID: <200809111602.30200.thomas.jarosch@intra2net.com>

On Thursday, 11. September 2008 14:40:03 you wrote:
> We are using Linux and
>
> uname -a
> Linux ms01 2.6.5-7.276.PTF.205323.0-bigsmp #1 SMP Mon Jul 24 10:45:31
> UTC 2006 i686
>
> At http://www.intevation.de/roundup/kolab/issue840 we saw that someone
> recomended
> a linux 2.6.19.2  http://www.intevation.de/roundup/kolab/msg12474

IIRC the fix was made in 2.6.20, so it's likely 2.6.19.2 contains it, too.

Judging from your long kernel version number, your linux distribution vendor 
might have already backported the fix. You'll have to check your kernel 
package sources for mmap() fixes or just upgrade to something recent.

Thomas


From thomas.jarosch at intra2net.com  Thu Sep 11 11:16:25 2008
From: thomas.jarosch at intra2net.com (Thomas Jarosch)
Date: Thu, 11 Sep 2008 17:16:25 +0200
Subject: Time for 2.3.13?
In-Reply-To: <1221096795.14593.1273338561@webmail.messagingengine.com>
References: <20080910064326.GC15156@brong.net>
	<200809101600.07571.thomas.jarosch@intra2net.com>
	<1221096795.14593.1273338561@webmail.messagingengine.com>
Message-ID: <200809111716.26370.thomas.jarosch@intra2net.com>

Hi Bron,

On Thursday, 11. September 2008 03:33:15 you wrote:
> Can you post the logic that your script was using?  That might help.

The problem is triggered by a perl script that syncs a list of users
against the actual user list in cyrus. It uses perl-IMAP-Admin to do this.

Unknown users get deleted from cyrus by recursivly deleting
the folders via $imap->delete($folder) starting at the lowest folder.
After that the user gets deleted from sasldb. Here's the relevant code:
------------------------------------------
    # Count number of "/" in a string.
    sub pntcnt
    {
        my($s)=@_;
        $n=0;

        for ($i=0; $i<length($s); $i++)
        {
            if (substr($s,$i,1) eq "/")
            {
                $n++;
            }
        }

        return $n;
    }

    # $x is something like "user/jhope"

    # delete subfolders
    @sublist = $imap->list($x . "/*");
    foreach $s (reverse sort { pntcnt($a) cmp pntcnt($b) } @sublist)
    {
        $imap->set_acl($s,$admin_name,"lrswipdca");
        $err=$imap->delete($s);
        if ($err)
        {
            print("error: $imap->{'Error'}\n");
        }
    }

    # delete inbox
    $imap->set_acl($x,$admin_name,"lrswipdca");
    $err=$imap->delete($x);
    if ($err)
    {
        print("error: $imap->{'Error'}\n");
    }

    # delete from sasldb
------------------------------------------

I've noticed there is a "delayed delete" feature in imapd. I don't know
the exact details about this feature, but if it delays the delete request 
until the connection is closed, that might be the vunerable point
for a timeout as I was deleting multiple GBs of mail.

The script doesn't do anything magic besides calling
$imap->delete and $imap->close.

> Doesn't look like a skiplist issue directly to me.  Possibly it's a
> "was in a transaction and rolled back" issue, but that's separate,
> and it doesn't explain the mails still being on disk!
>
> Also - do you have split meta?  I.e. are the mail files in a different
> location than the cyrus.* files?

The meta data is in the same folder as the mails.

Any idea what might be going on?

Thanks,
Thomas


From russek at win-rar.com  Thu Sep 11 12:25:11 2008
From: russek at win-rar.com (Johannes =?ISO-8859-1?Q?Ru=DFek?=)
Date: Thu, 11 Sep 2008 18:25:11 +0200
Subject: imapoptions
Message-ID: <1221150311.23779.56.camel@admc.win-rar.local>

Hello everybody,
I'm trying to extend the SSL client cert functionality to a point where
it's flexible to use and may fit any existing PKI. For that i need to
have an option in imapd.conf which would specify which part of the
certificate to use to get the username (right now it's hardcoded CN, i
need also UID, and for virtual-domains EMAIL or subjectAltName:email
would be useful).
So i took a look at how People used to write options in imapoptions and
it appeared to me that most options would use the STRINGLIST, even
though it seemed to me their options were exclusive. Why wouldn't they
use ENUM values?
I was thinking about using the ENUM values so i could use
switch(config_getenum(tls_username_attribute)) for easier programming
and no need to use strcasecmp etc. 
Does this make sense or is there a good reason to rather use
stringlists?
Best regards,
Johannes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 2616 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080911/57a83c9f/attachment.bin 

From pnfisher at berkeley.edu  Thu Sep 11 17:57:32 2008
From: pnfisher at berkeley.edu (Paul Fisher)
Date: Thu, 11 Sep 2008 14:57:32 -0700
Subject: Problem with skiplist
In-Reply-To: <20080911130229.GA28674@brong.net>
References: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
	<20080911130229.GA28674@brong.net>
Message-ID: <48C9944C.6080703@berkeley.edu>

Bron Gondwana wrote:
> 
> You really, really want a new version of cyrusdb_skiplist.c!  I
> recommend the attached one.  It's 2.3.12 plus the patches that will
> go into 2.3.13.

On a somewhat related note, we're planning to upgrade our production
Cyrus cluster tomorrow evening from 2.3.7 to 2.3.12p2 with a handful
of patches.  While skiplist has been mostly good to us in 2.3.7, we
had been planning to apply the following skiplist patches:

     cyrus-skiplist-locking-rework-2.3.12.diff
     cyrus-skiplist-safeunlock-2.3.12.diff

The code attached to your most recent message includes
"cyrus-skiplist-tuning-2.3.10.diff", which had not previously been
mentioned as an upstream candidate for 2.3.13, as well as a change to
"struct txn" to add "*next_txn" (and related code) that doesn't appear
to be part of the current set of FastMail patches or CVSHEAD.

Could you expand a bit on what skiplist code you're planning to push
upstream for 2.3.13?

Thanks,

Paul

From brong at fastmail.fm  Thu Sep 11 19:11:02 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 12 Sep 2008 09:11:02 +1000
Subject: Problem with skiplist
In-Reply-To: <B576524D0905194499F90736CAA2758A01D8ECE2@DEEXC1U03.de.lucent.com>
References: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
	<20080911130229.GA28674@brong.net>
	<B576524D0905194499F90736CAA2758A01D8ECE2@DEEXC1U03.de.lucent.com>
Message-ID: <20080911231102.GB9388@brong.net>

On Thu, Sep 11, 2008 at 03:48:25PM +0200, DEMBKOWSKI, Henryk (Henryk) wrote:
> Bron,
> 
> Thank you for help.

No problem.

You may have noticed, I sent the wrong version of cyrusdb_skiplist.c.
The one I sent was still stable, but had two more patches that aren't
candidates for upstream at this point.

I've posted the new version to the list now.

> Do you think that we should also change linux kernel to  2.6.19.2 or
> higher?

Yes.  That's getting pretty old!  We're running 2.6.26.1 on most of
our servers.  In particular, there's a nasty mmap bug on 64 bit for
quite a few earlier versions.  .22 => .25 are quite broken.  My bug
report to LKML triggered quite a flamewar about it, and Linus came up
with a patch that worked nicely.

Ahh, I see the thread you referenced has a link to my post.

On the other hand:

"VM: Fix nasty and subtle race in shared mmap'ed page writeback"

This is unlikely to be the cause.  Cyrus doesn't use mmap writes,
only reads.  While we don't run kernels that old any more, the
skiplist problems are much more likely to be related to bugs in
the older skiplist code than kernel issues.

Bron.
 

From brong at fastmail.fm  Thu Sep 11 21:56:42 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 12 Sep 2008 11:56:42 +1000
Subject: Time for 2.3.13?
In-Reply-To: <1F1680B6-DB1F-4FF8-BB87-E797728239FC@umich.edu>
References: <20080910064326.GC15156@brong.net>
	<1F1680B6-DB1F-4FF8-BB87-E797728239FC@umich.edu>
Message-ID: <20080912015642.GA12839@brong.net>

On Wed, Sep 10, 2008 at 09:45:07AM -0400, Wesley Craig wrote:
> I see 12 items in the bugzilla (one of which is the skiplist foreach  
> bug) which I think can be closed.  So, yes, I think we should get these 
> various fixes committed and begin rolling 2.3.13.  Can you make sure 
> bugzilla has the versions of these various patches that you think should 
> be applied to HEAD?

https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3077
https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3078
https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3079
https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3080

As well as 

https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3075

of course.

I think that's all I want into 2.3.13.  I have heaps more patches
that we're running in production, but they're not generalised
enough for core cyrus yet!

Bron.

From thomas.jarosch at intra2net.com  Fri Sep 12 04:39:50 2008
From: thomas.jarosch at intra2net.com (Thomas Jarosch)
Date: Fri, 12 Sep 2008 10:39:50 +0200
Subject: Time for 2.3.13?
In-Reply-To: <20080910064326.GC15156@brong.net>
References: <20080910064326.GC15156@brong.net>
Message-ID: <200809121039.50808.thomas.jarosch@intra2net.com>

Bron,

I've quickly reviewed your patches and put them in our production server.
Seems to work fine :-)

Maybe the comment of the "cyrus-always-unlink-2.3.12.diff" patch can be 
expanded a bit so I won't be removed in the future again: unlink() is normally 
not needed as fopen(stagefile, "w+") will truncate the file if it exists.
The patch is still good as it will remove files with wrong access permissions.

Cheers,
Thomas


From hdembkowski at alcatel-lucent.com  Fri Sep 12 09:48:36 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Fri, 12 Sep 2008 15:48:36 +0200
Subject: Last recovery
Message-ID: <B576524D0905194499F90736CAA2758A01D8F02B@DEEXC1U03.de.lucent.com>

Hi,

We are using tool validate-skiplist
(http://kolab.org/cgi-bin/viewcvs-kolab.cgi/utils/admin/)
to check if annotations.db is fine. Tool validate-skiplist returns
output like this

> validate-skiplist -v annotations.db
> Skiplist file Version 1.2
> Max level: 20
> Current level: 17
> 34415 active items
> Log start at 0x003C9ED8
> Last recovery on Wed Sep 10 01:10:40 2008
> 
> Checking log part...
> Found 3830 ADD, 5336 DELETE and 5431 COMMIT nodes in log.


We noticed that sometimes database is corrupted/repaired just after
"Last recovery" is changed. 
It seems that something is done which causes that database is
repaired/corrupted.  Do you know 
what is it?

Kind Regards,
Henryk



From brong at fastmail.fm  Thu Sep 11 09:02:29 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 11 Sep 2008 23:02:29 +1000
Subject: Problem with skiplist
In-Reply-To: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
References: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
Message-ID: <20080911130229.GA28674@brong.net>

On Thu, Sep 11, 2008 at 12:30:04PM +0200, DEMBKOWSKI, Henryk (Henryk) wrote:
> Hi,
> 
> We are using skiplist for our annotations.db.  And we have big problems
> with it. 
> 
> We are using tool validate-skiplist
> (http://kolab.org/cgi-bin/viewcvs-kolab.cgi/utils/admin/)
> to check if annotations.db is fine. Some time ago we got the following
> error:
> 
> validate-skiplist: Bad node: expected ADD or DELETE at 0x001CFDB4!
> (found0x78742F70)
> 
> Unfortunately we don't know exact scenario which could cause this
> problem. We just 
> run validate-skiplist in cron (every one hour) and from time to time we
> see such error.
> 
> So we found patches to fix some issues (our problems are the same as
> described at
> http://www.intevation.de/roundup/kolab/issue840) - source can be found
> here
> 
> http://permalink.gmane.org/gmane.mail.imap.cyrus/28033
> 
> However it doesn't help. We still have error 
> 
> validate-skiplist: Bad node: expected ADD or DELETE at 0x002A855C!
> (found 0x38373033)
> 
> 
> We noticed that this error/problem is self repaired. I mean after few
> hours everything is fine. 
> For example, we had errors  
> 
> validate-skiplist -v annotations.db
> Skiplist file Version 1.2
> Max level: 20
> Current level: 17
> 26549 active items
> Log start at 0x002A855C
> Last recovery on Tue Aug 26 22:18:33 2008
> 
> Checking log part...
> validate-skiplist: Bad node: expected ADD or DELETE at 0x002A855C!
> (found 0x38373033)
> 
> 
> But after few hours 
> 
> 
> validate-skiplist -v annotations.db
> Skiplist file Version 1.2
> Max level: 20
> Current level: 17
> 34415 active items
> Log start at 0x003C9ED8
> Last recovery on Wed Sep 10 01:10:40 2008
> 
> Checking log part...
> Found 3830 ADD, 5336 DELETE and 5431 COMMIT nodes in log.
> 
> It has been repaired without any actions from our side. Interesting
> thing is 
> that "Last recovery" has been changed. Do you know what is it? Who
> changed it?
> Why it is done? When it is done? "Last recovery" suggest that something 
> recovered annotations.db. 

You really, really want a new version of cyrusdb_skiplist.c!  I
recommend the attached one.  It's 2.3.12 plus the patches that will
go into 2.3.13.

Bron ( there won't be any compatibility issues, just drop it in your
       cyrus-imapd-2.3.12/lib/ directory and rebuild )
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrusdb_skiplist.c
Type: text/x-csrc
Size: 60380 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080911/4aa17df8/attachment-0001.bin 

From brong at fastmail.fm  Thu Sep 11 19:00:16 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 12 Sep 2008 09:00:16 +1000
Subject: Problem with skiplist
In-Reply-To: <48C9944C.6080703@berkeley.edu>
References: <B576524D0905194499F90736CAA2758A01D8EBD2@DEEXC1U03.de.lucent.com>
	<20080911130229.GA28674@brong.net> <48C9944C.6080703@berkeley.edu>
Message-ID: <20080911230016.GA9388@brong.net>

On Thu, Sep 11, 2008 at 02:57:32PM -0700, Paul Fisher wrote:
> Bron Gondwana wrote:
>>
>> You really, really want a new version of cyrusdb_skiplist.c!  I
>> recommend the attached one.  It's 2.3.12 plus the patches that will
>> go into 2.3.13.
>
> On a somewhat related note, we're planning to upgrade our production
> Cyrus cluster tomorrow evening from 2.3.7 to 2.3.12p2 with a handful
> of patches.  While skiplist has been mostly good to us in 2.3.7, we
> had been planning to apply the following skiplist patches:
>
>     cyrus-skiplist-locking-rework-2.3.12.diff
>     cyrus-skiplist-safeunlock-2.3.12.diff
>
> The code attached to your most recent message includes
> "cyrus-skiplist-tuning-2.3.10.diff", which had not previously been
> mentioned as an upstream candidate for 2.3.13, as well as a change to
> "struct txn" to add "*next_txn" (and related code) that doesn't appear
> to be part of the current set of FastMail patches or CVSHEAD.
>
> Could you expand a bit on what skiplist code you're planning to push
> upstream for 2.3.13?


Oops - you're right!  I'll repost with that taken out.  I must have
pushed the entire quilt series by mistake rather than just the patches
that are going upstream...  so much for doing stuff late at night.

I haven't updated the FastMail patches website yet - the new patch
in there is the nested-transactions, which isn't heavily tested yet.

That said - the one I posted is stable as well, we're running it in
production with all those patches - but what I'm pushing for 2.3.13
is just those two patches you mentioned above.  Here's a copy of
cyrusdb_skiplist.c with just those two.

Bron.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrusdb_skiplist.c
Type: text/x-csrc
Size: 59724 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080912/55c5a55f/attachment-0001.bin 

From brong at fastmail.fm  Fri Sep 12 10:59:43 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Sat, 13 Sep 2008 00:59:43 +1000
Subject: Time for 2.3.13?
In-Reply-To: <200809121039.50808.thomas.jarosch@intra2net.com>
References: <20080910064326.GC15156@brong.net>
	<200809121039.50808.thomas.jarosch@intra2net.com>
Message-ID: <1221231583.8380.1273639285@webmail.messagingengine.com>


On Fri, 12 Sep 2008 10:39:50 +0200, "Thomas Jarosch" <thomas.jarosch at intra2net.com> said:
> Bron,
> 
> I've quickly reviewed your patches and put them in our production server.
> Seems to work fine :-)
> 
> Maybe the comment of the "cyrus-always-unlink-2.3.12.diff" patch can be 
> expanded a bit so I won't be removed in the future again: unlink() is
> normally 
> not needed as fopen(stagefile, "w+") will truncate the file if it exists.
> The patch is still good as it will remove files with wrong access
> permissions.

Yeah, good point.  The problem with truncating the file of course is that
it also truncates the hard linked copy sitting in somebody's spool somewhere,
which causes random emails to appear where they shouldn't and is very
very bad!
-- 
  Bron Gondwana
  brong at fastmail.fm


From p.spadafora at gmail.com  Sun Sep 14 18:28:02 2008
From: p.spadafora at gmail.com (Paolo Spadafora)
Date: Mon, 15 Sep 2008 00:28:02 +0200
Subject: Time for 2.3.13?
In-Reply-To: <200809121039.50808.thomas.jarosch@intra2net.com>
References: <20080910064326.GC15156@brong.net>
	<200809121039.50808.thomas.jarosch@intra2net.com>
Message-ID: <3900058c0809141528k2c167d25x179e46833ad62be2@mail.gmail.com>

Hello,
I am trying to setup sharedseen on cyrus, but I have a doubt, how should I
set the annotation ?
documentation says:
"Added support for a shared \Seen flag on messages (must be enabled on a
per-mailbox basis with the /vendor/cmu/cyrus-imapd/sharedseen mailbox
annotation." ...
calling the following doesn't work:
A145 SETANNOTATION INBOX "/comment" ("value.shared"
"/vendor/cmu/cyrus-imapd/sharedseen")
A145 OK Completed

what am I doing wrong?

thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080915/479d5eab/attachment.html 

From p.spadafora at gmail.com  Sun Sep 14 18:34:11 2008
From: p.spadafora at gmail.com (Paolo Spadafora)
Date: Mon, 15 Sep 2008 00:34:11 +0200
Subject: sharedseen
Message-ID: <3900058c0809141534q11b2e384i4539fa25b147cf21@mail.gmail.com>

Hello,
I am trying to setup sharedseen on cyrus, but I have a doubt, how should I
set the annotation ?
documentation says:
"Added support for a shared \Seen flag on messages (must be enabled on a
per-mailbox basis with the /vendor/cmu/cyrus-imapd/sharedseen mailbox
annotation." ...
the following command gets executed correctly but sharedseen doesn't work:
A145 SETANNOTATION INBOX "/comment" ("value.shared"
"/vendor/cmu/cyrus-imapd/sharedseen")
A145 OK Completed

what am I doing wrong?

thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080915/28fa2d7e/attachment.html 

From sebastian.maus at bluemars.net  Mon Sep 15 03:33:07 2008
From: sebastian.maus at bluemars.net (Sebastian Maus)
Date: Mon, 15 Sep 2008 09:33:07 +0200
Subject: sharedseen
In-Reply-To: <3900058c0809141534q11b2e384i4539fa25b147cf21@mail.gmail.com>
References: <3900058c0809141534q11b2e384i4539fa25b147cf21@mail.gmail.com>
Message-ID: <F8E2198FD83C8F887445E706@sm01.wlan.bluemars.de>

Hello,

--On September 15, 2008 0:34:11 +0200 Paolo Spadafora
<p.spadafora at gmail.com> wrote:
> Hello,
> I am trying to setup sharedseen on cyrus, but I have a doubt, how should
> I set the annotation ?
> documentation says:
> "Added support for a shared \Seen flag on messages (must be enabled on a
> per-mailbox basis with the /vendor/cmu/cyrus-imapd/
> sharedseen mailbox annotation." ...
> the following command gets executed correctly but sharedseen doesn't work:
> A145 SETANNOTATION INBOX "/comment" ("value.shared"
> "/vendor/cmu/cyrus-imapd/sharedseen")
> A145 OK Completed
> 
> what am I doing wrong?

tracing what cyradm does with "mboxconfig mailbox sharedseen true|false",
it should work if you do it like this:

	SETANNOTATION "mailbox" \
		"/vendor/cmu/cyrus-imapd/sharedseen" ("value.shared" "true")

HTH,
 -S-

From p.spadafora at gmail.com  Mon Sep 15 06:38:07 2008
From: p.spadafora at gmail.com (Paolo Spadafora)
Date: Mon, 15 Sep 2008 12:38:07 +0200
Subject: R: sharedseen
In-Reply-To: <F8E2198FD83C8F887445E706@sm01.wlan.bluemars.de>
References: <3900058c0809141534q11b2e384i4539fa25b147cf21@mail.gmail.com>
	<F8E2198FD83C8F887445E706@sm01.wlan.bluemars.de>
Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAAEw7zz0vtwJAjsZqxzAxHq4CgQAAEAAAAO9pO3xmdYRGnt4OkmVAKlcBAAAAAA==@gmail.com>

It works.

Thank you.

 

-----Messaggio originale-----
Da: Sebastian Maus [mailto:sebastian.maus at bluemars.net] 
Inviato: luned? 15 settembre 2008 9.33
A: Paolo Spadafora; cyrus-devel at lists.andrew.cmu.edu
Oggetto: Re: sharedseen

Hello,

--On September 15, 2008 0:34:11 +0200 Paolo Spadafora
<p.spadafora at gmail.com> wrote:
> Hello,
> I am trying to setup sharedseen on cyrus, but I have a doubt, how should
> I set the annotation ?
> documentation says:
> "Added support for a shared \Seen flag on messages (must be enabled on a
> per-mailbox basis with the /vendor/cmu/cyrus-imapd/
> sharedseen mailbox annotation." ...
> the following command gets executed correctly but sharedseen doesn't work:
> A145 SETANNOTATION INBOX "/comment" ("value.shared"
> "/vendor/cmu/cyrus-imapd/sharedseen")
> A145 OK Completed
> 
> what am I doing wrong?

tracing what cyradm does with "mboxconfig mailbox sharedseen true|false",
it should work if you do it like this:

	SETANNOTATION "mailbox" \
		"/vendor/cmu/cyrus-imapd/sharedseen" ("value.shared" "true")

HTH,
 -S-


From jeblair at berkeley.edu  Mon Sep 15 20:47:32 2008
From: jeblair at berkeley.edu (James E. Blair)
Date: Mon, 15 Sep 2008 17:47:32 -0700
Subject: Corruption with delayed expunge and expire annotations
Message-ID: <87iqsw29y3.fsf@berkeley.edu>

We observed some corruption of cyrus.expunge when running 2.3.7 which
we think is still relevant now that we've upgrade to 2.3.12p2 (with
proposed 2.3.13 patches from FastMail) particularly in light of the
fact that the interaction of delayed expunge and expire annotations
seems poorly defined.

The corruption would result in cyrus.expunge files that may have only
had a few bytes or kilobytes of data, but were sparse files with an
apparent size of ~200GB.  In 2.3.7, when cyrus read this file it would
usually segfault.  In 2.3.12p2, cyrus manages to read the file as if
it were valid, and merrily starts writing a cyrus.expunge.NEW file
that is _actually_ 200GB, filling the partition very quickly.

Every user in our system has a spam folder with the expire mailbox
annotation set.  The idea is that spam that has sat in the folder for
more than a user-customizable value (say a week) will be automatically
expired (based on INTERNALDATE -- a local customization).  We also use
delayed expunge, and unusual interactions happen when these two meet.

cyr_expire is the program that performs both of these functions.  In
2.3.7, it was possible for it to call mailbox_expunge with two flags
set.  If an expire annotation for a given mailbox was set:

            expunge_flags |= EXPUNGE_FORCE;

And if delayed expunge is configured:

            expunge_flags |= EXPUNGE_CLEANUP;

During this period, we had several instances of corrupted expunge
files in spam folders -- the only mailboxes where cyr_expire was
dealing with delayed expunge cleanup and expire annotations.

When both of those flags are set, a unique code path through
mailbox_expunge is run.  Essentially, what happens is:

1) cyrus.expunge is opened.  (EXPUNGE_CLEANUP causes this)
2) cyrus.expunge is locked. 
3) cyrus.expunge is mmap-ed.
4) cyrus.expunge.NEW is opened for writing.
5) process_records is called with the expunge_fd argument set.
   This causes messages to be expunged according to the annotation,
   and their records to be written to cyrus.expunge (file).
6) Data from cyrus.expunge (mmap) is copied to cyrus.expunge.NEW (file).
7) process_records is called again, operating on cyrus.expunge (mmap)
   and cyrus.expunge.NEW (file).

There is potential for trouble between steps 5 and 6.  As recently
pointed out by Linus, it's dangerous to mix read/write and mmap
access:

http://kerneltrap.org/mailarchive/linux-kernel/2008/6/18/2162194

But in this case, we are writing to a file that was previously
mmap-ed, and then subsequently accessed.  The potential for corruption
there definitely seems to exist.

It seems that adding an unmap/mmap pair to ensure up to date data
after step 5 and before step 6 should add protection against this
situation.  Specifically, that would be after the first call to
process_records in mailbox_expunge.

In the mean time, cyr_expire has since changed so that it will no
longer call mailbox_expunge with both of those flags set, but (aside
from this bug) there seems no reason why both of those flags should
not be set.  It seems like a sensible way to handle a mailbox that is
subject to both delayed expunge and the expired annotation.

In fact, because cyr_expire only calls mailbox_expunge with only one
of the two flags set, it turns out that the annotation takes
precedence on our spam folders, and is never called with
EXPUNGE_CLEANUP.  We believe this means that the cyrus.expunge file
can grow without bound, and messages that people delete manually from
their spam folders never get removed.

It seems that there are two solutions to this problem:

1) Fix the corruption described above, and adjust cyr_expire so that
   it may once again call mailbox_expunge with both flags.

2) Change cyr_expire so that it calls mailbox_expunge twice in the
   case of an expire annotation on a system with delayed expunge --
   once to expire the messages, a second time to cleanup the expunged
   messages.

We would appreciate some feedback on this analysis, as well as
comments about the right way for cyr_expire to handle this case.

James E. Blair
Principal Email Systems Administrator
UC Berkeley - IST

From hdembkowski at alcatel-lucent.com  Thu Sep 18 11:00:39 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Thu, 18 Sep 2008 17:00:39 +0200
Subject: assert
Message-ID: <B576524D0905194499F90736CAA2758A01DD2DBE@DEEXC1U03.de.lucent.com>

Hi,

When I run tool hammer_skiplist to test new version of
cyrusdb_skiplist.c I got

root at ms02.borsuk# ./hammer_skiplist -C /etc/imapd_p.conf -n
/home/tsc/dhenryk/hammer.out
fatal error: Internal error: assertion failed: cyrusdb_skiplist.c: 739:
db->current_txn == NULL

In code I see

    724 static int lock_or_refresh(struct db *db, struct txn **tidptr)
    725 {
    726     int r;
    727
    728     assert(db != NULL && tidptr != NULL);
    729
    730     if (*tidptr) {
    731         /* check that the DB agrees that we're in this
transaction */
    732         assert(db->current_txn == *tidptr);
    733
    734         /* just update the active transaction */
    735         update_lock(db, *tidptr);
    736
    737     } else {
    738         /* check that the DB isn't in a transaction */
    739         assert(db->current_txn == NULL);
    740
    741         /* grab a r/w lock */
    742         if ((r = write_lock(db, NULL)) < 0) {
    743             return r;
    744         }
    745
    746         /* start the transaction */
    747         if (r = newtxn(db, tidptr)) {
    748             return r;
    749         }
    750     }
    751
    752     return 0;
    753 }



Is it OK. that I get such error?


Kind Regards,
Henryk

From wes at umich.edu  Thu Sep 18 17:30:39 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 18 Sep 2008 17:30:39 -0400
Subject: bug in the proxy module ...
In-Reply-To: <7EF5DBE4C76A7B4DA655334E9F2BFD26591E7131D5@FRSPX100.fr01.awl.atosorigin.net>
References: <BBDB6F8E46B86245820205AB03DE3D4A2B3421B4C4@FRSPX100.fr01.awl.atosorigin.net>
	<4B9DAB02-1117-4349-AD5C-38E4278AABE7@umich.edu>
	<484E8AAB.6000305@andrew.cmu.edu>
	<21FEDCA5-428A-4DDD-936B-F428731ABEC8@umich.edu>
	<7EF5DBE4C76A7B4DA655334E9F2BFD26591E7131D5@FRSPX100.fr01.awl.atosorigin.net>
Message-ID: <7E201256-E142-442A-9925-31CC9DDDDFE7@umich.edu>

I've tried the following in a production system for a while.  Seems  
to address at least the POP issue (as expected), without introducing  
any other obvious problems.  Independent confirmation would be  
helpful before I commit it, tho.  I'm also not sure what the original  
problem this:

	https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/ 
proxy.c.diff?r1=1.3;r2=1.4

was attempting to address, and whether my solution successfully  
addresses it.

:wes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrus-imapd-imap-proxy.diff
Type: application/octet-stream
Size: 487 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080918/8372b87e/attachment.obj 

From wes at umich.edu  Thu Sep 18 19:55:12 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 18 Sep 2008 19:55:12 -0400
Subject: ptclient & ldap changes
In-Reply-To: <4844B715.4000107@ypass.net>
References: <B8A882AE-344D-4D8F-9EBC-67949B3B5E32@umich.edu>
	<48403E71.2030603@ypass.net>
	<4D33877B-28E3-4294-BA99-30E38FE9371F@umich.edu>
	<4840CED4.70607@ypass.net>
	<21D95F9A-BAFE-4697-AFDA-0F849934D28C@umich.edu>
	<4841C239.6040807@ypass.net>
	<FB919FBA-EF7A-4569-AE2A-AF481D82260C@umich.edu>
	<4844B715.4000107@ypass.net>
Message-ID: <1B4BE9A7-588D-45E2-BC68-C5745C1DF920@umich.edu>

Wesley Craig wrote:
> We could make ldap_proxy_authz tri-valued: legacy, on, and off.   
> Legacy would be the default and would revert to the old behavior.   
> Of course, that means that it wouldn't support imapd.conf's typical  
> 0/1, on/off, t/f "switch" syntax.

> The complement is not true, tho: it's very practical in the  
> ldap_member_method: attribute scheme for there to be a lot of users  
> in a given group.  Perhaps the configurable size limit option  
> should be removed.  If ldap_member_method is filter, there should  
> be no (LDAP client imposed) size limit.  If ldap_member_method is  
> attribute, the limit should be 1.

Find the patch implementing the above (and the other stuff we  
discussed earlier) attached.

:wes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrus-imapd-ptclient-ldap.diff
Type: application/octet-stream
Size: 10620 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080918/d84f0bb7/attachment.obj 

From wes at umich.edu  Thu Sep 18 20:06:27 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 18 Sep 2008 20:06:27 -0400
Subject: referring  quota commands
In-Reply-To: <C42C26A0-BBCD-4D13-B34D-9652F527DDB1@umich.edu>
References: <C42C26A0-BBCD-4D13-B34D-9652F527DDB1@umich.edu>
Message-ID: <24000848-237F-49D9-901E-7B733A32FF76@umich.edu>

Find the patch implementing quota proxying below.  My analysis of the  
"won't retain that privilege" comment: if proxyadmin and admin are  
the same, then the admin *will* retain privs during proxy.  If not,  
then proxying is the wrong result, and the admin will need to connect  
directly to the backend, either because it knows which backend to  
connect to, or because it's following a referral.

This version checks "supports_referrals" to decide whether to send a  
referral, which is the same check that, for instance, CREATE for a  
top-level mailbox uses.  Another option might be appropriate, e.g.,  
proxyd_allow_admin_referrals, to specifically permit commands that  
won't retain privileges over proxy.  This version, however, is simpler.

Any comments before I commit it?

:wes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrus-imapd-quota-referral.diff
Type: application/octet-stream
Size: 2860 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080918/e5b2b120/attachment.obj 
-------------- next part --------------

On 14 Jul 2008, at 14:34, Wesley Craig wrote:
> Is there some reason why the quota commands aren't fully proxied?   
> E.g., this comment from cmd_getquotasroot:
>
>             /* If they are an admin, they won't retain that  
> privledge if we
>              * proxy for them, so we need to refer them -- even if  
> they haven't
>              * told us they're able to handle it. */
>             imapd_refer(tag, server, name);
>
> doesn't make much sense to me.  I guess I could make my quota  
> setting agent understand the referrals, but the proxy code in  
> imapd.c already has a nice framework for caching connections, etc.   
> It seems like implementing proxy for cmd_setquota, cmd_getquota,  
> and cmd_getquotaroot would be less work and allow provisioning  
> systems to work more seamlessly out of the box with Cyrus.  Any  
> comments?

From wes at umich.edu  Thu Sep 18 20:34:08 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 18 Sep 2008 20:34:08 -0400
Subject: patch to force cyrus to reload its configuration in a more
	rigorous way
In-Reply-To: <71fe4e760804060828r6338b94ep79e3d4d66f5f38d5@mail.gmail.com>
References: <71fe4e760804011007l21c93cf6i92dd028606464d36@mail.gmail.com>
	<33418.192.168.10.25.1207125018.squirrel@webmail.bi.corp.invoca.ch>
	<71fe4e760804020557q3c70d620x12f43a3d0ab03274@mail.gmail.com>
	<71fe4e760804060828r6338b94ep79e3d4d66f5f38d5@mail.gmail.com>
Message-ID: <39D9BEFB-E53A-470E-9358-86A0ADF4E79C@umich.edu>

On 06 Apr 2008, at 11:28, Alain Spineux wrote:
> I opened a new bug on bugzilla.
>
> https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2987

I think you mean:

	https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3053

I've reviewed the patch.  Please address my comments to get it  
committed.  Thanks for the contribution!

:wes

From wes at umich.edu  Thu Sep 18 20:51:33 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 18 Sep 2008 20:51:33 -0400
Subject: 2.3.8 traditional murder POP3 proxy
In-Reply-To: <4705792F.4090400@email.arizona.edu>
References: <4705792F.4090400@email.arizona.edu>
Message-ID: <1FDFC5A2-FAAB-4EA3-B283-F427BCAD9C1F@umich.edu>

On 04 Oct 2007, at 19:37, Shawn Nock wrote:
> 1. Looking at the code, if the client side of bitpipe is closed... I
> can't see a scenario where the backend of the bitpipe is kept  
> alive. The
> clean up code seems pretty much bullet proof, yet the behavior I am
> seeing is that the client side of bitppipe is closed and the  
> connection
> to the backend is left open until POP3 timeout (what I see is pop3d on
> the backend waiting on select(). What could cause this? (*note* I  
> doubt
> seriously that resolving this question will do anything to ease the
> symptoms of this bug in outlook).

This sounds a lot like:

	http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3066

Not so much like an outlook bug.

:wes

From wes at umich.edu  Thu Sep 18 21:03:49 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 18 Sep 2008 21:03:49 -0400
Subject: mbexamine and block size cache files
In-Reply-To: <20080820222536.GA87403@exuma.irbs.com>
References: <20080819235503.GA56814@exuma.irbs.com>
	<5C4D9A17-E7D9-40ED-AFB4-6C9B79F5EC63@umich.edu>
	<20080820135651.GA73439@exuma.irbs.com>
	<0F2DE84E-8A35-4432-B79B-D0875F044FBE@umich.edu>
	<20080820222536.GA87403@exuma.irbs.com>
Message-ID: <6F8A09A6-2A9E-480C-8A54-4B398E528B67@umich.edu>

On 20 Aug 2008, at 18:25, John Capo wrote:
> Tested patch attached.

I committed this.

:wes

From nock at email.arizona.edu  Thu Sep 18 22:42:56 2008
From: nock at email.arizona.edu (Shawn Nock)
Date: Thu, 18 Sep 2008 19:42:56 -0700
Subject: 2.3.8 traditional murder POP3 proxy
In-Reply-To: <1FDFC5A2-FAAB-4EA3-B283-F427BCAD9C1F@umich.edu>
References: <4705792F.4090400@email.arizona.edu>
	<1FDFC5A2-FAAB-4EA3-B283-F427BCAD9C1F@umich.edu>
Message-ID: <48D311B0.4060408@email.arizona.edu>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I followed the recent thread about this, and I agree. Looking back on
this thread there has been very little traffic since your last post.
I'll streamline application of this patch to our systems.

Thanks for the heads up,
Shawn

Wesley Craig wrote:
| On 04 Oct 2007, at 19:37, Shawn Nock wrote:
|> 1. Looking at the code, if the client side of bitpipe is closed... I
|> can't see a scenario where the backend of the bitpipe is kept alive. The
|> clean up code seems pretty much bullet proof, yet the behavior I am
|> seeing is that the client side of bitppipe is closed and the connection
|> to the backend is left open until POP3 timeout (what I see is pop3d on
|> the backend waiting on select(). What could cause this? (*note* I doubt
|> seriously that resolving this question will do anything to ease the
|> symptoms of this bug in outlook).
|
| This sounds a lot like:
|
|     http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3066
|
| Not so much like an outlook bug.
|
| :wes
|


- --
Shawn Nock (OpenPGP: 0xFF7D08A3)
Unix Systems Group; UITS
University of Arizona
nock at email.arizona.edu
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkjTEa8ACgkQUfD1hN5kJ6X2YgCfc58DcX9pXzvVRZyc/zT6f/q0
MXIAn19rg/+aNdJxPCD963veT+rB+3MZ
=4ev8
-----END PGP SIGNATURE-----

From wes at umich.edu  Thu Sep 18 22:48:08 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 18 Sep 2008 22:48:08 -0400
Subject: 2.3.8 traditional murder POP3 proxy
In-Reply-To: <48D311B0.4060408@email.arizona.edu>
References: <4705792F.4090400@email.arizona.edu>
	<1FDFC5A2-FAAB-4EA3-B283-F427BCAD9C1F@umich.edu>
	<48D311B0.4060408@email.arizona.edu>
Message-ID: <712985E1-386E-40FE-8F93-5266267A8FE5@umich.edu>

On 18 Sep 2008, at 22:42, Shawn Nock wrote:
> I followed the recent thread about this, and I agree. Looking back on
> this thread there has been very little traffic since your last post.
> I'll streamline application of this patch to our systems.

Please let me know if it corrects the issue you've seen.  Thanks,

:wes

From brong at fastmail.fm  Fri Sep 19 01:52:50 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 19 Sep 2008 15:52:50 +1000
Subject: assert
In-Reply-To: <B576524D0905194499F90736CAA2758A01DD2DBE@DEEXC1U03.de.lucent.com>
References: <B576524D0905194499F90736CAA2758A01DD2DBE@DEEXC1U03.de.lucent.com>
Message-ID: <20080919055250.GA27685@brong.net>

On Thu, Sep 18, 2008 at 05:00:39PM +0200, DEMBKOWSKI, Henryk (Henryk) wrote:
> Hi,
> 
> When I run tool hammer_skiplist to test new version of
> cyrusdb_skiplist.c I got
> 
> root at ms02.borsuk# ./hammer_skiplist -C /etc/imapd_p.conf -n
> /home/tsc/dhenryk/hammer.out
> fatal error: Internal error: assertion failed: cyrusdb_skiplist.c: 739:
> db->current_txn == NULL

        if (rop >= 999) {
          if (!r) r = DB_OLD->store(db, key, klen, value, vlen, tp);
          /* forget to commit */
        }

This would cause the bug you're seeing.  Basically it's an invalid usage
of the API deliberately to cause this crash.

It only happens once per 1000 cycles through the loop!

Bron.

From hdembkowski at alcatel-lucent.com  Fri Sep 19 03:54:24 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Fri, 19 Sep 2008 09:54:24 +0200
Subject: assert
In-Reply-To: <20080919055250.GA27685@brong.net>
References: <B576524D0905194499F90736CAA2758A01DD2DBE@DEEXC1U03.de.lucent.com>
	<20080919055250.GA27685@brong.net>
Message-ID: <B576524D0905194499F90736CAA2758A01DD2ECE@DEEXC1U03.de.lucent.com>

Bron,

Thank you very much for info.

Henryk 



-----Original Message-----
From: Bron Gondwana [mailto:brong at fastmail.fm] 
Sent: Friday, September 19, 2008 7:53 AM
To: DEMBKOWSKI, Henryk (Henryk)
Cc: cyrus-devel at lists.andrew.cmu.edu; info-cyrus at lists.andrew.cmu.edu
Subject: Re: assert

On Thu, Sep 18, 2008 at 05:00:39PM +0200, DEMBKOWSKI, Henryk (Henryk)
wrote:
> Hi,
> 
> When I run tool hammer_skiplist to test new version of 
> cyrusdb_skiplist.c I got
> 
> root at ms02.borsuk# ./hammer_skiplist -C /etc/imapd_p.conf -n 
> /home/tsc/dhenryk/hammer.out fatal error: Internal error: assertion 
> failed: cyrusdb_skiplist.c: 739:
> db->current_txn == NULL

        if (rop >= 999) {
          if (!r) r = DB_OLD->store(db, key, klen, value, vlen, tp);
          /* forget to commit */
        }

This would cause the bug you're seeing.  Basically it's an invalid usage
of the API deliberately to cause this crash.

It only happens once per 1000 cycles through the loop!

Bron.

From hdembkowski at alcatel-lucent.com  Tue Sep 23 09:58:03 2008
From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk))
Date: Tue, 23 Sep 2008 15:58:03 +0200
Subject: I can't remove message
Message-ID: <B576524D0905194499F90736CAA2758A01E133CE@DEEXC1U03.de.lucent.com>

Hi,

We have mailbox with one new message. Subscriber is not able to 
remove this message. We checked that cyrus (IMAP) doesn't allow 
to store any flags for message. I mean, subscriber can't save this
message. It is still marked as new.

I am wondering what can be wrong? Which of files can be corrupted:

-rw-------  2 cyrus mail 46667 Sep 23 15:07 211.

-rw-------  1 cyrus mail  1332 Sep 23 15:43 cyrus.cache
-rw-------  1 cyrus mail   289 Sep 23 15:08 cyrus.header
-rw-------  1 cyrus mail   176 Sep 23 15:43 cyrus.index

I thought that maybe reconstruct can help but it doesn't.
After reconstruct we still can't delete message. 


Kind Regards,
Henryk



From selsky at columbia.edu  Tue Sep 23 19:15:15 2008
From: selsky at columbia.edu (Matt Selsky)
Date: Tue, 23 Sep 2008 19:15:15 -0400
Subject: [Cyrus-CVS] src/cyrus/imap by murch
In-Reply-To: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
Message-ID: <20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>

Quoting murch at andrew.cmu.edu:

> Update of /afs/andrew/system/cvs/src/cyrus/imap
> In directory unix36.andrew.cmu.edu:/var/tmp/cvs-serv5457
>
> Modified Files:
> 	Makefile.in cyr_virusscan.c
> Log Message:
> aded code to append a notification message to user's INBOX when we remove
> infected messages from any of their mailboxes
>
>
> --- links to diffs follow ---
> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/Makefile.in.diff?r1=1.195&r2=1.196
> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/cyr_virusscan.c.diff?r1=1.4&r2=1.5

Why is this new feature being committed on the stable branch right  
before the (hopefully soon) 2.3.13 release?


-- 
Matt


From brong at fastmail.fm  Tue Sep 23 19:53:02 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Wed, 24 Sep 2008 09:53:02 +1000
Subject: [Cyrus-CVS] src/cyrus/imap by murch
In-Reply-To: <20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
Message-ID: <20080923235302.GA13895@brong.net>

On Tue, Sep 23, 2008 at 07:15:15PM -0400, Matt Selsky wrote:
> Quoting murch at andrew.cmu.edu:
>
>> Update of /afs/andrew/system/cvs/src/cyrus/imap
>> In directory unix36.andrew.cmu.edu:/var/tmp/cvs-serv5457
>>
>> Modified Files:
>> 	Makefile.in cyr_virusscan.c
>> Log Message:
>> aded code to append a notification message to user's INBOX when we remove
>> infected messages from any of their mailboxes
>>
>>
>> --- links to diffs follow ---
>> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/Makefile.in.diff?r1=1.195&r2=1.196
>> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/cyr_virusscan.c.diff?r1=1.4&r2=1.5
>
> Why is this new feature being committed on the stable branch right  
> before the (hopefully soon) 2.3.13 release?

It looks to me like a standalone daemon that doesn't run unless you
explicitly invoke it.  So long as the file compiles, it doesn't really
make much difference!

Bron.

From dave64 at andrew.cmu.edu  Tue Sep 23 20:08:26 2008
From: dave64 at andrew.cmu.edu (dave64)
Date: Tue, 23 Sep 2008 20:08:26 -0400
Subject: [Cyrus-CVS] src/cyrus/imap by murch
In-Reply-To: <20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
Message-ID: <48D984FA.3070309@andrew.cmu.edu>

Matt Selsky wrote:
> Quoting murch at andrew.cmu.edu:
> 
>> Update of /afs/andrew/system/cvs/src/cyrus/imap
>> In directory unix36.andrew.cmu.edu:/var/tmp/cvs-serv5457
>>
>> Modified Files:
>>     Makefile.in cyr_virusscan.c
>> Log Message:
>> aded code to append a notification message to user's INBOX when we remove
>> infected messages from any of their mailboxes
>>
>>
>> --- links to diffs follow ---
>> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/Makefile.in.diff?r1=1.195&r2=1.196 
>>
>> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/cyr_virusscan.c.diff?r1=1.4&r2=1.5 
>>
> 
> Why is this new feature being committed on the stable branch right 
> before the (hopefully soon) 2.3.13 release?

Because we need it ;)  As Bron pointed out, this is a standalone utility 
   so its ability to break anything else is limited.

Thanks,

Dave

From wes at umich.edu  Tue Sep 23 20:39:31 2008
From: wes at umich.edu (Wesley Craig)
Date: Tue, 23 Sep 2008 20:39:31 -0400
Subject: [Cyrus-CVS] src/cyrus/imap by murch
In-Reply-To: <20080923235302.GA13895@brong.net>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<20080923235302.GA13895@brong.net>
Message-ID: <363387F6-F81C-41C9-984E-7ED68638F454@umich.edu>

On 23 Sep 2008, at 19:53, Bron Gondwana wrote:
> It looks to me like a standalone daemon that doesn't run unless you
> explicitly invoke it.  So long as the file compiles, it doesn't really
> make much difference!

That's a good point, of course, but I think we've all agreed that  
significant new features should be discussed on the development list:

	http://cyrusimap.web.cmu.edu/bylaws.html

Most projects have a review-then-commit or a commit-then-review  
procedure, often switching between the two procedures by branch or as  
a release approaches.

I have no idea how significant the addition of cyr_virusscan is.  It  
would be a lot less work for me to know how significant the change is  
if someone mentioned what to expect when I'm looking over the changes.

:wes

From murch at andrew.cmu.edu  Tue Sep 23 20:51:53 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Tue, 23 Sep 2008 20:51:53 -0400
Subject: [Cyrus-CVS] src/cyrus/imap by murch
In-Reply-To: <20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
Message-ID: <48D98F29.6050505@andrew.cmu.edu>

Matt Selsky wrote:
> Quoting murch at andrew.cmu.edu:
> 
>> Update of /afs/andrew/system/cvs/src/cyrus/imap
>> In directory unix36.andrew.cmu.edu:/var/tmp/cvs-serv5457
>>
>> Modified Files:
>>     Makefile.in cyr_virusscan.c
>> Log Message:
>> aded code to append a notification message to user's INBOX when we remove
>> infected messages from any of their mailboxes
>>
>>
>> --- links to diffs follow ---
>> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/Makefile.in.diff?r1=1.195&r2=1.196 
>>
>> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/cyr_virusscan.c.diff?r1=1.4&r2=1.5 
>>
> 
> Why is this new feature being committed on the stable branch right 
> before the (hopefully soon) 2.3.13 release?

Its a separate tool with no changes to the rest of the codebase, and its 
a tool that my boss' boss asked for for the University.  I always do 
what gets me paid.


-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

From wes at umich.edu  Tue Sep 23 21:43:01 2008
From: wes at umich.edu (Wesley Craig)
Date: Tue, 23 Sep 2008 21:43:01 -0400
Subject: [Cyrus-CVS] src/cyrus/imap by murch
In-Reply-To: <48D98F29.6050505@andrew.cmu.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
Message-ID: <F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>

On 23 Sep 2008, at 20:51, Ken Murchison wrote:
> Its a separate tool with no changes to the rest of the codebase,  
> and its a tool that my boss' boss asked for for the University.  I  
> always do what gets me paid.

Me too!  ;)

Just to confirm, this tool doesn't get built unless a specific:

	make cyr_virusscan

is issued?  No changes to configure as well?  More communication is  
just better.

:wes

From murch at andrew.cmu.edu  Tue Sep 23 22:12:30 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Tue, 23 Sep 2008 22:12:30 -0400
Subject: [Cyrus-CVS] src/cyrus/imap by murch
In-Reply-To: <F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
	<F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
Message-ID: <48D9A20E.8080509@andrew.cmu.edu>

Wesley Craig wrote:
> On 23 Sep 2008, at 20:51, Ken Murchison wrote:
>> Its a separate tool with no changes to the rest of the codebase, and 
>> its a tool that my boss' boss asked for for the University.  I always 
>> do what gets me paid.
> 
> Me too!  ;)
> 
> Just to confirm, this tool doesn't get built unless a specific:
> 
>     make cyr_virusscan
> 
> is issued?  No changes to configure as well?  More communication is just 
> better.

Correct.

I started working my way through bugzilla today.  Should we discuss what 
we should include in 2.3.13?  I committed 3-4 of Bron's fixes today.


-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

From brong at fastmail.fm  Tue Sep 23 22:43:10 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Wed, 24 Sep 2008 12:43:10 +1000
Subject: Things for 2.3.13 (was: [Cyrus-CVS] src/cyrus/imap by murch)
In-Reply-To: <48D9A20E.8080509@andrew.cmu.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
	<F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
	<48D9A20E.8080509@andrew.cmu.edu>
Message-ID: <20080924024310.GA22349@brong.net>

On Tue, Sep 23, 2008 at 10:12:30PM -0400, Ken Murchison wrote:
> I started working my way through bugzilla today.  Should we discuss what  
> we should include in 2.3.13?  I committed 3-4 of Bron's fixes today.

Thanks, I saw the commits.

That's my entire wishlist for 2.3.13 (plus the buffer size patch you
already included)

I've got a couple of replication protocol items I'm interested in, but
they're not written yet.  Basically setseen_all and subscriptions.
They are the two things where we're still seeing occasional drift
between master and replica - setseen_all because it tries to merge
rather than just replace, and subscriptions I'm not sure yet.  Suspect
a race condition.  It's only occasionally that it seems to skip a
subscription on the replica, and sync_client -u fixes it.  We get a
handful of these per week, which isn't bad going.

It will be good to get the latest skiplist fixes out there.

Bron.

From murch at andrew.cmu.edu  Thu Sep 25 07:40:55 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Thu, 25 Sep 2008 07:40:55 -0400
Subject: Cyrus 2.3.13 RC1
Message-ID: <48DB78C7.9050903@andrew.cmu.edu>

I just put together a release candidate for Cyrus 2.3.13.  I'd
appreciate any independent testing before I release this to the masses.

http://www.contrib.andrew.cmu.edu/~murch/cyrus-imapd-2.3.13rc1.tar.gz


Noteworthy changes:

* Added an experimental "sql" backend for cyrusdb.  Currently MySQL,
   PostgreSQL, and SQLite are supported.
* Added support for IMAP [CAPABILITY] response code to client-side
   of Murder proxies.
* Added support for ManageSieve auto-capability response after
   STARTTLS and after AUTH with a SASL security layer.
* Made MAXWORD and MAXQUOTED sizes configurable via imapd.conf
* Rewrote cyrusdb_quotalegacy.c to use readir()
   rather than glob.c.  This avoids a potential crash due to
   conflicts between glibc and Heimdal implementations of glob().
* Added support for fulldirhash to 'ctl_mboxlist -v'
* Several skiplist bugfixes.

Check doc/changes.html for a complete list of changes.

If there are any outstanding issues that you believe still need to be
addressed in 2.3.13, please let me know.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University





From murch at andrew.cmu.edu  Thu Sep 25 09:01:26 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Thu, 25 Sep 2008 09:01:26 -0400
Subject: [Fwd: Re: Cyrus 2.3.13 RC1]
Message-ID: <48DB8BA6.20703@andrew.cmu.edu>

Anyone want to comment on these requests?  I had planned on getting to 
these patches, but then the push to get out 2.3.13 began.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University
-------------- next part --------------
An embedded message was scrubbed...
From: Michael Menge <michael.menge at zdv.uni-tuebingen.de>
Subject: Re: Cyrus 2.3.13 RC1
Date: Thu, 25 Sep 2008 14:52:20 +0200
Size: 10791
Url: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080925/702fbdc2/attachment.eml 

From vladimir.elisseev at gmail.com  Thu Sep 25 09:18:50 2008
From: vladimir.elisseev at gmail.com (vladimir.elisseev at gmail.com)
Date: Thu, 25 Sep 2008 15:18:50 +0200
Subject: How to set realm: cyrus-imap -> saslauthd?
Message-ID: <20080925151850.20654wm4w24lkybe@vovan.homelinux.org>

Hello,

I'm getting troubles with imap -> saslauthd authentication,  
particularly with passing realm to saslauthd.
Both "testsaslauthd -u test -p ***" and "testsaslauthd -u test -p ***  
-r TEST.LOCAL" work fine, but "imtest -u test -a test -m PLAIN" fails  
while saslauthd prints:
do_auth : auth failure: [user=test] [service=imap] [realm=test.local]  
[mech=kerberos5] [reason=saslauthd internal error].
The problem is clear: realm should be TEST.LOCAL.
I tried to set this realm with -r switch, but saslauthd uses always  
the same whatever I set with -r. Don't have any clue, how to deal with  
this issue and how to set realm for cyrus-imap.

Regards,
Vladimir.










From wes at umich.edu  Thu Sep 25 11:04:03 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 25 Sep 2008 11:04:03 -0400
Subject: Things for 2.3.13
In-Reply-To: <20080924024310.GA22349@brong.net>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
	<F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
	<48D9A20E.8080509@andrew.cmu.edu>
	<20080924024310.GA22349@brong.net>
Message-ID: <19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu>

On 23 Sep 2008, at 22:43, Bron Gondwana wrote:
> That's my entire wishlist for 2.3.13 (plus the buffer size patch you
> already included)

I think there should be an RC2.

I don't see:

	https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3075

included in RC1.  I gathered that *was* on your wishlist for 2.3.13.

I've marked a few other bugzilla items as blockers, as an indication  
of what I think should go into 2.3.13.  In general, unified murder  
continues to be broken for many uses.  I have fixes for 2914, but  
2915 requires some discussion.  There's also 2814, 2973, and 2976,  
all related to unified murder faults.

Stuff not in bugzilla (AFAIK) for which I have fixes running in  
production:

. force_sasl_client_mech and hostname_mechs can't be plain when  
backends require PLAIN+TLS.

. backend_connect edits prot->banner.is_capa.

. the mupdate master reports changes to mupdate slaves out of order  
if the change rate is higher that 1 every 10 seconds.

. master starts two threads for every connection.

. mupdate slave ignores errors when writing NOOPs (e.g., that the  
GSSAPI context has expired).  There's also the possibility to get our  
of sync with the mupdate master.

I also have tested code:

. which implements the various LDAP changes that I've discussed on  
the list.

. adds debugging for ctl_mboxlist, replacing the assert I added  
recently for cases where ctl_mboxlist was incorrectly removing local  
mailboxes.

. proxies quota commands when supports_referrals is 0.

. which looks for cases of MODSEQ being inappropriately set to 0 and  
correcting it to be 1.  Several buggy versions were released into the  
wild, so it's quite likely that there are MODSEQ 0 mailboxes out  
there.  Xfer-ing them to a machine with this code in place will  
correct the corruption.  Xfer-ing them to a machine without this code  
in place causes sync_client to die.

I'm also pretty confident there's a bug in prot_select() which causes  
it to return prematurely, indicating there's IO available on a socket  
when instead there's just an event that needs to be serviced.  I  
haven't worked on a fix, but I think it's likely there are deadlocks  
that happen as a result of this bug.  Not that I've seen any in the  
wild, yet.

There might also need to be something done about the sieve utf-7  
thing.  Not that I have a proposal...

:wes



From murch at andrew.cmu.edu  Thu Sep 25 11:35:51 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Thu, 25 Sep 2008 11:35:51 -0400
Subject: Things for 2.3.13
In-Reply-To: <19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
	<F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
	<48D9A20E.8080509@andrew.cmu.edu>
	<20080924024310.GA22349@brong.net>
	<19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu>
Message-ID: <48DBAFD7.3060007@andrew.cmu.edu>

Wesley Craig wrote:
> On 23 Sep 2008, at 22:43, Bron Gondwana wrote:
>> That's my entire wishlist for 2.3.13 (plus the buffer size patch you
>> already included)
> 
> I think there should be an RC2.
> 
> I don't see:
> 
>     https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3075
> 
> included in RC1.  I gathered that *was* on your wishlist for 2.3.13.

Isn't this the fix that Bron said needs more discussion?  Or am I 
confusing it with something else.


> I've marked a few other bugzilla items as blockers, as an indication of 
> what I think should go into 2.3.13.  In general, unified murder 
> continues to be broken for many uses.  I have fixes for 2914, but 2915 
> requires some discussion.  There's also 2814, 2973, and 2976, all 
> related to unified murder faults.

Unified Murder may have been a bad idea.  I'm all for removing the code, 
unless folks find it useful and want it fixed.


> Stuff not in bugzilla (AFAIK) for which I have fixes running in production:
> 
> . force_sasl_client_mech and hostname_mechs can't be plain when backends 
> require PLAIN+TLS.
> 
> . backend_connect edits prot->banner.is_capa.

Depending where you're looking, it might be by design.


Are the bugs below blockers, or can they wait?

> . the mupdate master reports changes to mupdate slaves out of order if 
> the change rate is higher that 1 every 10 seconds.
> 
> . master starts two threads for every connection.
> 
> . mupdate slave ignores errors when writing NOOPs (e.g., that the GSSAPI 
> context has expired).  There's also the possibility to get our of sync 
> with the mupdate master.
> 
> I also have tested code:
> 
> . which implements the various LDAP changes that I've discussed on the 
> list.
> 
> . adds debugging for ctl_mboxlist, replacing the assert I added recently 
> for cases where ctl_mboxlist was incorrectly removing local mailboxes.
> 
> . proxies quota commands when supports_referrals is 0.
> 
> . which looks for cases of MODSEQ being inappropriately set to 0 and 
> correcting it to be 1.  Several buggy versions were released into the 
> wild, so it's quite likely that there are MODSEQ 0 mailboxes out there.  
> Xfer-ing them to a machine with this code in place will correct the 
> corruption.  Xfer-ing them to a machine without this code in place 
> causes sync_client to die.
> 
> I'm also pretty confident there's a bug in prot_select() which causes it 
> to return prematurely, indicating there's IO available on a socket when 
> instead there's just an event that needs to be serviced.  I haven't 
> worked on a fix, but I think it's likely there are deadlocks that happen 
> as a result of this bug.  Not that I've seen any in the wild, yet.
> 
> There might also need to be something done about the sieve utf-7 thing.  
> Not that I have a proposal...
> 
> :wes
> 
> 
> 


-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

From wes at umich.edu  Thu Sep 25 11:46:41 2008
From: wes at umich.edu (Wesley Craig)
Date: Thu, 25 Sep 2008 11:46:41 -0400
Subject: Things for 2.3.13
In-Reply-To: <48DBAFD7.3060007@andrew.cmu.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
	<F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
	<48D9A20E.8080509@andrew.cmu.edu>
	<20080924024310.GA22349@brong.net>
	<19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu>
	<48DBAFD7.3060007@andrew.cmu.edu>
Message-ID: <81A29BCC-79D8-41CA-BD23-0C6819F57743@umich.edu>

On 25 Sep 2008, at 11:35, Ken Murchison wrote:
> Wesley Craig wrote:
>> On 23 Sep 2008, at 22:43, Bron Gondwana wrote:
>>> That's my entire wishlist for 2.3.13 (plus the buffer size patch you
>>> already included)
>> I think there should be an RC2.
>> I don't see:
>>     https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3075
>> included in RC1.  I gathered that *was* on your wishlist for 2.3.13.
>
> Isn't this the fix that Bron said needs more discussion?  Or am I  
> confusing it with something else.

No, the "more discussion" part was stacked transactions.  Which seems  
simple enough, but I'm not really prepared to evaluate the overall idea.

>> I've marked a few other bugzilla items as blockers, as an  
>> indication of what I think should go into 2.3.13.  In general,  
>> unified murder continues to be broken for many uses.  I have fixes  
>> for 2914, but 2915 requires some discussion.  There's also 2814,  
>> 2973, and 2976, all related to unified murder faults.
>
> Unified Murder may have been a bad idea.  I'm all for removing the  
> code, unless folks find it useful and want it fixed.

I think it's a good idea.  In particular, I think it's step in the  
right direction for small installations.  Wouldn't it be nice if you  
could run a full murder with replication (and automatic failover, but  
we're a long way from that) on a pair of machines?

>> . backend_connect edits prot->banner.is_capa.
>
> Depending where you're looking, it might be by design.

It causes crashes in some configurations.

> Are the bugs below blockers, or can they wait?
>
>> . the mupdate master reports changes to mupdate slaves out of  
>> order if the change rate is higher that 1 every 10 seconds.

This is a blocker, I think.  It likely the cause of certain  
corruption on frontend.

>> . master starts two threads for every connection.
>> . mupdate slave ignores errors when writing NOOPs (e.g., that the  
>> GSSAPI context has expired).  There's also the possibility to get  
>> our of sync with the mupdate master.

These two could wait.

>> . which implements the various LDAP changes that I've discussed on  
>> the list.

I guess this could wait.

>> . adds debugging for ctl_mboxlist, replacing the assert I added  
>> recently for cases where ctl_mboxlist was incorrectly removing  
>> local mailboxes.

This should be included, in hopes of gathering sufficient data to  
find whatever the bug is in mupdate master which causes the ordering  
problem.

>> . proxies quota commands when supports_referrals is 0.

Toss up.

>> . which looks for cases of MODSEQ being inappropriately set to 0  
>> and correcting it to be 1.  Several buggy versions were released  
>> into the wild, so it's quite likely that there are MODSEQ 0  
>> mailboxes out there.  Xfer-ing them to a machine with this code in  
>> place will correct the corruption.  Xfer-ing them to a machine  
>> without this code in place causes sync_client to die.

This could be a blocker, given the support traffic it's likely to  
generate.

>> I'm also pretty confident there's a bug in prot_select() which  
>> causes it to return prematurely, indicating there's IO available  
>> on a socket when instead there's just an event that needs to be  
>> serviced.  I haven't worked on a fix, but I think it's likely  
>> there are deadlocks that happen as a result of this bug.  Not that  
>> I've seen any in the wild, yet.

Probably this should wait.

>> There might also need to be something done about the sieve utf-7  
>> thing.  Not that I have a proposal...

I'd probably ask the people who are complaining about it.  They seem  
pretty aghast at the change.

:wes

From jc at irbs.com  Thu Sep 25 15:59:04 2008
From: jc at irbs.com (John Capo)
Date: Thu, 25 Sep 2008 15:59:04 -0400
Subject: Cyrus 2.3.13 RC1
In-Reply-To: <48DB78C7.9050903@andrew.cmu.edu>
References: <48DB78C7.9050903@andrew.cmu.edu>
Message-ID: <20080925195904.GA28965@exuma.irbs.com>

Quoting Ken Murchison (murch at andrew.cmu.edu):
> I just put together a release candidate for Cyrus 2.3.13.  I'd
> appreciate any independent testing before I release this to the masses.
> 
> http://www.contrib.andrew.cmu.edu/~murch/cyrus-imapd-2.3.13rc1.tar.gz
> 
> 
> Noteworthy changes:
> 
> * Added an experimental "sql" backend for cyrusdb.  Currently MySQL,
>   PostgreSQL, and SQLite are supported.
> * Added support for IMAP [CAPABILITY] response code to client-side
>   of Murder proxies.
> * Added support for ManageSieve auto-capability response after
>   STARTTLS and after AUTH with a SASL security layer.
> * Made MAXWORD and MAXQUOTED sizes configurable via imapd.conf
> * Rewrote cyrusdb_quotalegacy.c to use readir()
>   rather than glob.c.  This avoids a potential crash due to
>   conflicts between glibc and Heimdal implementations of glob().
> * Added support for fulldirhash to 'ctl_mboxlist -v'
> * Several skiplist bugfixes.
> 
> Check doc/changes.html for a complete list of changes.
> 
> If there are any outstanding issues that you believe still need to be
> addressed in 2.3.13, please let me know.


statuscache_db.c assumes an off_t is 32bits.

    if (p < dend) scdata->index_size = strtoul(p, &p, 10);

and in the printf that generates the string for the DB.

Attached is an autoconf patch that Lorenzo Catucci posted to the
mailing list in May.  The patch applies and re-running autoconf
does figure out that off_t is 64bits on BSD.  Autoconf fails badly
in other ways but it always does for me.

Here is my original post and the details I sent to Rudy Gevaert
about how the bug bites sometimes.

> Found a bug handling the off_t index_size value in the statuscache
> code.  The code assumes off_t is a 32 bits and the message count gets
> >written as 0.
> 
> (gdb) p *scdata
> $22 = {statusitems = 63, index_mtime = 1208893338, index_ino = 1974639,
> index_size = 6581528, messages = 74789, recent = 174, uidnext = 1547527,
>   uidvalidity = 1125360596, unseen = 74743, highestmodseq = 0}
> (gdb) n
> 247         r = DB->store(statuscachedb, key, keylen, data, datalen, NULL);
> (gdb) p data
> $23 = "3 63 1208893338 1974639 6581528 0 74789 174 1547527 1125360596
> 74743\000
> 
> The attached patch assumes off_t is 64 bits.  Are there any 32 bit
> off_t systems left?
> 

Here's the reason why testing didn't turn up this bug.

I suspected that this might be something to do with newer compilers
mucking with arguments to printf and friends and fixing argument
sizes based on printf paramaters.  This turns out not to be the
case and I would be astonished if the compiler did not do what it
was told to do with respect to printf arguments.

statuscache_db.c:

202         /* Sanity check the data */
203         if (!scdata->statusitems || !scdata->index_mtime || !scdata->index_ino ||
204             !scdata->index_size || !scdata->uidnext || !scdata->uidvalidity) {
205             return IMAP_NO_NOSUCHMSG;
206         }
207
(gdb) p *scdata
$11 = {statusitems = 63, index_mtime = 1208964427, index_ino = 48227,
   index_size = 1856, messages = 0, recent = 20, uidnext = 0,
   uidvalidity = 216164, unseen = 1166134651, highestmodseq = 17}

The test above will fail and the status cache data will not be used
if the bogus data in the status cache is 'just right'.  In the case
above the uidnext is 0 because the argument that was used to store
that value was not the right one due to printf argument alignment,
the 64 bit off_t when a 32 bit value was expected by printf.

My tests are with folders with many thousands of messages.  When
printf generates the string to store in the cache, bogus values are
stored that pass the test above. The bogus status cache data will
be used resulting in the wrong status.

* STATUS BK (MESSAGES 75904 RECENT 1620 UIDNEXT 1549555 UIDVALIDITY 1125360596 UNSEEN 75858)
* STATUS BK (MESSAGES 0 RECENT 75904 UIDNEXT 1620 UIDVALIDITY 1549555 UNSEEN 1125360596)

John Capo


-------------- next part --------------
diff -r f1c01f49ce58 configure.in
--- a/configure.in	Mon May 26 13:45:47 2008 +0200
+++ b/configure.in	Tue May 27 13:31:29 2008 +0200
@@ -49,7 +49,7 @@
 AC_INIT(imap/imapd.c)
 AC_PREREQ([2.54])
 AC_CONFIG_HEADER(config.h)
-AC_CANONICAL_HOST
+AC_CANONICAL_SYSTEM
 
 dnl Useful hook for distributions
 AC_ARG_WITH(extraident,[  --with-extraident=STRING   use STRING as extra version information],
@@ -110,6 +110,9 @@
   AC_DEFINE(HAVE_LONG_LONG_INT,[],[Does the compiler support long long int?])
   AC_C_BIGENDIAN
 fi
+
+dnl Check off_t size
+AC_CHECK_SIZEOF(off_t)
 
 CMU_C___ATTRIBUTE__
 CMU_C_FPIC
@@ -989,9 +992,17 @@
 dnl add perl cccdlflags when building libraries -- this ensures that the
 dnl libraries will be compiled as PIC if perl requires PIC objects
 dnl -- this is needed on NetBSD, but seems to cause problems on atleast Solaris --
-dnl    eval `${PERL} -V:cccdlflags`
-    PERL_CCCDLFLAGS="$cccdlflags"
-    AC_SUBST(PERL_CCCDLFLAGS)
+    case "${target_os}" in
+        linux*|netbsd*)
+            AC_MSG_CHECKING(for perl cccdlflags needed on "${target_os}")
+            eval `${PERL} -V:cccdlflags`
+            PERL_CCCDLFLAGS="$cccdlflags"
+            AC_SUBST(PERL_CCCDLFLAGS)
+            AC_MSG_RESULT($PERL_CCCDLFLAGS)
+            ;;
+        *)
+            AC_MSG_WARN(skipping check for perl cccdlflags on "${target_os}")
+    esac
 fi
 
 dnl for timsieved
diff -r f1c01f49ce58 imap/statuscache_db.c
--- a/imap/statuscache_db.c	Mon May 26 13:45:47 2008 +0200
+++ b/imap/statuscache_db.c	Tue May 27 13:31:29 2008 +0200
@@ -187,7 +187,11 @@
     if (p < dend) scdata->statusitems = (unsigned) strtol(p, &p, 10);
     if (p < dend) scdata->index_mtime = strtol(p, &p, 10);
     if (p < dend) scdata->index_ino = strtoul(p, &p, 10);
+#if SIZEOF_OFF_T == 8
+    if (p < dend) scdata->index_size = strtoull(p, &p, 10);
+#else
     if (p < dend) scdata->index_size = strtoul(p, &p, 10);
+#endif
     if (p < dend) scdata->messages = strtoul(p, &p, 10);
     if (p < dend) scdata->recent = (unsigned) strtoul(p, &p, 10);
     if (p < dend) scdata->uidnext = strtoul(p, &p, 10);
@@ -233,7 +237,11 @@
     char *key = statuscache_buildkey(mboxname, userid, &keylen);
 
     datalen = snprintf(data, sizeof(data),
+#if SIZEOF_OFF_T == 8
+		       "%u %u %ld %lu %llu %lu %u %lu %lu %u " MODSEQ_FMT,
+#else
 		       "%u %u %ld %lu %lu %lu %u %lu %lu %u " MODSEQ_FMT,
+#endif
 		       STATUSCACHE_VERSION, scdata->statusitems,
 		       scdata->index_mtime, scdata->index_ino,
 		       scdata->index_size, scdata->messages,

From dpc22 at cam.ac.uk  Thu Sep 25 16:47:04 2008
From: dpc22 at cam.ac.uk (David Carter)
Date: Thu, 25 Sep 2008 21:47:04 +0100 (BST)
Subject: Things for 2.3.13
In-Reply-To: <19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
	<F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
	<48D9A20E.8080509@andrew.cmu.edu>
	<20080924024310.GA22349@brong.net>
	<19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu>
Message-ID: <alpine.LSU.2.00.0809252142560.17135@hermes-1.csi.cam.ac.uk>

On Thu, 25 Sep 2008, Wesley Craig wrote:

> On 23 Sep 2008, at 22:43, Bron Gondwana wrote:
>> That's my entire wishlist for 2.3.13 (plus the buffer size patch you
>> already included)
>
> I think there should be an RC2.

I suggest that the final release candidate(s) should be advertised on the 
info-cyrus general list as well. That might pick some of the build 
problems that otherwise tend to crawl out of the woodwork after release.

-- 
David Carter                             Email: David.Carter at ucs.cam.ac.uk
University Computing Service,            Phone: (01223) 334502
New Museums Site, Pembroke Street,       Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

From brong at fastmail.fm  Thu Sep 25 18:21:27 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 26 Sep 2008 08:21:27 +1000
Subject: Things for 2.3.13
In-Reply-To: <48DBAFD7.3060007@andrew.cmu.edu>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
	<F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
	<48D9A20E.8080509@andrew.cmu.edu> <20080924024310.GA22349@brong.net>
	<19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu>
	<48DBAFD7.3060007@andrew.cmu.edu>
Message-ID: <1222381287.28663.1276016887@webmail.messagingengine.com>


On Thu, 25 Sep 2008 11:35:51 -0400, "Ken Murchison" <murch at andrew.cmu.edu> said:
> Wesley Craig wrote:
> > On 23 Sep 2008, at 22:43, Bron Gondwana wrote:
> >> That's my entire wishlist for 2.3.13 (plus the buffer size patch you
> >> already included)
> > 
> > I think there should be an RC2.
> > 
> > I don't see:
> > 
> >     https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3075
> > 
> > included in RC1.  I gathered that *was* on your wishlist for 2.3.13.
> 
> Isn't this the fix that Bron said needs more discussion?  Or am I 
> confusing it with something else.

As Wes said, this is the one that needs to go in!  I haven't put the
nested transaction stuff in Bugzilla yet, though it's in our running
builds now.  The biggest danger is that it allows unlimited depth,
which means something that didn't commit or abort correctly ONCE would
cause ALL later actions on that database to just be a nested item, and
it would never actually commit anything.  Subtle corruption bugs could
occur.

And I really want to put a "check no outstanding transactions" in the
db close function :)

> > . which looks for cases of MODSEQ being inappropriately set to 0 and 
> > correcting it to be 1.  Several buggy versions were released into the 
> > wild, so it's quite likely that there are MODSEQ 0 mailboxes out there.  
> > Xfer-ing them to a machine with this code in place will correct the 
> > corruption.  Xfer-ing them to a machine without this code in place 
> > causes sync_client to die.

Interesting..  I thought I pushed a fix for this into 2.3.7 already, but
maybe that was only to stop imapd crashing.

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm


From wes at umich.edu  Fri Sep 26 18:12:20 2008
From: wes at umich.edu (Wesley Craig)
Date: Fri, 26 Sep 2008 18:12:20 -0400
Subject: Things for 2.3.13
In-Reply-To: <1222381287.28663.1276016887@webmail.messagingengine.com>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu>
	<20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu>
	<48D98F29.6050505@andrew.cmu.edu>
	<F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu>
	<48D9A20E.8080509@andrew.cmu.edu>
	<20080924024310.GA22349@brong.net>
	<19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu>
	<48DBAFD7.3060007@andrew.cmu.edu>
	<1222381287.28663.1276016887@webmail.messagingengine.com>
Message-ID: <FC1C356C-D735-43BA-805D-D6ECBA276D28@umich.edu>

On 25 Sep 2008, at 18:21, Bron Gondwana wrote:
>>> . which looks for cases of MODSEQ being inappropriately set to 0 and
>>> correcting it to be 1.  Several buggy versions were released into  
>>> the
>>> wild, so it's quite likely that there are MODSEQ 0 mailboxes out  
>>> there.
>>> Xfer-ing them to a machine with this code in place will correct the
>>> corruption.  Xfer-ing them to a machine without this code in place
>>> causes sync_client to die.
>
> Interesting..  I thought I pushed a fix for this into 2.3.7  
> already, but
> maybe that was only to stop imapd crashing.

As I'm sure you recall, there were *many* fixes to the MODSEQ code.   
I'm not sure which versions had the bug which wrote MODSEQ 0, just  
that I've got a lot of mailboxes that have it.  A small percentage of  
around 1M mailboxes.  xfer-ing from a heavily patched 2.3.8 to  
2.3.12p2 showed the problem -- replication doesn't permit MODSEQ to  
be 0.  This fix just checks for MODSEQ 0 during index upgrade, since  
2.3.12 is an upgrade.  A subtlety is that all mailboxes get upgraded  
during xfer due to the change that sets mtime to INTERNAL during xfer.

I'll put this in bugzilla, anyway.  I've already added:

	https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2914
	https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3085
	https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3086
	https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3066

Rob suggests an imapd.conf option to control the new sieve utf-7  
behavior discussed here:

	http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-September/ 
029778.html

Someone working on that?  If there a bugzilla for it?

:wes

From robm at fastmail.fm  Fri Sep 26 23:35:25 2008
From: robm at fastmail.fm (Rob Mueller)
Date: Sat, 27 Sep 2008 13:35:25 +1000
Subject: Things for 2.3.13
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu><20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu><48D98F29.6050505@andrew.cmu.edu><F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu><48D9A20E.8080509@andrew.cmu.edu><20080924024310.GA22349@brong.net><19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu><48DBAFD7.3060007@andrew.cmu.edu><1222381287.28663.1276016887@webmail.messagingengine.com>
	<FC1C356C-D735-43BA-805D-D6ECBA276D28@umich.edu>
Message-ID: <AD374C8EA2744F0BA20490FE5C0AE835@robmhp>


> Rob suggests an imapd.conf option to control the new sieve utf-7
> behavior discussed here:
>
> http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-September/ 
> 029778.html
>
> Someone working on that?  If there a bugzilla for it?

Not that I know of for both of those. I was basically hoping the horde 
people (who seem most interested in it) would do it.

Rob


From wes at umich.edu  Mon Sep 29 11:09:05 2008
From: wes at umich.edu (Wesley Craig)
Date: Mon, 29 Sep 2008 11:09:05 -0400
Subject: Things for 2.3.13
In-Reply-To: <AD374C8EA2744F0BA20490FE5C0AE835@robmhp>
References: <200809231617.m8NGH92S005462@unix36.andrew.cmu.edu><20080923191515.lhzxulinz4kkos8w@cubmail.cc.columbia.edu><48D98F29.6050505@andrew.cmu.edu><F6E3B321-2733-43DD-A0B3-21E6D93AA739@umich.edu><48D9A20E.8080509@andrew.cmu.edu><20080924024310.GA22349@brong.net><19978FE1-771F-4F18-8A1F-B5AA3C4EB3FA@umich.edu><48DBAFD7.3060007@andrew.cmu.edu><1222381287.28663.1276016887@webmail.messagingengine.com>
	<FC1C356C-D735-43BA-805D-D6ECBA276D28@umich.edu>
	<AD374C8EA2744F0BA20490FE5C0AE835@robmhp>
Message-ID: <37A0266E-15D7-4CB4-856F-5DF6EE23C77D@umich.edu>

On 26 Sep 2008, at 23:35, Rob Mueller wrote:
>> Rob suggests an imapd.conf option to control the new sieve utf-7
>> behavior discussed here:
>>
>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-September/ 
>> 029778.html
>>
>> Someone working on that?  Is there a bugzilla for it?
>
> Not that I know of for both of those. I was basically hoping the  
> horde people (who seem most interested in it) would do it.

While I would support including such an option for 2.3.13, I'm not so  
interested that I'd write it.  If it's going to be added, someone  
needs to do it sooner rather than later.  Even better would be a  
tested upgrade path for existing sieve scripts.  And obviously  
something in the release notes about the change...

:wes

From brong at fastmail.fm  Tue Sep 30 07:59:37 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 30 Sep 2008 21:59:37 +1000
Subject: Cyrus 2.3.8 imapd process periodically sticks at 100% CPU
In-Reply-To: <48E0BDE8.7000505@siriusit.co.uk>
References: <48E0BDE8.7000505@siriusit.co.uk>
Message-ID: <20080930115937.GA16859@brong.net>

On Mon, Sep 29, 2008 at 12:37:12PM +0100, Mark Cave-Ayland wrote:
> What happens is that periodically (maybe around once a month?) we have 
> one particular user who contacts us complaining that they are unable 
> access their mailbox. Generally we always find the same thing: there is 
> an imapd process accessing his seen DB which is running at 100% CPU. 
> Once this process is killed then things go back to normal and the user 
> can log in.

The following patch fixes your issue and has the syslog line added as
well.

https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3088

Ken, Wes - I think this is a candidate for 2.3.13 since we haven't
released yet.  It's trivial, and infinite loops are bad!

Regards,

Bron ( also attaching the patch with this email )
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrus-inifiniteloop-2.3.12.diff
Type: text/x-diff
Size: 1380 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080930/b71c7630/attachment.bin 

From brong at fastmail.fm  Tue Sep 30 08:43:26 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 30 Sep 2008 22:43:26 +1000
Subject: Things for 2.3.13
In-Reply-To: <37A0266E-15D7-4CB4-856F-5DF6EE23C77D@umich.edu>
References: <FC1C356C-D735-43BA-805D-D6ECBA276D28@umich.edu>
	<AD374C8EA2744F0BA20490FE5C0AE835@robmhp>
	<37A0266E-15D7-4CB4-856F-5DF6EE23C77D@umich.edu>
Message-ID: <20080930124326.GA17332@brong.net>

On Mon, Sep 29, 2008 at 11:09:05AM -0400, Wesley Craig wrote:
> On 26 Sep 2008, at 23:35, Rob Mueller wrote:
>>> Rob suggests an imapd.conf option to control the new sieve utf-7
>>> behavior discussed here:
>>>
>>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-September/ 
>>> 029778.html
>>>
>>> Someone working on that?  Is there a bugzilla for it?
>>
>> Not that I know of for both of those. I was basically hoping the horde 
>> people (who seem most interested in it) would do it.
>
> While I would support including such an option for 2.3.13, I'm not so  
> interested that I'd write it.  If it's going to be added, someone needs 
> to do it sooner rather than later.  Even better would be a tested upgrade 
> path for existing sieve scripts.  And obviously something in the release 
> notes about the change...

Bunch of whingers the lot of you.

Here you go...

It was pretty trivial - most of the time was finding the original
diff to see how it got enabled:

https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/sieve/sieve.y.diff?r1=1.36;r2=1.37

Enjoy,

Bron.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrus-utf7-foldernames-2.3.12.diff
Type: text/x-diff
Size: 1801 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20080930/a3c42641/attachment.bin 

From brong at fastmail.fm  Tue Sep 30 08:48:51 2008
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 30 Sep 2008 22:48:51 +1000
Subject: Death by a thousand config items
Message-ID: <20080930124851.GB17332@brong.net>

I admit, I'm partially at fault here.  I just posted 
a patch creating yet another config switch, with the
usual tradeoff:

option (a) is better technically, whatever
option (b) is backwards compatible

I try, whenever doing this, to make (b) the default
so it doesn't break a zillion shmucks when they or
their distro upgrade to a shiny new stable Cyrus.

Unfortunately, over time, this means that the best
options for a _new_ cyrus installation include an
ever longer list of required options, just to avoid
the backwards compatibility cruft.

Oh the humanity.

I don't think there's anything we can do about this
right now, but I would like to plant the seed of the
idea in people's minds that we should flip the
defaults on a whole bunch of these options when 2.4
is ready to go stable.

This will have to be a very widely documented fact, 
and we'll advise packagers of the software to ship 
default configs that are backwards compatible if 
they have to - or if their format supports it,
automatically add the correct lines to the config -
but things like a saner sort order for mailboxes.db 
do have to switch over some time!

There's my random thought for the evening.

Bron.

From murch at andrew.cmu.edu  Tue Sep 30 09:36:47 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Tue, 30 Sep 2008 09:36:47 -0400
Subject: Things for 2.3.13
In-Reply-To: <20080930124326.GA17332@brong.net>
References: <FC1C356C-D735-43BA-805D-D6ECBA276D28@umich.edu>	<AD374C8EA2744F0BA20490FE5C0AE835@robmhp>	<37A0266E-15D7-4CB4-856F-5DF6EE23C77D@umich.edu>
	<20080930124326.GA17332@brong.net>
Message-ID: <48E22B6F.5050309@andrew.cmu.edu>

Given your follow-up email about too many config options, should we 
combine this with the existing rfc3028_strict option?


Bron Gondwana wrote:
> On Mon, Sep 29, 2008 at 11:09:05AM -0400, Wesley Craig wrote:
>> On 26 Sep 2008, at 23:35, Rob Mueller wrote:
>>>> Rob suggests an imapd.conf option to control the new sieve utf-7
>>>> behavior discussed here:
>>>>
>>>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-September/ 
>>>> 029778.html
>>>>
>>>> Someone working on that?  Is there a bugzilla for it?
>>> Not that I know of for both of those. I was basically hoping the horde 
>>> people (who seem most interested in it) would do it.
>> While I would support including such an option for 2.3.13, I'm not so  
>> interested that I'd write it.  If it's going to be added, someone needs 
>> to do it sooner rather than later.  Even better would be a tested upgrade 
>> path for existing sieve scripts.  And obviously something in the release 
>> notes about the change...
> 
> Bunch of whingers the lot of you.
> 
> Here you go...
> 
> It was pretty trivial - most of the time was finding the original
> diff to see how it got enabled:
> 
> https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/sieve/sieve.y.diff?r1=1.36;r2=1.37
> 
> Enjoy,
> 
> Bron.
> 


-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

From wes at umich.edu  Tue Sep 30 13:46:42 2008
From: wes at umich.edu (Wesley Craig)
Date: Tue, 30 Sep 2008 13:46:42 -0400
Subject: Death by a thousand config items
In-Reply-To: <20080930124851.GB17332@brong.net>
References: <20080930124851.GB17332@brong.net>
Message-ID: <79ECF39E-966F-4CBE-84E9-31CBA6F71AF0@umich.edu>

I agree, and I think that would make a good reason to release  
something called "2.4" -- it will break a bunch of stuff.  With that  
in mind, I'm about to propose another pair of config options.  These  
will control whether sieve should be compatible with draft 8 or 9 in  
terms of response to AUTHENTICATE & STARTTLS.  This will go right  
along with the sieve utf-7/-8 change, I suppose.

:wes

On 30 Sep 2008, at 08:48, Bron Gondwana wrote:
> I don't think there's anything we can do about this
> right now, but I would like to plant the seed of the
> idea in people's minds that we should flip the
> defaults on a whole bunch of these options when 2.4
> is ready to go stable.
>
> This will have to be a very widely documented fact,
> and we'll advise packagers of the software to ship
> default configs that are backwards compatible if
> they have to - or if their format supports it,
> automatically add the correct lines to the config -
> but things like a saner sort order for mailboxes.db
> do have to switch over some time!

From wes at umich.edu  Tue Sep 30 14:27:42 2008
From: wes at umich.edu (Wesley Craig)
Date: Tue, 30 Sep 2008 14:27:42 -0400
Subject: Things for 2.3.13
In-Reply-To: <48E22B6F.5050309@andrew.cmu.edu>
References: <FC1C356C-D735-43BA-805D-D6ECBA276D28@umich.edu>	<AD374C8EA2744F0BA20490FE5C0AE835@robmhp>	<37A0266E-15D7-4CB4-856F-5DF6EE23C77D@umich.edu>
	<20080930124326.GA17332@brong.net>
	<48E22B6F.5050309@andrew.cmu.edu>
Message-ID: <4D5D208C-F83D-4142-841F-0B54381B9AEA@umich.edu>

On 30 Sep 2008, at 09:36, Ken Murchison wrote:
> Given your follow-up email about too many config options, should we  
> combine this with the existing rfc3028_strict option?

I don't think the point of that comment was to say that there are  
"too many" per se.  Just that "technically better" does not trump  
"breaks clients".  I desire to move in the direction of "technically  
better", while being allowed to manage the transition.  You need the  
options.

Personally, I don't think it's a great idea to combine this option  
with rfc3028_strict.  I suspect most people leave rfc3028_strict on.   
Forcing them to relax address checking to get the legacy utf-7/-8  
behavior seems like the wrong approach.  But maybe it doesn't  
matter.  Kind of depends on what the clients do.  I notice FM has a  
nice chart of sieve clients.

And speaking of what sieve clients do, it seems like some expect  
managesieve to respond with a capability list after AUTHENTICATION,  
while some don't.  I gather the Internet-Draft for managesieve  
changed recently.  The Cyrus code implementing this seems to change  
between 2.3.12p1 and 2.3.12p2.  This breaks some significant clients,  
in addition to making it very challenging to upgrade a murder  
environment incrementally.  timsieved proxying and sivtest now expect  
servers to return the capability after authentication.  Since old  
servers don't, they just hang in a read.

I'm thinking of two new options: one which controls whether timsieved  
and sivtest expect draft-martin-managesieve-08 or draft-martin- 
managesieve-09 AUTHENTICATE behavior; and a second one which controls  
whether timsieved sends capability after AUTHENTICATE a la draft- 
martin-managesieve-09 or doesn't a la draft-martin-managesieve-08:

	sieve_expect_draft-martin-managesieve-08_authenticate:
	sieve_do_draft-martin-managesieve-08_authenticate:

Comments?

:wes

From murch at andrew.cmu.edu  Tue Sep 30 16:33:02 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Tue, 30 Sep 2008 16:33:02 -0400
Subject: More special chars in mailbox names
Message-ID: <48E28CFE.2020104@andrew.cmu.edu>

While looking over bug #3002, 
https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3002

I started wondering what other special characters we might want to 
allow.  The original authors were more restrictive with GOODCHARS than 
the RFC.  AFAICT the following characters are permitted by RFC 3501 but 
aren't included in GOODCHARS

! # $ & ' ; < > ? [ ^ ` } |


! and ^ have special meaning to Cyrus, so we may not want to add these 
to GOODCHARS.  & and ? might conflict with modified UTF-7 encoding.

Looking back at the list archives, there have been several requests to 
allow ' (single quote) and I'm included to do so.

Does anybody want to make a case for adding more?

Has anybody already added some of these in production without any 
adverse effects?

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

From wes at umich.edu  Tue Sep 30 17:06:01 2008
From: wes at umich.edu (Wesley Craig)
Date: Tue, 30 Sep 2008 17:06:01 -0400
Subject: More special chars in mailbox names
In-Reply-To: <48E28CFE.2020104@andrew.cmu.edu>
References: <48E28CFE.2020104@andrew.cmu.edu>
Message-ID: <28911D1A-E1F8-434A-8DCC-1EB68A9F4BF7@umich.edu>

On 30 Sep 2008, at 16:33, Ken Murchison wrote:
> While looking over bug #3002, https://bugzilla.andrew.cmu.edu/ 
> show_bug.cgi?id=3002
>
> I started wondering what other special characters we might want to  
> allow.  The original authors were more restrictive with GOODCHARS  
> than the RFC.  AFAICT the following characters are permitted by RFC  
> 3501 but aren't included in GOODCHARS
>
> ! # $ & ' ; < > ? [ ^ ` } |
>
>
> ! and ^ have special meaning to Cyrus, so we may not want to add  
> these to GOODCHARS.  & and ? might conflict with modified UTF-7  
> encoding.
>
> Looking back at the list archives, there have been several requests  
> to allow ' (single quote) and I'm included to do so.
>
> Does anybody want to make a case for adding more?
>
> Has anybody already added some of these in production without any  
> adverse effects?

UMich has added this list:

	#$%'()*;<>?[]^`{|}

for years without any reported issues.  These were added primarily to  
deal with transition from UWIMAP to Cyrus, I'm not making a case for  
any of them, personally.

:wes

From nock at email.arizona.edu  Tue Sep 30 17:14:09 2008
From: nock at email.arizona.edu (Shawn Nock)
Date: Tue, 30 Sep 2008 14:14:09 -0700
Subject: More special chars in mailbox names
In-Reply-To: <48E28CFE.2020104@andrew.cmu.edu>
References: <48E28CFE.2020104@andrew.cmu.edu>
Message-ID: <48E296A1.2000206@email.arizona.edu>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ken Murchison wrote:
|
| Has anybody already added some of these in production without any
| adverse effects?
|

We have been running the following for about a year on ~2.3.8 and a few
years before that on 2.2.x:

#define GOODCHARS "
+,-.'()[]{}<>!%$;?*^#\\\"`|0123456789:=@ABCDEFGHIJKLMNOP
QRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~"

We don't have any problems with it.

Peace,
Shawn


- --
Shawn Nock (OpenPGP: 0xFF7D08A3)
Unix Systems Group; UITS
University of Arizona
nock at email.arizona.edu
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFI4pahUfD1hN5kJ6URAq4SAKC0/8cMpMxfB+kqokBKo64p98pvSQCfSZTP
RXppHe/e4FGLi7ABAH1PTQA=
=/C6I
-----END PGP SIGNATURE-----

From murch at andrew.cmu.edu  Tue Sep 30 20:00:03 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Tue, 30 Sep 2008 20:00:03 -0400
Subject: More special chars in mailbox names
In-Reply-To: <28911D1A-E1F8-434A-8DCC-1EB68A9F4BF7@umich.edu>
References: <48E28CFE.2020104@andrew.cmu.edu>
	<28911D1A-E1F8-434A-8DCC-1EB68A9F4BF7@umich.edu>
Message-ID: <48E2BD83.90105@andrew.cmu.edu>

Wesley Craig wrote:
> On 30 Sep 2008, at 16:33, Ken Murchison wrote:
>> While looking over bug #3002, 
>> https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3002
>>
>> I started wondering what other special characters we might want to 
>> allow.  The original authors were more restrictive with GOODCHARS than 
>> the RFC.  AFAICT the following characters are permitted by RFC 3501 
>> but aren't included in GOODCHARS
>>
>> ! # $ & ' ; < > ? [ ^ ` } |
>>
>>
>> ! and ^ have special meaning to Cyrus, so we may not want to add these 
>> to GOODCHARS.  & and ? might conflict with modified UTF-7 encoding.
>>
>> Looking back at the list archives, there have been several requests to 
>> allow ' (single quote) and I'm included to do so.
>>
>> Does anybody want to make a case for adding more?
>>
>> Has anybody already added some of these in production without any 
>> adverse effects?
> 
> UMich has added this list:
> 
>     #$%'()*;<>?[]^`{|}
> 
> for years without any reported issues.  These were added primarily to 
> deal with transition from UWIMAP to Cyrus, I'm not making a case for any 
> of them, personally.


Unless I'm misreading the RFC 3501 grammar, the following are prohibited:

( ) { % * " \ ]

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

From wes at umich.edu  Tue Sep 30 20:57:51 2008
From: wes at umich.edu (Wesley Craig)
Date: Tue, 30 Sep 2008 20:57:51 -0400
Subject: More special chars in mailbox names
In-Reply-To: <48E2BD83.90105@andrew.cmu.edu>
References: <48E28CFE.2020104@andrew.cmu.edu>
	<28911D1A-E1F8-434A-8DCC-1EB68A9F4BF7@umich.edu>
	<48E2BD83.90105@andrew.cmu.edu>
Message-ID: <197F7E53-F3D3-4F12-A5C5-ACBC61C9D9C9@umich.edu>

On 30 Sep 2008, at 20:00, Ken Murchison wrote:
> Wesley Craig wrote:
>> UMich has added this list:
>>     #$%'()*;<>?[]^`{|}
>> for years without any reported issues.  These were added primarily  
>> to deal with transition from UWIMAP to Cyrus, I'm not making a  
>> case for any of them, personally.
>
> Unless I'm misreading the RFC 3501 grammar, the following are  
> prohibited:
>
> ( ) { % * " \ ]

Not that I'm not advocating a change similar to UMich's (I'm totally  
not).

What aspect of RFC3501 do you feel prohibits ( ) { % * " \ ] ?   
Anything that takes a mailbox can accept:

	mailbox         = "INBOX" / astring
	astring         = 1*ASTRING-CHAR / string
	string          = quoted / literal
	literal         = "{" number "}" CRLF *CHAR8
	CHAR8           = %x01-ff

I think UM's 1,000,000 rather yucky mailboxes offer some insight into  
how much latitude the protocol allows.  I'm pretty sure cyrus  
internals are more limiting that the protocol itself, e.g., I think  
the protocol would allow TAB in mailbox names.

:wes

From murch at andrew.cmu.edu  Tue Sep 30 21:23:07 2008
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Tue, 30 Sep 2008 21:23:07 -0400
Subject: More special chars in mailbox names
In-Reply-To: <197F7E53-F3D3-4F12-A5C5-ACBC61C9D9C9@umich.edu>
References: <48E28CFE.2020104@andrew.cmu.edu>
	<28911D1A-E1F8-434A-8DCC-1EB68A9F4BF7@umich.edu>
	<48E2BD83.90105@andrew.cmu.edu>
	<197F7E53-F3D3-4F12-A5C5-ACBC61C9D9C9@umich.edu>
Message-ID: <48E2D0FB.8040406@andrew.cmu.edu>

Wesley Craig wrote:
> On 30 Sep 2008, at 20:00, Ken Murchison wrote:
>> Wesley Craig wrote:
>>> UMich has added this list:
>>>     #$%'()*;<>?[]^`{|}
>>> for years without any reported issues.  These were added primarily to 
>>> deal with transition from UWIMAP to Cyrus, I'm not making a case for 
>>> any of them, personally.
>>
>> Unless I'm misreading the RFC 3501 grammar, the following are prohibited:
>>
>> ( ) { % * " \ ]
> 
> Not that I'm not advocating a change similar to UMich's (I'm totally not).
> 
> What aspect of RFC3501 do you feel prohibits ( ) { % * " \ ] ?  Anything 
> that takes a mailbox can accept:
> 
>     mailbox         = "INBOX" / astring
>     astring         = 1*ASTRING-CHAR / string
>     string          = quoted / literal
>     literal         = "{" number "}" CRLF *CHAR8
>     CHAR8           = %x01-ff

ASTRING-CHAR   = ATOM-CHAR / resp-specials

atom            = 1*ATOM-CHAR

ATOM-CHAR       = <any CHAR except atom-specials>

atom-specials   = "(" / ")" / "{" / SP / CTL / list-wildcards /
                   quoted-specials / resp-specials

list-wildcards  = "%" / "*"

quoted-specials = DQUOTE / "\"

resp-specials   = "]"


Maybe the Cyrus check is done AFTER parsing of a quoted string or 
literal, which puts just about everything on on the table.  Using 
list-wildcards would always be a bad idea IMHO.


-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

From wes at umich.edu  Tue Sep 30 22:15:19 2008
From: wes at umich.edu (Wesley Craig)
Date: Tue, 30 Sep 2008 22:15:19 -0400
Subject: More special chars in mailbox names
In-Reply-To: <48E2D0FB.8040406@andrew.cmu.edu>
References: <48E28CFE.2020104@andrew.cmu.edu>
	<28911D1A-E1F8-434A-8DCC-1EB68A9F4BF7@umich.edu>
	<48E2BD83.90105@andrew.cmu.edu>
	<197F7E53-F3D3-4F12-A5C5-ACBC61C9D9C9@umich.edu>
	<48E2D0FB.8040406@andrew.cmu.edu>
Message-ID: <DAA13B5F-6D37-47FF-A767-7E4C0AE937C9@umich.edu>

On 30 Sep 2008, at 21:23, Ken Murchison wrote:
> Using list-wildcards would always be a bad idea IMHO.

UMich has a lot of mailboxes with * in their names.  It's not a  
problem, tho you're probably right that it's a bad idea.  I guess  
IMAP implementations usually use literals to communicate.  Which is  
just what Cyrus does internally.  The rest of that optional syntax is  
gilding.

:wes