From carlos.horowicz at gmail.com  Mon Feb  9 17:53:54 2009
From: carlos.horowicz at gmail.com (Carlos Horowicz)
Date: Mon, 9 Feb 2009 20:53:54 -0200
Subject: Fwd: Huge header detection
In-Reply-To: <20090209110236.GA18585@brong.net>
References: <962968300902051655o3c591d5bt34d3feeba77bfdb7@mail.gmail.com>
	<20090206020224.3C9101F3EA9@spike.porcupine.org>
	<962968300902060819i3e96f355ud8eb75753863cf83@mail.gmail.com>
	<962968300902061034r5944d589p5e7954cb4450e955@mail.gmail.com>
	<20090206230220.GA31187@brong.net>
	<962968300902062036k26a695d2ka6ff142e12088c88@mail.gmail.com>
	<E6BE05427989E0B176BC5066@lewes.staff.uscs.susx.ac.uk>
	<20090209110236.GA18585@brong.net>
Message-ID: <962968300902091453h3e5f6a11jde4b90d3442f6659@mail.gmail.com>

On Mon, Feb 9, 2009 at 9:02 AM, Bron Gondwana <brong at fastmail.fm> wrote:
> On Mon, Feb 09, 2009 at 10:24:38AM +0000, Ian Eiloart wrote:
>>
>>
>> --On 7 February 2009 02:36:36 -0200 Carlos Horowicz
>> <carlos.horowicz at gmail.com> wrote:
>>
>>> I'm wondering what to do in a live system with may be hundreds of
>>> thousands of these strange e-mails already in users? mailboxes,
>
> My god, that many?  Nuke it from orbit.  Only way to be sure.
>
> (alternative plan, grep for the bogus files, unlink them and
> reconstruct the mailboxes - cheaper and less radioactive)

I have already done that (find ... -exec grep ...) but it took days to
scan entire volumes, and in this case I already knew which pattern to
look for.

>
>>> Should imapd be patched so that it just ignores the repetitions , both
>>> when building cyrus.cache and when it returns the headers to a client
>>> ? or should imapd really modify the original e-mail by stripping
>>> unnecessary/illegal headers and store a cleaned-up version ?
>>
>> It shouldn't be modifying messages. It should handle such messages
>> elegantly. Ignoring repetitions (beyond a threshold of repeats) seems the
>> most sensible option. However, failing to report them to a client could
>> cause confusion, so a threshold should be reasonably high. Of course some
>> headers are supposed to have multiple instances...
>
> Ditto with that.  This patch ignores repetitions beyond a threshold.  It
> turns out that ignoring specific headers separately is hard[tm], but
> just stopping parsing them after a count is way-easy.
>
>> Alerting the system administrator to the existence of such bogus messages
>> seems like a good idea, too. Perhaps through the logging system.
>
> Yeah, good point.  Allow me to add that.
>
>> If you don't want a particular message in the system, then it should not
>> be accepted by LMTP or by any IMAP message creation mechanism.
>
> Ditto.  Gosh.  That makes 3 tunables.  The gods of tunable
> non-proliferation will want my head for doing this:
>
> maxcacheheaders_warn = 500
> maxcacheheaders_skip = 1000 (same as the current patch)
> maxcacheheaders_reject = 2000
>
> Sound like reasonable defaults?  I'm tempted to make the _reject be '0'
> (don't reject) by default.


Is this patch intended to be also part of reconstruct ? 'cause this
would catch many other situations, like restores, spool rebuilds, and
even scanning imap spools.

>
> Bron.
>

Carlos

From powo at powo.priv.at  Tue Feb 10 02:40:29 2009
From: powo at powo.priv.at (Wolfgang Powisch (privat))
Date: Tue, 10 Feb 2009 08:40:29 +0100
Subject: 2.3.13 - fails to unlinking quota file
In-Reply-To: <498FF367.4010609@noc.uoa.gr>
References: <498D70B2.7080903@powo.priv.at> <20090208012007.GA20193@brong.net>
	<498FF367.4010609@noc.uoa.gr>
Message-ID: <49912F6D.3020407@powo.priv.at>

Am 09.02.2009 10:12, Aristotelis schrieb:
> Cyrus imap doesn't remove the quota from a folder when this folder is 
> deleted except if the folder is user.X folder.

This is the expected behaviour of cyrus-imapd without any patches ... correct?

If this would work it would be enough in the majority of all cases,
unfortunately it doesn't. Beside the fact that user.x Quota-Files remain
on disk after deleting a mailbox (which is rather no problem) I'm afraid
that something wrong is deleted, because the unlink is called with
a wrong (random?) quota-file-path.

In my case, the quota-root it tried to unlink was "u", so maybe it's
truncated somewhere to just the first letter of correct quota-root


BTW: thx for the hint of quota-patches

regards, Wolfgang



From carlos.horowicz at gmail.com  Tue Feb 10 07:50:11 2009
From: carlos.horowicz at gmail.com (Carlos Horowicz)
Date: Tue, 10 Feb 2009 10:50:11 -0200
Subject: Fwd: Huge header detection
In-Reply-To: <962968300902091453h3e5f6a11jde4b90d3442f6659@mail.gmail.com>
References: <962968300902051655o3c591d5bt34d3feeba77bfdb7@mail.gmail.com>
	<20090206020224.3C9101F3EA9@spike.porcupine.org>
	<962968300902060819i3e96f355ud8eb75753863cf83@mail.gmail.com>
	<962968300902061034r5944d589p5e7954cb4450e955@mail.gmail.com>
	<20090206230220.GA31187@brong.net>
	<962968300902062036k26a695d2ka6ff142e12088c88@mail.gmail.com>
	<E6BE05427989E0B176BC5066@lewes.staff.uscs.susx.ac.uk>
	<20090209110236.GA18585@brong.net>
	<962968300902091453h3e5f6a11jde4b90d3442f6659@mail.gmail.com>
Message-ID: <962968300902100450x3eac7f87vc6f7cc27f0ab8d32@mail.gmail.com>

On Mon, Feb 9, 2009 at 8:53 PM, Carlos Horowicz
<carlos.horowicz at gmail.com> wrote:
> On Mon, Feb 9, 2009 at 9:02 AM, Bron Gondwana <brong at fastmail.fm> wrote:
>> On Mon, Feb 09, 2009 at 10:24:38AM +0000, Ian Eiloart wrote:
>>>
>>>
>>> --On 7 February 2009 02:36:36 -0200 Carlos Horowicz
>>> <carlos.horowicz at gmail.com> wrote:
>>>
>>>> I'm wondering what to do in a live system with may be hundreds of
>>>> thousands of these strange e-mails already in users? mailboxes,
>>
>> My god, that many?  Nuke it from orbit.  Only way to be sure.
>>
>> (alternative plan, grep for the bogus files, unlink them and
>> reconstruct the mailboxes - cheaper and less radioactive)
>
> I have already done that (find ... -exec grep ...) but it took days to
> scan entire volumes, and in this case I already knew which pattern to
> look for.
>
>>
>>>> Should imapd be patched so that it just ignores the repetitions , both
>>>> when building cyrus.cache and when it returns the headers to a client
>>>> ? or should imapd really modify the original e-mail by stripping
>>>> unnecessary/illegal headers and store a cleaned-up version ?
>>>
>>> It shouldn't be modifying messages. It should handle such messages
>>> elegantly. Ignoring repetitions (beyond a threshold of repeats) seems the
>>> most sensible option. However, failing to report them to a client could
>>> cause confusion, so a threshold should be reasonably high. Of course some
>>> headers are supposed to have multiple instances...
>>
>> Ditto with that.  This patch ignores repetitions beyond a threshold.  It
>> turns out that ignoring specific headers separately is hard[tm], but
>> just stopping parsing them after a count is way-easy.
>>
>>> Alerting the system administrator to the existence of such bogus messages
>>> seems like a good idea, too. Perhaps through the logging system.
>>
>> Yeah, good point.  Allow me to add that.
>>
>>> If you don't want a particular message in the system, then it should not
>>> be accepted by LMTP or by any IMAP message creation mechanism.
>>
>> Ditto.  Gosh.  That makes 3 tunables.  The gods of tunable
>> non-proliferation will want my head for doing this:
>>
>> maxcacheheaders_warn = 500
>> maxcacheheaders_skip = 1000 (same as the current patch)
>> maxcacheheaders_reject = 2000
>>
>> Sound like reasonable defaults?  I'm tempted to make the _reject be '0'
>> (don't reject) by default.
>
>
> Is this patch intended to be also part of reconstruct ? 'cause this
> would catch many other situations, like restores, spool rebuilds, and
> even scanning imap spools.

I was thinking of reconstruct just for the sake of warning the admin.

for instanace in
syslog(LOG_ERR, "ERROR: message has %d header lines, not caching any
more", max_skip);

how can the message be identified so the admin can cat upon ?





>
>>
>> Bron.
>>
>
> Carlos
>

From whitverj at urbandale.k12.ia.us  Tue Feb 10 16:59:09 2009
From: whitverj at urbandale.k12.ia.us (Josh Whitver)
Date: Tue, 10 Feb 2009 15:59:09 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even after
	reconstructs
Message-ID: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>

Hello all,

I've got one user on my mail system that Cyrus seems to have it in  
for.  IMAP connections stall out, mail messages to her get stuck in  
the Postfix queue indefinitely, and things just generally aren't  
working for her.  Server is an Intel Xserve running Mac OS X 10.5.5  
Server, clients are 10.5.4 using Mail.app (but issues appear using any  
IMAP client, such as our RoundCube webmail system).  All other users  
(more than 500 of them) are fine.

Whenever an IMAP connection is attempted, I see this in the log:
Feb 10 09:14:17 urbandale imap[34508]: DBERROR: error fetching user. 
[username]: cyrusdb error

We've seen this kind of issue before, and usually doing the following  
clears it up:
sudo /usr/bin/cyrus/bin/reconstruct -r -f user/[username]

Under Tiger Server, you had to be sure to use sudo -u cyrusimap so  
that the cyrus files would have the correct permissions, but under  
Leopard, running it as root seems to do the job (the cyrus files are  
owned by _cyrus afterwards, like they should be).

At any rate, that didn't solve her issues, so I ended up deleting the  
cyrus.* files in all her mail boxes and her Inbox and running it  
again, creating fresh copies.  Still no dice.  So then I tried:
sudo /usr/bin/cyrus/bin/reconstruct -x -r user/[username]

Which again created fresh files, but the issues still persist.  I  
backed up the user's mailboxes and went into cyradm and deleted them  
altogether, and re-created them.  I didn't move any of the mail  
messages back because I wanted to see if it would work with a blank  
slate.  Same issue.

This seems to imply that something's borked in the main Cyrus  
database, if I'm not mistaken.  The question then becomes what to do  
about that.  Suggestions, anyone?  I'm rapidly running out of ideas -  
any help you can offer would be appreciated.  Thanks!
--
Josh Whitver
whitverj at urbandale.k12.ia.us / josh at whitver.net
Don't anthropomorphize computers.  They hate that.



From brong at fastmail.fm  Tue Feb 10 18:08:34 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Wed, 11 Feb 2009 10:08:34 +1100
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
Message-ID: <20090210230834.GA28780@brong.net>

On Tue, Feb 10, 2009 at 03:59:09PM -0600, Josh Whitver wrote:
> Hello all,
> 
> I've got one user on my mail system that Cyrus seems to have it in  
> for.  IMAP connections stall out, mail messages to her get stuck in  
> the Postfix queue indefinitely, and things just generally aren't  
> working for her.  Server is an Intel Xserve running Mac OS X 10.5.5  
> Server, clients are 10.5.4 using Mail.app (but issues appear using any  
> IMAP client, such as our RoundCube webmail system).  All other users  
> (more than 500 of them) are fine.
> 
> Whenever an IMAP connection is attempted, I see this in the log:
> Feb 10 09:14:17 urbandale imap[34508]: DBERROR: error fetching user. 
> [username]: cyrusdb error
> 
> We've seen this kind of issue before, and usually doing the following  
> clears it up:
> sudo /usr/bin/cyrus/bin/reconstruct -r -f user/[username]
> 
> Under Tiger Server, you had to be sure to use sudo -u cyrusimap so  
> that the cyrus files would have the correct permissions, but under  
> Leopard, running it as root seems to do the job (the cyrus files are  
> owned by _cyrus afterwards, like they should be).
> 
> At any rate, that didn't solve her issues, so I ended up deleting the  
> cyrus.* files in all her mail boxes and her Inbox and running it  
> again, creating fresh copies.  Still no dice.  So then I tried:
> sudo /usr/bin/cyrus/bin/reconstruct -x -r user/[username]
> 
> Which again created fresh files, but the issues still persist.  I  
> backed up the user's mailboxes and went into cyradm and deleted them  
> altogether, and re-created them.  I didn't move any of the mail  
> messages back because I wanted to see if it would work with a blank  
> slate.  Same issue.
> 
> This seems to imply that something's borked in the main Cyrus  
> database, if I'm not mistaken.  The question then becomes what to do  
> about that.  Suggestions, anyone?  I'm rapidly running out of ideas -  
> any help you can offer would be appreciated.  Thanks!

Yeah, you have a broken mailboxes.db I'd say!

Here's what I would do:

# stop cyrus
cyr_dbtool /var/imap/mailboxes.db skiplist show > mailboxes.dump
cyr_dbtool -n /var/lib/mailboxes_new.db skiplist set < mailboxes.dump
mv /var/imap/mailboxes.db /var/imap/mailboxes_old.db
mv /var/lib/mailboxes_new.db /var/imap/mailboxes.db
# start cyrus

This will give you a brand new clean mailboxes.db (but keep the old one
just in case)

Bron.

From whitverj at urbandale.k12.ia.us  Tue Feb 10 18:20:42 2009
From: whitverj at urbandale.k12.ia.us (Josh Whitver)
Date: Tue, 10 Feb 2009 17:20:42 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <20090210230834.GA28780@brong.net>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
	<20090210230834.GA28780@brong.net>
Message-ID: <69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>

On Feb 10, 2009, at 5:08 PM, Bron Gondwana wrote:
>>
>
> Yeah, you have a broken mailboxes.db I'd say!
>
> Here's what I would do:
>
> # stop cyrus
> cyr_dbtool /var/imap/mailboxes.db skiplist show > mailboxes.dump
> cyr_dbtool -n /var/lib/mailboxes_new.db skiplist set < mailboxes.dump
> mv /var/imap/mailboxes.db /var/imap/mailboxes_old.db
> mv /var/lib/mailboxes_new.db /var/imap/mailboxes.db
> # start cyrus
>
> This will give you a brand new clean mailboxes.db (but keep the old  
> one
> just in case)
>
> Bron.
>

Thanks - is this one of those operations that will kill the read/ 
unread status of my messages, or should all that stuff be preserved?
--
Josh Whitver
whitverj at urbandale.k12.ia.us / josh at whitver.net
Give a man a fish, and you'll feed him for a day. Teach a man to fish,  
and he'll buy a funny hat. Talk to a hungry man about fish, and you're  
a consultant.

From brong at fastmail.fm  Tue Feb 10 19:11:16 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Wed, 11 Feb 2009 11:11:16 +1100
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
	<20090210230834.GA28780@brong.net>
	<69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>
Message-ID: <20090211001116.GA540@brong.net>

On Tue, Feb 10, 2009 at 05:20:42PM -0600, Josh Whitver wrote:
> On Feb 10, 2009, at 5:08 PM, Bron Gondwana wrote:
>>>
>>
>> Yeah, you have a broken mailboxes.db I'd say!
>>
>> Here's what I would do:
>>
>> # stop cyrus
>> cyr_dbtool /var/imap/mailboxes.db skiplist show > mailboxes.dump
>> cyr_dbtool -n /var/lib/mailboxes_new.db skiplist set < mailboxes.dump
>> mv /var/imap/mailboxes.db /var/imap/mailboxes_old.db
>> mv /var/lib/mailboxes_new.db /var/imap/mailboxes.db
>> # start cyrus
>>
>> This will give you a brand new clean mailboxes.db (but keep the old  
>> one
>> just in case)
>>
>> Bron.
>>
>
> Thanks - is this one of those operations that will kill the read/unread 
> status of my messages, or should all that stuff be preserved?

No, it won't kill anything (hopefully).  You may want to compare the
output of show on the new file with show on the old file.  They _should_
be identical.

If you want to email me (privately) a copy of your old mailboxes.db, I'd
be happy to have a look through it and check for corruption.  Obviously
it contains private details (usernames, folder names and ACLs - no
actual email) of your users, so it's up to you.

Bron ( who knows the skiplist format far too intimately! )

From whitverj at urbandale.k12.ia.us  Tue Feb 10 22:52:29 2009
From: whitverj at urbandale.k12.ia.us (Josh Whitver)
Date: Tue, 10 Feb 2009 21:52:29 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <20090211001116.GA540@brong.net>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
	<20090210230834.GA28780@brong.net>
	<69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>
	<20090211001116.GA540@brong.net>
Message-ID: <89680FEE-9790-4EC3-B723-D1DE4BE488D9@urbandale.k12.ia.us>

On Feb 10, 2009, at 6:11 PM, Bron Gondwana wrote:

> On Tue, Feb 10, 2009 at 05:20:42PM -0600, Josh Whitver wrote:
>> On Feb 10, 2009, at 5:08 PM, Bron Gondwana wrote:
>>>>
>>>
>>> Yeah, you have a broken mailboxes.db I'd say!
>>>
>>> Here's what I would do:
>>>
>>> # stop cyrus
>>> cyr_dbtool /var/imap/mailboxes.db skiplist show > mailboxes.dump
>>> cyr_dbtool -n /var/lib/mailboxes_new.db skiplist set <  
>>> mailboxes.dump
>>> mv /var/imap/mailboxes.db /var/imap/mailboxes_old.db
>>> mv /var/lib/mailboxes_new.db /var/imap/mailboxes.db
>>> # start cyrus
>>>
>>> This will give you a brand new clean mailboxes.db (but keep the old
>>> one
>>> just in case)
>>>
>>> Bron.
>>>
>>
>> Thanks - is this one of those operations that will kill the read/ 
>> unread
>> status of my messages, or should all that stuff be preserved?
>
> No, it won't kill anything (hopefully).  You may want to compare the
> output of show on the new file with show on the old file.  They  
> _should_
> be identical.
>
> If you want to email me (privately) a copy of your old mailboxes.db,  
> I'd
> be happy to have a look through it and check for corruption.   
> Obviously
> it contains private details (usernames, folder names and ACLs - no
> actual email) of your users, so it's up to you.
>
> Bron ( who knows the skiplist format far too intimately! )

Well, I just tried it - no dice.  Same issues persist, even with a  
fresh mailboxes.db file.

Next steps?
--
Josh Whitver
whitverj at urbandale.k12.ia.us / josh at whitver.net
This message transmitted on 100% recycled electrons.

From whitverj at urbandale.k12.ia.us  Wed Feb 11 14:32:52 2009
From: whitverj at urbandale.k12.ia.us (Josh Whitver)
Date: Wed, 11 Feb 2009 13:32:52 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <89680FEE-9790-4EC3-B723-D1DE4BE488D9@urbandale.k12.ia.us>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
	<20090210230834.GA28780@brong.net>
	<69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>
	<20090211001116.GA540@brong.net>
	<89680FEE-9790-4EC3-B723-D1DE4BE488D9@urbandale.k12.ia.us>
Message-ID: <7D52B532-333C-4B1D-BA6E-E94D5EA6FA4A@urbandale.k12.ia.us>

On Feb 10, 2009, at 9:52 PM, Josh Whitver wrote:
>>>> Yeah, you have a broken mailboxes.db I'd say!
>>>>
>>>> Here's what I would do:
>>>>
>>>> # stop cyrus
>>>> cyr_dbtool /var/imap/mailboxes.db skiplist show > mailboxes.dump
>>>> cyr_dbtool -n /var/lib/mailboxes_new.db skiplist set <
>>>> mailboxes.dump
>>>> mv /var/imap/mailboxes.db /var/imap/mailboxes_old.db
>>>> mv /var/lib/mailboxes_new.db /var/imap/mailboxes.db
>>>> # start cyrus
>>>>
>>>> This will give you a brand new clean mailboxes.db (but keep the old
>>>> one
>>>> just in case)
>>>>
>>>> Bron.
> Well, I just tried it - no dice.  Same issues persist, even with a
> fresh mailboxes.db file.
>
> Next steps?

Anyone?  Bueller?
--
Josh Whitver
whitverj at urbandale.k12.ia.us / josh at whitver.net
"Give me ambiguity or give me something else!"


From blake at ispn.net  Wed Feb 11 14:49:30 2009
From: blake at ispn.net (Blake Hudson)
Date: Wed, 11 Feb 2009 13:49:30 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <7D52B532-333C-4B1D-BA6E-E94D5EA6FA4A@urbandale.k12.ia.us>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>	<20090210230834.GA28780@brong.net>	<69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>	<20090211001116.GA540@brong.net>	<89680FEE-9790-4EC3-B723-D1DE4BE488D9@urbandale.k12.ia.us>
	<7D52B532-333C-4B1D-BA6E-E94D5EA6FA4A@urbandale.k12.ia.us>
Message-ID: <49932BCA.3030400@ispn.net>

-------- Original Message  --------
Subject: Re: Cyrus DB Errors "DBERROR: error fetching user.username"
even    after reconstructs
From: Josh Whitver <whitverj at urbandale.k12.ia.us>
To: info-cyrus at lists.andrew.cmu.edu
Date: Wednesday, February 11, 2009 1:32:52 PM
> On Feb 10, 2009, at 9:52 PM, Josh Whitver wrote:
>   
>>>>> Yeah, you have a broken mailboxes.db I'd say!
>>>>>
>>>>> Here's what I would do:
>>>>>
>>>>> # stop cyrus
>>>>> cyr_dbtool /var/imap/mailboxes.db skiplist show > mailboxes.dump
>>>>> cyr_dbtool -n /var/lib/mailboxes_new.db skiplist set <
>>>>> mailboxes.dump
>>>>> mv /var/imap/mailboxes.db /var/imap/mailboxes_old.db
>>>>> mv /var/lib/mailboxes_new.db /var/imap/mailboxes.db
>>>>> # start cyrus
>>>>>
>>>>> This will give you a brand new clean mailboxes.db (but keep the old
>>>>> one
>>>>> just in case)
>>>>>
>>>>> Bron.
>>>>>           
>> Well, I just tried it - no dice.  Same issues persist, even with a
>> fresh mailboxes.db file.
>>
>> Next steps?
>>     
>
> Anyone?  Bueller?
>   
I agree it looks like a mailbox db problem. My inclination after a
failed reconstruct would be to remove and recreate the mailbox and
permissions (perhaps checking the mailbox db in between steps).

-Blake
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090211/31aafeb0/attachment.html 

From whitverj at urbandale.k12.ia.us  Wed Feb 11 14:52:25 2009
From: whitverj at urbandale.k12.ia.us (Josh Whitver)
Date: Wed, 11 Feb 2009 13:52:25 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <49932BCA.3030400@ispn.net>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>	<20090210230834.GA28780@brong.net>	<69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>	<20090211001116.GA540@brong.net>	<89680FEE-9790-4EC3-B723-D1DE4BE488D9@urbandale.k12.ia.us>
	<7D52B532-333C-4B1D-BA6E-E94D5EA6FA4A@urbandale.k12.ia.us>
	<49932BCA.3030400@ispn.net>
Message-ID: <AB3C1EBE-5CB0-47B6-8B0A-79FA55CFBFE7@urbandale.k12.ia.us>

On Feb 11, 2009, at 1:49 PM, Blake Hudson wrote:
>> On Feb 10, 2009, at 9:52 PM, Josh Whitver wrote:
>>
>>>>>> Yeah, you have a broken mailboxes.db I'd say!
>>>>>>
>>>>>> Here's what I would do:
>>>>>>
>>>>>> # stop cyrus
>>>>>> cyr_dbtool /var/imap/mailboxes.db skiplist show > mailboxes.dump
>>>>>> cyr_dbtool -n /var/lib/mailboxes_new.db skiplist set <
>>>>>> mailboxes.dump
>>>>>> mv /var/imap/mailboxes.db /var/imap/mailboxes_old.db
>>>>>> mv /var/lib/mailboxes_new.db /var/imap/mailboxes.db
>>>>>> # start cyrus
>>>>>>
>>>>>> This will give you a brand new clean mailboxes.db (but keep the  
>>>>>> old
>>>>>> one
>>>>>> just in case)
>>>>>>
>>>>>> Bron.
>>>>>>
>>> Well, I just tried it - no dice.  Same issues persist, even with a
>>> fresh mailboxes.db file.
>>>
>>> Next steps?
>>>
>> Anyone?  Bueller?
>>
> I agree it looks like a mailbox db problem. My inclination after a  
> failed reconstruct would be to remove and recreate the mailbox and  
> permissions (perhaps checking the mailbox db in between steps).
>
> -Blake


Thanks, but I've already gone in via cyradm and deleted the mailboxes  
and recreated them, and following the steps above, completely rebuilt  
the mailboxes.db file last night - same problems.  Thank you Mario,  
but I think my princess is in another castle! :)
--
Josh Whitver
whitverj at urbandale.k12.ia.us / josh at whitver.net
I once planted some birdseed.  A bird came up.
Now I don't know what to feed it.

From dwhite at olp.net  Wed Feb 11 16:32:27 2009
From: dwhite at olp.net (Dan White)
Date: Wed, 11 Feb 2009 15:32:27 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <AB3C1EBE-5CB0-47B6-8B0A-79FA55CFBFE7@urbandale.k12.ia.us>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>	<20090210230834.GA28780@brong.net>	<69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>	<20090211001116.GA540@brong.net>	<89680FEE-9790-4EC3-B723-D1DE4BE488D9@urbandale.k12.ia.us>	<7D52B532-333C-4B1D-BA6E-E94D5EA6FA4A@urbandale.k12.ia.us>	<49932BCA.3030400@ispn.net>
	<AB3C1EBE-5CB0-47B6-8B0A-79FA55CFBFE7@urbandale.k12.ia.us>
Message-ID: <499343EB.6020608@olp.net>

Josh Whitver wrote:
> Thanks, but I've already gone in via cyradm and deleted the mailboxes
> and recreated them, and following the steps above, completely rebuilt  
> the mailboxes.db file last night - same problems.  Thank you Mario,  
> but I think my princess is in another castle! :)
>   


Josh,

To get more details about what's going on underneath the hood, you can 
use the debug_command option (man imapd.conf) to obtain a back trace of 
the hanging imapd process.

For some configuration scenarios, see:

https://langhorst.com/cgi-bin/dwww//usr/share/doc/cyrus21-common/README.Debian.debug.gz

If you know the specific mailbox that is causing the problem, and you 
want to only debug specific IMAP connections, you could configure (in 
/etc/cyrus.conf):

        imap            cmd="proxyd -U 30" listen="1.2.3.4:imap" 
prefork=0 maxchild=200
        imaplh         cmd="imapd -U 30 -D" listen="127.0.0.1:imap" 
prefork=0 maxchild=200

where 1.2.3.4 is the address your users connect to. And then connect to 
the 127.0.0.1 imap port to debug that user's connection.

- Dan

From dwhite at olp.net  Wed Feb 11 16:39:53 2009
From: dwhite at olp.net (Dan White)
Date: Wed, 11 Feb 2009 15:39:53 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <499343EB.6020608@olp.net>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>	<20090210230834.GA28780@brong.net>	<69565EC3-8391-4BA7-BB00-B0884ECAD7A0@urbandale.k12.ia.us>	<20090211001116.GA540@brong.net>	<89680FEE-9790-4EC3-B723-D1DE4BE488D9@urbandale.k12.ia.us>	<7D52B532-333C-4B1D-BA6E-E94D5EA6FA4A@urbandale.k12.ia.us>	<49932BCA.3030400@ispn.net>	<AB3C1EBE-5CB0-47B6-8B0A-79FA55CFBFE7@urbandale.k12.ia.us>
	<499343EB.6020608@olp.net>
Message-ID: <499345A9.6010303@olp.net>

Dan White wrote:
>
>         imap            cmd="proxyd -U 30" listen="1.2.3.4:imap" 
> prefork=0 maxchild=200
>         imaplh         cmd="imapd -U 30 -D" listen="127.0.0.1:imap" 
> prefork=0 maxchild=200
>
>   

Typo. That's what I get for cutting and pasting from different systems. 
Both cmds should be imapd. I'm assuming you're not proxying.

- Dan

From brong at fastmail.fm  Wed Feb 11 21:40:47 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 12 Feb 2009 13:40:47 +1100
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
Message-ID: <1234406447.12704.1299910655@webmail.messagingengine.com>


On Tue, 10 Feb 2009 15:59:09 -0600, "Josh Whitver" <whitverj at urbandale.k12.ia.us> said:
> Whenever an IMAP connection is attempted, I see this in the log:
> Feb 10 09:14:17 urbandale imap[34508]: DBERROR: error fetching user. 
> [username]: cyrusdb error

OK - there are two spots in the code that can produce exactly that
error message.  One is in BDB, which I assume you're not using.  The
other is in mboxlist (latest CVS changes this one to "error fetching
mboxlist user.[username]: [error]" to be clearer!)

So, it's in mboxlist_mylookup.  The specific error is being returned
by:

    if (wrlock) {
        r = DB->fetchlock(mbdb, name, namelen, &data, &datalen, tid);
    } else {
        r = DB->fetch(mbdb, name, namelen, &data, &datalen, tid);
    }

    ...
    
    default:
        syslog(LOG_ERR, "DBERROR: error fetching mboxlist %s: %s",
               name, cyrusdb_strerror(r));
        return IMAP_IOERROR;
        break;

Sorry I didn't get back to you before.  I really don't think I can
debug this any further without a copy of your mailboxes.db file.

I assume you've ruled out filesystem corruption by checking syslog/dmesg/
whatever for signs of bad blocks?

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm


From whitverj at urbandale.k12.ia.us  Thu Feb 12 10:45:58 2009
From: whitverj at urbandale.k12.ia.us (Josh Whitver)
Date: Thu, 12 Feb 2009 09:45:58 -0600
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after reconstructs
In-Reply-To: <1234406447.12704.1299910655@webmail.messagingengine.com>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
	<1234406447.12704.1299910655@webmail.messagingengine.com>
Message-ID: <E0C5FF02-522A-4ADF-9EF0-A09D25585091@urbandale.k12.ia.us>

It's actually possible we are using BDB - earlier in the mail server's  
life we moved it from a PowerPC G5 Xserve to the Intel Xserve, and we  
needed to work around the endian issues inherent in some part of the  
mail server, so I recall converting a database in some direction,  
either from BDB to skiplist or vice-versa.  My memory escapes  me now  
as to which way we went, though.  How would I check?
--
Josh Whitver
whitverj at urbandale.k12.ia.us / josh at whitver.net
(If you can't hear me, it's because I'm in parentheses)


On Feb 11, 2009, at 8:40 PM, Bron Gondwana wrote:

>
> On Tue, 10 Feb 2009 15:59:09 -0600, "Josh Whitver" <whitverj at urbandale.k12.ia.us 
> > said:
>> Whenever an IMAP connection is attempted, I see this in the log:
>> Feb 10 09:14:17 urbandale imap[34508]: DBERROR: error fetching user.
>> [username]: cyrusdb error
>
> OK - there are two spots in the code that can produce exactly that
> error message.  One is in BDB, which I assume you're not using.  The
> other is in mboxlist (latest CVS changes this one to "error fetching
> mboxlist user.[username]: [error]" to be clearer!)
>
> So, it's in mboxlist_mylookup.  The specific error is being returned
> by:
>
>    if (wrlock) {
>        r = DB->fetchlock(mbdb, name, namelen, &data, &datalen, tid);
>    } else {
>        r = DB->fetch(mbdb, name, namelen, &data, &datalen, tid);
>    }
>
>    ...
>
>    default:
>        syslog(LOG_ERR, "DBERROR: error fetching mboxlist %s: %s",
>               name, cyrusdb_strerror(r));
>        return IMAP_IOERROR;
>        break;
>
> Sorry I didn't get back to you before.  I really don't think I can
> debug this any further without a copy of your mailboxes.db file.
>
> I assume you've ruled out filesystem corruption by checking syslog/ 
> dmesg/
> whatever for signs of bad blocks?
>
> Bron.
> -- 
>  Bron Gondwana
>  brong at fastmail.fm
>
>


From brong at fastmail.fm  Thu Feb 12 17:05:29 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 13 Feb 2009 09:05:29 +1100
Subject: Cyrus DB Errors "DBERROR: error fetching user.username" even
	after  reconstructs
In-Reply-To: <E0C5FF02-522A-4ADF-9EF0-A09D25585091@urbandale.k12.ia.us>
References: <0B5893E1-FBA7-48A9-90CC-4FE570224DDE@urbandale.k12.ia.us>
	<1234406447.12704.1299910655@webmail.messagingengine.com>
	<E0C5FF02-522A-4ADF-9EF0-A09D25585091@urbandale.k12.ia.us>
Message-ID: <20090212220529.GA11790@brong.net>

On Thu, Feb 12, 2009 at 09:45:58AM -0600, Josh Whitver wrote:
> It's actually possible we are using BDB - earlier in the mail server's  
> life we moved it from a PowerPC G5 Xserve to the Intel Xserve, and we  
> needed to work around the endian issues inherent in some part of the  
> mail server, so I recall converting a database in some direction, either 
> from BDB to skiplist or vice-versa.  My memory escapes  me now as to 
> which way we went, though.  How would I check?

grep mboxlist_db /etc/imapd.conf

The default is skiplist, so it will be skiplist unless you have
something else there.

(but then, I know it's skiplist because I have a copy of it now!)

Bron.

From jvoorhees1 at gmail.com  Thu Feb 12 17:49:29 2009
From: jvoorhees1 at gmail.com (Jason Voorhees)
Date: Thu, 12 Feb 2009 17:49:29 -0500
Subject: Security risk of POP3 & IMAP protocols
Message-ID: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>

Hi people:

A friend of mine is asking me about security risks of using IMAP &
POP3 protocols. Why? Because a sales person told my friend that IMAP
protocol is less secure than POP3 protocol. This assumption is not
related to Cyrus IMAP, instead is related only to the protocols.
I'm searching at Google something about POP3 & IMAP security but I'm
not pretty sure about comments I can found in forums or other sites.

Does anybody here know anything about security risk of these
protocols? Is it true that one of them is less secure than the other
one?

Thanks, bye

From octavian at corp.earthlink.net  Thu Feb 12 18:17:00 2009
From: octavian at corp.earthlink.net (Peter A. Friend)
Date: Thu, 12 Feb 2009 15:17:00 -0800
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
Message-ID: <9D1E51FD-1DC3-4A0F-8C85-BF3D4E276DF5@corp.earthlink.net>


On Feb 12, 2009, at 2:49 PM, Jason Voorhees wrote:

> Hi people:
>
> A friend of mine is asking me about security risks of using IMAP &
> POP3 protocols. Why? Because a sales person told my friend that IMAP
> protocol is less secure than POP3 protocol. This assumption is not
> related to Cyrus IMAP, instead is related only to the protocols.
> I'm searching at Google something about POP3 & IMAP security but I'm
> not pretty sure about comments I can found in forums or other sites.
>
> Does anybody here know anything about security risk of these
> protocols? Is it true that one of them is less secure than the other
> one?
>

I suppose that depends on one's definition of "security". There are  
secure authentication mechanisms available for both protocols, and you  
can use TLS. The more complex an application is the more opportunity  
there is for programmers to make mistakes or not properly validate  
inputs. Since IMAP is vastly more complicated that POP in it's  
operation, one could argue that an IMAP implementation is more likely  
to have exploitable bugs.

Peter

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090212/f985bf96/attachment.html 

From awilliam at whitemice.org  Thu Feb 12 19:58:34 2009
From: awilliam at whitemice.org (Adam Tauno Williams)
Date: Thu, 12 Feb 2009 19:58:34 -0500
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
Message-ID: <1234486714.7082.2.camel@linux-m3mt>

> A friend of mine is asking me about security risks of using IMAP &
> POP3 protocols. Why? Because a sales person told my friend that IMAP
> protocol is less secure than POP3 protocol. This assumption is not
> related to Cyrus IMAP, instead is related only to the protocols.
> I'm searching at Google something about POP3 & IMAP security but I'm
> not pretty sure about comments I can found in forums or other sites.

I'd write this claim off as bogus;  use GSSAPI authentication and TLS
and either is extremely secure.  Your more pressing security
vulnerabilities will certainly be elsewhere (the client OS and
configuration, most likely).


From brennan at columbia.edu  Thu Feb 12 20:49:53 2009
From: brennan at columbia.edu (Joseph Brennan)
Date: Thu, 12 Feb 2009 20:49:53 -0500
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <1234486714.7082.2.camel@linux-m3mt>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<1234486714.7082.2.camel@linux-m3mt>
Message-ID: <98119B7D6FA8AEB87A42FE23@[192.168.2.14]>


Adam Tauno Williams <awilliam at whitemice.org> wrote:

>> A friend of mine is asking me about security risks of using IMAP &
>> POP3 protocols. Why? Because a sales person told my friend that IMAP
>> protocol is less secure than POP3 protocol.


This reminds me of a concern that was raised about U Wash IMAP and storage
of mail in unix home directories.  In that setup IMAP access is based on
unix file system permissions, and IMAP will open files that are not mail
files if the user has unix file permissions to open them-- including
various system files.  This always struck me as a bogus concern since
the user could also telnet in and see the same files!

The protocol itself is no less secure than POP.  I don't understand why
POP is still around.

Joseph Brennan
Columbia University Information Technology



From ian.batten at uk.fujitsu.com  Fri Feb 13 04:23:10 2009
From: ian.batten at uk.fujitsu.com (Ian Batten)
Date: Fri, 13 Feb 2009 09:23:10 +0000
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <98119B7D6FA8AEB87A42FE23@[192.168.2.14]>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<1234486714.7082.2.camel@linux-m3mt>
	<98119B7D6FA8AEB87A42FE23@[192.168.2.14]>
Message-ID: <93ACAB20-B8D8-4C9E-9534-93E7E39A18D0@uk.fujitsu.com>


On 13 Feb 09, at 0149, Joseph Brennan wrote:
>
> The protocol itself is no less secure than POP.

Security isn't about protocols, it's about systems, and I suspect POP3  
vs IMAP is metonymic for local vs remote mail storage.

I can see an argument that says that one problem with IMAP is that  
your entire mail store, which is much more interesting to an attacker  
than a message in flight or your current mail pending collection a la  
POP3, is under someone else's control.  So if, say, you use a whole  
disk encryption product, mail delivered via traditional POP3 will be  
wrapped in the arms of the encryption immediately after collection,  
while mail stored on a remote server and accessed via IMAP will have  
whatever security features the server has.

If you control the IMAP server (for some suitable value of `you') then  
a risk assessment is the same task in both scenarios.  However, if, as  
is common in many situations, the IMAP server isn't within the scope  
of a risk assessment, then I can imagine that your 27001 life is a  
little easier if you don't have a large pool of potentially sensitive  
data under someone else's (for some value of `someone else')  
control.   Data at rest is a different class of problem to data in  
motion, and IMAP implies a _lot_ of data at rest.

To make this more concrete, imagine you're an HR department within a  
large enterprise, handling job applications, CVs, disciplinary  
processes, dismissals, etc.  You need to demonstrate your compliance  
with your local data protection regulations.  The theft of a day's  
email would be severely embarrassing, but is analogous to the theft of  
a day's postal mail: a risk which most businesses would accept.  It  
would expose limited amounts of information about a small subset of  
your employees.

However, the theft of a year's or a decade's email would expose  
substantial information about a large percentage of your employees,  
and would be analogous to allowing a few filing cabinets to be stolen.

Your email system is run by your corporation's IT function in another  
jurisdiction which has laxer data protection laws --- say, an EU  
company whose head office is in the USA.

Do you (a) store all your long term records in the other jurisdiction  
or (b) store them locally?

Now I'm not defending the argument, and indeed here we have ~4TB of  
email on our Cyrus servers.  But I don't think the position is  
entirely without merit, and having gone through the simplifying and  
distorting mirror of sales droids I can see where it's come from...

ian


From Duncan.Gibb at SiriusIT.co.uk  Fri Feb 13 08:17:47 2009
From: Duncan.Gibb at SiriusIT.co.uk (Duncan Gibb)
Date: Fri, 13 Feb 2009 13:17:47 +0000
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
Message-ID: <499572FB.10709@SiriusIT.co.uk>

Jason Voorhees wrote:

JV> a sales person told my friend that IMAP protocol is
JV> less secure than POP3 protocol.

Other people have covered the IMAP vs POP3 issues - Ian Batten most
comprehensively - but one comment I would add is that if you make either
service available to the open internet, even under SSL encryption,
password-based authentication is still susceptible to dictionary attack.
 So IMAP and/or POP3 (and/or SMTP AUTH) should be included in the list
of things you rate limit, monitor for bad password attempts, and lock
remote hosts out of if it they do things that look suspicious.


Cheers


Duncan

-- 
Duncan Gibb, Technical Director
Sirius Corporation plc - The Open Source Experts
http://www.siriusit.co.uk/ || +44 870 608 0063

From awilliam at whitemice.org  Fri Feb 13 09:13:40 2009
From: awilliam at whitemice.org (Adam Tauno Williams)
Date: Fri, 13 Feb 2009 09:13:40 -0500
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <499572FB.10709@SiriusIT.co.uk>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<499572FB.10709@SiriusIT.co.uk>
Message-ID: <1234534420.5264.32.camel@linux-m3mt>

On Fri, 2009-02-13 at 13:17 +0000, Duncan Gibb wrote:
> Jason Voorhees wrote:
> JV> a sales person told my friend that IMAP protocol is
> JV> less secure than POP3 protocol.
> Other people have covered the IMAP vs POP3 issues - Ian Batten most
> comprehensively - but one comment I would add is that if you make either
> service available to the open internet, even under SSL encryption,
> password-based authentication is still susceptible to dictionary attack.
>  So IMAP and/or POP3 (and/or SMTP AUTH) should be included in the list
> of things you rate limit, monitor for bad password attempts, and lock
> remote hosts out of if it they do things that look suspicious.

True;  but really none of those good practices is specific to any
protocol.   The exact same charge could be leveled against HTTP, FTP,
SSH, etc...  and if you use certificate/PKI authentication you run the
risk that someone could steal the private keys (and it isn't hard to
make a setup where that is comically easy).  It is really far and away
more about end-to-end security practices than it is the OSI layer 7
protocol(s) involved.

I stand by my assertion that the IMAP vs. POP issue is 100% bogosity. 


From Duncan.Gibb at SiriusIT.co.uk  Fri Feb 13 09:22:42 2009
From: Duncan.Gibb at SiriusIT.co.uk (Duncan Gibb)
Date: Fri, 13 Feb 2009 14:22:42 +0000
Subject: [OT] Re: Security risk of POP3 & IMAP protocols
In-Reply-To: <1234534420.5264.32.camel@linux-m3mt>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>	<499572FB.10709@SiriusIT.co.uk>
	<1234534420.5264.32.camel@linux-m3mt>
Message-ID: <49958232.8030209@SiriusIT.co.uk>

Adam Tauno Williams wrote:

JV> a sales person told my friend that IMAP protocol is
JV> less secure than POP3 protocol.

ATW> It is really far and away more about end-to-end security
ATW> practices than it is the OSI layer 7 protocol(s) involved.

Indeed.

ATW> I stand by my assertion that the IMAP vs. POP issue is 100% bogosity.

Yep; I agree.  Perhaps the sales person is pushing a mail system which
doesn't speak IMAP (if such a thing exists).


Duncan

-- 
Duncan Gibb, Technical Director
Sirius Corporation plc - The Open Source Experts
http://www.siriusit.co.uk/ || +44 870 608 0063

From addw at phcomp.co.uk  Fri Feb 13 09:35:43 2009
From: addw at phcomp.co.uk (Alain Williams)
Date: Fri, 13 Feb 2009 14:35:43 +0000
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <1234534420.5264.32.camel@linux-m3mt>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<499572FB.10709@SiriusIT.co.uk>
	<1234534420.5264.32.camel@linux-m3mt>
Message-ID: <20090213143543.GA4825@mint.phcomp.co.uk>

On Fri, Feb 13, 2009 at 09:13:40AM -0500, Adam Tauno Williams wrote:
> On Fri, 2009-02-13 at 13:17 +0000, Duncan Gibb wrote:
> > Jason Voorhees wrote:
> > JV> a sales person told my friend that IMAP protocol is
> > JV> less secure than POP3 protocol.
> > Other people have covered the IMAP vs POP3 issues - Ian Batten most
> > comprehensively - but one comment I would add is that if you make either
> > service available to the open internet, even under SSL encryption,
> > password-based authentication is still susceptible to dictionary attack.
> >  So IMAP and/or POP3 (and/or SMTP AUTH) should be included in the list
> > of things you rate limit, monitor for bad password attempts, and lock
> > remote hosts out of if it they do things that look suspicious.

That got me thinking ....
I rate limit ssh connections to try to prevent dictionary attacks (3 attempts/3 minutes/IP address).
If I were to do the same with IMAP would that cause problems with some clients,
ie are there some clients that to many connect/disconnects ?

> True;  but really none of those good practices is specific to any
> protocol.   The exact same charge could be leveled against HTTP, FTP,
> SSH, etc...  and if you use certificate/PKI authentication you run the
> risk that someone could steal the private keys (and it isn't hard to
> make a setup where that is comically easy).  It is really far and away
> more about end-to-end security practices than it is the OSI layer 7
> protocol(s) involved.

-- 
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256  http://www.phcomp.co.uk/
Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
Past chairman of UKUUG: http://www.ukuug.org/
#include <std_disclaimer.h>

From dave64 at andrew.cmu.edu  Fri Feb 13 09:40:54 2009
From: dave64 at andrew.cmu.edu (Dave McMurtrie)
Date: Fri, 13 Feb 2009 09:40:54 -0500
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <20090213143543.GA4825@mint.phcomp.co.uk>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>	<499572FB.10709@SiriusIT.co.uk>	<1234534420.5264.32.camel@linux-m3mt>
	<20090213143543.GA4825@mint.phcomp.co.uk>
Message-ID: <49958676.2010802@andrew.cmu.edu>

Alain Williams wrote:

> That got me thinking ....
> I rate limit ssh connections to try to prevent dictionary attacks (3 attempts/3 minutes/IP address).
> If I were to do the same with IMAP would that cause problems with some clients,
> ie are there some clients that to many connect/disconnects ?

Webmail is the first one that comes to mind.

Thanks,

Dave
-- 
Dave McMurtrie, SPE
Email Systems Team Leader
Carnegie Mellon University,
Computing Services

From iane at sussex.ac.uk  Fri Feb 13 10:21:06 2009
From: iane at sussex.ac.uk (Ian Eiloart)
Date: Fri, 13 Feb 2009 15:21:06 +0000
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <20090213143543.GA4825@mint.phcomp.co.uk>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<499572FB.10709@SiriusIT.co.uk>	<1234534420.5264.32.camel@linux-m3mt>
	<20090213143543.GA4825@mint.phcomp.co.uk>
Message-ID: <F8C5126C3F11E692F55118D7@lewes.staff.uscs.susx.ac.uk>



--On 13 February 2009 14:35:43 +0000 Alain Williams <addw at phcomp.co.uk> 
wrote:

> That got me thinking ....
> I rate limit ssh connections to try to prevent dictionary attacks (3
> attempts/3 minutes/IP address). If I were to do the same with IMAP would
> that cause problems with some clients, ie are there some clients that to
> many connect/disconnects ?

Yes. Anything that opens a bunch of mailboxes at the same time might be 
doing way more than that. You should be measuring "failed attempts", not 
"attempts".


-- 
Ian Eiloart
IT Services, University of Sussex
x3148

From jvoorhees1 at gmail.com  Fri Feb 13 10:23:58 2009
From: jvoorhees1 at gmail.com (Jason Voorhees)
Date: Fri, 13 Feb 2009 10:23:58 -0500
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
Message-ID: <c28863110902130723w27f35a10t252c7402c0b9323b@mail.gmail.com>

On Thu, Feb 12, 2009 at 5:49 PM, Jason Voorhees <jvoorhees1 at gmail.com> wrote:
> Hi people:
>
> A friend of mine is asking me about security risks of using IMAP &
> POP3 protocols. Why? Because a sales person told my friend that IMAP
> protocol is less secure than POP3 protocol. This assumption is not
> related to Cyrus IMAP, instead is related only to the protocols.
> I'm searching at Google something about POP3 & IMAP security but I'm
> not pretty sure about comments I can found in forums or other sites.
>
> Does anybody here know anything about security risk of these
> protocols? Is it true that one of them is less secure than the other
> one?
>
> Thanks, bye
>

Thanks everyone for your replies, they were good answers with
different points of view.
Actually, I made a mistake writing my post: My friend told me that the
sales person believes that POP3 has security problems and is
vulnerable so recommends IMAP as a replacement of use at final users.

Anyway, it doesn't matter what the sales person really said because I
can see now that the argument of using one protocol instead the other
one depends much of the context.
The POP3/IMAP server (now running Zimbra) is running at my friend's
office with all his users using POP3. I will migrate its mailserver to
Cyrus + MTA+other components...and they plan to use IMAP now.

I will explain him every point of view that you shared with me. Thanks again :)

From addw at phcomp.co.uk  Fri Feb 13 10:30:46 2009
From: addw at phcomp.co.uk (Alain Williams)
Date: Fri, 13 Feb 2009 15:30:46 +0000
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <F8C5126C3F11E692F55118D7@lewes.staff.uscs.susx.ac.uk>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<499572FB.10709@SiriusIT.co.uk>
	<1234534420.5264.32.camel@linux-m3mt>
	<20090213143543.GA4825@mint.phcomp.co.uk>
	<F8C5126C3F11E692F55118D7@lewes.staff.uscs.susx.ac.uk>
Message-ID: <20090213153046.GB4825@mint.phcomp.co.uk>

[23~On Fri, Feb 13, 2009 at 03:21:06PM +0000, Ian Eiloart wrote:
> 
> 
> --On 13 February 2009 14:35:43 +0000 Alain Williams <addw at phcomp.co.uk> 
> wrote:
> 
> >That got me thinking ....
> >I rate limit ssh connections to try to prevent dictionary attacks (3
> >attempts/3 minutes/IP address). If I were to do the same with IMAP would
> >that cause problems with some clients, ie are there some clients that to
> >many connect/disconnects ?
> 
> Yes. Anything that opens a bunch of mailboxes at the same time might be 
> doing way more than that. You should be measuring "failed attempts", not 
> "attempts".

Yes, but I do the rate limiting with iptables (Linux firewall).
I don't know how to feedback failed attempts to iptables.

-- 
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256  http://www.phcomp.co.uk/
Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
Past chairman of UKUUG: http://www.ukuug.org/
#include <std_disclaimer.h>

From list at joreybump.com  Fri Feb 13 10:41:10 2009
From: list at joreybump.com (Jorey Bump)
Date: Fri, 13 Feb 2009 10:41:10 -0500
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <20090213153046.GB4825@mint.phcomp.co.uk>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>	<499572FB.10709@SiriusIT.co.uk>	<1234534420.5264.32.camel@linux-m3mt>	<20090213143543.GA4825@mint.phcomp.co.uk>	<F8C5126C3F11E692F55118D7@lewes.staff.uscs.susx.ac.uk>
	<20090213153046.GB4825@mint.phcomp.co.uk>
Message-ID: <49959496.5020208@joreybump.com>

Alain Williams wrote, at 02/13/2009 10:30 AM:
> [23~On Fri, Feb 13, 2009 at 03:21:06PM +0000, Ian Eiloart wrote:
>>
>> --On 13 February 2009 14:35:43 +0000 Alain Williams <addw at phcomp.co.uk> 
>> wrote:
>>
>>> That got me thinking ....
>>> I rate limit ssh connections to try to prevent dictionary attacks (3
>>> attempts/3 minutes/IP address). If I were to do the same with IMAP would
>>> that cause problems with some clients, ie are there some clients that to
>>> many connect/disconnects ?
>> Yes. Anything that opens a bunch of mailboxes at the same time might be 
>> doing way more than that. You should be measuring "failed attempts", not 
>> "attempts".
> 
> Yes, but I do the rate limiting with iptables (Linux firewall).
> I don't know how to feedback failed attempts to iptables.

I have yet to encounter an automated brute force attack that negotiates
STARTTLS, SSL or any of the more secure SASL mechanisms. In time, this
will probably change, but you will get more bang for your buck now if
you enforce encrypted connections. You can still run an unencrypted port
on localhost (or restrict access another way) if you need it for webmail.



From iane at sussex.ac.uk  Fri Feb 13 10:45:11 2009
From: iane at sussex.ac.uk (Ian Eiloart)
Date: Fri, 13 Feb 2009 15:45:11 +0000
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <20090213153046.GB4825@mint.phcomp.co.uk>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<499572FB.10709@SiriusIT.co.uk>	<1234534420.5264.32.camel@linux-m3mt>
	<20090213143543.GA4825@mint.phcomp.co.uk>
	<F8C5126C3F11E692F55118D7@lewes.staff.uscs.susx.ac.uk>
	<20090213153046.GB4825@mint.phcomp.co.uk>
Message-ID: <4F98D411C5978FAC68FA813A@lewes.staff.uscs.susx.ac.uk>



--On 13 February 2009 15:30:46 +0000 Alain Williams <addw at phcomp.co.uk> 
wrote:

> [23~On Fri, Feb 13, 2009 at 03:21:06PM +0000, Ian Eiloart wrote:
>>
>>
>> --On 13 February 2009 14:35:43 +0000 Alain Williams <addw at phcomp.co.uk>
>> wrote:
>>
>> > That got me thinking ....
>> > I rate limit ssh connections to try to prevent dictionary attacks (3
>> > attempts/3 minutes/IP address). If I were to do the same with IMAP
>> > would that cause problems with some clients, ie are there some clients
>> > that to many connect/disconnects ?
>>
>> Yes. Anything that opens a bunch of mailboxes at the same time might be
>> doing way more than that. You should be measuring "failed attempts", not
>> "attempts".
>
> Yes, but I do the rate limiting with iptables (Linux firewall).
> I don't know how to feedback failed attempts to iptables.

Hmm, and for the webmail case, you'd want to do failed attempts per 
username per minute, not per IP address. Or, exempt your webmail server.

Apple Mail is a case in point, it checks for new mail in "your INBOX" or 
"all of your mailboxes" in parallel. I've seen it open dozens of 
connections from a single user, simultaneously.

-- 
Ian Eiloart
IT Services, University of Sussex
x3148

From D.H.Davis at bath.ac.uk  Fri Feb 13 11:17:57 2009
From: D.H.Davis at bath.ac.uk (Dennis Davis)
Date: Fri, 13 Feb 2009 16:17:57 +0000 (GMT)
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <20090213153046.GB4825@mint.phcomp.co.uk>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<499572FB.10709@SiriusIT.co.uk>
	<1234534420.5264.32.camel@linux-m3mt>
	<20090213143543.GA4825@mint.phcomp.co.uk>
	<F8C5126C3F11E692F55118D7@lewes.staff.uscs.susx.ac.uk>
	<20090213153046.GB4825@mint.phcomp.co.uk>
Message-ID: <alpine.BSO.2.00.0902131554160.3513@hinault.bath.ac.uk>

On Fri, 13 Feb 2009, Alain Williams wrote:

> From: Alain Williams <addw at phcomp.co.uk>
> To: Cyrus Mailing List <info-cyrus at lists.andrew.cmu.edu>
> Date: Fri, 13 Feb 2009 15:30:46 +0000
> Subject: Re: Security risk of POP3 & IMAP protocols

...

> > Yes. Anything that opens a bunch of mailboxes at the same time
> > might be doing way more than that. You should be measuring
> > "failed attempts", not "attempts".
>
> Yes, but I do the rate limiting with iptables (Linux firewall).  I
> don't know how to feedback failed attempts to iptables.

There are probably several ways to do this.  But, as a suggestion,
have a look at sshblack from:

http://www.pettingers.org/code/sshblack.html

It's intended for use against ssh brute-force attempts.  However
it's a perl script runnning tail on a log looking for suspicious
activity.  So should be easily adaptable for other purposes, along
with the iptables scripts included.  I expect the only wrinkle with
IMAP is that you'll want to block both port 143 and 993.

I fire up a small IMAP server with:

CYRUS_VERBOSE=1 ... 

and keep the logs separate.  Failed login attempts show up in the
logs as lines of the form:

Feb 13 15:42:25 bahamontes imap[10596]: badlogin: hinault.bath.ac.uk [138.38.56.28] PLAIN [SASL(-13): authentication failure: Password verification failed]

so it should be easy for a perl script to pick out the badly-behaved
client.

As others have pointed out, webmail servers are a particular
pain.  You'll probably need to whitelist your own webmail servers.
Otherwise the external blackhats will be able to persuade your IMAP
server to deny access to your webmail server(s).  A neat DOS attack
*and* lots of unhappy customers!

You should also consider how you'd harden up your webmail servers
against brute force attacks.  Not sure how you'd do that as many,
if not all webmail servers, rely on the IMAP server to validate the
connection.

Usual disclaimer: I've never tried doing this myself.  This advice
                  is worth what you paid for it.
-- 
Dennis Davis, BUCS, University of Bath, Bath, BA2 7AY, UK
D.H.Davis at bath.ac.uk               Phone: +44 1225 386101

From david.lang at digitalinsight.com  Fri Feb 13 12:22:02 2009
From: david.lang at digitalinsight.com (David Lang)
Date: Fri, 13 Feb 2009 09:22:02 -0800 (PST)
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <93ACAB20-B8D8-4C9E-9534-93E7E39A18D0@uk.fujitsu.com>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com><1234486714.7082.2.camel@linux-m3mt><98119B7D6FA8AEB87A42FE23@[192.168.2.14]>
	<93ACAB20-B8D8-4C9E-9534-93E7E39A18D0@uk.fujitsu.com>
Message-ID: <alpine.DEB.2.00.0902130918170.11190@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>

On Fri, 13 Feb 2009, Ian Batten wrote:

> On 13 Feb 09, at 0149, Joseph Brennan wrote:
>>
>> The protocol itself is no less secure than POP.
>
> Security isn't about protocols, it's about systems, and I suspect POP3
> vs IMAP is metonymic for local vs remote mail storage.
>
> I can see an argument that says that one problem with IMAP is that
> your entire mail store, which is much more interesting to an attacker
> than a message in flight or your current mail pending collection a la
> POP3, is under someone else's control.  So if, say, you use a whole
> disk encryption product, mail delivered via traditional POP3 will be
> wrapped in the arms of the encryption immediately after collection,
> while mail stored on a remote server and accessed via IMAP will have
> whatever security features the server has.
>
> If you control the IMAP server (for some suitable value of `you') then
> a risk assessment is the same task in both scenarios.  However, if, as
> is common in many situations, the IMAP server isn't within the scope
> of a risk assessment, then I can imagine that your 27001 life is a
> little easier if you don't have a large pool of potentially sensitive
> data under someone else's (for some value of `someone else')
> control.   Data at rest is a different class of problem to data in
> motion, and IMAP implies a _lot_ of data at rest.
>
> To make this more concrete, imagine you're an HR department within a
> large enterprise, handling job applications, CVs, disciplinary
> processes, dismissals, etc.  You need to demonstrate your compliance
> with your local data protection regulations.  The theft of a day's
> email would be severely embarrassing, but is analogous to the theft of
> a day's postal mail: a risk which most businesses would accept.  It
> would expose limited amounts of information about a small subset of
> your employees.
>
> However, the theft of a year's or a decade's email would expose
> substantial information about a large percentage of your employees,
> and would be analogous to allowing a few filing cabinets to be stolen.
>
> Your email system is run by your corporation's IT function in another
> jurisdiction which has laxer data protection laws --- say, an EU
> company whose head office is in the USA.
>
> Do you (a) store all your long term records in the other jurisdiction
> or (b) store them locally?
>
> Now I'm not defending the argument, and indeed here we have ~4TB of
> email on our Cyrus servers.  But I don't think the position is
> entirely without merit, and having gone through the simplifying and
> distorting mirror of sales droids I can see where it's come from...

the flip side of the complience issue is that it's a LOT easier to control 
retention policies (including backups) on a central server than on everybody's 
individual desktops/laptops.

as for the concerns about laxer data security in other juristictions, that's 
something that needs to be addressed when you outsource your mail (via contract 
with whoever you are having host your mail for you)

David Lang

From vbfox at ucdavis.edu  Fri Feb 13 12:46:26 2009
From: vbfox at ucdavis.edu (Vincent Fox)
Date: Fri, 13 Feb 2009 09:46:26 -0800
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <alpine.DEB.2.00.0902130918170.11190@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com><1234486714.7082.2.camel@linux-m3mt><98119B7D6FA8AEB87A42FE23@[192.168.2.14]>	<93ACAB20-B8D8-4C9E-9534-93E7E39A18D0@uk.fujitsu.com>
	<alpine.DEB.2.00.0902130918170.11190@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
Message-ID: <4995B1F2.7090300@ucdavis.edu>

David Lang wrote:
>
> the flip side of the complience issue is that it's a LOT easier to control 
> retention policies (including backups) on a central server than on everybody's 
> individual desktops/laptops.
>
> as for the concerns about laxer data security in other juristictions, that's 
> something that needs to be addressed when you outsource your mail (via contract 
> with whoever you are having host your mail for you)
>
>   
I worked at one organization that supported ONLY POP3.
No IMAP was offered.  Each client was configured to download
all messages and not leave a copy on the server.  This was
a policy that the University group I worked with that time
used for FOIA avoidance.  Want to see somebody's email
well you'll have to go see that individual.


From wes at umich.edu  Fri Feb 13 21:06:28 2009
From: wes at umich.edu (Wesley Craig)
Date: Fri, 13 Feb 2009 21:06:28 -0500
Subject: Security risk of POP3 & IMAP protocols
In-Reply-To: <93ACAB20-B8D8-4C9E-9534-93E7E39A18D0@uk.fujitsu.com>
References: <c28863110902121449h440b686cw88108232b32d125c@mail.gmail.com>
	<1234486714.7082.2.camel@linux-m3mt>
	<98119B7D6FA8AEB87A42FE23@[192.168.2.14]>
	<93ACAB20-B8D8-4C9E-9534-93E7E39A18D0@uk.fujitsu.com>
Message-ID: <55DBA796-BFC5-427A-A976-E6202CD750A3@umich.edu>

On 13 Feb 2009, at 04:23, Ian Batten wrote:
> Security isn't about protocols, it's about systems, and I suspect POP3
> vs IMAP is metonymic for local vs remote mail storage.

Also keep in mind that IMAP can be used just like POP, i.e., you can  
use IMAP to download & remove all mail from the server.

:wes

From schweizer.martin at gmail.com  Sat Feb 14 11:30:59 2009
From: schweizer.martin at gmail.com (Martin Schweizer)
Date: Sat, 14 Feb 2009 17:30:59 +0100
Subject: Problem with sieve login since update to 2.3.13
In-Reply-To: <380ccfd60902140823w36d1f4d0h406108023a17d600@mail.gmail.com>
References: <380ccfd60902140823w36d1f4d0h406108023a17d600@mail.gmail.com>
Message-ID: <380ccfd60902140830l4fc3e1eeo14a59c54c245024e@mail.gmail.com>

Hello

I have the following setup:

- FreeBSD acsvfbsd02.acutronic.ch 7.0-RELEASE FreeBSD 7.0-RELEASE #2:
Fri Aug 22 15:37:32 CEST 2008
martin at acsvfbsd02.acutronic.ch:/usr/obj/usr/src/sys/GENERIC  i386
- Cyrus IMAP v2.3.13
- Squirrelmail
- Websieve

Since the update to .13 I can no longer login to sieve as a regular
user (also not by sieveshell). Only root can login by sieveshell. If I
want to login as a regular user by sieveshell the password prompt
cames up and I can type in the password, that is all. No login,
nothing. I only see this message in /var/log/console.log:

Feb 14 16:50:54 acsvfbsd02 kernel: Feb 14 16:50:54 acsvfbsd02
pop3[75351]: realm changed: authentication aborted
Feb 14 16:51:14 acsvfbsd02 kernel: Feb 14 16:51:14 acsvfbsd02
sieve[75353]: no user in db
Feb 14 16:51:31 acsvfbsd02 kernel: Feb 14 16:51:31 acsvfbsd02 su:
martin to root on /dev/ttyp1
Feb 14 16:52:20 acsvfbsd02 kernel: Feb 14 16:52:20 acsvfbsd02
sieve[75378]: no user in db
Feb 14 16:53:07 acsvfbsd02 kernel: Feb 14 16:53:07 acsvfbsd02
sieve[75388]: no user in db
Feb 14 16:53:07 acsvfbsd02 kernel: Feb 14 16:53:07 acsvfbsd02
sieve[75388]: client response doesn't match what we generated
Feb 14 16:53:21 acsvfbsd02 kernel: Feb 14 16:53:21 acsvfbsd02
sieve[75401]: no user in db
Feb 14 16:53:21 acsvfbsd02 kernel: Feb 14 16:53:21 acsvfbsd02
sieve[75401]: user martin is not allowed to proxy
Feb 14 16:53:29 acsvfbsd02 kernel: Feb 14 16:53:29 acsvfbsd02
sieve[75411]: no user in db
Feb 14 16:53:29 acsvfbsd02 kernel: Feb 14 16:53:29 acsvfbsd02
sieve[75411]: user martin is not allowed to proxy
Feb 14 16:53:43 acsvfbsd02 kernel: Feb 14 16:53:43 acsvfbsd02
sieve[75421]: no user in db
Feb 14 16:53:43 acsvfbsd02 kernel: Feb 14 16:53:43 acsvfbsd02
sieve[75421]: client response doesn't match what we generated
Feb 14 16:53:48 acsvfbsd02 kernel: Feb 14 16:53:48 acsvfbsd02
sieve[75421]: Password verification failed
Feb 14 16:59:01 acsvfbsd02 kernel: Feb 14 16:59:01 acsvfbsd02
pop3[75464]: realm changed: authentication aborted

Any ideas?

Kind regards,



Attached my config files:

/usr/local/etc/cyrus.conf
==================

# standard standalone server implementation

START {
 # do not delete this entry!
 recover       cmd="ctl_cyrusdb -r"

 syncclient   cmd="/usr/local/cyrus/bin/sync_client -r"

 # this is only necessary if using idled for IMAP IDLE
 # idled               cmd="idled"
}

# UNIX sockets start with a slash and are put into /var/imap/socket
SERVICES {
 # add or remove based on preferences
 imap          cmd="imapd" listen="imap" prefork=0 provide_uuid=1
# imaps         cmd="imapd -s" listen="imaps" prefork=0
 pop3          cmd="pop3d" listen="pop3" prefork=0
# pop3s         cmd="pop3d -s" listen="pop3s" prefork=0
 sieve         cmd="timsieved" listen="sieve" prefork=0
#  syncserver    cmd="/usr/local/cyrus/bin/sync_server" listen="csync"
#  syncclient   cmd="/usr/local/cyrus/bin/sync_client -r"

 # these are only necessary if receiving/exporting usenet via NNTP
#  nntp         cmd="nntpd" listen="nntp" prefork=0
#  nntps                cmd="nntpd -s" listen="nntps" prefork=0

 # at least one LMTP is required for delivery
 lmtp          cmd="lmtpd" listen="lmtp" prefork=0
 lmtpunix      cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0

 # this is only necessary if using notifications
#  notify       cmd="notifyd" listen="/var/imap/socket/notify"
proto="udp" prefork=1

  fud cmd="/usr/local/cyrus/bin/fud" listen="fud" prefork=1 proto="udp"
}

EVENTS {
 # this is required
 checkpoint    cmd="ctl_cyrusdb -c" period=30

 # this is only necessary if using duplicate delivery suppression,
 # Sieve or NNTP
 delprune      cmd="cyr_expire -E 3" at=0400

 # this is only necessary if caching TLS sessions
 tlsprune      cmd="tls_prune" at=0400

 # SQUAT
 squatter      cmd="squatter -r user" period=1440


/usr/local/etc/imapd.conf
==================

configdirectory: /var/imap
partition-default: /var/spool/imap
admins:root cyrus

# sasl_pwcheck_method: auxprop

sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5


# sasl_sasldb_path: /usr/local/etc/sasldb2.db
sasl_log_level: 3

lmtpsocket:/var/imap/socket/lmtp

allowplaintext: yes

sync_host: my.host.ch
sync_authname: testuser
sync_password: blabla

sync_machineid: 1
sync_log: 1
sync_repeat_interval: 1



--
Martin Schweizer
schweizer.martin at gmail.com
Tel.: +41 32 512 48 54 (VoIP)
Fax: +1 619 3300587

From list at joreybump.com  Sat Feb 14 12:13:25 2009
From: list at joreybump.com (Jorey Bump)
Date: Sat, 14 Feb 2009 12:13:25 -0500
Subject: Problem with sieve login since update to 2.3.13
In-Reply-To: <380ccfd60902140830l4fc3e1eeo14a59c54c245024e@mail.gmail.com>
References: <380ccfd60902140823w36d1f4d0h406108023a17d600@mail.gmail.com>
	<380ccfd60902140830l4fc3e1eeo14a59c54c245024e@mail.gmail.com>
Message-ID: <4996FBB5.90805@joreybump.com>

Martin Schweizer wrote, at 02/14/2009 11:30 AM:

> Since the update to .13 I can no longer login to sieve as a regular
> user (also not by sieveshell). Only root can login by sieveshell. If I
> want to login as a regular user by sieveshell the password prompt
> cames up and I can type in the password, that is all. No login,
> nothing.

Add this line to imap.conf:

sieve_sasl_send_unsolicited_capability: 1


From list at joreybump.com  Sat Feb 14 12:32:02 2009
From: list at joreybump.com (Jorey Bump)
Date: Sat, 14 Feb 2009 12:32:02 -0500
Subject: Problem with sieve login since update to 2.3.13
In-Reply-To: <4996FBB5.90805@joreybump.com>
References: <380ccfd60902140823w36d1f4d0h406108023a17d600@mail.gmail.com>	<380ccfd60902140830l4fc3e1eeo14a59c54c245024e@mail.gmail.com>
	<4996FBB5.90805@joreybump.com>
Message-ID: <49970012.6060106@joreybump.com>

Jorey Bump wrote, at 02/14/2009 12:13 PM:
> Martin Schweizer wrote, at 02/14/2009 11:30 AM:
> 
>> Since the update to .13 I can no longer login to sieve as a regular
>> user (also not by sieveshell). Only root can login by sieveshell. If I
>> want to login as a regular user by sieveshell the password prompt
>> cames up and I can type in the password, that is all. No login,
>> nothing.
> 
> Add this line to imap.conf:
> 
> sieve_sasl_send_unsolicited_capability: 1

Sorry, obviously that should be imapd.conf. :P

From schweizer.martin at gmail.com  Sun Feb 15 12:13:32 2009
From: schweizer.martin at gmail.com (Martin Schweizer)
Date: Sun, 15 Feb 2009 18:13:32 +0100
Subject: Problem with sieve login since update to 2.3.13
In-Reply-To: <49970012.6060106@joreybump.com>
References: <380ccfd60902140823w36d1f4d0h406108023a17d600@mail.gmail.com>
	<380ccfd60902140830l4fc3e1eeo14a59c54c245024e@mail.gmail.com>
	<4996FBB5.90805@joreybump.com> <49970012.6060106@joreybump.com>
Message-ID: <380ccfd60902150913r7e485489m3fac0563923ff5c2@mail.gmail.com>

Hello Jorey

Thanks! It works now.

PS: Did I over readed something in the doc directory?

Regards,

2009/2/14 Jorey Bump <list at joreybump.com>:
> Jorey Bump wrote, at 02/14/2009 12:13 PM:
>> Martin Schweizer wrote, at 02/14/2009 11:30 AM:
>>
>>> Since the update to .13 I can no longer login to sieve as a regular
>>> user (also not by sieveshell). Only root can login by sieveshell. If I
>>> want to login as a regular user by sieveshell the password prompt
>>> cames up and I can type in the password, that is all. No login,
>>> nothing.
>>
>> Add this line to imap.conf:
>>
>> sieve_sasl_send_unsolicited_capability: 1
>
> Sorry, obviously that should be imapd.conf. :P
>



-- 
Martin Schweizer
schweizer.martin at gmail.com
Tel.: +41 32 512 48 54 (VoIP)
Fax: +1 619 3300587

From hans.moser at ofd-sth.niedersachsen.de  Wed Feb 18 04:16:01 2009
From: hans.moser at ofd-sth.niedersachsen.de (Marc Patermann)
Date: Wed, 18 Feb 2009 10:16:01 +0100
Subject: rpms for SLES 10
Message-ID: <499BD1D1.4040601@ofd-sth.niedersachsen.de>

Hi,

does anyone have a useful repository with latest IMAPd packages for SuSE 
Linux Enterprise Server 10 (SLES10)?
The official packages are still 2.2.x.
http://www.novell.com/products/linuxpackages/server10/i386/cyrus-imapd.html

On the SuSE build service I found nothing which is directly compiled 
on/with SLES, only OpenSUSE or "factory".


Marc

From dev.mansonthomas at gmail.com  Wed Feb 18 06:32:59 2009
From: dev.mansonthomas at gmail.com (Thomas Manson)
Date: Wed, 18 Feb 2009 12:32:59 +0100
Subject: UNSUBSCRIBE
Message-ID: <f43eb7e60902180332h4451f13dt761bfbb84537cfeb@mail.gmail.com>

UNSUBSCRIBE

From edwin.boersma at secureoffice.net  Wed Feb 18 10:49:23 2009
From: edwin.boersma at secureoffice.net (Edwin Boersma)
Date: Wed, 18 Feb 2009 16:49:23 +0100
Subject: virtdomains and defaultdomain issue
Message-ID: <499C2E03.7000409@secureoffice.net>

Hi,

To be able to have user names like <user>@<our.domain> and
<sameuser>@<another.domain>, I have changed our IMAP config to use
virtual domains. To be able to access the existing mailboxes, I added
the "defaultdomain" option to imapd.conf. The user names were then
renamed from <user> to <user>@<our.domain>, so I would think we could
login.
But we couldn't. We use a sql lookup for authentication and so does
Postfix to find the local users. Now, here I have found a problem.

What happens? At authentication, the (default) domain name appears to
get replaced with the *server's hostname* and the authentication fails.
If I change the user name to <user>@<server name>, it works. But then
Postfix requires the email address to be present in the virtual users
table, pointing to the local user name again. And this is only the case
for the default domain, not for foreign domains.

The server's local name should not be used here, as it is totally
arbitrary. It would make it impossible to e.g. migrate cyrus and Postfix
to another server.

(I hope you still follow)

Here's the imapd.conf:
configdirectory: /var/lib/imap
partition-default: /var/spool/imap
sievedir: /var/lib/sieve
admins: cyrus root

allowanonymouslogin: no
autocreate_users: anyone
autocreatequota: 1000000
reject8bit: no
quotawarn: 90
timeout: 30
poptimeout: 10
dracinterval: 0
drachost: localhost
allowplaintext: yes
lmtp_overquota_perm_failure: no
lmtp_downcase_rcpt: yes
createonpost: yes
unixhierarchysep: yes
virtdomains: yes
defaultdomain: secureoffice.net

sasl_pwcheck_method: auxprop
sasl_auxprop_plugin: sql
sasl_sql_engine: mysql
sasl_mech_list: login
sasl_sql_hostnames: localhost
sasl_sql_user: mail
sasl_sql_passwd: Love32Home15Should50Travel
sasl_sql_database: maildb
sasl_sql_verbose: yes
sasl_sql_select: SELECT clear AS password FROM users WHERE id='%u@%r'
AND Active='Y'

And this is from the logfile:
Feb 18 16:26:07 reindeer imap[14741]: sql plugin create statement from
userPassword edwin.boersma reindeer
Feb 18 16:26:07 reindeer imap[14741]: sql plugin doing query SELECT
clear AS password FROM users WHERE id='edwin.boersma at reindeer' AND
Active='Y';

My username is edwin.boersma at secureoffice.net, and the server's local
name is reindeer. Is there something wrong in my config, or is this
works-as-designed?

-- 
Kind regards,

Edwin Boersma
Lead Developer Web Applications

SecureOffice Europe AB
Ideon Science Park B2 floor 2
Scheelev?gen 17
22363 Lund
Sweden

W: http://www.secureoffice.net
T: +46 462868773
M: +46 709726431

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090218/dc25af65/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: edwin_boersma.vcf
Type: text/x-vcard
Size: 134 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090218/dc25af65/attachment.vcf 

From aspineux at gmail.com  Thu Feb 19 01:27:04 2009
From: aspineux at gmail.com (Alain Spineux)
Date: Thu, 19 Feb 2009 07:27:04 +0100
Subject: virtdomains and defaultdomain issue
In-Reply-To: <499C2E03.7000409@secureoffice.net>
References: <499C2E03.7000409@secureoffice.net>
Message-ID: <71fe4e760902182227s40bb0b3dwf4e2bee1471a2463@mail.gmail.com>

2009/2/18 Edwin Boersma <edwin.boersma at secureoffice.net>:
> Hi,
>
> To be able to have user names like <user>@<our.domain> and
> <sameuser>@<another.domain>, I have changed our IMAP config to use virtual
> domains. To be able to access the existing mailboxes, I added the
> "defaultdomain" option to imapd.conf.

It looks correct !

> The user names were then renamed from
> <user> to <user>@<our.domain>,

Which user ? Not in cyrus imap ! In SQL then ?
What happen if you don't rename it !

> so I would think we could login.
> But we couldn't. We use a sql lookup for authentication and so does Postfix
> to find the local users. Now, here I have found a problem.
>
> What happens? At authentication, the (default) domain name appears to get
> replaced with the server's hostname and the authentication fails. If I
> change the user name to <user>@<server name>, it works. But then Postfix
> requires the email address to be present in the virtual users table,
> pointing to the local user name again. And this is only the case for the
> default domain, not for foreign domains.
>
> The server's local name should not be used here, as it is totally arbitrary.
> It would make it impossible to e.g. migrate cyrus and Postfix to another
> server.
>
> (I hope you still follow)
>
> Here's the imapd.conf:
> configdirectory: /var/lib/imap
> partition-default: /var/spool/imap
> sievedir: /var/lib/sieve
> admins: cyrus root
>
> allowanonymouslogin: no
> autocreate_users: anyone
> autocreatequota: 1000000
> reject8bit: no
> quotawarn: 90
> timeout: 30
> poptimeout: 10
> dracinterval: 0
> drachost: localhost
> allowplaintext: yes
> lmtp_overquota_perm_failure: no
> lmtp_downcase_rcpt: yes
> createonpost: yes
> unixhierarchysep: yes
> virtdomains: yes
> defaultdomain: secureoffice.net
>
> sasl_pwcheck_method: auxprop
> sasl_auxprop_plugin: sql
> sasl_sql_engine: mysql
> sasl_mech_list: login
> sasl_sql_hostnames: localhost
> sasl_sql_user: mail
> sasl_sql_passwd: Love32Home15Should50Travel
> sasl_sql_database: maildb
> sasl_sql_verbose: yes
> sasl_sql_select: SELECT clear AS password FROM users WHERE id='%u@%r' AND
> Active='Y'
>
> And this is from the logfile:
> Feb 18 16:26:07 reindeer imap[14741]: sql plugin create statement from
> userPassword edwin.boersma reindeer
> Feb 18 16:26:07 reindeer imap[14741]: sql plugin doing query SELECT clear AS
> password FROM users WHERE id='edwin.boersma at reindeer' AND Active='Y';
>
> My username is edwin.boersma at secureoffice.net, and the server's local name
> is reindeer. Is there something wrong in my config, or is this
> works-as-designed?
>
> --
> Kind regards,
>
> Edwin Boersma
> Lead Developer Web Applications
>
> SecureOffice Europe AB
> Ideon Science Park B2 floor 2
> Scheelev?gen 17
> 22363 Lund
> Sweden
>
> W: http://www.secureoffice.net
> T: +46 462868773
> M: +46 709726431
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



-- 
Alain Spineux
aspineux gmail com
May the sources be with you

From shwaltz at cabm.rutgers.edu  Fri Feb 20 12:00:50 2009
From: shwaltz at cabm.rutgers.edu (Shelley Waltz)
Date: Fri, 20 Feb 2009 12:00:50 -0500 (EST)
Subject: Replication rolling or cron?
Message-ID: <48093.192.76.178.13.1235149250.squirrel@webmailnew.cabm.rutgers.edu>

I have cyrus-imapd-2.3.7 running on RHEL5 with a single replica server.
Replication works fine for a while, then after a day or two it dies.
Cannot find a failure in the logs.

I am using the rolling replication in impad.conf with
sync_authname: replica
sync_log: 1
sync_host: replica.edu
sync_repeat_interval: 30
sync_password: xxxx
sync_shutdown_file: /var/lib/imap/NOSYNC

I can get things working with a reboot, however, this is disruptive.
Question is - should I abandon rolling and just force a replication via a
cron job every 10 min?  I have low volume and only 200 accounts.

Does anyone have a monitoring script or a script to run sync_client by cron?
thanks
S


From andreas.moroder at gmx.net  Fri Feb 20 12:27:54 2009
From: andreas.moroder at gmx.net (andreas moroder)
Date: Fri, 20 Feb 2009 18:27:54 +0100
Subject: imp webmail, cyrus imap and virus filtering
Message-ID: <499EE81A.1070602@gmx.net>

Hello,

we have a imp/postfix/cyrus system. The mail that goes out and comes in
from/to our system via smtp is checked by trendmicros virus wall.

The problem is mail that remains in our mailsystem and is accessed via
imap from imp. Viruswall has no way to check this mail.

Is it possible to write a sieve filter to start a virusscan on this mail
( by scanning the specific file in the users mail directory ? ) every
time a mail is created or read via imap ?

Thanks
Andreas


From ian.batten at uk.fujitsu.com  Fri Feb 20 12:31:02 2009
From: ian.batten at uk.fujitsu.com (Ian Batten)
Date: Fri, 20 Feb 2009 17:31:02 +0000
Subject: imp webmail, cyrus imap and virus filtering
In-Reply-To: <499EE81A.1070602@gmx.net>
References: <499EE81A.1070602@gmx.net>
Message-ID: <390C0676-D12E-40E7-8B9B-85388130B6DD@uk.fujitsu.com>


On 20 Feb 09, at 1727, andreas moroder wrote:

> Hello,
>
> we have a imp/postfix/cyrus system. The mail that goes out and comes  
> in
> from/to our system via smtp is checked by trendmicros virus wall.
>
> The problem is mail that remains in our mailsystem and is accessed via
> imap from imp. Viruswall has no way to check this mail.
>
> Is it possible to write a sieve filter to start a virusscan on this  
> mail
> ( by scanning the specific file in the users mail directory ? ) every
> time a mail is created or read via imap ?

You need to integrate virus scanning into your postfix environment.   
Mail sent between users of your IMP environment presumably is handed  
off to postfix and thence to lmtpd or deliver, and if your virus  
scanning were performed there you'd be sorted.  If your virus scanning  
is instead an SMTP relay outside your internal system then you have a  
problem.

We use clamav-milter alongside every sendmail instance and therefore  
scan every piece of mail at every point of handling.

ian


From aspineux at gmail.com  Fri Feb 20 17:41:00 2009
From: aspineux at gmail.com (Alain Spineux)
Date: Fri, 20 Feb 2009 23:41:00 +0100
Subject: imp webmail, cyrus imap and virus filtering
In-Reply-To: <499EE81A.1070602@gmx.net>
References: <499EE81A.1070602@gmx.net>
Message-ID: <71fe4e760902201441i794a6d90p5e5dc04b721b121e@mail.gmail.com>

On Fri, Feb 20, 2009 at 6:27 PM, andreas moroder
<andreas.moroder at gmx.net> wrote:
> Hello,
>
> we have a imp/postfix/cyrus system. The mail that goes out and comes in
> from/to our system via smtp is checked by trendmicros virus wall.
>
> The problem is mail that remains in our mailsystem and is accessed via
> imap from imp. Viruswall has no way to check this mail.


You mean mail already already in your INBOXes received before you have
installed your trendmicros filter, or mail sent internally by your
user ?

In the last case the simple solution is to ask your user to send email
directly to your trendmicro !
If this is not possible you can configure your trendmicros as a filter
for your postfix !
But if you want keep your trendmicro in front for your incoming email,
and have postfix in front for your local users, this is an unusual
configuration, ask the postfix mailing list for information to do that
!

Regards


>
> Is it possible to write a sieve filter to start a virusscan on this mail
> ( by scanning the specific file in the users mail directory ? ) every
> time a mail is created or read via imap ?
>
> Thanks
> Andreas
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



-- 
Alain Spineux
aspineux gmail com
May the sources be with you

From wes at umich.edu  Fri Feb 20 22:21:38 2009
From: wes at umich.edu (Wesley Craig)
Date: Fri, 20 Feb 2009 22:21:38 -0500
Subject: Replication rolling or cron?
In-Reply-To: <48093.192.76.178.13.1235149250.squirrel@webmailnew.cabm.rutgers.edu>
References: <48093.192.76.178.13.1235149250.squirrel@webmailnew.cabm.rutgers.edu>
Message-ID: <64AFC65A-D827-4F09-AE9E-8D36911BE746@umich.edu>

On 20 Feb 2009, at 12:00, Shelley Waltz wrote:
> I can get things working with a reboot, however, this is disruptive.
> Question is - should I abandon rolling and just force a replication  
> via a
> cron job every 10 min?  I have low volume and only 200 accounts.
>
> Does anyone have a monitoring script or a script to run sync_client  
> by cron?

Running sync_client out of cyrus.conf is a mistake.  It can be  
started stand-alone, and that's the best way to do it.  If it dies,  
review why it died, fix whatever problems the review reveals, and  
restart just sync_client.  Rolling replication is better than simply  
running sync_client from cron.

My other bit of advice is to upgrade.  2.3.7 has a lot of replication  
problems.  Many, many of the changes that have happened since 2.3.7  
are improvements to replication.

:wes

From brong at fastmail.fm  Sat Feb 21 02:28:49 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Sat, 21 Feb 2009 18:28:49 +1100
Subject: Replication rolling or cron?
In-Reply-To: <64AFC65A-D827-4F09-AE9E-8D36911BE746@umich.edu>
References: <48093.192.76.178.13.1235149250.squirrel@webmailnew.cabm.rutgers.edu>
	<64AFC65A-D827-4F09-AE9E-8D36911BE746@umich.edu>
Message-ID: <20090221072849.GA12552@brong.net>

On Fri, Feb 20, 2009 at 10:21:38PM -0500, Wesley Craig wrote:
> On 20 Feb 2009, at 12:00, Shelley Waltz wrote:
> > I can get things working with a reboot, however, this is disruptive.
> > Question is - should I abandon rolling and just force a replication  
> > via a
> > cron job every 10 min?  I have low volume and only 200 accounts.
> >
> > Does anyone have a monitoring script or a script to run sync_client  
> > by cron?
> 
> Running sync_client out of cyrus.conf is a mistake.  It can be  
> started stand-alone, and that's the best way to do it.  If it dies,  
> review why it died, fix whatever problems the review reveals, and  
> restart just sync_client.  Rolling replication is better than simply  
> running sync_client from cron.

We should probably document that...

(at FastMail we start it from cyrus.conf, but also have a job that
runs from cron and checks that there's a sync_client running every 10
minutes.  It also runs any log_$pid files that are lying around in
the sync directory without a matching pid (to clean up from failures)
and emails us about anything it had to do.  We run a separate task that
counts up the number of bytes in all files in the sync directory and
emails us if it goes over 100kb - about 15 minutes worth, and shows
up in status commands if it gets above 10kb)
 
> My other bit of advice is to upgrade.  2.3.7 has a lot of replication  
> problems.  Many, many of the changes that have happened since 2.3.7  
> are improvements to replication.

Yes, definitely, and there will probably be a 2.3.14 soonish, once I
get around to harassing Ken about it and double-checking the bugzilla
for blockers.

(I don't consider non-regression bugs to be a release blocker.  We
can always do another release once they're fixed)

Bron.

From cl at 1984.is  Sun Feb 22 19:18:09 2009
From: cl at 1984.is (Mordur Ingolfsson)
Date: Mon, 23 Feb 2009 00:18:09 +0000
Subject: Admin user cyrus not seeing mailboxes
Message-ID: <49A1EB41.7000905@1984.is>

Dear list,

I have a cyrus imapd 2.3.8-1 running on Debian GNU/Linux. I have in
/etc/imapd.conf the following :

admins: cyrus

 to be able to use the cyradm interface to perform administrative tasks.
I am able to log into cyradm with the user cyrus's credentials, but I
when I issue listmailbox I only get a listing of the cyrus users
mailboxes (which get created on logon) but not of all the other users
mailboxes, which is what I need. 

I have another, older installantion of Cyrus, a 2.2.13-3 version where
the behavior of cyradm is the way I expected it to be on the new Cyrus
installation, so I am at a loss here. Is there a difference in the
behaviour of cyradm between these versions that explains this?

Thanks,

Mordur Ingolfsson

From ian.batten at uk.fujitsu.com  Mon Feb 23 04:43:08 2009
From: ian.batten at uk.fujitsu.com (Ian Batten)
Date: Mon, 23 Feb 2009 09:43:08 +0000
Subject: Replication rolling or cron?
In-Reply-To: <20090221072849.GA12552@brong.net>
References: <48093.192.76.178.13.1235149250.squirrel@webmailnew.cabm.rutgers.edu>
	<64AFC65A-D827-4F09-AE9E-8D36911BE746@umich.edu>
	<20090221072849.GA12552@brong.net>
Message-ID: <535ADDAF-12B2-4AA5-9FF2-ED7D48DD93D9@uk.fujitsu.com>

>
> (at FastMail we start it from cyrus.conf, but also have a job that
> runs from cron and checks that there's a sync_client running every 10
> minutes.

It also works well with SMF if you're a Solaris site.

$ svcs -x sync_client
svc:/network/sync_client:default (?)
  State: online since Thu Feb 12 22:53:26 2009
    See: /var/svc/log/network-sync_client:default.log
Impact: None.
$

That way it restarts on failure.

ian

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/ 
service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
   <service name='network/sync_client' type='service' version='0'>
     <dependency name='filesystem_minimal' grouping='require_all'  
restart_on='none' type='service'>
       <service_fmri value='svc:/system/filesystem/local'/>
     </dependency>
     <dependency name='loopback' grouping='require_any'  
restart_on='error' type='service'>
       <service_fmri value='svc:/network/loopback'/>
     </dependency>
     <dependency name='network' grouping='optional_all'  
restart_on='error' type='service'>
       <service_fmri value='svc:/milestone/network'/>
     </dependency>
     <dependency name='cyrus' grouping='require_all'  
restart_on='error' type='service'>
       <service_fmri value='svc:/network/cyrus:default'/>
     </dependency>
     <exec_method name='stop' type='method' exec=':kill'  
timeout_seconds='60'>
       <method_context/>
     </exec_method>
     <property_group name='tm_common_name' type='template'/>
     <instance name='default' enabled='true'>
       <exec_method name='start' type='method' exec='/opt/cyrus/bin/ 
sync_client -r' timeout_seconds='60'>
         <method_context project='cyrus'>
           <method_credential user='cyrus' group='mail'/>
         </method_context>
       </exec_method>
     </instance>
     <stability value='Unstable'/>
   </service>
</service_bundle>


From brong at fastmail.fm  Mon Feb 23 06:33:25 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Mon, 23 Feb 2009 22:33:25 +1100
Subject: Replication rolling or cron?
In-Reply-To: <535ADDAF-12B2-4AA5-9FF2-ED7D48DD93D9@uk.fujitsu.com>
References: <48093.192.76.178.13.1235149250.squirrel@webmailnew.cabm.rutgers.edu>
	<64AFC65A-D827-4F09-AE9E-8D36911BE746@umich.edu>
	<20090221072849.GA12552@brong.net>
	<535ADDAF-12B2-4AA5-9FF2-ED7D48DD93D9@uk.fujitsu.com>
Message-ID: <20090223113325.GB7971@brong.net>

On Mon, Feb 23, 2009 at 09:43:08AM +0000, Ian Batten wrote:
>>
>> (at FastMail we start it from cyrus.conf, but also have a job that
>> runs from cron and checks that there's a sync_client running every 10
>> minutes.
>
> It also works well with SMF if you're a Solaris site.
>
> $ svcs -x sync_client
> svc:/network/sync_client:default (?)
>  State: online since Thu Feb 12 22:53:26 2009
>    See: /var/svc/log/network-sync_client:default.log
> Impact: None.
> $
>
> That way it restarts on failure.

Or good old "init" on pretty much anything unixy.

Our monitoring script also scans for other issues though,
and retries the failed log files of past sync_clients.
I like layers of stuff checking on each other!

Bron.

From edwin.boersma at secureoffice.net  Mon Feb 23 07:43:20 2009
From: edwin.boersma at secureoffice.net (Edwin Boersma)
Date: Mon, 23 Feb 2009 13:43:20 +0100
Subject: virtdomains and defaultdomain issue
In-Reply-To: <71fe4e760902182227s40bb0b3dwf4e2bee1471a2463@mail.gmail.com>
References: <499C2E03.7000409@secureoffice.net>
	<71fe4e760902182227s40bb0b3dwf4e2bee1471a2463@mail.gmail.com>
Message-ID: <49A299E8.4050506@secureoffice.net>

Hi,

Just to make it clear: the problem only occurs with the default domain,
not with other virtual domains. All user are in the SQL database, and
cyrus does a correct translation to the mailbox for all the others. The
only problem is that the default domain is replaced with the local
computer name.

I made another user without domain extension, and then I can't login.
Cyrus tries to lookup the user with the domain extension from %r. So
only if I have a user "user@<local computer name>, i can login to imap.
But then Postfix requires an alias from <user>@default.domain to <user>,
else it won't accept any email.

Why is this such a big problem? It is, because the database is fed with
information from a script, where we add users to the mail system. Now I
have to tell the script to perform special actions when the user is in
the default domain, and this is not a good situation.

In my opinion (can you give me yours, Andrew?), cyrus should not rewrite
the default domain when using %r, but internally redirect to the local
mailbox (so after login). Or provide a mechanism where the local mailbox
is transformed into a virtual domain box.

Kind regards,

Edwin Boersma
Lead Developer Web Applications

SecureOffice Europe AB
Ideon Science Park B2 floor 2
Scheelev?gen 17
22363 Lund
Sweden

W: http://www.secureoffice.net
T: +46 462868773
M: +46 709726431



Alain Spineux wrote:
> 2009/2/18 Edwin Boersma <edwin.boersma at secureoffice.net>:
>   
>> Hi,
>>
>> To be able to have user names like <user>@<our.domain> and
>> <sameuser>@<another.domain>, I have changed our IMAP config to use virtual
>> domains. To be able to access the existing mailboxes, I added the
>> "defaultdomain" option to imapd.conf.
>>     
>
> It looks correct !
>
>   
>> The user names were then renamed from
>> <user> to <user>@<our.domain>,
>>     
>
> Which user ? Not in cyrus imap ! In SQL then ?
> What happen if you don't rename it !
>
>   
>> so I would think we could login.
>> But we couldn't. We use a sql lookup for authentication and so does Postfix
>> to find the local users. Now, here I have found a problem.
>>
>> What happens? At authentication, the (default) domain name appears to get
>> replaced with the server's hostname and the authentication fails. If I
>> change the user name to <user>@<server name>, it works. But then Postfix
>> requires the email address to be present in the virtual users table,
>> pointing to the local user name again. And this is only the case for the
>> default domain, not for foreign domains.
>>
>> The server's local name should not be used here, as it is totally arbitrary.
>> It would make it impossible to e.g. migrate cyrus and Postfix to another
>> server.
>>
>> (I hope you still follow)
>>
>> Here's the imapd.conf:
>> configdirectory: /var/lib/imap
>> partition-default: /var/spool/imap
>> sievedir: /var/lib/sieve
>> admins: cyrus root
>>
>> allowanonymouslogin: no
>> autocreate_users: anyone
>> autocreatequota: 1000000
>> reject8bit: no
>> quotawarn: 90
>> timeout: 30
>> poptimeout: 10
>> dracinterval: 0
>> drachost: localhost
>> allowplaintext: yes
>> lmtp_overquota_perm_failure: no
>> lmtp_downcase_rcpt: yes
>> createonpost: yes
>> unixhierarchysep: yes
>> virtdomains: yes
>> defaultdomain: secureoffice.net
>>
>> sasl_pwcheck_method: auxprop
>> sasl_auxprop_plugin: sql
>> sasl_sql_engine: mysql
>> sasl_mech_list: login
>> sasl_sql_hostnames: localhost
>> sasl_sql_user: mail
>> sasl_sql_passwd: xxxxxxxxxxxxxxxxxxxxxxx
>> sasl_sql_database: maildb
>> sasl_sql_verbose: yes
>> sasl_sql_select: SELECT clear AS password FROM users WHERE id='%u@%r' AND
>> Active='Y'
>>
>> And this is from the logfile:
>> Feb 18 16:26:07 reindeer imap[14741]: sql plugin create statement from
>> userPassword edwin.boersma reindeer
>> Feb 18 16:26:07 reindeer imap[14741]: sql plugin doing query SELECT clear AS
>> password FROM users WHERE id='edwin.boersma at reindeer' AND Active='Y';
>>
>> My username is edwin.boersma at secureoffice.net, and the server's local name
>> is reindeer. Is there something wrong in my config, or is this
>> works-as-designed?
>>
>> --
>> Kind regards,
>>
>> Edwin Boersma
>> Lead Developer Web Applications
>>
>> SecureOffice Europe AB
>> Ideon Science Park B2 floor 2
>> Scheelev?gen 17
>> 22363 Lund
>> Sweden
>>
>> W: http://www.secureoffice.net
>> T: +46 462868773
>> M: +46 709726431
>>
>> ----
>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>
>>     
>
>
>
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090223/5f1e6621/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: edwin_boersma.vcf
Type: text/x-vcard
Size: 134 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090223/5f1e6621/attachment.vcf 

From list at joreybump.com  Mon Feb 23 08:14:56 2009
From: list at joreybump.com (Jorey Bump)
Date: Mon, 23 Feb 2009 08:14:56 -0500
Subject: virtdomains and defaultdomain issue
In-Reply-To: <49A299E8.4050506@secureoffice.net>
References: <499C2E03.7000409@secureoffice.net>	<71fe4e760902182227s40bb0b3dwf4e2bee1471a2463@mail.gmail.com>
	<49A299E8.4050506@secureoffice.net>
Message-ID: <49A2A150.80106@joreybump.com>

Edwin Boersma wrote, at 02/23/2009 07:43 AM:
> Hi,
> 
> Just to make it clear: the problem only occurs with the default domain,
> not with other virtual domains. All user are in the SQL database, and
> cyrus does a correct translation to the mailbox for all the others. The
> only problem is that the default domain is replaced with the local
> computer name.
[snip]
> In my opinion (can you give me yours, Andrew?), cyrus should not rewrite
> the default domain when using %r, but internally redirect to the local
> mailbox (so after login). Or provide a mechanism where the local mailbox
> is transformed into a virtual domain box.


>> 2009/2/18 Edwin Boersma <edwin.boersma at secureoffice.net>:
>>   
>>> Hi,
>>>
>>> To be able to have user names like <user>@<our.domain> and
>>> <sameuser>@<another.domain>, I have changed our IMAP config to use virtual
>>> domains. To be able to access the existing mailboxes, I added the
>>> "defaultdomain" option to imapd.conf.

You will probably also want to set servername to prevent cyrus from
using gethostname:

>>> Here's the imapd.conf:

>>> defaultdomain: secureoffice.net

  servername: secureoffice.net


Is there a problem you are trying to solve with user at domain logins? In
most cases, this is done to support similar logins across multiple
domains (support at example.com, support at example.net, etc.). However, I
find that this confuses clients, who will try to use alias addresses as
logins, and prefer to assign unique logins across all domains
(foosupport, barsupport, etc.). This way, I don't need to enable
virtdomains in Cyrus IMAPd, and just put everyone in the same realm (a
single arbitrary domain, it doesn't even need to exist in DNS or accept
email). Then I set defaultdomain and servername to that realm in
imapd.conf along with smtpd_sasl_local_domain in the Postfix main.cf. As
a result, all lookups are done against this single realm and users can
authenticate with a bare login without appending the realm. This
approach still supports multiple email domains, but simplifies
configuration and may even improve portability (but I'm using sasldb,
not SQL, so there may be other issues I'm not considering). The only
caveat is that all logins must be unique; two users of different
accounts can't each login as "support". On the other hand, this
arrangement has come in handy when we've had to replace heavily spammed
public addresses like info at example.com with information at example.com,
because it isn't necessary to change login credentials in the client. I
only mention this as an alternative, in case you really don't need to
support full user at domain logins.



From edwin.boersma at secureoffice.net  Mon Feb 23 08:31:11 2009
From: edwin.boersma at secureoffice.net (Edwin Boersma)
Date: Mon, 23 Feb 2009 14:31:11 +0100
Subject: virtdomains and defaultdomain issue
In-Reply-To: <49A2A150.80106@joreybump.com>
References: <499C2E03.7000409@secureoffice.net>	<71fe4e760902182227s40bb0b3dwf4e2bee1471a2463@mail.gmail.com>
	<49A299E8.4050506@secureoffice.net> <49A2A150.80106@joreybump.com>
Message-ID: <49A2A51F.8050207@secureoffice.net>

Thanks, Jorey, this was the missing link! Now it works as I expected it
to (good work, Andrew!).

Kind regards,

Edwin Boersma
Lead Developer Web Applications

SecureOffice Europe AB
Ideon Science Park B2 floor 2
Scheelev?gen 17
22363 Lund
Sweden

W: http://www.secureoffice.net
T: +46 462868773
M: +46 709726431



Jorey Bump wrote:
> Edwin Boersma wrote, at 02/23/2009 07:43 AM:
>   
>> Hi,
>>
>> Just to make it clear: the problem only occurs with the default domain,
>> not with other virtual domains. All user are in the SQL database, and
>> cyrus does a correct translation to the mailbox for all the others. The
>> only problem is that the default domain is replaced with the local
>> computer name.
>>     
> [snip]
>   
>> In my opinion (can you give me yours, Andrew?), cyrus should not rewrite
>> the default domain when using %r, but internally redirect to the local
>> mailbox (so after login). Or provide a mechanism where the local mailbox
>> is transformed into a virtual domain box.
>>     
>
>
>   
>>> 2009/2/18 Edwin Boersma <edwin.boersma at secureoffice.net>:
>>>   
>>>       
>>>> Hi,
>>>>
>>>> To be able to have user names like <user>@<our.domain> and
>>>> <sameuser>@<another.domain>, I have changed our IMAP config to use virtual
>>>> domains. To be able to access the existing mailboxes, I added the
>>>> "defaultdomain" option to imapd.conf.
>>>>         
>
> You will probably also want to set servername to prevent cyrus from
> using gethostname:
>
>   
>>>> Here's the imapd.conf:
>>>>         
>
>   
>>>> defaultdomain: secureoffice.net
>>>>         
>
>   servername: secureoffice.net
>
>
> Is there a problem you are trying to solve with user at domain logins? In
> most cases, this is done to support similar logins across multiple
> domains (support at example.com, support at example.net, etc.). However, I
> find that this confuses clients, who will try to use alias addresses as
> logins, and prefer to assign unique logins across all domains
> (foosupport, barsupport, etc.). This way, I don't need to enable
> virtdomains in Cyrus IMAPd, and just put everyone in the same realm (a
> single arbitrary domain, it doesn't even need to exist in DNS or accept
> email). Then I set defaultdomain and servername to that realm in
> imapd.conf along with smtpd_sasl_local_domain in the Postfix main.cf. As
> a result, all lookups are done against this single realm and users can
> authenticate with a bare login without appending the realm. This
> approach still supports multiple email domains, but simplifies
> configuration and may even improve portability (but I'm using sasldb,
> not SQL, so there may be other issues I'm not considering). The only
> caveat is that all logins must be unique; two users of different
> accounts can't each login as "support". On the other hand, this
> arrangement has come in handy when we've had to replace heavily spammed
> public addresses like info at example.com with information at example.com,
> because it isn't necessary to change login credentials in the client. I
> only mention this as an alternative, in case you really don't need to
> support full user at domain logins.
>
>
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090223/0fd6930b/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: edwin_boersma.vcf
Type: text/x-vcard
Size: 134 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090223/0fd6930b/attachment-0001.vcf 

From abrink at gmail.com  Mon Feb 23 22:27:44 2009
From: abrink at gmail.com (Andrew Brink)
Date: Mon, 23 Feb 2009 21:27:44 -0600
Subject: Sieve Errors
Message-ID: <89e7a5860902231927l3960d301q5045ad8db3925eeb@mail.gmail.com>

All -

I have been running 2.2.11 in production for over a year now.  I recently
found the need to enable sieve for vacation support.  After enabling, these
are my log messages:

Feb 23 20:04:17 mail master[18638]: about to exec /usr/cyrus/bin/timsieved
Feb 23 20:04:17 mail sieve[18638]: incorrect version of Berkeley db:
compiled against 4.1.25, linked against 4.3.29
Feb 23 20:04:17 mail master[12029]: process 18638 exited, signaled to death
by 11
Feb 23 20:04:17 mail master[12029]: service sieve pid 18637 in READY state:
terminated abnormally
Please let me know if there is an easy way to fix.  I am slighly confused
about the BDB message, as all other deamons function correctly.

Thanks,

Andrew Brink
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090223/df04225f/attachment.html 

From peo at intersonic.se  Tue Feb 24 02:39:55 2009
From: peo at intersonic.se (Per olof Ljungmark)
Date: Tue, 24 Feb 2009 08:39:55 +0100
Subject: Replication rolling or cron?
In-Reply-To: <20090221072849.GA12552@brong.net>
References: <48093.192.76.178.13.1235149250.squirrel@webmailnew.cabm.rutgers.edu>	<64AFC65A-D827-4F09-AE9E-8D36911BE746@umich.edu>
	<20090221072849.GA12552@brong.net>
Message-ID: <49A3A44B.8020706@intersonic.se>



Bron Gondwana wrote:
> On Fri, Feb 20, 2009 at 10:21:38PM -0500, Wesley Craig wrote:
>> On 20 Feb 2009, at 12:00, Shelley Waltz wrote:
>>> I can get things working with a reboot, however, this is disruptive.
>>> Question is - should I abandon rolling and just force a replication  
>>> via a
>>> cron job every 10 min?  I have low volume and only 200 accounts.
>>>
>>> Does anyone have a monitoring script or a script to run sync_client  
>>> by cron?
>> Running sync_client out of cyrus.conf is a mistake.  It can be  
>> started stand-alone, and that's the best way to do it.  If it dies,  
>> review why it died, fix whatever problems the review reveals, and  
>> restart just sync_client.  Rolling replication is better than simply  
>> running sync_client from cron.
> 
> We should probably document that...

Aha! We are tricked!

install-replication.html:

Add a line similar to the following in the STARTUP section of cyrus.conf:
  syncclient       cmd="/usr/cyrus/bin/sync_client -r"

I thought I was a regular reader of the mailing list but I sure missed
this one, bet there are a bunch of people running rolling replication
out of cyrus.conf.

That said, it's been working just fine...

Cheers,

--
per

From cl at 1984.is  Tue Feb 24 09:51:32 2009
From: cl at 1984.is (Mordur Ingolfsson)
Date: Tue, 24 Feb 2009 14:51:32 +0000
Subject: Admin user cyrus not seeing mailboxes
In-Reply-To: <ac90d4f80902230908g243259bdgc2a475989b3e5abf@mail.gmail.com>
References: <49A1EB41.7000905@1984.is>
	<ac90d4f80902230908g243259bdgc2a475989b3e5abf@mail.gmail.com>
Message-ID: <49A40974.1000307@1984.is>

brian wrote:
> On Sun, Feb 22, 2009 at 7:18 PM, Mordur Ingolfsson <cl at 1984.is> wrote:
>   
>> Dear list,
>>
>> I have a cyrus imapd 2.3.8-1 running on Debian GNU/Linux. I have in
>> /etc/imapd.conf the following :
>>
>> admins: cyrus
>>
>>  to be able to use the cyradm interface to perform administrative tasks.
>> I am able to log into cyradm with the user cyrus's credentials, but I
>> when I issue listmailbox I only get a listing of the cyrus users
>> mailboxes (which get created on logon) but not of all the other users
>> mailboxes, which is what I need.
>>
>> I have another, older installantion of Cyrus, a 2.2.13-3 version where
>> the behavior of cyradm is the way I expected it to be on the new Cyrus
>> installation, so I am at a loss here. Is there a difference in the
>> behaviour of cyradm between these versions that explains this?
>>
>>     
>
> These mailboxes were copied from another machine? Check the group
> ownership of the files in your spool dir. Also, perhaps the ownership
> is set to UIDs, not 'cyrus' and they don't match your current cyrus
>   
> user.
>   
Dear Brian,

Thank you for your reply.

The mailboxes were not copied from another machine, the were (and are
still being) created by cyrus on this machine. The ownership of
/var/spool/cyrus/....  is recursively user cyrus and group mail, exactly
like on the older installation.

Still have not figured this out.

Thank


From pobox at verysmall.org  Tue Feb 24 17:56:18 2009
From: pobox at verysmall.org (Iv Ray)
Date: Tue, 24 Feb 2009 23:56:18 +0100
Subject: initial slowness
Message-ID: <EFA6B8CC-D282-444A-A592-A4FA96DD0F1A@verysmall.org>

Hello everyone,

A few weeks ago I wrote to the list about slow Cyrus IMAP, and I  
received a couple of answers, however I am still struggling.

What I noticed now is that even the command line login -

cyradm -u cyrus localhost

takes about 25 seconds, after the password is entered, to show the  
localhost> prompt (and the server load is close to zero).

I did reconstruct several times and once quota -f, but this does not  
seem to help.

I read about rebuilding of the mailboxes.db - but I am a bit scared.

The server is running FreeBSD 6.x with latest Cyrus IMAP and has,  
usually, load close to zero.

I also noticed that the add/edit mailbox operations done with a php  
script take very long (perhaps around 25 seconds, as well).

Some hints what might be wrong and/or steps to clean up/improve the  
situation will be greatly appreciated!

Many thanks,
Iv



From morgan at orst.edu  Tue Feb 24 18:02:54 2009
From: morgan at orst.edu (Andrew Morgan)
Date: Tue, 24 Feb 2009 15:02:54 -0800 (PST)
Subject: initial slowness
In-Reply-To: <EFA6B8CC-D282-444A-A592-A4FA96DD0F1A@verysmall.org>
References: <EFA6B8CC-D282-444A-A592-A4FA96DD0F1A@verysmall.org>
Message-ID: <alpine.DEB.2.00.0902241502020.19740@shell.onid.oregonstate.edu>

On Tue, 24 Feb 2009, Iv Ray wrote:

> Hello everyone,
>
> A few weeks ago I wrote to the list about slow Cyrus IMAP, and I
> received a couple of answers, however I am still struggling.
>
> What I noticed now is that even the command line login -
>
> cyradm -u cyrus localhost
>
> takes about 25 seconds, after the password is entered, to show the
> localhost> prompt (and the server load is close to zero).
>
> I did reconstruct several times and once quota -f, but this does not
> seem to help.
>
> I read about rebuilding of the mailboxes.db - but I am a bit scared.
>
> The server is running FreeBSD 6.x with latest Cyrus IMAP and has,
> usually, load close to zero.
>
> I also noticed that the add/edit mailbox operations done with a php
> script take very long (perhaps around 25 seconds, as well).
>
> Some hints what might be wrong and/or steps to clean up/improve the
> situation will be greatly appreciated!

Are all operations slow, or only authentication?

Your best tool for testing is imtest.  You can issue raw IMAP commands and 
see how long they take.

 	Andy

From raymond at sundland.com  Tue Feb 24 18:28:40 2009
From: raymond at sundland.com (Raymond T. Sundland)
Date: Tue, 24 Feb 2009 18:28:40 -0500
Subject: initial slowness
In-Reply-To: <EFA6B8CC-D282-444A-A592-A4FA96DD0F1A@verysmall.org>
References: <EFA6B8CC-D282-444A-A592-A4FA96DD0F1A@verysmall.org>
Message-ID: <49A482A8.4020303@sundland.com>

I have two possible fixes, not sure if either will work...

The first possible is if your system is configured for IPv6, but Cyrus 
is not opening a port on any IPv6 addresses, if you just type 
'localhost', it will likely try the IPv6 address first and the timeout 
is probably in the 20-25 second range.

The other option I ran into some time ago... and it had to due to 
defining multiple authentication schemes in the SASL configuration.  
When you define multiple, it will try each one in order.  I stripped it 
down to just PLAIN, which is what I wanted to use, and it was much 
quicker.  The cyradm script will prompt for the password before actually 
trying to connect, which is why it's slow AFTER the password is entered.

Iv Ray wrote:
> Hello everyone,
>
> A few weeks ago I wrote to the list about slow Cyrus IMAP, and I  
> received a couple of answers, however I am still struggling.
>
> What I noticed now is that even the command line login -
>
> cyradm -u cyrus localhost
>
> takes about 25 seconds, after the password is entered, to show the  
> localhost> prompt (and the server load is close to zero).
>
> I did reconstruct several times and once quota -f, but this does not  
> seem to help.
>
> I read about rebuilding of the mailboxes.db - but I am a bit scared.
>
> The server is running FreeBSD 6.x with latest Cyrus IMAP and has,  
> usually, load close to zero.
>
> I also noticed that the add/edit mailbox operations done with a php  
> script take very long (perhaps around 25 seconds, as well).
>
> Some hints what might be wrong and/or steps to clean up/improve the  
> situation will be greatly appreciated!
>
> Many thanks,
> Iv
>
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>   

From pobox at verysmall.org  Tue Feb 24 19:14:15 2009
From: pobox at verysmall.org (Iv Ray)
Date: Wed, 25 Feb 2009 01:14:15 +0100
Subject: initial slowness
In-Reply-To: <alpine.DEB.2.00.0902241502020.19740@shell.onid.oregonstate.edu>
References: <EFA6B8CC-D282-444A-A592-A4FA96DD0F1A@verysmall.org>
	<alpine.DEB.2.00.0902241502020.19740@shell.onid.oregonstate.edu>
Message-ID: <BC1D8E10-803B-4333-9BFF-D27BCE13D428@verysmall.org>


On 25.02.2009, at 00:02, Andrew Morgan wrote:

> Are all operations slow, or only authentication?

Only the coming of the localhost> prompt.

I assume this is the authentication...?

> Your best tool for testing is imtest.  You can issue raw IMAP  
> commands and see how long they take.

Oh, I'll have a look.

Thanks a lot,
Iv

From pobox at verysmall.org  Tue Feb 24 19:15:32 2009
From: pobox at verysmall.org (Iv Ray)
Date: Wed, 25 Feb 2009 01:15:32 +0100
Subject: initial slowness
In-Reply-To: <49A482A8.4020303@sundland.com>
References: <EFA6B8CC-D282-444A-A592-A4FA96DD0F1A@verysmall.org>
	<49A482A8.4020303@sundland.com>
Message-ID: <1075531C-41AB-4610-9A52-7CF95EAD3977@verysmall.org>

On 25.02.2009, at 00:28, Raymond T. Sundland wrote:

> I have two possible fixes, not sure if either will work...
>
> The first possible is if your system is configured for IPv6, but  
> Cyrus is not opening a port on any IPv6 addresses, if you just type  
> 'localhost', it will likely try the IPv6 address first and the  
> timeout is probably in the 20-25 second range.
>
> The other option I ran into some time ago... and it had to due to  
> defining multiple authentication schemes in the SASL configuration.   
> When you define multiple, it will try each one in order.  I stripped  
> it down to just PLAIN, which is what I wanted to use, and it was  
> much quicker.  The cyradm script will prompt for the password before  
> actually trying to connect, which is why it's slow AFTER the  
> password is entered.

Raymond,

Thanks a lot for the hints.

I'll try them and write back if success.

Iv

From jvoorhees1 at gmail.com  Wed Feb 25 17:45:25 2009
From: jvoorhees1 at gmail.com (Jason Voorhees)
Date: Wed, 25 Feb 2009 17:45:25 -0500
Subject: How to copy messages between mailboxes?
Message-ID: <c28863110902251445p2ed6224r7f30cbfe800e6871@mail.gmail.com>

Hi people:

I'm trying to create a script to copy "certain" messages from a
mailbox of user A to mailbox of user B. These are the steps I have in
mind:

1. Select what messages to copy from mailbox of user A. They can
message files located at different folders within its mailbox.

2. Choose where to copy messages in mailbox of user B.

3. Make the copy with cp and assign appropiate perms.

4. Reconstruct the mailbox of user B

These would be the commands:

# mkdir /var/spool/imap/a/user/B/Recent
# cd /var/spool/imap/a/user/A
# cp 234. 235. 236. Drafts/12* /var/spool/imap/a/user/B/Recent
# chown -R cyrus:mail /var/spool/imap/a/user/B/Recent
# su cyrus -c '/usr/lib/cyrus-imapd/reconstruct -r user/B'

I thought this it would be enough, but I know I'm missing some things:

1. What name should I assign to message files being copied from A to
B? I know that their name is equal to its UID. So, what's the value of
the major UID? Does every folder mantain its own UID sequence?
2. There are no cyrus.header, cyrus.index nor cyrus.cache files at
/var/spool/imap/a/user/B/Recent because I just created that directory.
So... I don't need to copy those files from A because reconstruct will
create them... is this right?

I know my English isn't very good yet to explain my question
correctly, but I hope someone can understands me to help me a bit with
this.

Thanks, bye

From atif.ghaffar at gmail.com  Wed Feb 25 17:47:43 2009
From: atif.ghaffar at gmail.com (Atif Ghaffar)
Date: Wed, 25 Feb 2009 23:47:43 +0100
Subject: Separating master processes for different services to utilise 
	multiple cores/cpus
Message-ID: <f240b640902251447u2a23a50egb506df74eba922a6@mail.gmail.com>

Hello,

I am a long time cyrus user but have not been hands on for a while.
At my current job we are running cyrus to manage all mailstore for our ISP.

Recently I have migrate a couple of boxes to newer ones with more cores per
cpu.

when running htop -u cyrus,  i see that only one core is being used for most
of the work and that seems to be a waste of the rest of the cpus.

We are using the following services (imap, pop3, lmtp, sieve)

One thought that comes to mind is to run different master processes which do
one service at a time.

I am thinking on consolidating into

1. cyrus-master-main (running imap/sieve)
2. cyrus-master-pop3 (running pop3)
3. cyrus-master-lmtp (running lmtp)

Does this makes any sense?
Would this balance the usage of the CPUs.

If yes, I will post the config files for inspection.


best regards
Atif
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090225/b4735dcb/attachment.html 

From raymond at sundland.com  Wed Feb 25 18:09:27 2009
From: raymond at sundland.com (Raymond T. Sundland)
Date: Wed, 25 Feb 2009 18:09:27 -0500
Subject: Separating master processes for different services to utilise
	multiple cores/cpus
In-Reply-To: <f240b640902251447u2a23a50egb506df74eba922a6@mail.gmail.com>
References: <f240b640902251447u2a23a50egb506df74eba922a6@mail.gmail.com>
Message-ID: <49A5CFA7.6030807@sundland.com>

When master receives a connection, it spawns a child process to manage 
that connection.  That would essentially make it multi-threaded making 
use of the multiple CPUs when needed, I would think.  Is that not the 
case?  You can launch multiple master processes, but that wouldn't have 
any better guarantee to use multiple CPUs, either.


Atif Ghaffar wrote:
> Hello,
>
> I am a long time cyrus user but have not been hands on for a while.
> At my current job we are running cyrus to manage all mailstore for our 
> ISP.
>
> Recently I have migrate a couple of boxes to newer ones with more 
> cores per cpu.
>
> when running htop -u cyrus,  i see that only one core is being used 
> for most of the work and that seems to be a waste of the rest of the cpus.
>
> We are using the following services (imap, pop3, lmtp, sieve)
>
> One thought that comes to mind is to run different master processes 
> which do one service at a time.
>
> I am thinking on consolidating into
>
> 1. cyrus-master-main (running imap/sieve)
> 2. cyrus-master-pop3 (running pop3)
> 3. cyrus-master-lmtp (running lmtp)
>
> Does this makes any sense?
> Would this balance the usage of the CPUs.
>
> If yes, I will post the config files for inspection.
>
>
> best regards
> Atif
> ------------------------------------------------------------------------
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090225/bc82c6fa/attachment.html 

From raymond at sundland.com  Wed Feb 25 18:12:55 2009
From: raymond at sundland.com (Raymond T. Sundland)
Date: Wed, 25 Feb 2009 18:12:55 -0500
Subject: How to copy messages between mailboxes?
In-Reply-To: <c28863110902251445p2ed6224r7f30cbfe800e6871@mail.gmail.com>
References: <c28863110902251445p2ed6224r7f30cbfe800e6871@mail.gmail.com>
Message-ID: <49A5D077.4070209@sundland.com>

I think it would be (better|easier|more effective) to:

1) Assign necessary permissions to mailbox A and mailbox B to user doing 
the moving.
2) Login to the imap server and move the mail
3) Unassign the permissions, if no longer needed.

It may even be possible to do this with sieve, automatically if needed?  
I haven't looked, I'm just guessing this is possible, maybe not.


Jason Voorhees wrote:
> Hi people:
>
> I'm trying to create a script to copy "certain" messages from a
> mailbox of user A to mailbox of user B. These are the steps I have in
> mind:
>
> 1. Select what messages to copy from mailbox of user A. They can
> message files located at different folders within its mailbox.
>
> 2. Choose where to copy messages in mailbox of user B.
>
> 3. Make the copy with cp and assign appropiate perms.
>
> 4. Reconstruct the mailbox of user B
>
> These would be the commands:
>
> # mkdir /var/spool/imap/a/user/B/Recent
> # cd /var/spool/imap/a/user/A
> # cp 234. 235. 236. Drafts/12* /var/spool/imap/a/user/B/Recent
> # chown -R cyrus:mail /var/spool/imap/a/user/B/Recent
> # su cyrus -c '/usr/lib/cyrus-imapd/reconstruct -r user/B'
>
> I thought this it would be enough, but I know I'm missing some things:
>
> 1. What name should I assign to message files being copied from A to
> B? I know that their name is equal to its UID. So, what's the value of
> the major UID? Does every folder mantain its own UID sequence?
> 2. There are no cyrus.header, cyrus.index nor cyrus.cache files at
> /var/spool/imap/a/user/B/Recent because I just created that directory.
> So... I don't need to copy those files from A because reconstruct will
> create them... is this right?
>
> I know my English isn't very good yet to explain my question
> correctly, but I hope someone can understands me to help me a bit with
> this.
>
> Thanks, bye
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>   

From brong at fastmail.fm  Wed Feb 25 18:16:40 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 26 Feb 2009 10:16:40 +1100
Subject: How to copy messages between mailboxes?
In-Reply-To: <c28863110902251445p2ed6224r7f30cbfe800e6871@mail.gmail.com>
References: <c28863110902251445p2ed6224r7f30cbfe800e6871@mail.gmail.com>
Message-ID: <20090225231640.GA7716@brong.net>

On Wed, Feb 25, 2009 at 05:45:25PM -0500, Jason Voorhees wrote:
> Hi people:
> 
> I'm trying to create a script to copy "certain" messages from a
> mailbox of user A to mailbox of user B. These are the steps I have in
> mind:
 
Wrong steps.  That's not a good way to do it.

> # mkdir /var/spool/imap/a/user/B/Recent
> # cd /var/spool/imap/a/user/A
> # cp 234. 235. 236. Drafts/12* /var/spool/imap/a/user/B/Recent
> # chown -R cyrus:mail /var/spool/imap/a/user/B/Recent
> # su cyrus -c '/usr/lib/cyrus-imapd/reconstruct -r user/B'

Use IMAP.

a) Connect as an admin (with permissions on both user.A and user.B)

b) CREATE user.B.Recent

c) SELECT user.A
d) UID COPY 234,235,236 user.B.Recent

e) SELECT user.A.Drafts
f) UID COPY 12:* user.B.Recent

------------

Much cleaner, and you get all the locking goodness of doing it the
RIGHT way.

An alternative to being admin is just adding read ACLs for user
B on user A's folders (as user A), then logging in as user B and
doing these steps.

You can use any IMAP client library.  We use Perl Mail::ImapTalk,
but whatever language you're familiar with is fine.

Bron.

From raymond at sundland.com  Wed Feb 25 18:17:36 2009
From: raymond at sundland.com (Raymond T. Sundland)
Date: Wed, 25 Feb 2009 18:17:36 -0500
Subject: Separating master processes for different services to utilise
	multiple cores/cpus
In-Reply-To: <49A5CFA7.6030807@sundland.com>
References: <f240b640902251447u2a23a50egb506df74eba922a6@mail.gmail.com>
	<49A5CFA7.6030807@sundland.com>
Message-ID: <49A5D190.5080400@sundland.com>

Just for demonstration, my system has relative low usage, but you can 
see the output of top:

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
83831 cyrus       1  44    0 34068K   824K select 3  16:31  0.00% master
83834 cyrus       1  44    0 55296K   480K select 3  15:44  0.00% idled
28343 cyrus       1  44    0 93048K  5240K select 0   0:03  0.00% imapd
28383 cyrus       1  44    0 82208K  4920K select 1   0:02  0.00% imapd
28382 cyrus       1  44    0 80120K  4812K select 0   0:01  0.00% imapd
28350 cyrus       1  44    0 66420K  4300K select 3   0:01  0.00% imapd
28375 cyrus       1  44    0 66488K  4108K select 0   0:00  0.00% imapd
11942 cyrus       1   4    0 66392K  5060K accept 0   0:00  0.00% pop3d
12157 cyrus       1  44    0 66988K  5688K select 1   0:00  0.00% pop3d
12159 cyrus       1  44    0 66708K  5656K select 3   0:00  0.00% pop3d
12158 cyrus       1  44    0 66580K  5648K select 0   0:00  0.00% pop3d
12238 cyrus       1   4    0 66408K  5452K accept 0   0:00  0.00% lmtpd

As you can see, it's using CPUs 0, 1 and 3 at this point.

Raymond T. Sundland wrote:
> When master receives a connection, it spawns a child process to manage 
> that connection.  That would essentially make it multi-threaded making 
> use of the multiple CPUs when needed, I would think.  Is that not the 
> case?  You can launch multiple master processes, but that wouldn't 
> have any better guarantee to use multiple CPUs, either.
>
>
> Atif Ghaffar wrote:
>> Hello,
>>
>> I am a long time cyrus user but have not been hands on for a while.
>> At my current job we are running cyrus to manage all mailstore for 
>> our ISP.
>>
>> Recently I have migrate a couple of boxes to newer ones with more 
>> cores per cpu.
>>
>> when running htop -u cyrus,  i see that only one core is being used 
>> for most of the work and that seems to be a waste of the rest of the 
>> cpus.
>>
>> We are using the following services (imap, pop3, lmtp, sieve)
>>
>> One thought that comes to mind is to run different master processes 
>> which do one service at a time.
>>
>> I am thinking on consolidating into
>>
>> 1. cyrus-master-main (running imap/sieve)
>> 2. cyrus-master-pop3 (running pop3)
>> 3. cyrus-master-lmtp (running lmtp)
>>
>> Does this makes any sense?
>> Would this balance the usage of the CPUs.
>>
>> If yes, I will post the config files for inspection.
>>
>>
>> best regards
>> Atif
>> ------------------------------------------------------------------------
>>
>> ----
>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
> ------------------------------------------------------------------------
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090225/24c54c8f/attachment-0001.html 

From jvoorhees1 at gmail.com  Thu Feb 26 09:42:04 2009
From: jvoorhees1 at gmail.com (Jason Voorhees)
Date: Thu, 26 Feb 2009 09:42:04 -0500
Subject: How to copy messages between mailboxes?
In-Reply-To: <20090225231640.GA7716@brong.net>
References: <c28863110902251445p2ed6224r7f30cbfe800e6871@mail.gmail.com>
	<20090225231640.GA7716@brong.net>
Message-ID: <c28863110902260642q1a4ba443y4d3f6e502c7b4f3c@mail.gmail.com>

On Wed, Feb 25, 2009 at 6:16 PM, Bron Gondwana <brong at fastmail.fm> wrote:
> On Wed, Feb 25, 2009 at 05:45:25PM -0500, Jason Voorhees wrote:
>> Hi people:
>>
>> I'm trying to create a script to copy "certain" messages from a
>> mailbox of user A to mailbox of user B. These are the steps I have in
>> mind:
>
> Wrong steps. ?That's not a good way to do it.
>
Well, I just based on the example shown in the cyrus documentation. It
says that a message file recently copied it will no be visible until
reconstruct command is executed.


>> # mkdir /var/spool/imap/a/user/B/Recent
>> # cd /var/spool/imap/a/user/A
>> # cp 234. 235. 236. Drafts/12* /var/spool/imap/a/user/B/Recent
>> # chown -R cyrus:mail /var/spool/imap/a/user/B/Recent
>> # su cyrus -c '/usr/lib/cyrus-imapd/reconstruct -r user/B'
>
> Use IMAP.
>
> a) Connect as an admin (with permissions on both user.A and user.B)
>
> b) CREATE user.B.Recent
>
> c) SELECT user.A
> d) UID COPY 234,235,236 user.B.Recent
>
> e) SELECT user.A.Drafts
> f) UID COPY 12:* user.B.Recent
>
> ------------
>
> Much cleaner, and you get all the locking goodness of doing it the
> RIGHT way.
>
> An alternative to being admin is just adding read ACLs for user
> B on user A's folders (as user A), then logging in as user B and
> doing these steps.
>
> You can use any IMAP client library. ?We use Perl Mail::ImapTalk,
> but whatever language you're familiar with is fine.
>
> Bron.
>
Unfortunately, I know almost nothing about programming :( It doesn't
seem a good idea to learn programming now just to solve this little
issue.
Anyway I know that programming is necessary to me so I'll learn as
soon as I have enough free time.

Thanks anyway people, I'll read about Perl::ImapTalk

Bye

From tschlabach at gmx.net  Thu Feb 26 11:35:49 2009
From: tschlabach at gmx.net (Torsten Schlabach)
Date: Thu, 26 Feb 2009 17:35:49 +0100
Subject: reconstruct: Authoritative ACL?
Message-ID: <49A6C4E5.3010300@gmx.net>

Dear list!

Just a question:

Cyrus keeps a backup copy of the ACL for each mail folder in the 
cyrus.header file.

When doing a reconstruct -r -f, the ACL from the cyrus.header file is 
taken over and copied over to the mailboxes.db.

I can imagine situations where this is not what you want.

Unfortunately, deleting the cyrus.header file is not an option, for two 
reasons:

1. One needs to preserve the UID of the folder to make sure the xxx.seen 
and xxx.sub databases can still be matched.
2. Without a cyrus.header file, the folder will not be reconstructed at all.

Would it make sense to prepare a patch which would introduce an 
--ignore-acl or --acl=mailboxesdb|filesystem switch to reconstruct?

Regards,
Torsten

From brong at fastmail.fm  Thu Feb 26 17:19:51 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 27 Feb 2009 09:19:51 +1100
Subject: reconstruct: Authoritative ACL?
In-Reply-To: <49A6C4E5.3010300@gmx.net>
References: <49A6C4E5.3010300@gmx.net>
Message-ID: <20090226221951.GB8099@brong.net>

On Thu, Feb 26, 2009 at 05:35:49PM +0100, Torsten Schlabach wrote:
> Dear list!
> 
> Just a question:
> 
> Cyrus keeps a backup copy of the ACL for each mail folder in the 
> cyrus.header file.
> 
> When doing a reconstruct -r -f, the ACL from the cyrus.header file is 
> taken over and copied over to the mailboxes.db.
> 
> I can imagine situations where this is not what you want.

Why are they out of sync?  I can imagine situations, but they're
almost always caused by you doing the wrong thing in the first place.

Bron ( who wants to the put the UniqueId in the mailboxes.db so you
       don't need a cyrus.header, but that's like - effort and stuff.
       Format change.  We hate backwards incompatible format changes! )

From brong at fastmail.fm  Thu Feb 26 17:17:50 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 27 Feb 2009 09:17:50 +1100
Subject: How to copy messages between mailboxes?
In-Reply-To: <c28863110902260642q1a4ba443y4d3f6e502c7b4f3c@mail.gmail.com>
References: <c28863110902251445p2ed6224r7f30cbfe800e6871@mail.gmail.com>
	<20090225231640.GA7716@brong.net>
	<c28863110902260642q1a4ba443y4d3f6e502c7b4f3c@mail.gmail.com>
Message-ID: <20090226221749.GA8099@brong.net>

On Thu, Feb 26, 2009 at 09:42:04AM -0500, Jason Voorhees wrote:
> On Wed, Feb 25, 2009 at 6:16 PM, Bron Gondwana <brong at fastmail.fm> wrote:
> > On Wed, Feb 25, 2009 at 05:45:25PM -0500, Jason Voorhees wrote:
> >> Hi people:
> >>
> >> I'm trying to create a script to copy "certain" messages from a
> >> mailbox of user A to mailbox of user B. These are the steps I have in
> >> mind:
> >
> > Wrong steps. ?That's not a good way to do it.
> >
> Well, I just based on the example shown in the cyrus documentation. It
> says that a message file recently copied it will no be visible until
> reconstruct command is executed.

True.  Doing it this way really isn't very nice though, and as you
noticed, you probably should be renaming the files as well so that
the UIDs match up.  If the same UID was used both in user.A and 
user.A.Drafts, then your idea would not work anyway.
 
> > Use IMAP.
>
> Unfortunately, I know almost nothing about programming :( It doesn't
> seem a good idea to learn programming now just to solve this little
> issue.

Learn programming.  This is a fairly simple piece of programming, and
being able to automate tasks is worth the effort.

> Anyway I know that programming is necessary to me so I'll learn as
> soon as I have enough free time.

Hey - you can always do it by hand with an IMAP client, which is no
harder than doing it by hand using the 'cp' command.  Just log in as
the admin user.
 
> Thanks anyway people, I'll read about Perl::ImapTalk

Yeah, it's pretty good.  Mail::ImapTalk on CPAN.  It's written by Rob,
who sits about 1 metre away from me, and he reads this list too, so
if you have any problems you know where to ask!

Bron.

From reinaldoc at gmail.com  Thu Feb 26 18:03:39 2009
From: reinaldoc at gmail.com (Reinaldo de Carvalho)
Date: Thu, 26 Feb 2009 20:03:39 -0300
Subject: How to copy messages between mailboxes?
In-Reply-To: <c28863110902260642q1a4ba443y4d3f6e502c7b4f3c@mail.gmail.com>
References: <c28863110902251445p2ed6224r7f30cbfe800e6871@mail.gmail.com>
	<20090225231640.GA7716@brong.net>
	<c28863110902260642q1a4ba443y4d3f6e502c7b4f3c@mail.gmail.com>
Message-ID: <4a5881460902261503p33bb5c6eqe9f41fe0a0912e17@mail.gmail.com>

On Thu, Feb 26, 2009 at 11:42 AM, Jason Voorhees <jvoorhees1 at gmail.com> wrote:
> Unfortunately, I know almost nothing about programming :( It doesn't
> seem a good idea to learn programming now just to solve this little
> issue.
> Anyway I know that programming is necessary to me so I'll learn as
> soon as I have enough free time.
>

If you don't want write a code, configure a MUA with cyrus user admin
(at admins:) and drag and drop message. Cyrus admin see all messages
for all users.

-- 
Reinaldo de Carvalho
http://korreio.sf.net (Now available in English)
http://python-cyrus.sf.net

From wes at umich.edu  Thu Feb 26 20:01:07 2009
From: wes at umich.edu (Wesley Craig)
Date: Thu, 26 Feb 2009 20:01:07 -0500
Subject: reconstruct: Authoritative ACL?
In-Reply-To: <49A6C4E5.3010300@gmx.net>
References: <49A6C4E5.3010300@gmx.net>
Message-ID: <6D47A59A-9ECE-4A09-8A8F-224BDB5DE06B@umich.edu>

On 26 Feb 2009, at 11:35, Torsten Schlabach wrote:
> I can imagine situations where this is not what you want.

I can't.  Please share a couple.

:wes

From tschlabach at gmx.net  Fri Feb 27 06:18:00 2009
From: tschlabach at gmx.net (Torsten Schlabach)
Date: Fri, 27 Feb 2009 12:18:00 +0100
Subject: reconstruct: Authoritative ACL?
In-Reply-To: <6D47A59A-9ECE-4A09-8A8F-224BDB5DE06B@umich.edu>
References: <49A6C4E5.3010300@gmx.net>
	<6D47A59A-9ECE-4A09-8A8F-224BDB5DE06B@umich.edu>
Message-ID: <49A7CBE8.7090100@gmx.net>

Hi!

 >> I can imagine situations where this is not what you want.
 >
 > I can't.  Please share a couple.

If they are currupted for whatever reason?

I mean, by the books, reconstruct is a utility which you should not 
need, because if nothing ever goes wrong, you will never have a need to 
reconstruct mailboxes.

Also if I follow the subject recently discussed in a thread here, which 
basically says: "Hands off the files, if you need to move things around, 
do it through IMAP" you could argue that you shouldn't need the -f 
switch as well.

Would it hurt to have a finer grain control of how reconstruct works?

Regards,
Torsten


From brong at fastmail.fm  Fri Feb 27 07:21:29 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 27 Feb 2009 23:21:29 +1100
Subject: reconstruct: Authoritative ACL?
In-Reply-To: <49A7CBE8.7090100@gmx.net>
References: <49A6C4E5.3010300@gmx.net>
	<6D47A59A-9ECE-4A09-8A8F-224BDB5DE06B@umich.edu>
	<49A7CBE8.7090100@gmx.net>
Message-ID: <20090227122129.GB20551@brong.net>

On Fri, Feb 27, 2009 at 12:18:00PM +0100, Torsten Schlabach wrote:
> Hi!
> 
>  >> I can imagine situations where this is not what you want.
>  >
>  > I can't.  Please share a couple.
> 
> If they are currupted for whatever reason?
 
50/50 chance that one or the other is corrupted, of course.  Assuming
that you haven't done anything clever like hand copy a cyrus.header
from the wrong directory in (in which case the UniqueId is broken
anyway, so all bets are off)

(and assuming the corruption didn't bother the magic at the start of 
the file, without which reconstruct will ignore it)

> I mean, by the books, reconstruct is a utility which you should not 
> need, because if nothing ever goes wrong, you will never have a need to 
> reconstruct mailboxes.

Yeah, pretty much - which is why you'll see some diffs in the current
CVS where I've put in logging options to let you know if they found
anything that needed fixing...

> Also if I follow the subject recently discussed in a thread here, which 
> basically says: "Hands off the files, if you need to move things around, 
> do it through IMAP" you could argue that you shouldn't need the -f 
> switch as well.

Strictly it's "if you don't understand what's going on really very
quite well, you'll just make yourself more hurt.  Here's the way
that actually keeps things sane".

Now, there's restoring from backups.  I still use reconstruct for that
at the moment because I can't be arsed building a full index file
reconstruction toolkit.  But I'll probably fix that one day.  It's
annoying that we're losing seen state.  Though seen state sucks.
Seen state for the OWNER of the mailbox should be stored in the
cyrus.index file anyway - it's the 99.9999% case.  Worth optimising.
Bloody seen files.  </rant>

> Would it hurt to have a finer grain control of how reconstruct works?

No, probably not.  We're just jumping on your "can imagine situations"
which basically boils down to "I've munted my files".

Bron ( but then our "FixUser" tool resets the ACLs just after
       reconstructing the mailboxes anyway, so I wouldn't notice
       either way )

From gmane-2006-04-16 at jt-socal.com  Fri Feb 27 07:49:43 2009
From: gmane-2006-04-16 at jt-socal.com (John Thomas)
Date: Fri, 27 Feb 2009 04:49:43 -0800
Subject: List to Spam Harvest
Message-ID: <49A7E167.6030905@jt-socal.com>

I know little, so please forgive if this is wrong.
The following link seems to be crawled by Google and exposes our email 
addresses to spam harvesters.  I wonder if it makes sense and is 
possible to not do this or obfuscate the addresses?
http://cyrusimap.web.cmu.edu/archive/mailbox.php?mailbox=archive.info-cyrus
-- 
Sincerely,
John Thomas

From tschlabach at gmx.net  Fri Feb 27 08:08:49 2009
From: tschlabach at gmx.net (Torsten Schlabach)
Date: Fri, 27 Feb 2009 14:08:49 +0100
Subject: reconstruct: Authoritative ACL?
In-Reply-To: <20090227122129.GB20551@brong.net>
References: <49A6C4E5.3010300@gmx.net>
	<6D47A59A-9ECE-4A09-8A8F-224BDB5DE06B@umich.edu>
	<49A7CBE8.7090100@gmx.net> <20090227122129.GB20551@brong.net>
Message-ID: <49A7E5E1.4070406@gmx.net>

Hi!

 > but then our "FixUser" tool resets the ACLs just after
 > reconstructing the mailboxes anyway

Where's that? I am not aware of this utility. What does it do? Just give 
the mailbox owner the default ACLs back?

 > hand copy a cyrus.header from the wrong directory in (in which case
 > the UniqueId is broken anyway

No, I wouldn't do that. But I might have restored mailboxes from server 
A onto server B. In that case I would assume that I can rely on UniqueId 
to be random enought to make a conflict *very* unlikely, can't I?

BTW: I think, a reason why people sometimes copy cyrus.header files 
around is that this is what makes a folder recognized as such by 
reconstruct at all. Why does reconstruct ignore a folder with messages 
in it but no cyrus.* files?

 > It's annoying that we're losing seen state.

Are you mixing subjects here? You can restore seen state (as well as 
subscriptions) if you saved the xxx.seen file (and xxx.sub file) and as 
long as the UniqueId still match.

You are loosing flags (replied, forwarded, ...), though, as they are in 
cyrus.index, aren't they?

 > Now, there's restoring from backups.  I still use reconstruct for that
 > at the moment because I can't be arsed building a full index file
 > reconstruction toolkit.  But I'll probably fix that one day.

If you are restoring from backups, why don't you have a cyrus.index file?

Regards,
Torsten

Bron Gondwana schrieb:
> On Fri, Feb 27, 2009 at 12:18:00PM +0100, Torsten Schlabach wrote:
>> Hi!
>>
>>  >> I can imagine situations where this is not what you want.
>>  >
>>  > I can't.  Please share a couple.
>>
>> If they are currupted for whatever reason?
>  
> 50/50 chance that one or the other is corrupted, of course.  Assuming
> that you haven't done anything clever like hand copy a cyrus.header
> from the wrong directory in (in which case the UniqueId is broken
> anyway, so all bets are off)
> 
> (and assuming the corruption didn't bother the magic at the start of 
> the file, without which reconstruct will ignore it)
> 
>> I mean, by the books, reconstruct is a utility which you should not 
>> need, because if nothing ever goes wrong, you will never have a need to 
>> reconstruct mailboxes.
> 
> Yeah, pretty much - which is why you'll see some diffs in the current
> CVS where I've put in logging options to let you know if they found
> anything that needed fixing...
> 
>> Also if I follow the subject recently discussed in a thread here, which 
>> basically says: "Hands off the files, if you need to move things around, 
>> do it through IMAP" you could argue that you shouldn't need the -f 
>> switch as well.
> 
> Strictly it's "if you don't understand what's going on really very
> quite well, you'll just make yourself more hurt.  Here's the way
> that actually keeps things sane".
> 
> Now, there's restoring from backups.  I still use reconstruct for that
> at the moment because I can't be arsed building a full index file
> reconstruction toolkit.  But I'll probably fix that one day.  It's
> annoying that we're losing seen state.  Though seen state sucks.
> Seen state for the OWNER of the mailbox should be stored in the
> cyrus.index file anyway - it's the 99.9999% case.  Worth optimising.
> Bloody seen files.  </rant>
> 
>> Would it hurt to have a finer grain control of how reconstruct works?
> 
> No, probably not.  We're just jumping on your "can imagine situations"
> which basically boils down to "I've munted my files".
> 
> Bron ( but then our "FixUser" tool resets the ACLs just after
>        reconstructing the mailboxes anyway, so I wouldn't notice
>        either way )

From david.lang at digitalinsight.com  Fri Feb 27 13:03:10 2009
From: david.lang at digitalinsight.com (David Lang)
Date: Fri, 27 Feb 2009 10:03:10 -0800 (PST)
Subject: List to Spam Harvest
In-Reply-To: <49A7E167.6030905@jt-socal.com>
References: <49A7E167.6030905@jt-socal.com>
Message-ID: <alpine.DEB.2.00.0902271002220.5961@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>

On Fri, 27 Feb 2009, John Thomas wrote:

> I know little, so please forgive if this is wrong.
> The following link seems to be crawled by Google and exposes our email
> addresses to spam harvesters.  I wonder if it makes sense and is
> possible to not do this or obfuscate the addresses?
> http://cyrusimap.web.cmu.edu/archive/mailbox.php?mailbox=archive.info-cyrus

if you send mail to a public mailing list it can be harvested by spammers.

David Lang

From whitverj at urbandale.k12.ia.us  Fri Feb 27 13:24:55 2009
From: whitverj at urbandale.k12.ia.us (Josh Whitver)
Date: Fri, 27 Feb 2009 12:24:55 -0600
Subject: List to Spam Harvest
In-Reply-To: <alpine.DEB.2.00.0902271002220.5961@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
References: <49A7E167.6030905@jt-socal.com>
	<alpine.DEB.2.00.0902271002220.5961@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
Message-ID: <A27CC78B-55C9-4FA2-B001-F453DC61210F@urbandale.k12.ia.us>

Most web archives of mailing lists hide or otherwise obfuscate the  
email addresses, however.

See:
http://lists.apple.com/archives/macos-x-server/2009/Feb/threads.html

Or:
http://lists.psu.edu/cgi-bin/wa?A1=ind0902&L=macenterprise

Or:
http://lists.roundcube.net/protect/identify.php
--
Josh Whitver
whitverj at urbandale.k12.ia.us / josh at whitver.net
Don't worry if you're a kleptomaniac, you can always take something  
for it.




On Feb 27, 2009, at 12:03 PM, David Lang wrote:

> On Fri, 27 Feb 2009, John Thomas wrote:
>
>> I know little, so please forgive if this is wrong.
>> The following link seems to be crawled by Google and exposes our  
>> email
>> addresses to spam harvesters.  I wonder if it makes sense and is
>> possible to not do this or obfuscate the addresses?
>> http://cyrusimap.web.cmu.edu/archive/mailbox.php?mailbox=archive.info-cyrus
>
> if you send mail to a public mailing list it can be harvested by  
> spammers.
>
> David Lang
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>


From adam at morrison-ind.com  Fri Feb 27 13:46:00 2009
From: adam at morrison-ind.com (Adam Tauno Williams)
Date: Fri, 27 Feb 2009 13:46:00 -0500
Subject: List to Spam Harvest
In-Reply-To: <alpine.DEB.2.00.0902271002220.5961@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
References: <49A7E167.6030905@jt-socal.com>
	<alpine.DEB.2.00.0902271002220.5961@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
Message-ID: <1235760360.5271.9.camel@linux-m3mt>

On Fri, 2009-02-27 at 10:03 -0800, David Lang wrote:
> On Fri, 27 Feb 2009, John Thomas wrote:
> > I know little, so please forgive if this is wrong.
> > The following link seems to be crawled by Google and exposes our email
> > addresses to spam harvesters.  I wonder if it makes sense and is
> > possible to not do this or obfuscate the addresses?
> > http://cyrusimap.web.cmu.edu/archive/mailbox.php?mailbox=archive.info-cyrus
> if you send mail to a public mailing list it can be harvested by spammers.

Ugh, please no more obfustication than already [foolishly] exists.  It
does not work and defeats the very point of e-mail (facilitating
communication).

I'm posting this message as adam at morrison-ind.com,  although I
frequently use awilliam at whitemice.org and
awilliam at whitemiceconsulting.com as well.   And posting these here in
plain text unobfusticated will have no measurable effect on the amount
of SPAM I receive.   I've been using these addresses for years [a
decade?] go ahead and google them;  if someone is harvesting they've
certainly picked them up already - and the amount of SPAM I receive is
very manageable.


From dave64 at andrew.cmu.edu  Fri Feb 27 14:04:29 2009
From: dave64 at andrew.cmu.edu (Dave McMurtrie)
Date: Fri, 27 Feb 2009 14:04:29 -0500
Subject: List to Spam Harvest
In-Reply-To: <1235760360.5271.9.camel@linux-m3mt>
References: <49A7E167.6030905@jt-socal.com>	<alpine.DEB.2.00.0902271002220.5961@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
	<1235760360.5271.9.camel@linux-m3mt>
Message-ID: <49A8393D.6010608@andrew.cmu.edu>

We're considering just getting rid of the
http://cyrusimap.web.cmu.edu/archive/mailbox.php?mailbox=archive.info-cyrus
archive interface, since we also have the mailman pipermail archives at
http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Since that's more of a long-term project, I just hacked the php script
that generates the archives you're talking about to obfuscate the
address in the exact manner that pipermail does for mailman archives.
Essentially, replace "@" with " at ".

I seriously doubt this would thwart even the laziest, most dim-witted
spammer, but I also don't feel very strongly about the value of
obfuscation anyway.  Hopefully this will be a compromise that everyone
can live with.

Thanks,

Dave
-- 
Dave McMurtrie, SPE
Email Systems Team Leader
Carnegie Mellon University,
Computing Services

From brennan at columbia.edu  Fri Feb 27 15:38:16 2009
From: brennan at columbia.edu (Joseph Brennan)
Date: Fri, 27 Feb 2009 15:38:16 -0500
Subject: List to Spam Harvest
In-Reply-To: <1235760360.5271.9.camel@linux-m3mt>
References: <49A7E167.6030905@jt-socal.com>
	<alpine.DEB.2.00.0902271002220.5961@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
	<1235760360.5271.9.camel@linux-m3mt>
Message-ID: <DE319EAE2B529E366E585F4F@sodor.cc.columbia.edu>



--On Friday, February 27, 2009 13:46 -0500 Adam Tauno Williams 
<adam at morrison-ind.com> wrote:

> And posting these here in
> plain text unobfusticated will have no measurable effect on the amount
> of SPAM I receive.   I've been using these addresses for years [a
> decade?] go ahead and google them


My address has been on the net since 1989 and Google tells me it is
on 729 web pages.  I know for a fact that there are people here who
get a lot more spam than I do, because I follow up on spam reports.
Web harvesting certainly exists but I don't like going into hiding.

The compromise with " at " sounds pretty good though.  No argument.

Joseph Brennan
Lead Email Systems Engineer
Columbia University Information Technology


From adam at morrison-ind.com  Fri Feb 27 15:49:28 2009
From: adam at morrison-ind.com (Adam Tauno Williams)
Date: Fri, 27 Feb 2009 15:49:28 -0500
Subject: List to Spam Harvest
In-Reply-To: <DE319EAE2B529E366E585F4F@sodor.cc.columbia.edu>
References: <49A7E167.6030905@jt-socal.com>
	<alpine.DEB.2.00.0902271002220.5961@qynat-qrfxgbc.qvtvgnyvafvtug.pbz>
	<1235760360.5271.9.camel@linux-m3mt>
	<DE319EAE2B529E366E585F4F@sodor.cc.columbia.edu>
Message-ID: <1235767768.5271.27.camel@linux-m3mt>

On Fri, 2009-02-27 at 15:38 -0500, Joseph Brennan wrote:
> --On Friday, February 27, 2009 13:46 -0500 Adam Tauno Williams 
> <adam at morrison-ind.com> wrote:
> > And posting these here in
> > plain text unobfusticated will have no measurable effect on the amount
> > of SPAM I receive.   I've been using these addresses for years [a
> > decade?] go ahead and google them
> My address has been on the net since 1989 and Google tells me it is
> on 729 web pages.  I know for a fact that there are people here who
> get a lot more spam than I do, because I follow up on spam reports.
> Web harvesting certainly exists but I don't like going into hiding.

It exists but it is by no means the primary source of SPAM;  there has
been some research on how to attract SPAM and the clear far and away
winner is to sign up / register with a consumer products or services
company.  The last study I read indicated that doing so produced 50x
more SPAM than any other action.  Using a public e-mail list produced
essentially none.

> The compromise with " at " sounds pretty good though.  No argument.

Fine with me.


From morgan at orst.edu  Fri Feb 27 19:56:35 2009
From: morgan at orst.edu (Andrew Morgan)
Date: Fri, 27 Feb 2009 16:56:35 -0800 (PST)
Subject: reconstruct: Authoritative ACL?
In-Reply-To: <49A7E5E1.4070406@gmx.net>
References: <49A6C4E5.3010300@gmx.net>
	<6D47A59A-9ECE-4A09-8A8F-224BDB5DE06B@umich.edu>
	<49A7CBE8.7090100@gmx.net> <20090227122129.GB20551@brong.net>
	<49A7E5E1.4070406@gmx.net>
Message-ID: <alpine.DEB.2.00.0902271652040.19740@shell.onid.oregonstate.edu>

On Fri, 27 Feb 2009, Torsten Schlabach wrote:

> No, I wouldn't do that. But I might have restored mailboxes from server
> A onto server B. In that case I would assume that I can rely on UniqueId
> to be random enought to make a conflict *very* unlikely, can't I?
>
> BTW: I think, a reason why people sometimes copy cyrus.header files
> around is that this is what makes a folder recognized as such by
> reconstruct at all. Why does reconstruct ignore a folder with messages
> in it but no cyrus.* files?

If you want to have a folder recognized by reconstruct, create a new, 
empty cyrus.header file.  Reconstruct will populate it with the correct 
information for that mailbox.  Don't forget to set the ownership and perms 
as well.  Here is what I do when restoring messages for users into a 
"RESTORE" folder:

touch RESTORE/cyrus.header
chown cyrus:mail RESTORE/cyrus.header
chmod 600 RESTORE/cyrus.header
reconstruct -x -f user.username
quota -f user.username

 	Andy