possible problems with deduplication database

[gavin.gray at ed.ac.uk]

We run our cyrus 2.4 murder with deduplication switched on.

Generally speaking this works well, but every so often user's complain 
about receiving duplicate messages.

We've been taking a look at our delivery logs and our delivery db and it 
does seem to sometimes behave oddly

e.g a series of messages gets delivered with the same Message-ID and if 
dump deliver.db

we can see this sort of thing:

id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1411970492  uid: 24022
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1411974127  uid: 24023
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1411977693  uid: 24026
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1411988514  uid: 24031
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1411992092  uid: 24033
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1412059168  uid: 24042
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1412060492  uid: 24043
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1412070549  uid: 24051
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1412073074  uid: 24053
id: <E4Aa6n7XGPzsf%2FKcSmgv9w%3D%3D at kiwwimail.co.uk>    to: user.xxxxxxxx 
at: 1412075354  uid: 24055


these are quite isolated incidents, but surely this shouldn't happen at 
all? I suppose I'm asking if there are any known reasons why the
deduplication process might behave in this way?


We run cyr_expire -E 1 every night

Gavin Gray
Edinburgh University Information Services
Rm 2013 JCMB
Kings Buildings
Edinburgh
EH9 3JZ
UK
tel +44 (0)131 650 5987
email gavin.gray at ed.ac.uk

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.