cyrus: delete previously deleted messages(after backup recovery)

[Y B]

Thank you for your help, I managed to write a cyrus.index parser that
checks all the effective mail UIDs (=filenames) and erases all the
unnecessary files (previously deleted and thus not referenced in the index
file).

Just in case someone faces the same issue someday, here is the script :
(instead of directly deleting the emails, it creates a bash file with all
the rm commands, so it can be reviewed before, just in case something goes
wrong...)

In my case, I used the script (cleanMailbox.sh) this way (to clean only the
INBOXes) :

for userdir in `find /data/mail/imap/spool/*/user/ -maxdepth 1 -mindepth 1
-name "*"`; do echo $userdir; ~/adminScripts/cleanMailbox.sh "$userdir";
done;

and the script itself:
*************************************************************************************
#!/bin/bash
# cleanMailbox.sh : remove all mailfiles not referenced in the cyrus.index
file

if [ ! -d "$1" ]; then
    echo "USAGE: argument must be a valid cyrus user mail directory"
       exit 0
fi

INDEXPATH="$1/cyrus.index"

if [ ! -f "$INDEXPATH" ]; then
    echo "No cyrus.index in $1. ABORT"
    exit 0
fi

#1- read start offset in cyrus.index at position 0x0c ==12 (4 bytes)
STARTOFFSET=`od --skip-bytes=12 --read-bytes=4 -An -t x1 "$INDEXPATH" | tr
-d ' '`
STARTOFFSET_10=`printf '%d\n' 0x"$STARTOFFSET"`
echo "messages start offset =0x$STARTOFFSET (=$STARTOFFSET_10)"

#2- read record size in cyrus.index at position 0x10 ==16 (4 bytes)
RECORDSIZE=`od --skip-bytes=16 --read-bytes=4 -An -t x1 "$INDEXPATH" | tr
-d ' '`
RECORDSIZE_10=`printf '%d\n' 0x"$RECORDSIZE"`
echo "message record size =0x$RECORDSIZE (=$RECORDSIZE_10)"

#3- read number of emails in cyrus.index at position 0x14 ==20 (4 bytes)
NBMESSAGES=`od --skip-bytes=20 --read-bytes=4 -An -t x1 "$INDEXPATH" | tr
-d ' '`
NBMESSAGES_10=`printf '%d\n' 0x"$NBMESSAGES"`
echo "number of emails in metadata=0x$NBMESSAGES (=$NBMESSAGES_10)"

# ALL message UID (=filename) will be in cyrus.index at STARTOFFSET + i *
RECORDSIZE
# with i integer in [1..NBMESSAGES]
# Build a reference file to keep track of mail filenames (UID.) that will
NOT be deleted
for i in `seq 0 $((NBMESSAGES_10 - 1))`;
  do
    MESSAGEOFFSET=$((STARTOFFSET_10 + i * RECORDSIZE_10))
    UIDMESSAGE=`od --skip-bytes=$MESSAGEOFFSET --read-bytes=4 -An -t x1
"$INDEXPATH" | tr -d ' '`
    UIDMESSAGE_10=`printf '%d\n' 0x"$UIDMESSAGE"`
    echo "$UIDMESSAGE_10". >> "$1"/KEEP_THIS.tmp
done

# instead of deleting directly the undesirable files, build a command file
with all the rm commands:
echo cd "$1" >> /tmp/eraseOldMails.sh

cd "$1"
# for all email files
for mailfile in `ls *\.`;
  do
    # if mailfile not found in KEEP_THIS list, delete it:
    grep "^$mailfile$" "$1"/KEEP_THIS.tmp >/dev/null || echo rm "$mailfile"
>> /tmp/eraseOldMails.sh
done

chmod +x /tmp/eraseOldMails.sh
rm "$1"/KEEP_THIS.tmp
*************************************************************************************

Yannick Barbeaux



2014-04-23 17:06 GMT+02:00 Michael Menge <michael.menge at zdv.uni-tuebingen.de
>:

> Hi
>
>
> Quoting Y B <ybpubs at gmail.com>:
>
>  OK, I managed to reconstruct my test mailbox. The files that were in my
>> directory structure but not visible in the mailbox are now all visible in
>> my email client.
>> Unfortunately, the cyrreconstruct command is not verbose at all and I
>> cannot see the filenames corresponding to rediscovered emails (so I cannot
>> delete them...).
>> Any clue?
>>
>>
>>  ...
>
>
>>> My version of cyrus : 2.2.13
>>>
>>
>
>
> Cyrus 2.2.13 is very old. AFSIR the -n feature was added in 2.4.x.
> Updating in this situation might do harm, as the metadata format
> changed in 2.4.x
>
> I think there are no reconstruct options or other tools, to remove the
> deleted mails out of the box, but the information which mails where in
> the mailbox are stored in the cyrus.index file.
> See http://git.cyrusimap.org/cyrus-imapd/tree/doc/internal/
> mailbox-format.html?h=cyrus-imapd-2.2
>
> So you may be able to write a tool to parse the file.
>
> Regards
>
>     Michael Menge
>
> ------------------------------------------------------------
> --------------------
> M.Menge                                Tel.: (49) 7071/29-70316
> Universität Tübingen                   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung          mail: michael.menge at zdv.uni-
> tuebingen.de
> Wächterstraße 76
> 72074 Tübingen
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20140425/1061476e/attachment.html