From egoitz at sarenet.es  Tue Jul  9 08:10:49 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 9 Jul 2019 14:10:49 +0200
Subject: Issues with replication and folder/Sieve subscription
Message-ID: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>

Good morning,


After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas didn't have some folders (or all) subscribed the same way they had in previous env in Cyrus 2.3. Same happened for some users with Sieve scripts. It seemed the content itself was perfectly copied. It was like, if the copy between versions would not had fully succeed. We fixed it easily by creating some scripts for checking folder subscriptions and Sieve scripts existence. We though it could perhaps had something to do with some issue replicating folders subscriptions and sieve scripts from 2.3 to 3.0.8. After that, as we fixed it easily and was nothing related to content, we just didn?t go in deep in this topic.

When we started running couples of servers in Cyrus 3.0.8 (upgraded and apparently with all the change process finished) we have seen in some cases that folder subscription was not properly replicated again. Both members of the couple of servers where running same Cyrus version, the 3.0.8 and this issue, was not seen in all users? just in some of them? Due to this reason I have started checking the git repo logs? for trying to see some perhaps related change or similar? some commit that could very clearly affect to this issue (that could have caused it). I have had no success.

So after that, and after not being able to reproduce it, have taken a look at the code. By the nature of the things seen, I supposed that perhaps a META operation failed could be involved. I say it because in sync_do_meta() I can read : 

r = sync_response_parse(sync_be->in, "META", NULL,
                            replica_subs, replica_sieve, replica_seen, NULL);
if (!r) r = sync_do_user_seen(userid, replica_seen, sync_be, flags);
if (!r) r = sync_do_user_sub(userid, replica_subs, sync_be, flags);
if (!r) r = sync_do_user_sieve(userid, replica_sieve, sync_be, flags);

Have been looking for some hours around it but have not been able to see nothing strange? nothing that could have caused this...

Have you ever heard about this issue?.


Best regards,



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190709/5e247cde/attachment.html>

From egoitz at sarenet.es  Tue Jul  9 08:25:15 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 9 Jul 2019 14:25:15 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
Message-ID: <B885EADC-4F7E-49F7-964F-BE2E062BE2E0@sarenet.es>

Could perhaps, some of this something to do, with having intermediate folders subscribed/unsubscribed in the middle of the tree? And that to cause something like it that perhaps is not caused when the mailbox is being accessed by the user instead of being replicated (when the change is applied by synchronization)?.



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 9 jul 2019, a las 14:10, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> Good morning,
> 
> 
> After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas didn't have some folders (or all) subscribed the same way they had in previous env in Cyrus 2.3. Same happened for some users with Sieve scripts. It seemed the content itself was perfectly copied. It was like, if the copy between versions would not had fully succeed. We fixed it easily by creating some scripts for checking folder subscriptions and Sieve scripts existence. We though it could perhaps had something to do with some issue replicating folders subscriptions and sieve scripts from 2.3 to 3.0.8. After that, as we fixed it easily and was nothing related to content, we just didn?t go in deep in this topic.
> 
> When we started running couples of servers in Cyrus 3.0.8 (upgraded and apparently with all the change process finished) we have seen in some cases that folder subscription was not properly replicated again. Both members of the couple of servers where running same Cyrus version, the 3.0.8 and this issue, was not seen in all users? just in some of them? Due to this reason I have started checking the git repo logs? for trying to see some perhaps related change or similar? some commit that could very clearly affect to this issue (that could have caused it). I have had no success.
> 
> So after that, and after not being able to reproduce it, have taken a look at the code. By the nature of the things seen, I supposed that perhaps a META operation failed could be involved. I say it because in sync_do_meta() I can read : 
> 
> r = sync_response_parse(sync_be->in, "META", NULL,
>                             replica_subs, replica_sieve, replica_seen, NULL);
> if (!r) r = sync_do_user_seen(userid, replica_seen, sync_be, flags);
> if (!r) r = sync_do_user_sub(userid, replica_subs, sync_be, flags);
> if (!r) r = sync_do_user_sieve(userid, replica_sieve, sync_be, flags);
> 
> Have been looking for some hours around it but have not been able to see nothing strange? nothing that could have caused this...
> 
> Have you ever heard about this issue?.
> 
> 
> Best regards,
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190709/b2c298ce/attachment-0001.html>

From Albert.Shih at obspm.fr  Tue Jul  9 09:03:41 2019
From: Albert.Shih at obspm.fr (Albert Shih)
Date: Tue, 9 Jul 2019 15:03:41 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
Message-ID: <20190709130341.GD2120@io.chezmoi.fr>

Le 09/07/2019 ? 14:10:49+0200, Egoitz Aurrekoetxea a ?crit
> Good morning,
>
>
> After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas didn't
> have some folders (or all) subscribed the same way they had in previous env in
> Cyrus 2.3. Same happened for some users with Sieve scripts. It seemed the
> content itself was perfectly copied. It was like, if the copy between versions
> would not had fully succeed. We fixed it easily by creating some scripts for
> checking folder subscriptions and Sieve scripts existence. We though it could
> perhaps had something to do with some issue replicating folders subscriptions
> and sieve scripts from 2.3 to 3.0.8. After that, as we fixed it easily and was
> nothing related to content, we just didn?t go in deep in this topic.

After my transfering all mail from my old server (dovecot) to the new one (under
cyrus), I try to initialize the sync, so I launch the synchronization, and
find out everytime the user got a sieve, the sync processus crash, but....event the
it crash it still create ? something ?, so the next time I launch the sync
it pass (and email a actually synchronize). So for the first synchro I
juste launch a infinite loop in bash to synchronize all user.

I known it's not a very satisfying method but it work.

With new user I don't have any problem.

I already send a email here but don't get any solution

    https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-May/040186.html

Don't know if that help

Regards
--
Albert SHIH
DIO b?timent 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
France
xmpp: jas at obspm.fr
Heure local/Local time:
Tue 09 Jul 2019 02:57:46 PM CEST

From egoitz at sarenet.es  Tue Jul  9 16:44:19 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 9 Jul 2019 22:44:19 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <20190709130341.GD2120@io.chezmoi.fr>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
 <20190709130341.GD2120@io.chezmoi.fr>
Message-ID: <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>

Hi Albert,

If instead of -A you used -u for each of your users did it worked? Or did it crashed in the same user as with -A?. Which Cyrus version were you running?. 

Cheers,



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:egoitz at sarenet.es>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 9 jul 2019, a las 15:03, Albert Shih <Albert.Shih at obspm.fr> escribi?:
> 
> Le 09/07/2019 ? 14:10:49+0200, Egoitz Aurrekoetxea a ?crit
>> Good morning,
>> 
>> 
>> After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas didn't
>> have some folders (or all) subscribed the same way they had in previous env in
>> Cyrus 2.3. Same happened for some users with Sieve scripts. It seemed the
>> content itself was perfectly copied. It was like, if the copy between versions
>> would not had fully succeed. We fixed it easily by creating some scripts for
>> checking folder subscriptions and Sieve scripts existence. We though it could
>> perhaps had something to do with some issue replicating folders subscriptions
>> and sieve scripts from 2.3 to 3.0.8. After that, as we fixed it easily and was
>> nothing related to content, we just didn?t go in deep in this topic.
> 
> After my transfering all mail from my old server (dovecot) to the new one (under
> cyrus), I try to initialize the sync, so I launch the synchronization, and
> find out everytime the user got a sieve, the sync processus crash, but....event the
> it crash it still create ? something ?, so the next time I launch the sync
> it pass (and email a actually synchronize). So for the first synchro I
> juste launch a infinite loop in bash to synchronize all user.
> 
> I known it's not a very satisfying method but it work.
> 
> With new user I don't have any problem.
> 
> I already send a email here but don't get any solution
> 
>    https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-May/040186.html
> 
> Don't know if that help
> 
> Regards
> --
> Albert SHIH
> DIO b?timent 15
> Observatoire de Paris
> 5 Place Jules Janssen
> 92195 Meudon Cedex
> France
> xmpp: jas at obspm.fr
> Heure local/Local time:
> Tue 09 Jul 2019 02:57:46 PM CEST
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190709/d2b11305/attachment.html>

From egoitz at sarenet.es  Tue Jul  9 16:49:01 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 9 Jul 2019 22:49:01 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
 <20190709130341.GD2120@io.chezmoi.fr>
 <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>
Message-ID: <EF26144F-E613-4660-B9EB-239A7C3F4BE8@sarenet.es>

By the way, for your case I would recommend doing a script that does a get from dovecot and a put to Cyrus instead of copying Sieve files directly? it?s a much more cleaner way?

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:egoitz at sarenet.es>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 9 jul 2019, a las 22:44, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> Hi Albert,
> 
> If instead of -A you used -u for each of your users did it worked? Or did it crashed in the same user as with -A?. Which Cyrus version were you running?. 
> 
> Cheers,
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 9 jul 2019, a las 15:03, Albert Shih <Albert.Shih at obspm.fr <mailto:Albert.Shih at obspm.fr>> escribi?:
>> 
>> Le 09/07/2019 ? 14:10:49+0200, Egoitz Aurrekoetxea a ?crit
>>> Good morning,
>>> 
>>> 
>>> After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas didn't
>>> have some folders (or all) subscribed the same way they had in previous env in
>>> Cyrus 2.3. Same happened for some users with Sieve scripts. It seemed the
>>> content itself was perfectly copied. It was like, if the copy between versions
>>> would not had fully succeed. We fixed it easily by creating some scripts for
>>> checking folder subscriptions and Sieve scripts existence. We though it could
>>> perhaps had something to do with some issue replicating folders subscriptions
>>> and sieve scripts from 2.3 to 3.0.8. After that, as we fixed it easily and was
>>> nothing related to content, we just didn?t go in deep in this topic.
>> 
>> After my transfering all mail from my old server (dovecot) to the new one (under
>> cyrus), I try to initialize the sync, so I launch the synchronization, and
>> find out everytime the user got a sieve, the sync processus crash, but....event the
>> it crash it still create ? something ?, so the next time I launch the sync
>> it pass (and email a actually synchronize). So for the first synchro I
>> juste launch a infinite loop in bash to synchronize all user.
>> 
>> I known it's not a very satisfying method but it work.
>> 
>> With new user I don't have any problem.
>> 
>> I already send a email here but don't get any solution
>> 
>>    https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-May/040186.html <https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-May/040186.html>
>> 
>> Don't know if that help
>> 
>> Regards
>> --
>> Albert SHIH
>> DIO b?timent 15
>> Observatoire de Paris
>> 5 Place Jules Janssen
>> 92195 Meudon Cedex
>> France
>> xmpp: jas at obspm.fr <mailto:jas at obspm.fr>
>> Heure local/Local time:
>> Tue 09 Jul 2019 02:57:46 PM CEST
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190709/58c8f3b6/attachment-0001.html>

From Albert.Shih at obspm.fr  Wed Jul 10 03:20:04 2019
From: Albert.Shih at obspm.fr (Albert Shih)
Date: Wed, 10 Jul 2019 09:20:04 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
 <20190709130341.GD2120@io.chezmoi.fr>
 <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>
Message-ID: <20190710072004.GA5650@io.chezmoi.fr>

Le 09/07/2019 ? 22:44:19+0200, Egoitz Aurrekoetxea a ?crit
Hi,

>
> If instead of -A you used -u for each of your users did it worked? Or did it

If I remember correctly (but I not sure) this is how I find out the
problem.

  Try time -A, notice it crash,

    so try -u first_user, notice it work

      try again -A, notice it crash again (second user)

        try -u second_user, notice it work,

          try -A, notice it crash but for the nth>>1 user

            try -u (n+1)th user notice it work

  try to find the difference between nth user who crash and the (n-1)th
  user who work, find out the only difference is the presence of sieve

  Try the infinite loop with -A.

So in fact I never really try the -u with a non working user.


> crashed in the same user as with -A?. Which Cyrus version were you running?.

Don't sure but something like 3.0.4 (or 3.0.5)

--
Albert SHIH
DIO b?timent 15
Observatoire de Paris
France
xmpp: jas at obspm.fr
Heure local/Local time:
Wed 10 Jul 2019 09:14:51 AM CEST

From Albert.Shih at obspm.fr  Wed Jul 10 03:22:26 2019
From: Albert.Shih at obspm.fr (Albert Shih)
Date: Wed, 10 Jul 2019 09:22:26 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <EF26144F-E613-4660-B9EB-239A7C3F4BE8@sarenet.es>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
 <20190709130341.GD2120@io.chezmoi.fr>
 <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>
 <EF26144F-E613-4660-B9EB-239A7C3F4BE8@sarenet.es>
Message-ID: <20190710072226.GB5650@io.chezmoi.fr>

Le 09/07/2019 ? 22:49:01+0200, Egoitz Aurrekoetxea a ?crit
> By the way, for your case I would recommend doing a script that does a get from
> dovecot and a put to Cyrus instead of copying Sieve files directly? it?s a much
> more cleaner way?

Yes, it is what I did, before I try de sync I event do a
/usr/local/cyrsus/sievec on each file to by absolute sure the sieve file
compile correctly

Regards.


--
Albert SHIH
DIO b?timent 15
Observatoire de Paris
xmpp: jas at obspm.fr
Heure local/Local time:
Wed 10 Jul 2019 09:20:09 AM CEST

From egoitz at sarenet.es  Wed Jul 10 04:01:56 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 10 Jul 2019 10:01:56 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <20190710072226.GB5650@io.chezmoi.fr>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
 <20190709130341.GD2120@io.chezmoi.fr>
 <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>
 <EF26144F-E613-4660-B9EB-239A7C3F4BE8@sarenet.es>
 <20190710072226.GB5650@io.chezmoi.fr>
Message-ID: <D1FA50AB-395B-4BC7-A64A-3A671A24D56E@sarenet.es>

It would be better to just talk daemons IMHO?. That should work? but sometimes? perhaps another things could be implied? so? the most clear way, should be to just put as a normal client with a expect script for instance all the source scripts to Cyrus? but using sieveshell? and not using files directly (not copying) anything to Cyrus partitions ?by hand??

As said IMHO?

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 10 jul 2019, a las 9:22, Albert Shih <Albert.Shih at obspm.fr> escribi?:
> 
> Le 09/07/2019 ? 22:49:01+0200, Egoitz Aurrekoetxea a ?crit
>> By the way, for your case I would recommend doing a script that does a get from
>> dovecot and a put to Cyrus instead of copying Sieve files directly? it?s a much
>> more cleaner way?
> 
> Yes, it is what I did, before I try de sync I event do a
> /usr/local/cyrsus/sievec on each file to by absolute sure the sieve file
> compile correctly
> 
> Regards.
> 
> 
> --
> Albert SHIH
> DIO b?timent 15
> Observatoire de Paris
> xmpp: jas at obspm.fr
> Heure local/Local time:
> Wed 10 Jul 2019 09:20:09 AM CEST

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190710/626a01cc/attachment.html>

From egoitz at sarenet.es  Wed Jul 10 04:03:35 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 10 Jul 2019 10:03:35 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <20190710072226.GB5650@io.chezmoi.fr>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
 <20190709130341.GD2120@io.chezmoi.fr>
 <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>
 <EF26144F-E613-4660-B9EB-239A7C3F4BE8@sarenet.es>
 <20190710072226.GB5650@io.chezmoi.fr>
Message-ID: <D166FF7F-D48C-4660-A18A-8D91F3AC5C93@sarenet.es>

The subject of this email is not properly set? it should be . Issues in replication with folder subscription and Sieve

As I think I discovered something I reopen a new thread with the title properly

 



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 10 jul 2019, a las 9:22, Albert Shih <Albert.Shih at obspm.fr> escribi?:
> 
> Le 09/07/2019 ? 22:49:01+0200, Egoitz Aurrekoetxea a ?crit
>> By the way, for your case I would recommend doing a script that does a get from
>> dovecot and a put to Cyrus instead of copying Sieve files directly? it?s a much
>> more cleaner way?
> 
> Yes, it is what I did, before I try de sync I event do a
> /usr/local/cyrsus/sievec on each file to by absolute sure the sieve file
> compile correctly
> 
> Regards.
> 
> 
> --
> Albert SHIH
> DIO b?timent 15
> Observatoire de Paris
> xmpp: jas at obspm.fr
> Heure local/Local time:
> Wed 10 Jul 2019 09:20:09 AM CEST

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190710/2cb515d8/attachment.html>

From egoitz at sarenet.es  Wed Jul 10 05:11:27 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 10 Jul 2019 11:11:27 +0200
Subject: Folder subscription issue
Message-ID: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>

Good morning,

In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let?s forget about it (for me at least) as it has never reproduced again since then.

About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed ?should? (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the ? move_subscription? condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I?m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.

An example : 

The folder domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.

May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com!user.parta^partb.Trash.XINGANG
May 16 16:16:50 mx6c imap[83976]: Rename: domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com!user.parta^partb.Trash.XINGANG
May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG

In the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the ?____.sub? file.

A diff between the master (replication client) .sub file and the slave?s one is (mx5 is the master, the client and 6 the slave): 

--- subscripciones-mx5c/parta.partb.sub	2019-07-10 08:47:29.000000000 +0200
+++ subscripciones-mx6c/parta.partb.sub	2019-07-10 08:48:08.000000000 +0200

+domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG	
-domain.com!user.parta^partb.Trash.XINGANG	

Perhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird? both mua? perhaps by RFC they should send the SUB command but? it?s the only theory I could arrive to? I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but?.)..

What do you think?.

Best regards,



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190710/198c9e26/attachment-0001.html>

From Hagedorn at uni-koeln.de  Wed Jul 10 05:34:47 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Wed, 10 Jul 2019 11:34:47 +0200
Subject: Folder subscription issue
In-Reply-To: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
Message-ID: <aaf617f2-d3db-2e0b-f776-837caa21d8a7@uni-koeln.de>

Hi,

I'm curious if this only happens for rename to trash, or for all renames
of subscribed folders. IMHO it makes no sense to automatically subscribe
to a folder in the trash. So perhaps the bug isn't in the replication
code but rather in the handling of rename to trash?


Am 10.07.19 um 11:11 Uhr schrieb Egoitz Aurrekoetxea:
> About the folder subscription issue, I think I got something, at least a
> close approximation... When a user causes in mua a rename mailbox (a
> rename for a folder caused by a folder move in hierarchy), after the own
> rename, if folders were subscribed ?should? (for the "plain user" at
> least) become subscribed in the new path. It seems that after a user
> rename in Cyrus the new folder is automatically subscribed (even if no
> subscribe command is sent by the mua). But this, does not cause in the
> replica (in the slave, if SUB is not sent by the client)
> a?sync_apply_changesub() or something like entering in the
> ??move_subscription? condition in mboxlist_renamemailbox(), and then,
> the folder is properly renamed but not subscribed in the slave. I think
> this is what I?m suffering. Obviously, if after a rename the mua sends a
> subscribe too, no issue is seen. I think the problem happens when a
> mailbox rename happens and a SUB is not send later.
> 
> An example :?
> 
> The folder domain.com
> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> is moved (renamed) to trash.
> 
> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed
> domain.com
> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> to domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com
> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> -> domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com
> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> 
> In the master (sync), the folder in Trash is subscribed but in the slave
> it is not, and remains subscribed the folder in the original location in
> the ?____.sub? file.
> 
> A diff between the master (replication client) .sub file and the slave?s
> one is (mx5 is the master, the client and 6 the slave):?
> 
> --- subscripciones-mx5c/parta.partb.sub2019-07-10 08:47:29.000000000 +0200
> +++ subscripciones-mx6c/parta.partb.sub2019-07-10 08:48:08.000000000 +0200
> 
> +domain.com
> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> -domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
> 
> Perhaps a move_subscription to one or sync_apply_changesub should be
> forced in order to fix it?. I have seen issues with Outlook 2016 and
> Thunderbird? both mua? perhaps by RFC they should send the SUB command
> but? it?s the only theory I could arrive to? I have a ton more cases
> like this one.. Data gets properly handled but subscriptions have this
> issue (perhaps we could say is a mua issue but?.)..
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0x197B06994D105B45.asc
Type: application/pgp-keys
Size: 17099 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190710/84a37e5c/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Hagedorn.vcf
Type: text/x-vcard
Size: 333 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190710/84a37e5c/attachment-0001.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5367 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190710/84a37e5c/attachment-0001.p7s>

From egoitz at sarenet.es  Wed Jul 10 05:39:24 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 10 Jul 2019 11:39:24 +0200
Subject: Folder subscription issue
In-Reply-To: <aaf617f2-d3db-2e0b-f776-837caa21d8a7@uni-koeln.de>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <aaf617f2-d3db-2e0b-f776-837caa21d8a7@uni-koeln.de>
Message-ID: <8CC73940-7B87-43D2-AEF7-148FF97B862A@sarenet.es>

Hi!

It happens with any folder? in fact Trash is not the folder we would announce in special-use as Trash? it?s just a normal folder really here?. It?s a generally happening thing with rename of folders inside the own hierarchy inside the own user? (don?t really know if a rename mailbox for changing partition would have the same issue). Not something related to the Trash folder...

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 10 jul 2019, a las 11:34, Sebastian Hagedorn <Hagedorn at uni-koeln.de> escribi?:
> 
> Hi,
> 
> I'm curious if this only happens for rename to trash, or for all renames
> of subscribed folders. IMHO it makes no sense to automatically subscribe
> to a folder in the trash. So perhaps the bug isn't in the replication
> code but rather in the handling of rename to trash?
> 
> 
> Am 10.07.19 um 11:11 Uhr schrieb Egoitz Aurrekoetxea:
>> About the folder subscription issue, I think I got something, at least a
>> close approximation... When a user causes in mua a rename mailbox (a
>> rename for a folder caused by a folder move in hierarchy), after the own
>> rename, if folders were subscribed ?should? (for the "plain user" at
>> least) become subscribed in the new path. It seems that after a user
>> rename in Cyrus the new folder is automatically subscribed (even if no
>> subscribe command is sent by the mua). But this, does not cause in the
>> replica (in the slave, if SUB is not sent by the client)
>> a sync_apply_changesub() or something like entering in the
>> ? move_subscription? condition in mboxlist_renamemailbox(), and then,
>> the folder is properly renamed but not subscribed in the slave. I think
>> this is what I?m suffering. Obviously, if after a rename the mua sends a
>> subscribe too, no issue is seen. I think the problem happens when a
>> mailbox rename happens and a SUB is not send later.
>> 
>> An example : 
>> 
>> The folder domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> is moved (renamed) to trash.
>> 
>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed
>> domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> to domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> -> domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> 
>> In the master (sync), the folder in Trash is subscribed but in the slave
>> it is not, and remains subscribed the folder in the original location in
>> the ?____.sub? file.
>> 
>> A diff between the master (replication client) .sub file and the slave?s
>> one is (mx5 is the master, the client and 6 the slave): 
>> 
>> --- subscripciones-mx5c/parta.partb.sub2019-07-10 08:47:29.000000000 +0200
>> +++ subscripciones-mx6c/parta.partb.sub2019-07-10 08:48:08.000000000 +0200
>> 
>> +domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> -domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>> 
>> Perhaps a move_subscription to one or sync_apply_changesub should be
>> forced in order to fix it?. I have seen issues with Outlook 2016 and
>> Thunderbird? both mua? perhaps by RFC they should send the SUB command
>> but? it?s the only theory I could arrive to? I have a ton more cases
>> like this one.. Data gets properly handled but subscriptions have this
>> issue (perhaps we could say is a mua issue but?.)..
> <0x197B06994D105B45.asc><Hagedorn.vcf>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190710/c315ab87/attachment.html>

From egoitz at sarenet.es  Fri Jul 12 09:26:40 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Fri, 12 Jul 2019 15:26:40 +0200
Subject: Folder subscription issue
In-Reply-To: <8CC73940-7B87-43D2-AEF7-148FF97B862A@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <aaf617f2-d3db-2e0b-f776-837caa21d8a7@uni-koeln.de>
 <8CC73940-7B87-43D2-AEF7-148FF97B862A@sarenet.es>
Message-ID: <2A2CBDD7-9E2D-4E64-8B6B-635BBD926C63@sarenet.es>

By the way I think I found some more?..

When a sync_client in user mode? creates a non existing user in the replica? if the user has a dot? the quota gets properly created, but seen, sub files are wrongly created? for instance?.

-rw-------   1 cyrus  cyrus  2576528 Jul 12 13:03 f.veiga.conversations
-rw-------   1 cyrus  cyrus       88 Jul 12 13:03 f.veiga.counters
-rw-------   1 cyrus  cyrus      451 Jul 12 10:43 f.veiga.sub
-rw-------   1 cyrus  cyrus       16 Jul 12 01:32 f.veiga.xapianactive
-rw-------   1 cyrus  cyrus     2584 Apr  3 01:48 f^veiga.seen
-rw-------   1 cyrus  cyrus      316 Apr  3 01:48 f^veiga.sub

The Apr 3 files are from the initial sync when the box was empty? the first ones (with the dot and not ^) are the actual files being used by Cyrus?. And updated with the replication and so?

It seems this to be the reason because I didn?t see in the initial sync any subscribed folders? but later when set them from a mua where properly replicated?

With Sieve seemed to happen something similar? but in this case with the ?^? and ?.? In the directory name?.

It?s like the name used for creating the subscribing and seen databases in a user mode replication was not properly handled by Cyrus? because it does the same as with quota database with indeed with ?^? is properly created?.

Thanks! Bye!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 10 jul 2019, a las 11:39, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> Hi!
> 
> It happens with any folder? in fact Trash is not the folder we would announce in special-use as Trash? it?s just a normal folder really here?. It?s a generally happening thing with rename of folders inside the own hierarchy inside the own user? (don?t really know if a rename mailbox for changing partition would have the same issue). Not something related to the Trash folder...
> 
> Cheers!
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 10 jul 2019, a las 11:34, Sebastian Hagedorn <Hagedorn at uni-koeln.de <mailto:Hagedorn at uni-koeln.de>> escribi?:
>> 
>> Hi,
>> 
>> I'm curious if this only happens for rename to trash, or for all renames
>> of subscribed folders. IMHO it makes no sense to automatically subscribe
>> to a folder in the trash. So perhaps the bug isn't in the replication
>> code but rather in the handling of rename to trash?
>> 
>> 
>> Am 10.07.19 um 11:11 Uhr schrieb Egoitz Aurrekoetxea:
>>> About the folder subscription issue, I think I got something, at least a
>>> close approximation... When a user causes in mua a rename mailbox (a
>>> rename for a folder caused by a folder move in hierarchy), after the own
>>> rename, if folders were subscribed ?should? (for the "plain user" at
>>> least) become subscribed in the new path. It seems that after a user
>>> rename in Cyrus the new folder is automatically subscribed (even if no
>>> subscribe command is sent by the mua). But this, does not cause in the
>>> replica (in the slave, if SUB is not sent by the client)
>>> a sync_apply_changesub() or something like entering in the
>>> ? move_subscription? condition in mboxlist_renamemailbox(), and then,
>>> the folder is properly renamed but not subscribed in the slave. I think
>>> this is what I?m suffering. Obviously, if after a rename the mua sends a
>>> subscribe too, no issue is seen. I think the problem happens when a
>>> mailbox rename happens and a SUB is not send later.
>>> 
>>> An example : 
>>> 
>>> The folder domain.com <http://domain.com/>
>>> <http://domain.com <http://domain.com/>>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> is moved (renamed) to trash.
>>> 
>>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed
>>> domain.com <http://domain.com/>
>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> to domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com
>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> -> domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com
>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> 
>>> In the master (sync), the folder in Trash is subscribed but in the slave
>>> it is not, and remains subscribed the folder in the original location in
>>> the ?____.sub? file.
>>> 
>>> A diff between the master (replication client) .sub file and the slave?s
>>> one is (mx5 is the master, the client and 6 the slave): 
>>> 
>>> --- subscripciones-mx5c/parta.partb.sub2019-07-10 08:47:29.000000000 +0200
>>> +++ subscripciones-mx6c/parta.partb.sub2019-07-10 08:48:08.000000000 +0200
>>> 
>>> +domain.com
>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> -domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>> 
>>> Perhaps a move_subscription to one or sync_apply_changesub should be
>>> forced in order to fix it?. I have seen issues with Outlook 2016 and
>>> Thunderbird? both mua? perhaps by RFC they should send the SUB command
>>> but? it?s the only theory I could arrive to? I have a ton more cases
>>> like this one.. Data gets properly handled but subscriptions have this
>>> issue (perhaps we could say is a mua issue but?.)..
>> <0x197B06994D105B45.asc><Hagedorn.vcf>
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190712/97f5d149/attachment-0001.html>

From brong at fastmailteam.com  Sun Jul 14 08:36:53 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Sun, 14 Jul 2019 22:36:53 +1000
Subject: Folder subscription issue
In-Reply-To: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
Message-ID: <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>

On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
> Good morning,
> 
> In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let?s forget about it (for me at least) as it has never reproduced again since then.

Sorry, I missed the previous thread. Did you mention which version of Cyrus you are running? There's clearly a bug and it would be good to know which version(s) it affects.

> About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed ?should? (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the ? move_subscription? condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I?m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.

Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename. Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state. If not, that's a bug for sure!

> An example : 
> 
> The folder domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.
> 
> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com!user.parta^partb.Trash.XINGANG
> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com!user.parta^partb.Trash.XINGANG
> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> 
> In the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the ?____.sub? file.

This might just be a matter of failing to sync_log the subscription in that codepath. Hmm...

But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear. I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around!

> A diff between the master (replication client) .sub file and the slave?s one is (mx5 is the master, the client and 6 the slave): 
> 
> --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.000000000 +0200
> +++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.000000000 +0200
> 
> +domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> -domain.com!user.parta^partb.Trash.XINGANG
> 
> Perhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird? both mua? perhaps by RFC they should send the SUB command but? it?s the only theory I could arrive to? I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but?.)..

Bron.
--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190714/95bf125b/attachment.html>

From brong at fastmail.fm  Sun Jul 14 08:58:04 2019
From: brong at fastmail.fm (Bron Gondwana)
Date: Sun, 14 Jul 2019 22:58:04 +1000
Subject: Folder subscription issue
In-Reply-To: <2A2CBDD7-9E2D-4E64-8B6B-635BBD926C63@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <aaf617f2-d3db-2e0b-f776-837caa21d8a7@uni-koeln.de>
 <8CC73940-7B87-43D2-AEF7-148FF97B862A@sarenet.es>
 <2A2CBDD7-9E2D-4E64-8B6B-635BBD926C63@sarenet.es>
Message-ID: <70830b13-c7fc-435f-8646-48b5c00b1bac@www.fastmail.com>

And this is even more exciting and I'd love to know the version!

Bron.

On Fri, Jul 12, 2019, at 23:33, Egoitz Aurrekoetxea wrote:
> By the way I think I found some more?..
> 
> When a sync_client in user mode? creates a non existing user in the replica? if the user has a dot? the quota gets properly created, but seen, sub files are wrongly created? for instance?.
> 
> -rw------- 1 cyrus cyrus 2576528 Jul 12 13:03 f.veiga.conversations
> -rw------- 1 cyrus cyrus 88 Jul 12 13:03 f.veiga.counters
> -rw------- 1 cyrus cyrus 451 Jul 12 10:43 f.veiga.sub
> -rw------- 1 cyrus cyrus 16 Jul 12 01:32 f.veiga.xapianactive
> -rw------- 1 cyrus cyrus 2584 Apr 3 01:48 f^veiga.seen
> -rw------- 1 cyrus cyrus 316 Apr 3 01:48 f^veiga.sub
> 
> The Apr 3 files are from the initial sync when the box was empty? the first ones (with the dot and not ^) are the actual files being used by Cyrus?. And updated with the replication and so?
> 
> It seems this to be the reason because I didn?t see in the initial sync any subscribed folders? but later when set them from a mua where properly replicated?
> 
> With Sieve seemed to happen something similar? but in this case with the ?^? and ?.? In the directory name?.
> 
> It?s like the name used for creating the subscribing and seen databases in a user mode replication was not properly handled by Cyrus? because it does the same as with quota database with indeed with ?^? is properly created?.
> 
> Thanks! Bye!
> 
> 
> sarenet
> *Egoitz Aurrekoetxea*
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 10 jul 2019, a las 11:39, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>> 
>> Hi!
>> 
>> It happens with any folder? in fact Trash is not the folder we would announce in special-use as Trash? it?s just a normal folder really here?. It?s a generally happening thing with rename of folders inside the own hierarchy inside the own user? (don?t really know if a rename mailbox for changing partition would have the same issue). Not something related to the Trash folder...
>> 
>> Cheers!
>> 
>> 
>> sarenet
>> *Egoitz Aurrekoetxea*
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es
>> www.sarenet.es
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 10 jul 2019, a las 11:34, Sebastian Hagedorn <Hagedorn at uni-koeln.de> escribi?:
>>> 
>>> Hi,
>>> 
>>> I'm curious if this only happens for rename to trash, or for all renames
>>> of subscribed folders. IMHO it makes no sense to automatically subscribe
>>> to a folder in the trash. So perhaps the bug isn't in the replication
>>> code but rather in the handling of rename to trash?
>>> 
>>> 
>>> Am 10.07.19 um 11:11 Uhr schrieb Egoitz Aurrekoetxea:
>>>> About the folder subscription issue, I think I got something, at least a
>>>> close approximation... When a user causes in mua a rename mailbox (a
>>>> rename for a folder caused by a folder move in hierarchy), after the own
>>>> rename, if folders were subscribed ?should? (for the "plain user" at
>>>> least) become subscribed in the new path. It seems that after a user
>>>> rename in Cyrus the new folder is automatically subscribed (even if no
>>>> subscribe command is sent by the mua). But this, does not cause in the
>>>> replica (in the slave, if SUB is not sent by the client)
>>>> a sync_apply_changesub() or something like entering in the
>>>> ? move_subscription? condition in mboxlist_renamemailbox(), and then,
>>>> the folder is properly renamed but not subscribed in the slave. I think
>>>> this is what I?m suffering. Obviously, if after a rename the mua sends a
>>>> subscribe too, no issue is seen. I think the problem happens when a
>>>> mailbox rename happens and a SUB is not send later.
>>>> 
>>>> An example : 
>>>> 
>>>> The folder domain.com
>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>> is moved (renamed) to trash.
>>>> 
>>>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed
>>>> domain.com
>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>> to domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com
>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>> -> domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com
>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>> 
>>>> In the master (sync), the folder in Trash is subscribed but in the slave
>>>> it is not, and remains subscribed the folder in the original location in
>>>> the ?____.sub? file.
>>>> 
>>>> A diff between the master (replication client) .sub file and the slave?s
>>>> one is (mx5 is the master, the client and 6 the slave): 
>>>> 
>>>> --- subscripciones-mx5c/parta.partb.sub2019-07-10 08:47:29.000000000 +0200
>>>> +++ subscripciones-mx6c/parta.partb.sub2019-07-10 08:48:08.000000000 +0200
>>>> 
>>>> +domain.com
>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>> -domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>>> 
>>>> Perhaps a move_subscription to one or sync_apply_changesub should be
>>>> forced in order to fix it?. I have seen issues with Outlook 2016 and
>>>> Thunderbird? both mua? perhaps by RFC they should send the SUB command
>>>> but? it?s the only theory I could arrive to? I have a ton more cases
>>>> like this one.. Data gets properly handled but subscriptions have this
>>>> issue (perhaps we could say is a mua issue but?.)..
>>> <0x197B06994D105B45.asc><Hagedorn.vcf>
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
 Bron Gondwana
 brong at fastmail.fm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190714/2d39210d/attachment-0001.html>

From ellie at fastmail.com  Sun Jul 14 22:06:26 2019
From: ellie at fastmail.com (ellie timoney)
Date: Mon, 15 Jul 2019 12:06:26 +1000
Subject: Folder subscription issue
In-Reply-To: <70830b13-c7fc-435f-8646-48b5c00b1bac@www.fastmail.com>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <aaf617f2-d3db-2e0b-f776-837caa21d8a7@uni-koeln.de>
 <8CC73940-7B87-43D2-AEF7-148FF97B862A@sarenet.es>
 <2A2CBDD7-9E2D-4E64-8B6B-635BBD926C63@sarenet.es>
 <70830b13-c7fc-435f-8646-48b5c00b1bac@www.fastmail.com>
Message-ID: <af44ce47-cfdc-4c55-8836-21e989bec473@www.fastmail.com>

I think they said 3.0.8, which is not the latest 3.0, but I also don't think anything about replication/subscriptions/etc has changed since then, so it might as well be

It would also be very interesting to know whether the master and replica definitely both have the same value for "unixhierarchysep". I don't think it should matter? But maybe there's a bug hiding there that's making it matter.

On Sun, Jul 14, 2019, at 11:03 PM, Bron Gondwana wrote:
> And this is even more exciting and I'd love to know the version!
> 
> Bron.
> 
> On Fri, Jul 12, 2019, at 23:33, Egoitz Aurrekoetxea wrote:
>> By the way I think I found some more?..
>> 
>> When a sync_client in user mode? creates a non existing user in the replica? if the user has a dot? the quota gets properly created, but seen, sub files are wrongly created? for instance?.
>> 
>> -rw------- 1 cyrus cyrus 2576528 Jul 12 13:03 f.veiga.conversations
>> -rw------- 1 cyrus cyrus 88 Jul 12 13:03 f.veiga.counters
>> -rw------- 1 cyrus cyrus 451 Jul 12 10:43 f.veiga.sub
>> -rw------- 1 cyrus cyrus 16 Jul 12 01:32 f.veiga.xapianactive
>> -rw------- 1 cyrus cyrus 2584 Apr 3 01:48 f^veiga.seen
>> -rw------- 1 cyrus cyrus 316 Apr 3 01:48 f^veiga.sub
>> 
>> The Apr 3 files are from the initial sync when the box was empty? the first ones (with the dot and not ^) are the actual files being used by Cyrus?. And updated with the replication and so?
>> 
>> It seems this to be the reason because I didn?t see in the initial sync any subscribed folders? but later when set them from a mua where properly replicated?
>> 
>> With Sieve seemed to happen something similar? but in this case with the ?^? and ?.? In the directory name?.
>> 
>> It?s like the name used for creating the subscribing and seen databases in a user mode replication was not properly handled by Cyrus? because it does the same as with quota database with indeed with ?^? is properly created?.
>> 
>> Thanks! Bye!
>> 
>> 
>> sarenet
>> *Egoitz Aurrekoetxea*
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es
>> www.sarenet.es
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 10 jul 2019, a las 11:39, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>> 
>>> Hi!
>>> 
>>> It happens with any folder? in fact Trash is not the folder we would announce in special-use as Trash? it?s just a normal folder really here?. It?s a generally happening thing with rename of folders inside the own hierarchy inside the own user? (don?t really know if a rename mailbox for changing partition would have the same issue). Not something related to the Trash folder...
>>> 
>>> Cheers!
>>> 
>>> 
>>> sarenet
>>> *Egoitz Aurrekoetxea*
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnol?gico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> egoitz at sarenet.es
>>> www.sarenet.es
>>> 
>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>> 
>>>> El 10 jul 2019, a las 11:34, Sebastian Hagedorn <Hagedorn at uni-koeln.de> escribi?:
>>>> 
>>>> Hi,
>>>> 
>>>> I'm curious if this only happens for rename to trash, or for all renames
>>>> of subscribed folders. IMHO it makes no sense to automatically subscribe
>>>> to a folder in the trash. So perhaps the bug isn't in the replication
>>>> code but rather in the handling of rename to trash?
>>>> 
>>>> 
>>>> Am 10.07.19 um 11:11 Uhr schrieb Egoitz Aurrekoetxea:
>>>>> About the folder subscription issue, I think I got something, at least a
>>>>> close approximation... When a user causes in mua a rename mailbox (a
>>>>> rename for a folder caused by a folder move in hierarchy), after the own
>>>>> rename, if folders were subscribed ?should? (for the "plain user" at
>>>>> least) become subscribed in the new path. It seems that after a user
>>>>> rename in Cyrus the new folder is automatically subscribed (even if no
>>>>> subscribe command is sent by the mua). But this, does not cause in the
>>>>> replica (in the slave, if SUB is not sent by the client)
>>>>> a sync_apply_changesub() or something like entering in the
>>>>> ? move_subscription? condition in mboxlist_renamemailbox(), and then,
>>>>> the folder is properly renamed but not subscribed in the slave. I think
>>>>> this is what I?m suffering. Obviously, if after a rename the mua sends a
>>>>> subscribe too, no issue is seen. I think the problem happens when a
>>>>> mailbox rename happens and a SUB is not send later.
>>>>> 
>>>>> An example : 
>>>>> 
>>>>> The folder domain.com
>>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>> is moved (renamed) to trash.
>>>>> 
>>>>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed
>>>>> domain.com
>>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>> to domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com
>>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>> -> domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>>>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com
>>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>> 
>>>>> In the master (sync), the folder in Trash is subscribed but in the slave
>>>>> it is not, and remains subscribed the folder in the original location in
>>>>> the ?____.sub? file.
>>>>> 
>>>>> A diff between the master (replication client) .sub file and the slave?s
>>>>> one is (mx5 is the master, the client and 6 the slave): 
>>>>> 
>>>>> --- subscripciones-mx5c/parta.partb.sub2019-07-10 08:47:29.000000000 +0200
>>>>> +++ subscripciones-mx6c/parta.partb.sub2019-07-10 08:48:08.000000000 +0200
>>>>> 
>>>>> +domain.com
>>>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>> -domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>>>> 
>>>>> Perhaps a move_subscription to one or sync_apply_changesub should be
>>>>> forced in order to fix it?. I have seen issues with Outlook 2016 and
>>>>> Thunderbird? both mua? perhaps by RFC they should send the SUB command
>>>>> but? it?s the only theory I could arrive to? I have a ton more cases
>>>>> like this one.. Data gets properly handled but subscriptions have this
>>>>> issue (perhaps we could say is a mua issue but?.)..
>>>> <0x197B06994D105B45.asc><Hagedorn.vcf>
>>> 
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> -- 
>  Bron Gondwana
>  brong at fastmail.fm
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190715/d66cd749/attachment-0001.html>

From egoitz at sarenet.es  Mon Jul 15 04:36:45 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Mon, 15 Jul 2019 10:36:45 +0200
Subject: Folder subscription issue
In-Reply-To: <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
Message-ID: <C4A673AD-6034-4189-AA22-CC39215DC8B5@sarenet.es>

Hi all :) :),


First of all, thanks a lot for your time :) .

At present moment? I?m slightly confused due to all the tests done, attempts to reproduce the issue and all the Cyrus code read... I?ll tell all the history although I think some parts are not relevant but in an attempt to not omit something useful : 

We were running Cyrus 2.3.18. We moved then to 2.4 as an intermediate version (it converted databases on the fly and so? and although this caused us to work more at nights due to the high load we preferred to do it this way) , in order to be able to finally replicate to a 3.0.8 version. From 2.3 to 2.4 we used sync_client and sync_server and from 2.4 to 3.0 too (if I remember correctly too) because I think Imap replication was not working between 2.4 and 3.0.8. Finally in 3.0.8, we set up another slave for having a couple in 3.0.8 replicating in rolling mode through imap.

The first thing we noticed was that in 3.0.8 (when setting this version as master, giving the service), some accounts had the folders not subscribed and the sieve script didn?t exist (created by each user from Roundcube plugin). As when we were in the intermediate phase between 2.4 and 3.0 (a couple of days) some issues appeared (for instance, in some migrations, while we were using 2.4 for the service and replicating with sync_client in user mode to 3.0.8 some Cyrus database locks happened and so?) we though it was due to that version difference. We finally wrote some scripts for copying manually the Sieve scripts manually. We created too another one for massive folder subscription for those customers telling us they were not seeing the folders. We didn?t worry more then, due to the version difference as commented.

If all that happened were that, we would be relaxed, but?. New problems come again (from 3.0.8 to 3.0.8)?. We needed to say, that we setup new slaves by just syncing with sync_client in user mode?. Finally run in rolling replication mode with a last user mode replication? Here they are : 

- When failing over from a 3.0.8 to another 3.0.8 (master-slave change) we saw again the subscription issue mainly (perhaps the Sieve issue could still be there).
- When setting up new slaves (for moving vm from storage for instance, we created a new slave) with sync_client in user mode? we didn?t appreciate the Sieve issue (although it could exist), but yes, the subscriptions one?.

I started checking what could be happening and I saw some examples as the one described in the mail below. Later, I think I found something more concrete. Please take a look : 

% ls -al
total 104
drwx------  2 cyrus  cyrus    512 Jul 14 03:12 .
drwx------  7 cyrus  cyrus    512 Feb 11 14:52 ..
-rw-------  1 cyrus  cyrus  77072 Jul 15 04:36 a.saizar.conversations
-rw-------  1 cyrus  cyrus     88 Jul  3 10:00 a.saizar.counters
-rw-------  1 cyrus  cyrus     36 Mar 15 05:47 a.saizar.sub
-rw-------  1 cyrus  cyrus     12 Jul 14 03:12 a.saizar.xapianactive
-rw-------  1 cyrus  cyrus    944 Feb 11 00:41 a^saizar.seen
-rw-------  1 cyrus  cyrus    203 Feb 11 00:41 a^saizar.sub
% pwd
/expert/correo/imap/domain/l/xxxxxxxxxxxxx/user/a

It seems, that (at least some) of the mailboxes that had a dot in their name, the subscriptions, sieve (I?m pretty sure too) and seen were created in the slave (sync destination) with the Cyrus mailbox internal name (replacing dots with ^)? instead with just a dot, as it?s seen it works normally. It seems, sync_client in user mode was creating the file with the ^ instead the dot. Then when Cyrus was trying to find the file, with the dots (as it was expected) it didn?t find them. So creates new one and then as that new subs database is empty, it?s the same effect as if subscriptions were ?not copied? with sync_client. I think, although it?s a theory that all the meta files (quota, sieve, seen, subs) are created the same way, all of them with ?^? when only quota file has to have that name format.

So, for summarizing I think I have observed two important issues : 

1 - When you replicate from a 3.0.8 master to 3.0.8 slave sometimes subscriptions (at least, I assume to seen and probably Sieve) are not copied properly (are copied with the dot) and then as the file is not present when Cyrus tries to access with the proper name? creates a new database for subscriptions and considers it?s, just empty? no subscriptions?. I?d say the problem affects just to users who have a dot in the username part (without the domain) of the email. Happens in user mode replication.
2 - Seen a lot at differences of subscribed folders in a master in a slave (both in 3.0.8 and in rolling replication) I saw some differences in some enumerated accounts of some customers :

Jul 12 10:13:26 mx13c imap[53014]: conversations_rename_folder: renamed xxxxxxx.com!user.gl-p^expooort.Sent.OFERTAS.AGENTES VENDA.Gabana Logistics to xxxxxxx.com!user.gl-p^expooort.Sent.OFERTAS.Gabana Logistics
Jul 12 10:13:26 mx13c imap[53014]: Rename: xxxxxxx.com!user.gl-p^expooort.Sent.OFERTAS.AGENTES VENDA.Gabana Logistics -> xxxxxxx.com!user.gl-p^expooort.Sent.OFERTAS.Gabana Logistics
Jul 12 10:13:26 mx13c imap[53014]: Deleted mailbox xxxxxxx.com!user.gl-p^expooort.Sent.OFERTAS.AGENTES VENDA.Gabana Logistics

Gabana Logistics (xxxxxxx.com!user.gl-p^expooort.Sent.OFERTAS.AGENTES VENDA.Gabana Logistics) is still subscribed in the actual slave (and in the slave in that moment).

My suspects are : 

- For the first commented issue (number 1), sync_client when populating dlists, seems to perform an incorrect name conversion (conversion to internal name when shouldn?t) and then when creating quota, sieve, seen and subs, quota is created properly but not he other ones?
- For the second commented one (number 2)? I?d say a mboxlist_changesub is missing if RFC says the behavior of the mua is correct. 

I do attach the files of a master and slave in Cyrus 3.0.8. If you needed a cyrus.conf file too please tell me. I?m at your disposal for whatever you needed for fixing this. 


Best regards,







> El 14 jul 2019, a las 14:36, Bron Gondwana <brong at fastmailteam.com> escribi?:
> 
> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>> Good morning,
>> 
>> In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let?s forget about it (for me at least) as it has never reproduced again since then.
> 
> Sorry, I missed the previous thread.  Did you mention which version of Cyrus you are running?  There's clearly a bug and it would be good to know which version(s) it affects.
> 
>> About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed ?should? (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the ? move_subscription? condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I?m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.
> 
> Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename.  Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state.  If not, that's a bug for sure!
> 
>> An example : 
>> 
>> The folder domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.
>> 
>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> 
>> In the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the ?____.sub? file.
> 
> This might just be a matter of failing to sync_log the subscription in that codepath.  Hmm...
> 
> But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear.  I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around!
> 
>> A diff between the master (replication client) .sub file and the slave?s one is (mx5 is the master, the client and 6 the slave): 
>> 
>> --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.000000000 +0200
>> +++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.000000000 +0200
>> 
>> +domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> -domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>> 
>> Perhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird? both mua? perhaps by RFC they should send the SUB command but? it?s the only theory I could arrive to? I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but?.)..
> 
> Bron.
> --
>   Bron Gondwana, CEO, Fastmail Pty Ltd
>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190715/cbfb67fc/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: master-file
Type: application/octet-stream
Size: 14200 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190715/cbfb67fc/attachment-0002.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190715/cbfb67fc/attachment-0004.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: slave-file
Type: application/octet-stream
Size: 14369 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190715/cbfb67fc/attachment-0003.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190715/cbfb67fc/attachment-0005.html>

From egoitz at sarenet.es  Mon Jul 15 12:27:31 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Mon, 15 Jul 2019 18:27:31 +0200
Subject: Folder subscription issue
In-Reply-To: <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
Message-ID: <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>

Hi Bron!

Sorry for answering so late I had the thought I answered you before?. I?m slightly stressed these days...

I answer below in green bold for instance?..



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 14 jul 2019, a las 14:36, Bron Gondwana <brong at fastmailteam.com> escribi?:
> 
> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>> Good morning,
>> 
>> In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let?s forget about it (for me at least) as it has never reproduced again since then.
> 
> Sorry, I missed the previous thread.  Did you mention which version of Cyrus you are running?  There's clearly a bug and it would be good to know which version(s) it affects.


I think I have specified all these details (including what the previous thread was saying) in the answered mail this morning? I think it's all there? the version was 3.0.8?. If is not all or you need something, please tell me and I?ll try my bests for providing you all the info you need..


> 
>> About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed ?should? (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the ? move_subscription? condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I?m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.
> 
> Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename.  Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state.  If not, that's a bug for sure!

This tests are some days ago? I?m slightly confused now because I don?t remember it exactly? 

> 
>> An example : 
>> 
>> The folder domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.
>> 
>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> 
>> In the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the ?____.sub? file.
> 
> This might just be a matter of failing to sync_log the subscription in that codepath.  Hmm...
> 
> But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear.  I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around!


That?s it Bron..  it was strange? anyway? you know, when you try to debug an issue like the commented you do a closer inspection to all this behaviour, you know?. Although as said before, perhaps it?s better to talk about today's examples? I got them more recent?.


> 
>> A diff between the master (replication client) .sub file and the slave?s one is (mx5 is the master, the client and 6 the slave): 
>> 
>> --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.000000000 +0200
>> +++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.000000000 +0200
>> 
>> +domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> -domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>> 
>> Perhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird? both mua? perhaps by RFC they should send the SUB command but? it?s the only theory I could arrive to? I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but?.)..
> 
> Bron.


Thanks a lot!
> --
>   Bron Gondwana, CEO, Fastmail Pty Ltd
>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190715/096eff46/attachment-0001.html>

From brong at fastmail.fm  Mon Jul 15 20:04:23 2019
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 16 Jul 2019 10:04:23 +1000
Subject: Folder subscription issue
In-Reply-To: <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
Message-ID: <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>

Sorry for not getting back to you yesterday. I was on my way back from vacation and had family commitments all last night.

Regarding contribution - the very best thing to do for a case like this is to build Cassandane tests which isolate the issue :) I'll see if I can get that done today.

Cheers,

Bron.

On Tue, Jul 16, 2019, at 02:34, Egoitz Aurrekoetxea wrote:
> Hi Bron!
> 
> Sorry for answering so late I had the thought I answered you before?. I?m slightly stressed these days...
> 
> I answer below in green bold for instance?..
> 
> 
> sarenet
> *Egoitz Aurrekoetxea*
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 14 jul 2019, a las 14:36, Bron Gondwana <brong at fastmailteam.com> escribi?:
>> 
>> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>>> Good morning,
>>> 
>>> In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let?s forget about it (for me at least) as it has never reproduced again since then.
>> 
>> Sorry, I missed the previous thread. Did you mention which version of Cyrus you are running? There's clearly a bug and it would be good to know which version(s) it affects.
> 
> 
> *I think I have specified all these details (including what the previous thread was saying) in the answered mail this morning? I think it's all there? the version was 3.0.8?. If is not all or you need something, please tell me and I?ll try my bests for providing you all the info you need..*
> 
> 
>> 
>>> About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed ?should? (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the ? move_subscription? condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I?m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.
>> 
>> Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename. Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state. If not, that's a bug for sure!
> 
> *This tests are some days ago? I?m slightly confused now because I don?t remember it exactly? *
> 
>> 
>>> An example : 
>>> 
>>> The folder domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.
>>> 
>>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com!user.parta^partb.Trash.XINGANG
>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com!user.parta^partb.Trash.XINGANG
>>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> 
>>> In the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the ?____.sub? file.
>> 
>> This might just be a matter of failing to sync_log the subscription in that codepath. Hmm...
>> 
>> But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear. I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around!
> 
> 
> *That?s it Bron.. it was strange? anyway? you know, when you try to debug an issue like the commented you do a closer inspection to all this behaviour, you know?. **Although as said before, perhaps it?s better to talk about today's examples? I got them more recent?.*
> 
> 
>> 
>>> A diff between the master (replication client) .sub file and the slave?s one is (mx5 is the master, the client and 6 the slave): 
>>> 
>>> --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.000000000 +0200
>>> +++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.000000000 +0200
>>> 
>>> +domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> -domain.com!user.parta^partb.Trash.XINGANG
>>> 
>>> Perhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird? both mua? perhaps by RFC they should send the SUB command but? it?s the only theory I could arrive to? I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but?.)..
>> 
>> Bron.
> 
> 
> *Thanks a lot!*
>> --
>>  Bron Gondwana, CEO, Fastmail Pty Ltd
>> brong at fastmailteam.com
>> 
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
 Bron Gondwana
 brong at fastmail.fm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190716/a5674bc5/attachment-0001.html>

From egoitz at sarenet.es  Tue Jul 16 04:10:09 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 16 Jul 2019 10:10:09 +0200
Subject: Folder subscription issue
In-Reply-To: <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
Message-ID: <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>

No problem Bron!!

Very thankful for your time and your help!!. 

I have some ideas/questions about the synchronous replication too for commenting you? I have never heard about Cassandane (only to Ellie in this list or Github commit comments) :) but will check it :) :) although I?d need to wait the holidays to come, in order to be more free for all these :)

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 16 jul 2019, a las 2:04, Bron Gondwana <brong at fastmail.fm> escribi?:
> 
> Sorry for not getting back to you yesterday.  I was on my way back from vacation and had family commitments all last night.
> 
> Regarding contribution - the very best thing to do for a case like this is to build Cassandane tests which isolate the issue :)  I'll see if I can get that done today.
> 
> Cheers,
> 
> Bron.
> 
> On Tue, Jul 16, 2019, at 02:34, Egoitz Aurrekoetxea wrote:
>> Hi Bron!
>> 
>> Sorry for answering so late I had the thought I answered you before?. I?m slightly stressed these days...
>> 
>> I answer below in green bold for instance?..
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es
>> www.sarenet.es <http://www.sarenet.es/>
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 14 jul 2019, a las 14:36, Bron Gondwana <brong at fastmailteam.com <mailto:brong at fastmailteam.com>> escribi?:
>>> 
>>> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>>>> Good morning,
>>>> 
>>>> In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let?s forget about it (for me at least) as it has never reproduced again since then.
>>> 
>>> Sorry, I missed the previous thread.  Did you mention which version of Cyrus you are running?  There's clearly a bug and it would be good to know which version(s) it affects.
>> 
>> 
>> I think I have specified all these details (including what the previous thread was saying) in the answered mail this morning? I think it's all there? the version was 3.0.8?. If is not all or you need something, please tell me and I?ll try my bests for providing you all the info you need..
>> 
>> 
>>> 
>>>> About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed ?should? (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the ? move_subscription? condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I?m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.
>>> 
>>> Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename.  Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state.  If not, that's a bug for sure!
>> 
>> This tests are some days ago? I?m slightly confused now because I don?t remember it exactly? 
>> 
>>> 
>>>> An example : 
>>>> 
>>>> The folder domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.
>>>> 
>>>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>> 
>>>> In the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the ?____.sub? file.
>>> 
>>> This might just be a matter of failing to sync_log the subscription in that codepath.  Hmm...
>>> 
>>> But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear.  I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around!
>> 
>> 
>> That?s it Bron..  it was strange? anyway? you know, when you try to debug an issue like the commented you do a closer inspection to all this behaviour, you know?. Although as said before, perhaps it?s better to talk about today's examples? I got them more recent?.
>> 
>> 
>>> 
>>>> A diff between the master (replication client) .sub file and the slave?s one is (mx5 is the master, the client and 6 the slave): 
>>>> 
>>>> --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.000000000 +0200
>>>> +++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.000000000 +0200
>>>> 
>>>> +domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>> -domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>> 
>>>> Perhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird? both mua? perhaps by RFC they should send the SUB command but? it?s the only theory I could arrive to? I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but?.)..
>>> 
>>> Bron.
>> 
>> 
>> Thanks a lot!
>>> --
>>>   Bron Gondwana, CEO, Fastmail Pty Ltd
>>>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
>>> 
>>> 
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> -- 
>   Bron Gondwana
>   brong at fastmail.fm <mailto:brong at fastmail.fm>
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190716/ab5f0f5e/attachment-0001.html>

From egoitz at sarenet.es  Tue Jul 16 12:41:34 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 16 Jul 2019 18:41:34 +0200
Subject: Folder subscription issue
In-Reply-To: <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
Message-ID: <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>

Hi!

One question are this records in the mailbox database (seen with a ctl_mboxlist -d) normal ?

xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017. NOVIEMBRE 2017     16 (null) 
xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017. OCTUBRE 2017       16 (null) 
xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.AGOSTO 2017 16 (null) 
xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.DICIEMBRE 2017      16 (null) 
xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.SEPTIEMBRE 2017     16 (null) 
xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.SEPTIEMBRE 2017.JULIO 2017  16 (null) 
xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.SEPTIEMBRE 2017.JUNIO 2017  16 (null) 
xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.SEPTIEMBRE 2017.MAYO 2017   16 (null) 

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 16 jul 2019, a las 10:10, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> No problem Bron!!
> 
> Very thankful for your time and your help!!. 
> 
> I have some ideas/questions about the synchronous replication too for commenting you? I have never heard about Cassandane (only to Ellie in this list or Github commit comments) :) but will check it :) :) although I?d need to wait the holidays to come, in order to be more free for all these :)
> 
> Cheers!
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 16 jul 2019, a las 2:04, Bron Gondwana <brong at fastmail.fm <mailto:brong at fastmail.fm>> escribi?:
>> 
>> Sorry for not getting back to you yesterday.  I was on my way back from vacation and had family commitments all last night.
>> 
>> Regarding contribution - the very best thing to do for a case like this is to build Cassandane tests which isolate the issue :)  I'll see if I can get that done today.
>> 
>> Cheers,
>> 
>> Bron.
>> 
>> On Tue, Jul 16, 2019, at 02:34, Egoitz Aurrekoetxea wrote:
>>> Hi Bron!
>>> 
>>> Sorry for answering so late I had the thought I answered you before?. I?m slightly stressed these days...
>>> 
>>> I answer below in green bold for instance?..
>>> 
>>> 
>>> 
>>> Egoitz Aurrekoetxea
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnol?gico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>> www.sarenet.es <http://www.sarenet.es/>
>>> 
>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>> 
>>>> El 14 jul 2019, a las 14:36, Bron Gondwana <brong at fastmailteam.com <mailto:brong at fastmailteam.com>> escribi?:
>>>> 
>>>> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>>>>> Good morning,
>>>>> 
>>>>> In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let?s forget about it (for me at least) as it has never reproduced again since then.
>>>> 
>>>> Sorry, I missed the previous thread.  Did you mention which version of Cyrus you are running?  There's clearly a bug and it would be good to know which version(s) it affects.
>>> 
>>> 
>>> I think I have specified all these details (including what the previous thread was saying) in the answered mail this morning? I think it's all there? the version was 3.0.8?. If is not all or you need something, please tell me and I?ll try my bests for providing you all the info you need..
>>> 
>>> 
>>>> 
>>>>> About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed ?should? (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the ? move_subscription? condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I?m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.
>>>> 
>>>> Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename.  Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state.  If not, that's a bug for sure!
>>> 
>>> This tests are some days ago? I?m slightly confused now because I don?t remember it exactly? 
>>> 
>>>> 
>>>>> An example : 
>>>>> 
>>>>> The folder domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.
>>>>> 
>>>>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>> 
>>>>> In the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the ?____.sub? file.
>>>> 
>>>> This might just be a matter of failing to sync_log the subscription in that codepath.  Hmm...
>>>> 
>>>> But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear.  I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around!
>>> 
>>> 
>>> That?s it Bron..  it was strange? anyway? you know, when you try to debug an issue like the commented you do a closer inspection to all this behaviour, you know?. Although as said before, perhaps it?s better to talk about today's examples? I got them more recent?.
>>> 
>>> 
>>>> 
>>>>> A diff between the master (replication client) .sub file and the slave?s one is (mx5 is the master, the client and 6 the slave): 
>>>>> 
>>>>> --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.000000000 +0200
>>>>> +++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.000000000 +0200
>>>>> 
>>>>> +domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>> -domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>>> 
>>>>> Perhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird? both mua? perhaps by RFC they should send the SUB command but? it?s the only theory I could arrive to? I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but?.)..
>>>> 
>>>> Bron.
>>> 
>>> 
>>> Thanks a lot!
>>>> --
>>>>   Bron Gondwana, CEO, Fastmail Pty Ltd
>>>>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
>>>> 
>>>> 
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>> 
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>> 
>> -- 
>>   Bron Gondwana
>>   brong at fastmail.fm <mailto:brong at fastmail.fm>
>> 
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190716/2bcc918d/attachment-0001.html>

From egoitz at sarenet.es  Tue Jul 16 14:20:49 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 16 Jul 2019 20:20:49 +0200
Subject: Folder subscription issue
In-Reply-To: <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
Message-ID: <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>

Hi!

Mmm perhaps I?m telling this excessively sure but? 

In sync_do_user() function, when your scripts syncs user by user instead of using MODE_ALLUSER (which perhaps is more normal? but our scripts are done with a for loop with each user) shouldn?t the userid be converted : 

   /* we don't hold locks while sending commands */
    mailbox_close(&mailbox);
    r = do_user_main(userid, topart, replica_folders, replica_quota,
                     sync_be, channelp, flags);
    if (r) goto done;
    
  /* COMMENTED CHANGE */
  char *user_without_dot = mbname_userid(userid)
    r = sync_do_user_sub(user_without_dot, replica_subs, sync_be, flags);
    if (r) goto done;
    r = sync_do_user_sieve(user_without_dot, replica_sieve, sync_be, flags);
    if (r) goto done;
    r = sync_do_user_seen(user_without_dot, replica_seen, sync_be, flags);
  /* END COMMENTED CHANGE */

I got tons of cases like this that could are explained this way. :

It?s like if sync_do_user wan?t giving the proper user for later creating the file and so to sync_do_user_sub , sync_do_user_seen? etc etc?

Perhaps I have not reproduced (I realized now, although not tested) this because I didn?t create folders in my testing env (the one for reproducing this)? not at least this way? 

I think fix this issue?. What do you think about it?. At least that, shouldn?t damage? I mean at least the fact of replacing the ^ with a dot? and subscripciones would work?. Seen seems to be handled now perhaps in conversations database or similar? but subscriptions? the problem I?m seeing is this ?^? in the file name?.

What do you think mates?




Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 16 jul 2019, a las 18:41, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> Hi!
> 
> One question are this records in the mailbox database (seen with a ctl_mboxlist -d) normal ?
> 
> xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017. NOVIEMBRE 2017     16 (null) 
> xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017. OCTUBRE 2017       16 (null) 
> xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.AGOSTO 2017 16 (null) 
> xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.DICIEMBRE 2017      16 (null) 
> xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.SEPTIEMBRE 2017     16 (null) 
> xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.SEPTIEMBRE 2017.JULIO 2017  16 (null) 
> xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.SEPTIEMBRE 2017.JUNIO 2017  16 (null) 
> xxxxxxxxxx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017.SEPTIEMBRE 2017.MAYO 2017   16 (null) 
> 
> Cheers!
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 16 jul 2019, a las 10:10, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>> 
>> No problem Bron!!
>> 
>> Very thankful for your time and your help!!. 
>> 
>> I have some ideas/questions about the synchronous replication too for commenting you? I have never heard about Cassandane (only to Ellie in this list or Github commit comments) :) but will check it :) :) although I?d need to wait the holidays to come, in order to be more free for all these :)
>> 
>> Cheers!
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es <mailto:undefined>
>> www.sarenet.es <http://www.sarenet.es/>
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 16 jul 2019, a las 2:04, Bron Gondwana <brong at fastmail.fm <mailto:brong at fastmail.fm>> escribi?:
>>> 
>>> Sorry for not getting back to you yesterday.  I was on my way back from vacation and had family commitments all last night.
>>> 
>>> Regarding contribution - the very best thing to do for a case like this is to build Cassandane tests which isolate the issue :)  I'll see if I can get that done today.
>>> 
>>> Cheers,
>>> 
>>> Bron.
>>> 
>>> On Tue, Jul 16, 2019, at 02:34, Egoitz Aurrekoetxea wrote:
>>>> Hi Bron!
>>>> 
>>>> Sorry for answering so late I had the thought I answered you before?. I?m slightly stressed these days...
>>>> 
>>>> I answer below in green bold for instance?..
>>>> 
>>>> 
>>>> 
>>>> Egoitz Aurrekoetxea
>>>> Dpto. de sistemas
>>>> 944 209 470
>>>> Parque Tecnol?gico. Edificio 103
>>>> 48170 Zamudio (Bizkaia)
>>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>>> www.sarenet.es <http://www.sarenet.es/>
>>>> 
>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>> 
>>>>> El 14 jul 2019, a las 14:36, Bron Gondwana <brong at fastmailteam.com <mailto:brong at fastmailteam.com>> escribi?:
>>>>> 
>>>>> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>>>>>> Good morning,
>>>>>> 
>>>>>> In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let?s forget about it (for me at least) as it has never reproduced again since then.
>>>>> 
>>>>> Sorry, I missed the previous thread.  Did you mention which version of Cyrus you are running?  There's clearly a bug and it would be good to know which version(s) it affects.
>>>> 
>>>> 
>>>> I think I have specified all these details (including what the previous thread was saying) in the answered mail this morning? I think it's all there? the version was 3.0.8?. If is not all or you need something, please tell me and I?ll try my bests for providing you all the info you need..
>>>> 
>>>> 
>>>>> 
>>>>>> About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed ?should? (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the ? move_subscription? condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I?m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.
>>>>> 
>>>>> Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename.  Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state.  If not, that's a bug for sure!
>>>> 
>>>> This tests are some days ago? I?m slightly confused now because I don?t remember it exactly? 
>>>> 
>>>>> 
>>>>>> An example : 
>>>>>> 
>>>>>> The folder domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.
>>>>>> 
>>>>>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>>>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>>> 
>>>>>> In the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the ?____.sub? file.
>>>>> 
>>>>> This might just be a matter of failing to sync_log the subscription in that codepath.  Hmm...
>>>>> 
>>>>> But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear.  I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around!
>>>> 
>>>> 
>>>> That?s it Bron..  it was strange? anyway? you know, when you try to debug an issue like the commented you do a closer inspection to all this behaviour, you know?. Although as said before, perhaps it?s better to talk about today's examples? I got them more recent?.
>>>> 
>>>> 
>>>>> 
>>>>>> A diff between the master (replication client) .sub file and the slave?s one is (mx5 is the master, the client and 6 the slave): 
>>>>>> 
>>>>>> --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.000000000 +0200
>>>>>> +++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.000000000 +0200
>>>>>> 
>>>>>> +domain.com <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>>>>> -domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>>>>>> 
>>>>>> Perhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird? both mua? perhaps by RFC they should send the SUB command but? it?s the only theory I could arrive to? I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but?.)..
>>>>> 
>>>>> Bron.
>>>> 
>>>> 
>>>> Thanks a lot!
>>>>> --
>>>>>   Bron Gondwana, CEO, Fastmail Pty Ltd
>>>>>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
>>>>> 
>>>>> 
>>>>> ----
>>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>>> To Unsubscribe:
>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>>> 
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>> 
>>> -- 
>>>   Bron Gondwana
>>>   brong at fastmail.fm <mailto:brong at fastmail.fm>
>>> 
>>> 
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190716/93b62a66/attachment-0001.html>

From egoitz at sarenet.es  Tue Jul 16 14:23:32 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 16 Jul 2019 20:23:32 +0200
Subject: Folder subscription issue
In-Reply-To: <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
Message-ID: <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>

Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> mbname_userid

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190716/ba79c7d6/attachment.html>

From egoitz at sarenet.es  Wed Jul 17 06:51:55 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 17 Jul 2019 12:51:55 +0200
Subject: Folder subscription issue
In-Reply-To: <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
Message-ID: <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>

Anyway, not being able to reproduce it? working on reproducing?.



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>> 
>> mbname_userid
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190717/576c60f5/attachment.html>

From egoitz at sarenet.es  Wed Jul 17 09:55:36 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 17 Jul 2019 15:55:36 +0200
Subject: Folder subscription issue
In-Reply-To: <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
Message-ID: <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>

Hi mates,

Reproduced and properly located (not a Cyrus bug). We have to apologize for the generated noise. Cyrus replication works just fine. 

As I explained before, we fetch from Mysql the accounts each server has (instead of using -A) for no reason? just for historical reason? 

So, when we fetched the emails, as in very old (now) versions of Cyrus a conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o even older), we were doing the following in the Perl script :

while (@fila = $query->fetchrow()) {
        $fila[0] =~ s/\./^/g;
        $email = $fila[0]."@".$fila[1];
        system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S xxxxxx.sarenet.es -v -u ".$email);

This explained the existence of .seen and .sub files with ?^? and Sieve dirs with ?^? too. That didn?t in fact explain, why they had non zero size. The reason of the non zero size (of files with ^ and ending in .sub and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4 perhaps do it) used the ?^? in the seen and sub filenames and Sieve names. So if you do a sync_client of a user with ?^? , even in 3.0 it copies you that file content of .sub and .seen from the master to the slave (because it exists! because it has copied with sync_client too but from a 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart the own mail account content is always copied and sync_client doesn?t complain about it (whether you use ^ or a dot for the user).

So mates, thanks a lot again for your time and I honestly think I?ll sleep better today :)

We have just discovered these, so we wanted to clarify it.

Thanks again,
Regards,



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> Anyway, not being able to reproduce it? working on reproducing?.
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>> 
>> Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es <mailto:undefined>
>> www.sarenet.es <http://www.sarenet.es/>
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>> 
>>> mbname_userid
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190717/f06ae4f3/attachment-0001.html>

From brong at fastmailteam.com  Wed Jul 17 17:58:41 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Thu, 18 Jul 2019 07:58:41 +1000
Subject: Folder subscription issue
In-Reply-To: <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
 <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
Message-ID: <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>

Oh awesome - thanks for letting us know :) This will save ellie spending more time on it, though it's good to have tests for it anyway.

Bron.

On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
> Hi mates,
> 
> Reproduced and properly located (*_not a Cyrus bug_*). We have to apologize for the generated noise. *_Cyrus replication works just fine_*. 
> 
> As I explained before, we fetch from Mysql the accounts each server has (instead of using -A) for no reason? just for historical reason? 
> 
> So, when we fetched the emails, as in very old (now) versions of Cyrus a conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o even older), we were doing the following in the Perl script :
> 
> while (@fila = $query->fetchrow()) {
> $fila[0] =~ s/\./^/g;
>  $email = $fila[0]."@".$fila[1];
>  system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S xxxxxx.sarenet.es -v -u ".$email);
> 
> This explained the existence of .seen and .sub files with ?^? and Sieve dirs with ?^? too. That didn?t in fact explain, why they had non zero size. The reason of the non zero size (of files with ^ and ending in .sub and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4 perhaps do it) used the ?^? in the seen and sub filenames and Sieve names. So if you do a sync_client of a user with ?^? , even in 3.0 it copies you that file content of .sub and .seen from the master to the slave (because it exists! because it has copied with sync_client too but from a 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart the own mail account content is always copied and sync_client doesn?t complain about it (whether you use ^ or a dot for the user).
> 
> So mates, thanks a lot again for your time and I honestly think I?ll sleep better today :)
> 
> We have just discovered these, so we wanted to clarify it.
> 
> Thanks again,
> Regards,
> 
> 
> sarenet
> *Egoitz Aurrekoetxea*
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>> 
>> Anyway, not being able to reproduce it? working on reproducing?.
>> 
>> 
>> sarenet
>> *Egoitz Aurrekoetxea*
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es
>> www.sarenet.es
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>> 
>>> Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?
>>> 
>>> 
>>> sarenet
>>> *Egoitz Aurrekoetxea*
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnol?gico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> egoitz at sarenet.es
>>> www.sarenet.es
>>> 
>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>> 
>>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>>> 
>>>> mbname_userid
>>> 
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/5785b536/attachment-0001.html>

From egoitz at sarenet.es  Thu Jul 18 03:16:24 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 18 Jul 2019 09:16:24 +0200
Subject: Folder subscription issue
In-Reply-To: <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
 <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
 <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>
Message-ID: <635EE199-CE7A-4CAA-8098-3E217F26650F@sarenet.es>

Hi!,

Yes, I?m so sorry for the noise? but you know?. It?s an un modified script in more than 8 years, not done by me? which you assume it works? because indeed, it copies mailbox content (sync_client -u aaa^aaaa at bbb.es <mailto:aaa%5Eaaaa at bbb.es> seems to copy mailbox content as aaa.aaaa at bbb.es <mailto:aaa.aaaa at bbb.es> but not subs) but not ?properly? the subs, sieve? Basically taking a look at some stored consoles (ssh sessions) of the migration?. I noted that were in fact those USER ___^____ at ____ so I did? I?m doing it by me!! Then looked at that script and saw?.

Sorry for the time wasted Ellie.. same Bron,

Cheers!


Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 17 jul 2019, a las 23:58, Bron Gondwana <brong at fastmailteam.com> escribi?:
> 
> Oh awesome - thanks for letting us know :)  This will save ellie spending more time on it, though it's good to have tests for it anyway.
> 
> Bron.
> 
> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>> Hi mates,
>> 
>> Reproduced and properly located (not a Cyrus bug). We have to apologize for the generated noise. Cyrus replication works just fine. 
>> 
>> As I explained before, we fetch from Mysql the accounts each server has (instead of using -A) for no reason? just for historical reason? 
>> 
>> So, when we fetched the emails, as in very old (now) versions of Cyrus a conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o even older), we were doing the following in the Perl script :
>> 
>> while (@fila = $query->fetchrow()) {
>>         $fila[0] =~ s/\./^/g;
>>         $email = $fila[0]."@".$fila[1];
>>         system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S xxxxxx.sarenet.es <http://xxxxxx.sarenet.es/> -v -u ".$email);
>> 
>> This explained the existence of .seen and .sub files with ?^? and Sieve dirs with ?^? too. That didn?t in fact explain, why they had non zero size. The reason of the non zero size (of files with ^ and ending in .sub and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4 perhaps do it) used the ?^? in the seen and sub filenames and Sieve names. So if you do a sync_client of a user with ?^? , even in 3.0 it copies you that file content of .sub and .seen from the master to the slave (because it exists! because it has copied with sync_client too but from a 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart the own mail account content is always copied and sync_client doesn?t complain about it (whether you use ^ or a dot for the user).
>> 
>> So mates, thanks a lot again for your time and I honestly think I?ll sleep better today :)
>> 
>> We have just discovered these, so we wanted to clarify it.
>> 
>> Thanks again,
>> Regards,
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es
>> www.sarenet.es <http://www.sarenet.es/>
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>> 
>>> Anyway, not being able to reproduce it? working on reproducing?.
>>> 
>>> 
>>> 
>>> Egoitz Aurrekoetxea
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnol?gico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> egoitz at sarenet.es
>>> www.sarenet.es <http://www.sarenet.es/>
>>> 
>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>> 
>>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>>> 
>>>> Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?
>>>> 
>>>> 
>>>> 
>>>> Egoitz Aurrekoetxea
>>>> Dpto. de sistemas
>>>> 944 209 470
>>>> Parque Tecnol?gico. Edificio 103
>>>> 48170 Zamudio (Bizkaia)
>>>> egoitz at sarenet.es
>>>> www.sarenet.es <http://www.sarenet.es/>
>>>> 
>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>> 
>>>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>>>> 
>>>>> mbname_userid
>>>> 
>>>> 
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>> 
>>> 
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> --
>   Bron Gondwana, CEO, Fastmail Pty Ltd
>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/94518bf6/attachment-0001.html>

From egoitz at sarenet.es  Thu Jul 18 05:09:07 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 18 Jul 2019 11:09:07 +0200
Subject: Suggested feature and contribution
Message-ID: <6595AD29-BECE-45F9-9DFE-3FB65B74C099@sarenet.es>

Good morning,

When using delete_delayed if someone removes a big folder (that one with more than 20 subfolders anywhere below it) in mboxlist_delayed_deletemailbox() only last 20 are preserved. We think it could be a good idea to preserve all and to have a parameter for configuring it. The reason for that, is that we use delete_delayed for storing the removed content remotely with the customer hired retention period in slow disk space. Perhaps could be a good idea something like : 

In mboxlist_delayed_deletemailbox() : 

If (!preserve_delete_delayed_folders_always)
{
    /* keep the last 19, so the new one is the 20th */
    for (i = 0; i < (int)existing.count - 19; i++) {
        const char *subname = strarray_nth(&existing, i);
        syslog(LOG_NOTICE, "too many subfolders for %s, deleting %s (%d / %d)",
               newname, subname, i+1, (int)existing.count);
        r = mboxlist_deletemailbox(subname, 1, userid, auth_state, NULL, 0, 1, 1,
                                   keep_intermediaries);
        if (r) goto done;
    }
}

In Cyrus imapd.conf : 

preserve_delete_delayed_folders_always: true or false

Obviously, the part of code implied in reading the config file for contemplating the possibility?.

At Sarenet, we have done some scripts for managing the copy of removed content (using delete delayed) and uploading it to a remote imap server. We are using them successfully some years ago now. Is Cyrus project interested in that scripts (that manage removed content) for later allowing the possibility of restoring removed elements from another different IMAP store?. I have asked to my superiors and would be happy to contribute it to Cyrus project? What do you think Bron, Ellie?

Cheers!





Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/e831da20/attachment.html>

From egoitz at sarenet.es  Thu Jul 18 05:13:15 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 18 Jul 2019 11:13:15 +0200
Subject: Issues with replication and folder/Sieve subscription
In-Reply-To: <D166FF7F-D48C-4660-A18A-8D91F3AC5C93@sarenet.es>
References: <ED143D59-1786-4250-9458-41E2E7F1CF5D@sarenet.es>
 <20190709130341.GD2120@io.chezmoi.fr>
 <934506DD-F1A7-4CF0-AAC8-E8AEA6032EAB@sarenet.es>
 <EF26144F-E613-4660-B9EB-239A7C3F4BE8@sarenet.es>
 <20190710072226.GB5650@io.chezmoi.fr>
 <D166FF7F-D48C-4660-A18A-8D91F3AC5C93@sarenet.es>
Message-ID: <D5C5A0F3-B2BC-41AF-A4CA-6F0D2ED69558@sarenet.es>

That thread is clarified. Was an issue from a script from Sarenet?. It has been hard to find (as described in the new thread) but Cyrus was just fine.



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 10 jul 2019, a las 10:03, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> The subject of this email is not properly set? it should be . Issues in replication with folder subscription and Sieve
> 
> As I think I discovered something I reopen a new thread with the title properly
> 
>  
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 10 jul 2019, a las 9:22, Albert Shih <Albert.Shih at obspm.fr <mailto:Albert.Shih at obspm.fr>> escribi?:
>> 
>> Le 09/07/2019 ? 22:49:01+0200, Egoitz Aurrekoetxea a ?crit
>>> By the way, for your case I would recommend doing a script that does a get from
>>> dovecot and a put to Cyrus instead of copying Sieve files directly? it?s a much
>>> more cleaner way?
>> 
>> Yes, it is what I did, before I try de sync I event do a
>> /usr/local/cyrsus/sievec on each file to by absolute sure the sieve file
>> compile correctly
>> 
>> Regards.
>> 
>> 
>> --
>> Albert SHIH
>> DIO b?timent 15
>> Observatoire de Paris
>> xmpp: jas at obspm.fr <mailto:jas at obspm.fr>
>> Heure local/Local time:
>> Wed 10 Jul 2019 09:20:09 AM CEST
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/0efc8ae0/attachment-0001.html>

From brong at fastmail.fm  Thu Jul 18 06:22:32 2019
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 18 Jul 2019 20:22:32 +1000
Subject: Folder subscription issue
In-Reply-To: <635EE199-CE7A-4CAA-8098-3E217F26650F@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
 <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
 <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>
 <635EE199-CE7A-4CAA-8098-3E217F26650F@sarenet.es>
Message-ID: <bb36c021-cb42-4ec4-bd50-3eb393c2892a@www.fastmail.com>

Hmmm.... so sync_client isn't validating the user match properly. Fun. That's definitely a bug that we should fix.

I'll ask ellie (CC'd) to play with the test on that, or guide you through the process of adding one to Cassandane!

Cheers,

Bron.

On Thu, Jul 18, 2019, at 17:22, Egoitz Aurrekoetxea wrote:
> Hi!,
> 
> Yes, I?m so sorry for the noise? but you know?. It?s an un modified script in more than 8 years, not done by me? which you assume it works? because indeed, it copies mailbox content (sync_client -u aaa^aaaa at bbb.es <mailto:aaa%5Eaaaa at bbb.es> seems to copy mailbox content as aaa.aaaa at bbb.es but not subs) but not ?properly? the subs, sieve? Basically taking a look at some stored consoles (ssh sessions) of the migration?. I noted that were in fact those USER ___^____ at ____ so I did? I?m doing it by me!! Then looked at that script and saw?.
> 
> Sorry for the time wasted Ellie.. same Bron,
> 
> Cheers!
> 
> sarenet
> *Egoitz Aurrekoetxea*
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 17 jul 2019, a las 23:58, Bron Gondwana <brong at fastmailteam.com> escribi?:
>> 
>> Oh awesome - thanks for letting us know :) This will save ellie spending more time on it, though it's good to have tests for it anyway.
>> 
>> Bron.
>> 
>> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>>> Hi mates,
>>> 
>>> Reproduced and properly located (*_not a Cyrus bug_*). We have to apologize for the generated noise. *_Cyrus replication works just fine_*. 
>>> 
>>> As I explained before, we fetch from Mysql the accounts each server has (instead of using -A) for no reason? just for historical reason? 
>>> 
>>> So, when we fetched the emails, as in very old (now) versions of Cyrus a conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o even older), we were doing the following in the Perl script :
>>> 
>>> while (@fila = $query->fetchrow()) {
>>> $fila[0] =~ s/\./^/g;
>>>  $email = $fila[0]."@".$fila[1];
>>>  system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S xxxxxx.sarenet.es -v -u ".$email);
>>> 
>>> This explained the existence of .seen and .sub files with ?^? and Sieve dirs with ?^? too. That didn?t in fact explain, why they had non zero size. The reason of the non zero size (of files with ^ and ending in .sub and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4 perhaps do it) used the ?^? in the seen and sub filenames and Sieve names. So if you do a sync_client of a user with ?^? , even in 3.0 it copies you that file content of .sub and .seen from the master to the slave (because it exists! because it has copied with sync_client too but from a 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart the own mail account content is always copied and sync_client doesn?t complain about it (whether you use ^ or a dot for the user).
>>> 
>>> So mates, thanks a lot again for your time and I honestly think I?ll sleep better today :)
>>> 
>>> We have just discovered these, so we wanted to clarify it.
>>> 
>>> Thanks again,
>>> Regards,
>>> 
>>> 
>>> sarenet
>>> *Egoitz Aurrekoetxea*
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnol?gico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> egoitz at sarenet.es
>>> www.sarenet.es
>>> 
>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>> 
>>>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>>> 
>>>> Anyway, not being able to reproduce it? working on reproducing?.
>>>> 
>>>> 
>>>> sarenet
>>>> *Egoitz Aurrekoetxea*
>>>> Dpto. de sistemas
>>>> 944 209 470
>>>> Parque Tecnol?gico. Edificio 103
>>>> 48170 Zamudio (Bizkaia)
>>>> egoitz at sarenet.es
>>>> www.sarenet.es
>>>> 
>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>> 
>>>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>>>> 
>>>>> Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?
>>>>> 
>>>>> 
>>>>> sarenet
>>>>> *Egoitz Aurrekoetxea*
>>>>> Dpto. de sistemas
>>>>> 944 209 470
>>>>> Parque Tecnol?gico. Edificio 103
>>>>> 48170 Zamudio (Bizkaia)
>>>>> egoitz at sarenet.es
>>>>> www.sarenet.es
>>>>> 
>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>> 
>>>>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>>>>> 
>>>>>> mbname_userid
>>>>> 
>>>>> ----
>>>>> Cyrus Home Page: http://www.cyrusimap.org/
>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>> To Unsubscribe:
>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>> 
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> 
>> --
>>  Bron Gondwana, CEO, Fastmail Pty Ltd
>> brong at fastmailteam.com
>> 
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
 Bron Gondwana
 brong at fastmail.fm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/adfd5018/attachment-0001.html>

From egoitz at sarenet.es  Thu Jul 18 06:30:10 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 18 Jul 2019 12:30:10 +0200
Subject: Folder subscription issue
In-Reply-To: <bb36c021-cb42-4ec4-bd50-3eb393c2892a@www.fastmail.com>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
 <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
 <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>
 <635EE199-CE7A-4CAA-8098-3E217F26650F@sarenet.es>
 <bb36c021-cb42-4ec4-bd50-3eb393c2892a@www.fastmail.com>
Message-ID: <7A5E1847-C444-474C-A344-7E41BBDFEA1E@sarenet.es>

Thanks :) :)

Take a look please at what I told you too in a new thread about the contribution with our deleted mail restoring system? perhaps it?s interesting for Cyrus?. For us it?s pretty useful at least? it allows users to restore deleted mail by their own?

Cheers!! 



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 18 jul 2019, a las 12:22, Bron Gondwana <brong at fastmail.fm> escribi?:
> 
> Hmmm.... so sync_client isn't validating the user match properly.  Fun.  That's definitely a bug that we should fix.
> 
> I'll ask ellie (CC'd) to play with the test on that, or guide you through the process of adding one to Cassandane!
> 
> Cheers,
> 
> Bron.
> 
> On Thu, Jul 18, 2019, at 17:22, Egoitz Aurrekoetxea wrote:
>> Hi!,
>> 
>> Yes, I?m so sorry for the noise? but you know?. It?s an un modified script in more than 8 years, not done by me? which you assume it works? because indeed, it copies mailbox content (sync_client -u aaa^aaaa at bbb.es <mailto:aaa%5Eaaaa at bbb.es> seems to copy mailbox content as aaa.aaaa at bbb.es <mailto:aaa.aaaa at bbb.es> but not subs) but not ?properly? the subs, sieve? Basically taking a look at some stored consoles (ssh sessions) of the migration?. I noted that were in fact those USER ___^____ at ____ so I did? I?m doing it by me!! Then looked at that script and saw?.
>> 
>> Sorry for the time wasted Ellie.. same Bron,
>> 
>> Cheers!
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es
>> www.sarenet.es <http://www.sarenet.es/>
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 17 jul 2019, a las 23:58, Bron Gondwana <brong at fastmailteam.com <mailto:brong at fastmailteam.com>> escribi?:
>>> 
>>> Oh awesome - thanks for letting us know :)  This will save ellie spending more time on it, though it's good to have tests for it anyway.
>>> 
>>> Bron.
>>> 
>>> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>>>> Hi mates,
>>>> 
>>>> Reproduced and properly located (not a Cyrus bug). We have to apologize for the generated noise. Cyrus replication works just fine. 
>>>> 
>>>> As I explained before, we fetch from Mysql the accounts each server has (instead of using -A) for no reason? just for historical reason? 
>>>> 
>>>> So, when we fetched the emails, as in very old (now) versions of Cyrus a conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o even older), we were doing the following in the Perl script :
>>>> 
>>>> while (@fila = $query->fetchrow()) {
>>>>         $fila[0] =~ s/\./^/g;
>>>>         $email = $fila[0]."@".$fila[1];
>>>>         system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S xxxxxx.sarenet.es <http://xxxxxx.sarenet.es/> -v -u ".$email);
>>>> 
>>>> This explained the existence of .seen and .sub files with ?^? and Sieve dirs with ?^? too. That didn?t in fact explain, why they had non zero size. The reason of the non zero size (of files with ^ and ending in .sub and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4 perhaps do it) used the ?^? in the seen and sub filenames and Sieve names. So if you do a sync_client of a user with ?^? , even in 3.0 it copies you that file content of .sub and .seen from the master to the slave (because it exists! because it has copied with sync_client too but from a 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart the own mail account content is always copied and sync_client doesn?t complain about it (whether you use ^ or a dot for the user).
>>>> 
>>>> So mates, thanks a lot again for your time and I honestly think I?ll sleep better today :)
>>>> 
>>>> We have just discovered these, so we wanted to clarify it.
>>>> 
>>>> Thanks again,
>>>> Regards,
>>>> 
>>>> 
>>>> 
>>>> Egoitz Aurrekoetxea
>>>> Dpto. de sistemas
>>>> 944 209 470
>>>> Parque Tecnol?gico. Edificio 103
>>>> 48170 Zamudio (Bizkaia)
>>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>>> www.sarenet.es <http://www.sarenet.es/>
>>>> 
>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>> 
>>>>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>>>> 
>>>>> Anyway, not being able to reproduce it? working on reproducing?.
>>>>> 
>>>>> 
>>>>> 
>>>>> Egoitz Aurrekoetxea
>>>>> Dpto. de sistemas
>>>>> 944 209 470
>>>>> Parque Tecnol?gico. Edificio 103
>>>>> 48170 Zamudio (Bizkaia)
>>>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>>>> www.sarenet.es <http://www.sarenet.es/>
>>>>> 
>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>> 
>>>>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>>>>> 
>>>>>> Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Egoitz Aurrekoetxea
>>>>>> Dpto. de sistemas
>>>>>> 944 209 470
>>>>>> Parque Tecnol?gico. Edificio 103
>>>>>> 48170 Zamudio (Bizkaia)
>>>>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>>>>> www.sarenet.es <http://www.sarenet.es/>
>>>>>> 
>>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>>> 
>>>>>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>>>>>> 
>>>>>>> mbname_userid
>>>>>> 
>>>>>> 
>>>>>> ----
>>>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>>>> To Unsubscribe:
>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>>>> 
>>>>> 
>>>>> ----
>>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>>> To Unsubscribe:
>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>>> 
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>> 
>>> --
>>>   Bron Gondwana, CEO, Fastmail Pty Ltd
>>>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
>>> 
>>> 
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
> 
> -- 
>   Bron Gondwana
>   brong at fastmail.fm <mailto:brong at fastmail.fm>
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/e18cfa1d/attachment-0001.html>

From brong at fastmail.fm  Thu Jul 18 08:09:50 2019
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 18 Jul 2019 22:09:50 +1000
Subject: Folder subscription issue
In-Reply-To: <7A5E1847-C444-474C-A344-7E41BBDFEA1E@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
 <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
 <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>
 <635EE199-CE7A-4CAA-8098-3E217F26650F@sarenet.es>
 <bb36c021-cb42-4ec4-bd50-3eb393c2892a@www.fastmail.com>
 <7A5E1847-C444-474C-A344-7E41BBDFEA1E@sarenet.es>
Message-ID: <cd76fdd8-757b-45e9-beff-6bad482ab8e8@www.fastmail.com>

Awesome, I'll take a look. We're just talking about having some level of self-service restore in JMAP as well, so it'll be great to look at what you've done.

Cheers,

Bron.

On Thu, Jul 18, 2019, at 20:30, Egoitz Aurrekoetxea wrote:
> Thanks :) :)
> 
> Take a look please at what I told you too in a new thread about the contribution with our deleted mail restoring system? perhaps it?s interesting for Cyrus?. For us it?s pretty useful at least? it allows users to restore deleted mail by their own?
> 
> Cheers!! 
> 
> 
> sarenet
> *Egoitz Aurrekoetxea*
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 18 jul 2019, a las 12:22, Bron Gondwana <brong at fastmail.fm> escribi?:
>> 
>> Hmmm.... so sync_client isn't validating the user match properly. Fun. That's definitely a bug that we should fix.
>> 
>> I'll ask ellie (CC'd) to play with the test on that, or guide you through the process of adding one to Cassandane!
>> 
>> Cheers,
>> 
>> Bron.
>> 
>> On Thu, Jul 18, 2019, at 17:22, Egoitz Aurrekoetxea wrote:
>>> Hi!,
>>> 
>>> Yes, I?m so sorry for the noise? but you know?. It?s an un modified script in more than 8 years, not done by me? which you assume it works? because indeed, it copies mailbox content (sync_client -u aaa^aaaa at bbb.es <mailto:aaa%5Eaaaa at bbb.es> seems to copy mailbox content as aaa.aaaa at bbb.es but not subs) but not ?properly? the subs, sieve? Basically taking a look at some stored consoles (ssh sessions) of the migration?. I noted that were in fact those USER ___^____ at ____ so I did? I?m doing it by me!! Then looked at that script and saw?.
>>> 
>>> Sorry for the time wasted Ellie.. same Bron,
>>> 
>>> Cheers!
>>> 
>>> sarenet
>>> *Egoitz Aurrekoetxea*
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnol?gico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> egoitz at sarenet.es
>>> www.sarenet.es
>>> 
>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>> 
>>>> El 17 jul 2019, a las 23:58, Bron Gondwana <brong at fastmailteam.com> escribi?:
>>>> 
>>>> Oh awesome - thanks for letting us know :) This will save ellie spending more time on it, though it's good to have tests for it anyway.
>>>> 
>>>> Bron.
>>>> 
>>>> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>>>>> Hi mates,
>>>>> 
>>>>> Reproduced and properly located (*_not a Cyrus bug_*). We have to apologize for the generated noise. *_Cyrus replication works just fine_*. 
>>>>> 
>>>>> As I explained before, we fetch from Mysql the accounts each server has (instead of using -A) for no reason? just for historical reason? 
>>>>> 
>>>>> So, when we fetched the emails, as in very old (now) versions of Cyrus a conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o even older), we were doing the following in the Perl script :
>>>>> 
>>>>> while (@fila = $query->fetchrow()) {
>>>>> $fila[0] =~ s/\./^/g;
>>>>>  $email = $fila[0]."@".$fila[1];
>>>>>  system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S xxxxxx.sarenet.es -v -u ".$email);
>>>>> 
>>>>> This explained the existence of .seen and .sub files with ?^? and Sieve dirs with ?^? too. That didn?t in fact explain, why they had non zero size. The reason of the non zero size (of files with ^ and ending in .sub and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4 perhaps do it) used the ?^? in the seen and sub filenames and Sieve names. So if you do a sync_client of a user with ?^? , even in 3.0 it copies you that file content of .sub and .seen from the master to the slave (because it exists! because it has copied with sync_client too but from a 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart the own mail account content is always copied and sync_client doesn?t complain about it (whether you use ^ or a dot for the user).
>>>>> 
>>>>> So mates, thanks a lot again for your time and I honestly think I?ll sleep better today :)
>>>>> 
>>>>> We have just discovered these, so we wanted to clarify it.
>>>>> 
>>>>> Thanks again,
>>>>> Regards,
>>>>> 
>>>>> 
>>>>> sarenet
>>>>> *Egoitz Aurrekoetxea*
>>>>> Dpto. de sistemas
>>>>> 944 209 470
>>>>> Parque Tecnol?gico. Edificio 103
>>>>> 48170 Zamudio (Bizkaia)
>>>>> egoitz at sarenet.es
>>>>> www.sarenet.es
>>>>> 
>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>> 
>>>>>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>>>>> 
>>>>>> Anyway, not being able to reproduce it? working on reproducing?.
>>>>>> 
>>>>>> 
>>>>>> sarenet
>>>>>> *Egoitz Aurrekoetxea*
>>>>>> Dpto. de sistemas
>>>>>> 944 209 470
>>>>>> Parque Tecnol?gico. Edificio 103
>>>>>> 48170 Zamudio (Bizkaia)
>>>>>> egoitz at sarenet.es
>>>>>> www.sarenet.es
>>>>>> 
>>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>>> 
>>>>>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>>>>>> 
>>>>>>> Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?
>>>>>>> 
>>>>>>> 
>>>>>>> sarenet
>>>>>>> *Egoitz Aurrekoetxea*
>>>>>>> Dpto. de sistemas
>>>>>>> 944 209 470
>>>>>>> Parque Tecnol?gico. Edificio 103
>>>>>>> 48170 Zamudio (Bizkaia)
>>>>>>> egoitz at sarenet.es
>>>>>>> www.sarenet.es
>>>>>>> 
>>>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>>>> 
>>>>>>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
>>>>>>>> 
>>>>>>>> mbname_userid
>>>>>>> 
>>>>>>> ----
>>>>>>> Cyrus Home Page: http://www.cyrusimap.org/
>>>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>>>> To Unsubscribe:
>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>>>> 
>>>>>> ----
>>>>>> Cyrus Home Page: http://www.cyrusimap.org/
>>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>>> To Unsubscribe:
>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>>> ----
>>>>> Cyrus Home Page: http://www.cyrusimap.org/
>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>> To Unsubscribe:
>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>> 
>>>> --
>>>>  Bron Gondwana, CEO, Fastmail Pty Ltd
>>>> brong at fastmailteam.com
>>>> 
>>>> 
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> 
>> -- 
>>  Bron Gondwana
>> brong at fastmail.fm
>> 
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
 Bron Gondwana
 brong at fastmail.fm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/72d4bd4a/attachment-0001.html>

From egoitz at sarenet.es  Thu Jul 18 10:12:58 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 18 Jul 2019 16:12:58 +0200
Subject: Folder subscription issue
In-Reply-To: <cd76fdd8-757b-45e9-beff-6bad482ab8e8@www.fastmail.com>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
 <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
 <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>
 <635EE199-CE7A-4CAA-8098-3E217F26650F@sarenet.es>
 <bb36c021-cb42-4ec4-bd50-3eb393c2892a@www.fastmail.com>
 <7A5E1847-C444-474C-A344-7E41BBDFEA1E@sarenet.es>
 <cd76fdd8-757b-45e9-beff-6bad482ab8e8@www.fastmail.com>
Message-ID: <8435EBDA-5314-43AE-A1FF-90F9616F364D@sarenet.es>

Hi!!,

Fine! Very happy sharing then :) :) . It only handles email. For Calendars/Contacts we have been long time now, using Davical (to which we contributed in it?s day https://wiki.davical.org/index.php/DAViCal-cli <https://wiki.davical.org/index.php/DAViCal-cli>) . We don?t refuse to use Caldav with Cyrus, it?s just we did the system previous to Cyrus Caldav system.

I attach the code in this email. I explain how we use it. We have each mailbox server running this code as a cron job and we have too some servers with Cyrus IMAP for just storing removed content (without the cron obviously). Each user in the restore server (a normal mailbox server but just for storing deleted content) is something like : user_domain at recuperaciones.saremail.com <mailto:user_domain at recuperaciones.saremail.com>. All our servers have autocreate feature (although in our mailbox servers is not being used nowadays). So, we keep track of what has been removed in a mailbox server with two elements? the Cyrus log and cyradm command. With cyradm command we keep track of deleted ?folders" in each user account. With the log, we know where expunges had been run. Later, we take the DELETED mailboxes (the folders of each user) and upload them to Saremail-Restore. After that, we check the log (from some hours before till the present moment). Then we ask unexpunge to see what has been removed in each place. We upload them. We keep track in a database of what exactly has dealed with and what is remaining to deal with, so in the case a fail over to a slave is produced unexpunges can then be run there, even if there?s nothing in the logs that say that (because it?s obviously a slave).


If you think it could be useful, perhaps could be uploaded to contrib directory?

Cheers!












Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 18 jul 2019, a las 14:09, Bron Gondwana <brong at fastmail.fm> escribi?:
> 
> Awesome, I'll take a look.  We're just talking about having some level of self-service restore in JMAP as well, so it'll be great to look at what you've done.
> 
> Cheers,
> 
> Bron.
> 
> On Thu, Jul 18, 2019, at 20:30, Egoitz Aurrekoetxea wrote:
>> Thanks :) :)
>> 
>> Take a look please at what I told you too in a new thread about the contribution with our deleted mail restoring system? perhaps it?s interesting for Cyrus?. For us it?s pretty useful at least? it allows users to restore deleted mail by their own?
>> 
>> Cheers!! 
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es
>> www.sarenet.es <http://www.sarenet.es/>
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 
>>> El 18 jul 2019, a las 12:22, Bron Gondwana <brong at fastmail.fm <mailto:brong at fastmail.fm>> escribi?:
>>> 
>>> Hmmm.... so sync_client isn't validating the user match properly.  Fun.  That's definitely a bug that we should fix.
>>> 
>>> I'll ask ellie (CC'd) to play with the test on that, or guide you through the process of adding one to Cassandane!
>>> 
>>> Cheers,
>>> 
>>> Bron.
>>> 
>>> On Thu, Jul 18, 2019, at 17:22, Egoitz Aurrekoetxea wrote:
>>>> Hi!,
>>>> 
>>>> Yes, I?m so sorry for the noise? but you know?. It?s an un modified script in more than 8 years, not done by me? which you assume it works? because indeed, it copies mailbox content (sync_client -u aaa^aaaa at bbb.es <mailto:aaa%5Eaaaa at bbb.es> seems to copy mailbox content as aaa.aaaa at bbb.es <mailto:aaa.aaaa at bbb.es> but not subs) but not ?properly? the subs, sieve? Basically taking a look at some stored consoles (ssh sessions) of the migration?. I noted that were in fact those USER ___^____ at ____ so I did? I?m doing it by me!! Then looked at that script and saw?.
>>>> 
>>>> Sorry for the time wasted Ellie.. same Bron,
>>>> 
>>>> Cheers!
>>>> 
>>>> 
>>>> Egoitz Aurrekoetxea
>>>> Dpto. de sistemas
>>>> 944 209 470
>>>> Parque Tecnol?gico. Edificio 103
>>>> 48170 Zamudio (Bizkaia)
>>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>>> www.sarenet.es <http://www.sarenet.es/>
>>>> 
>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>> 
>>>>> El 17 jul 2019, a las 23:58, Bron Gondwana <brong at fastmailteam.com <mailto:brong at fastmailteam.com>> escribi?:
>>>>> 
>>>>> Oh awesome - thanks for letting us know :)  This will save ellie spending more time on it, though it's good to have tests for it anyway.
>>>>> 
>>>>> Bron.
>>>>> 
>>>>> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>>>>>> Hi mates,
>>>>>> 
>>>>>> Reproduced and properly located (not a Cyrus bug). We have to apologize for the generated noise. Cyrus replication works just fine. 
>>>>>> 
>>>>>> As I explained before, we fetch from Mysql the accounts each server has (instead of using -A) for no reason? just for historical reason? 
>>>>>> 
>>>>>> So, when we fetched the emails, as in very old (now) versions of Cyrus a conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o even older), we were doing the following in the Perl script :
>>>>>> 
>>>>>> while (@fila = $query->fetchrow()) {
>>>>>>         $fila[0] =~ s/\./^/g;
>>>>>>         $email = $fila[0]."@".$fila[1];
>>>>>>         system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S xxxxxx.sarenet.es <http://xxxxxx.sarenet.es/> -v -u ".$email);
>>>>>> 
>>>>>> This explained the existence of .seen and .sub files with ?^? and Sieve dirs with ?^? too. That didn?t in fact explain, why they had non zero size. The reason of the non zero size (of files with ^ and ending in .sub and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4 perhaps do it) used the ?^? in the seen and sub filenames and Sieve names. So if you do a sync_client of a user with ?^? , even in 3.0 it copies you that file content of .sub and .seen from the master to the slave (because it exists! because it has copied with sync_client too but from a 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart the own mail account content is always copied and sync_client doesn?t complain about it (whether you use ^ or a dot for the user).
>>>>>> 
>>>>>> So mates, thanks a lot again for your time and I honestly think I?ll sleep better today :)
>>>>>> 
>>>>>> We have just discovered these, so we wanted to clarify it.
>>>>>> 
>>>>>> Thanks again,
>>>>>> Regards,
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Egoitz Aurrekoetxea
>>>>>> Dpto. de sistemas
>>>>>> 944 209 470
>>>>>> Parque Tecnol?gico. Edificio 103
>>>>>> 48170 Zamudio (Bizkaia)
>>>>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>>>>> www.sarenet.es <http://www.sarenet.es/>
>>>>>> 
>>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>>> 
>>>>>>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>>>>>> 
>>>>>>> Anyway, not being able to reproduce it? working on reproducing?.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Egoitz Aurrekoetxea
>>>>>>> Dpto. de sistemas
>>>>>>> 944 209 470
>>>>>>> Parque Tecnol?gico. Edificio 103
>>>>>>> 48170 Zamudio (Bizkaia)
>>>>>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>>>>>> www.sarenet.es <http://www.sarenet.es/>
>>>>>>> 
>>>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>>>> 
>>>>>>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>>>>>>> 
>>>>>>>> Perhaps mbname_userid it?s not the exact function? perhaps we could just call something like _append_extbuf() in this case but?. That?s what I was trying to explain? could that be?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Egoitz Aurrekoetxea
>>>>>>>> Dpto. de sistemas
>>>>>>>> 944 209 470
>>>>>>>> Parque Tecnol?gico. Edificio 103
>>>>>>>> 48170 Zamudio (Bizkaia)
>>>>>>>> egoitz at sarenet.es <mailto:egoitz at sarenet.es>
>>>>>>>> www.sarenet.es <http://www.sarenet.es/>
>>>>>>>> 
>>>>>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>>>>>> 
>>>>>>>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea <egoitz at sarenet.es <mailto:egoitz at sarenet.es>> escribi?:
>>>>>>>>> 
>>>>>>>>> mbname_userid
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ----
>>>>>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>>>>>> To Unsubscribe:
>>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>>>>>> 
>>>>>>> 
>>>>>>> ----
>>>>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>>>>> To Unsubscribe:
>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>>>>> 
>>>>>> ----
>>>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>>>> To Unsubscribe:
>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>>>> 
>>>>> --
>>>>>   Bron Gondwana, CEO, Fastmail Pty Ltd
>>>>>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
>>>>> 
>>>>> 
>>>>> ----
>>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>>> To Unsubscribe:
>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>>> 
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>>> 
>>> -- 
>>>   Bron Gondwana
>>>   brong at fastmail.fm <mailto:brong at fastmail.fm>
>>> 
>>> 
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
> 
> -- 
>   Bron Gondwana
>   brong at fastmail.fm <mailto:brong at fastmail.fm>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/8abce83c/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: saremail-restore.zip
Type: application/zip
Size: 14185 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/8abce83c/attachment-0001.zip>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/8abce83c/attachment-0003.html>

From tecnologia at unnoba.edu.ar  Thu Jul 18 10:27:14 2019
From: tecnologia at unnoba.edu.ar (Infraestructura TIC - UNNOBA)
Date: Thu, 18 Jul 2019 11:27:14 -0300
Subject: Folder subscription issue
In-Reply-To: <8435EBDA-5314-43AE-A1FF-90F9616F364D@sarenet.es>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
 <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
 <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>
 <635EE199-CE7A-4CAA-8098-3E217F26650F@sarenet.es>
 <bb36c021-cb42-4ec4-bd50-3eb393c2892a@www.fastmail.com>
 <7A5E1847-C444-474C-A344-7E41BBDFEA1E@sarenet.es>
 <cd76fdd8-757b-45e9-beff-6bad482ab8e8@www.fastmail.com>
 <8435EBDA-5314-43AE-A1FF-90F9616F364D@sarenet.es>
Message-ID: <e5f71459-8cf0-7636-b4f4-539279cf0d31@unnoba.edu.ar>

Much?simas gracias, Egoitz!


El 18/7/19 a las 11:12, Egoitz Aurrekoetxea escribi?:
> Hi!!,
>
> Fine! Very happy sharing then :) :) . It only handles email. For
> Calendars/Contacts we have been long time now, using Davical (to which
> we contributed in it?s
> day?https://wiki.davical.org/index.php/DAViCal-cli) . We don?t refuse
> to use Caldav with Cyrus, it?s just we did the system previous to
> Cyrus Caldav system.
>
> I attach the code in this email. I explain how we use it. We have each
> mailbox server running this code as a cron job and we have too some
> servers with Cyrus IMAP for just storing removed content (without the
> cron obviously). Each user in the restore server (a normal mailbox
> server but just for storing deleted content) is something like :
> user_domain at recuperaciones.saremail.com
> <mailto:user_domain at recuperaciones.saremail.com>. All our servers have
> autocreate feature (although in our mailbox servers is not being used
> nowadays). So, we keep track of what has been removed in a mailbox
> server with two elements? the Cyrus log and cyradm command. With
> cyradm command we keep track of deleted ?folders" in each user
> account. With the log, we know where expunges had been run. Later, we
> take the DELETED mailboxes (the folders of each user) and upload them
> to Saremail-Restore. After that, we check the log (from some hours
> before till the present moment). Then we ask unexpunge to see what has
> been removed in each place. We upload them. We keep track in a
> database of what exactly has dealed with and what is remaining to deal
> with, so in the case a fail over to a slave is produced unexpunges can
> then be run there, even if there?s nothing in the logs that say that
> (because it?s obviously a slave).
>
>
> If you think it could be useful, perhaps could be uploaded to contrib
> directory?
>
> Cheers!


-- 
Lic. Javier Charne
Responsable Infraestructura Tecnol?gica
Prosecretar?a de TIC | UNNOBA
Jun?n, Buenos Aires, Argentina
javier at unnoba.edu.ar
Tel: +54 (0236) 4407750 int 11712
Cel: +5492364542182

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/5352421b/attachment.html>

From egoitz at sarenet.es  Thu Jul 18 10:32:33 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 18 Jul 2019 16:32:33 +0200
Subject: Folder subscription issue
In-Reply-To: <e5f71459-8cf0-7636-b4f4-539279cf0d31@unnoba.edu.ar>
References: <3A79EDA4-59CF-4472-B7BF-F0823E1F0D7D@sarenet.es>
 <5894f148-5fab-479b-a0ec-72c1680c5d16@www.fastmail.com>
 <B7D1A95A-D9A8-4D11-A71F-0750F84815D9@sarenet.es>
 <babd7232-d3cf-46fe-b329-705318747fb2@beta.fastmail.com>
 <7A0EEC1B-AC15-4FD9-8F77-1EDF5BB5301E@sarenet.es>
 <29AB96F4-D0F9-4F8A-B468-1C0AF86BED38@sarenet.es>
 <242498A8-F723-4409-A22B-1C6AE4492B10@sarenet.es>
 <217F3314-66BE-4F3F-A643-F1A1AABC77EC@sarenet.es>
 <1D63115C-8FA1-466A-8479-024D8B6D7F8F@sarenet.es>
 <A8B2AC91-1E32-4E71-8227-1446B75DACE1@sarenet.es>
 <be23d64f-0634-403e-be42-7a089576be67@www.fastmail.com>
 <635EE199-CE7A-4CAA-8098-3E217F26650F@sarenet.es>
 <bb36c021-cb42-4ec4-bd50-3eb393c2892a@www.fastmail.com>
 <7A5E1847-C444-474C-A344-7E41BBDFEA1E@sarenet.es>
 <cd76fdd8-757b-45e9-beff-6bad482ab8e8@www.fastmail.com>
 <8435EBDA-5314-43AE-A1FF-90F9616F364D@sarenet.es>
 <e5f71459-8cf0-7636-b4f4-539279cf0d31@unnoba.edu.ar>
Message-ID: <B8B8000A-4796-4635-8962-9A112B5E62A7@sarenet.es>

A ti Javier :)



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 18 jul 2019, a las 16:27, Infraestructura TIC - UNNOBA <tecnologia at unnoba.edu.ar> escribi?:
> 
> Much?simas gracias, Egoitz!
> 
> 
> El 18/7/19 a las 11:12, Egoitz Aurrekoetxea escribi?:
>> Hi!!,
>> 
>> Fine! Very happy sharing then :) :) . It only handles email. For Calendars/Contacts we have been long time now, using Davical (to which we contributed in it?s day https://wiki.davical.org/index.php/DAViCal-cli <https://wiki.davical.org/index.php/DAViCal-cli>) . We don?t refuse to use Caldav with Cyrus, it?s just we did the system previous to Cyrus Caldav system.
>> 
>> I attach the code in this email. I explain how we use it. We have each mailbox server running this code as a cron job and we have too some servers with Cyrus IMAP for just storing removed content (without the cron obviously). Each user in the restore server (a normal mailbox server but just for storing deleted content) is something like : user_domain at recuperaciones.saremail.com <mailto:user_domain at recuperaciones.saremail.com>. All our servers have autocreate feature (although in our mailbox servers is not being used nowadays). So, we keep track of what has been removed in a mailbox server with two elements? the Cyrus log and cyradm command. With cyradm command we keep track of deleted ?folders" in each user account. With the log, we know where expunges had been run. Later, we take the DELETED mailboxes (the folders of each user) and upload them to Saremail-Restore. After that, we check the log (from some hours before till the present moment). Then we ask unexpunge to see what has been removed in each place. We upload them. We keep track in a database of what exactly has dealed with and what is remaining to deal with, so in the case a fail over to a slave is produced unexpunges can then be run there, even if there?s nothing in the logs that say that (because it?s obviously a slave).
>> 
>> 
>> If you think it could be useful, perhaps could be uploaded to contrib directory?
>> 
>> Cheers!
> 
> -- 
> Lic. Javier Charne
> Responsable Infraestructura Tecnol?gica
> Prosecretar?a de TIC | UNNOBA
> Jun?n, Buenos Aires, Argentina
> javier at unnoba.edu.ar <mailto:javier at unnoba.edu.ar>
> Tel: +54 (0236) 4407750 int 11712
> Cel: +5492364542182
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190718/484eefad/attachment.html>

From brong at fastmailteam.com  Thu Jul 18 20:09:24 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Fri, 19 Jul 2019 10:09:24 +1000
Subject: Suggested feature and contribution
In-Reply-To: <6595AD29-BECE-45F9-9DFE-3FB65B74C099@sarenet.es>
References: <6595AD29-BECE-45F9-9DFE-3FB65B74C099@sarenet.es>
Message-ID: <420856de-6401-4591-b9b4-595f9c4f97bb@www.fastmail.com>

On Thu, Jul 18, 2019, at 19:09, Egoitz Aurrekoetxea wrote:
> Good morning,
> 
> When using delete_delayed if someone removes a big folder (that one with more than 20 subfolders anywhere below it) in mboxlist_delayed_deletemailbox() only last 20 are preserved. We think it could be a good idea to preserve all and to have a parameter for configuring it. The reason for that, is that we use delete_delayed for storing the removed content remotely with the customer hired retention period in slow disk space. Perhaps could be a good idea something like : 
> 
> In mboxlist_delayed_deletemailbox() : 
> 
> *If (!preserve_delete_delayed_folders_always)*
> *{*
>  /* keep the last 19, so the new one is the 20th */
>  for (i = 0; i < (int)existing.count - 19; i++) {
>  const char *subname = strarray_nth(&existing, i);
>  syslog(LOG_NOTICE, "too many subfolders for %s, deleting %s (%d / %d)",
>  newname, subname, i+1, (int)existing.count);
>  r = mboxlist_deletemailbox(subname, 1, userid, auth_state, NULL, 0, 1, 1,
>  keep_intermediaries);
>  if (r) goto done;
>  }
> *}*

Hmm.... yeah, OK. This is actually buggy in that case! The intended behaviour was to avoid a Denial of Service attack where you would create and delete the same mailbox name millions of times - however, the whole concept is bogus because there's nothing stopping somebody creating and deleting folder000001 through folderFFFFFF and creating the same attack.

I suggest that we just remove this whole silly check entirely, and if we want a similar level of attack protection we do something smarter like a quota for total folders+deleted folders that haven't been cleaned up yet - set it high enough that anybody hitting that is clearly doing something wrong, and require the administrator semi-manually clean up the deleted folders in order to re-allow folder creation.

Cheers,

Bron.

-- 
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190719/3f724840/attachment.html>

From brong at fastmailteam.com  Thu Jul 18 20:13:12 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Fri, 19 Jul 2019 10:13:12 +1000
Subject: Suggested feature and contribution
In-Reply-To: <420856de-6401-4591-b9b4-595f9c4f97bb@www.fastmail.com>
References: <6595AD29-BECE-45F9-9DFE-3FB65B74C099@sarenet.es>
 <420856de-6401-4591-b9b4-595f9c4f97bb@www.fastmail.com>
Message-ID: <770665f2-3960-46ba-b408-4501754796b4@www.fastmail.com>



On Fri, Jul 19, 2019, at 10:09, Bron Gondwana wrote:
> On Thu, Jul 18, 2019, at 19:09, Egoitz Aurrekoetxea wrote:
>> Good morning,
>> 
>> When using delete_delayed if someone removes a big folder (that one with more than 20 subfolders anywhere below it) in mboxlist_delayed_deletemailbox() only last 20 are preserved. We think it could be a good idea to preserve all and to have a parameter for configuring it. The reason for that, is that we use delete_delayed for storing the removed content remotely with the customer hired retention period in slow disk space. Perhaps could be a good idea something like : 
>> 
>> In mboxlist_delayed_deletemailbox() : 
>> 
>> *If (!preserve_delete_delayed_folders_always)*
>> *{*
>>  /* keep the last 19, so the new one is the 20th */
>>  for (i = 0; i < (int)existing.count - 19; i++) {
>>  const char *subname = strarray_nth(&existing, i);
>>  syslog(LOG_NOTICE, "too many subfolders for %s, deleting %s (%d / %d)",
>>  newname, subname, i+1, (int)existing.count);
>>  r = mboxlist_deletemailbox(subname, 1, userid, auth_state, NULL, 0, 1, 1,
>>  keep_intermediaries);
>>  if (r) goto done;
>>  }
>> *}*
> 
> Hmm.... yeah, OK. This is actually buggy in that case! The intended behaviour was to avoid a Denial of Service attack where you would create and delete the same mailbox name millions of times - however, the whole concept is bogus because there's nothing stopping somebody creating and deleting folder000001 through folderFFFFFF and creating the same attack.
> 
> I suggest that we just remove this whole silly check entirely, and if we want a similar level of attack protection we do something smarter like a quota for total folders+deleted folders that haven't been cleaned up yet - set it high enough that anybody hitting that is clearly doing something wrong, and require the administrator semi-manually clean up the deleted folders in order to re-allow folder creation.
> 
> Cheers,
> 
> Bron.

I've pushed commits to master and 3.0 to remove this check.

--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190719/87da0a14/attachment.html>

From egoitz at sarenet.es  Fri Jul 19 03:47:54 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Fri, 19 Jul 2019 09:47:54 +0200
Subject: Suggested feature and contribution
In-Reply-To: <770665f2-3960-46ba-b408-4501754796b4@www.fastmail.com>
References: <6595AD29-BECE-45F9-9DFE-3FB65B74C099@sarenet.es>
 <420856de-6401-4591-b9b4-595f9c4f97bb@www.fastmail.com>
 <770665f2-3960-46ba-b408-4501754796b4@www.fastmail.com>
Message-ID: <B4A34E33-B138-4753-BCA1-E5A6D4FE9E24@sarenet.es>

Thanks a lot Bron!

Yeah I saw your commit in 2015? I think the idea is fine but perhaps an alert should be better for that instead of a restriction?. Normally people is not going to do this kind of silly things? And when done is normally a mua driven crazy (we know everyone which mua I?m talking about :p )?.

I?ll tell about this fix to a customer of us who removed tons of folder? weren?t at saremail_restore (our deleted items recover system) and I started taking a look at what was going and arrived to this lines of mboxlist.c (I think it was..)?


Cheers!


Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 19 jul 2019, a las 2:13, Bron Gondwana <brong at fastmailteam.com> escribi?:
> 
> 
> 
> On Fri, Jul 19, 2019, at 10:09, Bron Gondwana wrote:
>> On Thu, Jul 18, 2019, at 19:09, Egoitz Aurrekoetxea wrote:
>>> Good morning,
>>> 
>>> When using delete_delayed if someone removes a big folder (that one with more than 20 subfolders anywhere below it) in mboxlist_delayed_deletemailbox() only last 20 are preserved. We think it could be a good idea to preserve all and to have a parameter for configuring it. The reason for that, is that we use delete_delayed for storing the removed content remotely with the customer hired retention period in slow disk space. Perhaps could be a good idea something like : 
>>> 
>>> In mboxlist_delayed_deletemailbox() : 
>>> 
>>> If (!preserve_delete_delayed_folders_always)
>>> {
>>>     /* keep the last 19, so the new one is the 20th */
>>>     for (i = 0; i < (int)existing.count - 19; i++) {
>>>         const char *subname = strarray_nth(&existing, i);
>>>         syslog(LOG_NOTICE, "too many subfolders for %s, deleting %s (%d / %d)",
>>>                newname, subname, i+1, (int)existing.count);
>>>         r = mboxlist_deletemailbox(subname, 1, userid, auth_state, NULL, 0, 1, 1,
>>>                                    keep_intermediaries);
>>>         if (r) goto done;
>>>     }
>>> }
>> 
>> Hmm.... yeah, OK.  This is actually buggy in that case!  The intended behaviour was to avoid a Denial of Service attack where you would create and delete the same mailbox name millions of times - however, the whole concept is bogus because there's nothing stopping somebody creating and deleting folder000001 through folderFFFFFF and creating the same attack.
>> 
>> I suggest that we just remove this whole silly check entirely, and if we want a similar level of attack protection we do something smarter like a quota for total folders+deleted folders that haven't been cleaned up yet - set it high enough that anybody hitting that is clearly doing something wrong, and require the administrator semi-manually clean up the deleted folders in order to re-allow folder creation.
>> 
>> Cheers,
>> 
>> Bron.
> 
> I've pushed commits to master and 3.0 to remove this check.
> 
> --
>   Bron Gondwana, CEO, Fastmail Pty Ltd
>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190719/d5d52903/attachment-0001.html>

From egoitz at sarenet.es  Fri Jul 19 03:48:41 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Fri, 19 Jul 2019 09:48:41 +0200
Subject: Suggested feature and contribution
In-Reply-To: <B4A34E33-B138-4753-BCA1-E5A6D4FE9E24@sarenet.es>
References: <6595AD29-BECE-45F9-9DFE-3FB65B74C099@sarenet.es>
 <420856de-6401-4591-b9b4-595f9c4f97bb@www.fastmail.com>
 <770665f2-3960-46ba-b408-4501754796b4@www.fastmail.com>
 <B4A34E33-B138-4753-BCA1-E5A6D4FE9E24@sarenet.es>
Message-ID: <CC31094E-73D3-456C-A316-7C6212CB6C83@sarenet.es>

When said an alert I meant a Nagios alert for instance?

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia)
egoitz at sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electr?nico piense si es necesario hacerlo.

> El 19 jul 2019, a las 9:47, Egoitz Aurrekoetxea <egoitz at sarenet.es> escribi?:
> 
> Thanks a lot Bron!
> 
> Yeah I saw your commit in 2015? I think the idea is fine but perhaps an alert should be better for that instead of a restriction?. Normally people is not going to do this kind of silly things? And when done is normally a mua driven crazy (we know everyone which mua I?m talking about :p )?.
> 
> I?ll tell about this fix to a customer of us who removed tons of folder? weren?t at saremail_restore (our deleted items recover system) and I started taking a look at what was going and arrived to this lines of mboxlist.c (I think it was..)?
> 
> 
> Cheers!
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 
>> El 19 jul 2019, a las 2:13, Bron Gondwana <brong at fastmailteam.com <mailto:brong at fastmailteam.com>> escribi?:
>> 
>> 
>> 
>> On Fri, Jul 19, 2019, at 10:09, Bron Gondwana wrote:
>>> On Thu, Jul 18, 2019, at 19:09, Egoitz Aurrekoetxea wrote:
>>>> Good morning,
>>>> 
>>>> When using delete_delayed if someone removes a big folder (that one with more than 20 subfolders anywhere below it) in mboxlist_delayed_deletemailbox() only last 20 are preserved. We think it could be a good idea to preserve all and to have a parameter for configuring it. The reason for that, is that we use delete_delayed for storing the removed content remotely with the customer hired retention period in slow disk space. Perhaps could be a good idea something like : 
>>>> 
>>>> In mboxlist_delayed_deletemailbox() : 
>>>> 
>>>> If (!preserve_delete_delayed_folders_always)
>>>> {
>>>>     /* keep the last 19, so the new one is the 20th */
>>>>     for (i = 0; i < (int)existing.count - 19; i++) {
>>>>         const char *subname = strarray_nth(&existing, i);
>>>>         syslog(LOG_NOTICE, "too many subfolders for %s, deleting %s (%d / %d)",
>>>>                newname, subname, i+1, (int)existing.count);
>>>>         r = mboxlist_deletemailbox(subname, 1, userid, auth_state, NULL, 0, 1, 1,
>>>>                                    keep_intermediaries);
>>>>         if (r) goto done;
>>>>     }
>>>> }
>>> 
>>> Hmm.... yeah, OK.  This is actually buggy in that case!  The intended behaviour was to avoid a Denial of Service attack where you would create and delete the same mailbox name millions of times - however, the whole concept is bogus because there's nothing stopping somebody creating and deleting folder000001 through folderFFFFFF and creating the same attack.
>>> 
>>> I suggest that we just remove this whole silly check entirely, and if we want a similar level of attack protection we do something smarter like a quota for total folders+deleted folders that haven't been cleaned up yet - set it high enough that anybody hitting that is clearly doing something wrong, and require the administrator semi-manually clean up the deleted folders in order to re-allow folder creation.
>>> 
>>> Cheers,
>>> 
>>> Bron.
>> 
>> I've pushed commits to master and 3.0 to remove this check.
>> 
>> --
>>   Bron Gondwana, CEO, Fastmail Pty Ltd
>>   brong at fastmailteam.com <mailto:brong at fastmailteam.com>
>> 
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190719/b2978485/attachment.html>

From awilliam at whitemice.org  Mon Jul 22 08:59:25 2019
From: awilliam at whitemice.org (Adam Tauno Williams)
Date: Mon, 22 Jul 2019 08:59:25 -0400
Subject: Suggested feature and contribution
In-Reply-To: <CC31094E-73D3-456C-A316-7C6212CB6C83@sarenet.es>
References: <6595AD29-BECE-45F9-9DFE-3FB65B74C099@sarenet.es>
 <420856de-6401-4591-b9b4-595f9c4f97bb@www.fastmail.com>
 <770665f2-3960-46ba-b408-4501754796b4@www.fastmail.com>
 <B4A34E33-B138-4753-BCA1-E5A6D4FE9E24@sarenet.es>
 <CC31094E-73D3-456C-A316-7C6212CB6C83@sarenet.es>
Message-ID: <1563800365.2962.6.camel@whitemice.org>

On Fri, 2019-07-19 at 09:48 +0200, Egoitz Aurrekoetxea wrote:
> When said an alert I meant a Nagios alert for instance?

Almost every NMS could catch this if there was a distinct enough log
[syslog] message.   It would be nice if more subsystems produced
distinct log messages on suspicious|strange events.

-- 
Executive Committee Chair
Michigan Association of Railroad Passengers
537 Shirley St NE Grand Rapids, MI 49503-1754 Phone: 616.581.8010
E-mail: awilliam at whitemice.org GPG#D95ED383 


From falon at ruparpiemonte.it  Tue Jul 23 09:52:02 2019
From: falon at ruparpiemonte.it (Marco)
Date: Tue, 23 Jul 2019 15:52:02 +0200
Subject: Suggested feature and contribution
In-Reply-To: <420856de-6401-4591-b9b4-595f9c4f97bb@www.fastmail.com>
References: <6595AD29-BECE-45F9-9DFE-3FB65B74C099@sarenet.es>
 <420856de-6401-4591-b9b4-595f9c4f97bb@www.fastmail.com>
Message-ID: <8dfd6fce-e956-3eab-7f13-9b4dfa94b1c6@ruparpiemonte.it>

Il 19/07/2019 02:09, Bron Gondwana ha scritto:
> Hmm.... yeah, OK.? This is actually buggy in that case!? The intended 
> behaviour was to avoid a Denial of Service attack where you would create 
> and delete the same mailbox name millions of times

Hello,

  I had never seen this with mailbox, but I saw something similar with 
messages and delayed expunge enabled.

Some broken clients, such as Open-Xchange, in some obscure circumstances 
enjoy to recreate and delete the same mail in the Drafts folder forever 
and ever, until the partition fills up.

Cheers
Marco

From chentaocredungtao at yahoo.com  Mon Jul 29 08:23:34 2019
From: chentaocredungtao at yahoo.com (Chentao Credungtao)
Date: Mon, 29 Jul 2019 13:23:34 +0100
Subject: how to list all mailboxes having a specific annotation ?
Message-ID: <da002475-44e0-3da2-8193-1e411fd40bb3@yahoo.com>

Hi,

For some mailboxes, I set the "squat" annotation to "false".

But I can't remember on which mailboxes I did that...

Is there a way to list all mailboxes having that annotation set to false ?

Thanks


From ellie at fastmail.com  Mon Jul 29 21:13:15 2019
From: ellie at fastmail.com (ellie timoney)
Date: Tue, 30 Jul 2019 11:13:15 +1000
Subject: Cyrus IMAP 3.0.11 released
Message-ID: <e2b87dbd-e195-4bb3-971f-c9cb456adc40@www.fastmail.com>

The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.0.11

I'm trialling hosting the release files using Github's releases feature.  Please use the Github download links if possible, and advise if you have any problems!  (It may even download faster due to Github's content delivery network.)

This release contains a fix for a potential data loss issue when upgrading mail servers containing messages that existed prior to Cyrus 2.3.  For details, please see the release notes.  If upgrading a service that has been around a long time, consider upgrading to 3.0.11 rather than one of the earlier 3.0 releases.

Download URLs:

    https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.11/cyrus-imapd-3.0.11.tar.gz
    https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.11/cyrus-imapd-3.0.11.tar.gz.sig

    https://www.cyrusimap.org/releases/cyrus-imapd-3.0.11.tar.gz
    https://www.cyrusimap.org/releases/cyrus-imapd-3.0.11.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 3.0.11:

    https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.11.html
    https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

From chentaocredungtao at yahoo.com  Tue Jul 30 09:22:41 2019
From: chentaocredungtao at yahoo.com (Chentao Credungtao)
Date: Tue, 30 Jul 2019 14:22:41 +0100
Subject: is it safe to manually remove DELETED folders ?
Message-ID: <20854ccd-6f33-4b2c-c77d-c11ab0beebc0@yahoo.com>

Hello

My main question : is it safe to manually remove (rm -rf) a folder like?? 
/domain/e/example.net/*u/DELETED/user/someuser/Trash/INBOX/5BCA0EB9* ?

The folder doesn't show with cyradm ??? listmailboxes.


As a side question, and out of curiosity : any idea how come a DELETED 
folder doesn't show with cyradm ??? listmailboxes ?

I know exactly what happened, October last year our webmail (SOGo) had a 
bug (fixed since) : when a user pressed on <Delete> without selecting an 
e-mail, the whole INBOX ended up in trash. So I'm not surprise to have 
this /u/DELETED/user/someuser/Trash/INBOX/5BCA0EB9 folder, what 
surprises me is that it's not listed with cyradm (other DELETED folders 
_are_ listed with cyradm > lm DELETED*).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190730/fe7e3941/attachment.html>

From Hagedorn at uni-koeln.de  Tue Jul 30 09:32:23 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Tue, 30 Jul 2019 15:32:23 +0200
Subject: is it safe to manually remove DELETED folders ?
In-Reply-To: <20854ccd-6f33-4b2c-c77d-c11ab0beebc0@yahoo.com>
References: <20854ccd-6f33-4b2c-c77d-c11ab0beebc0@yahoo.com>
Message-ID: <590ae43d-88db-e6ad-22b8-3404bf5965d5@uni-koeln.de>

You shpuld check the output of

ctl_mboxlist -d

Only if the DELETED mailbox(es) aren't included there is it safe to just
remove them.
-- 
    .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
   .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.

Am 30.07.19 um 15:22 Uhr schrieb Chentao Credungtao via Info-cyrus:
> Hello
> 
> My main question : is it safe to manually remove (rm -rf) a folder
> like??
> /domain/e/example.net/*u/DELETED/user/someuser/Trash/INBOX/5BCA0EB9* ?
> 
> The folder doesn't show with cyradm ??? listmailboxes.
> 
> 
> As a side question, and out of curiosity : any idea how come a DELETED
> folder doesn't show with cyradm ??? listmailboxes ?
> 
> I know exactly what happened, October last year our webmail (SOGo) had a
> bug (fixed since) : when a user pressed on <Delete> without selecting an
> e-mail, the whole INBOX ended up in trash. So I'm not surprise to have
> this /u/DELETED/user/someuser/Trash/INBOX/5BCA0EB9 folder, what
> surprises me is that it's not listed with cyradm (other DELETED folders
> _are_ listed with cyradm > lm DELETED*).
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0x197B06994D105B45.asc
Type: application/pgp-keys
Size: 17099 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190730/3f237dd4/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Hagedorn.vcf
Type: text/x-vcard
Size: 333 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190730/3f237dd4/attachment-0001.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5367 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190730/3f237dd4/attachment-0001.p7s>

From chentaocredungtao at yahoo.com  Tue Jul 30 10:05:30 2019
From: chentaocredungtao at yahoo.com (Chentao Credungtao)
Date: Tue, 30 Jul 2019 15:05:30 +0100
Subject: is it safe to manually remove DELETED folders ?
In-Reply-To: <590ae43d-88db-e6ad-22b8-3404bf5965d5@uni-koeln.de>
References: <20854ccd-6f33-4b2c-c77d-c11ab0beebc0@yahoo.com>
 <590ae43d-88db-e6ad-22b8-3404bf5965d5@uni-koeln.de>
Message-ID: <dcaa0a27-ef26-3e0e-9c2c-5d0a3d3752aa@yahoo.com>

Thanks. And done (they weren't included),

Out of curiosity : does your answer imply that in some cases some 
mailboxes will show with ctl_mboxlist, but not with cyradm lm ? In which 
case ?


On 07/30/2019 02:32 PM, Sebastian Hagedorn wrote:
> You shpuld check the output of
>
> ctl_mboxlist -d
>
> Only if the DELETED mailbox(es) aren't included there is it safe to just
> remove them.


From gbulfon at sonicle.com  Wed Jul 31 04:42:18 2019
From: gbulfon at sonicle.com (Gabriele Bulfon)
Date: Wed, 31 Jul 2019 10:42:18 +0200 (CEST)
Subject: Cyrus 2.5.11 default domain on auth
Message-ID: <1243978856.960.1564562538571@www>

Hello,
?
we have Cyrus installations with virtual domains for years.
Users usually authenticate via imap as
name.lastname at domain.com
?and get to their inboxes.
?
I have now a situation where I need to let users authenticate just with name.lastname and let the system fill in the default domain.
Here is our saslauthd.conf:
?
ldap_servers: ldap://localhost/
ldap_default_domain: sonicle.com
ldap_search_base: ou=People,dc=%2,dc=%1
ldap_auth_method: bind
ldap_filter: uid=%u
and this imapd.conf:
allowusermoves: 1
lmtp_downcase_rcpt: 1
configdirectory: /sonicle/var/imap
partition-default: /sonicle/var/spool/imap
sievedir: /sonicle/var/sieve
sieve_extensions: fileinto reject vacation vacation-seconds imapflags imap4flags notify envelope relational regex subaddress copy date
sieve_utf8fileinto: 1
sieve_vacation_min_response: 0
admins: sonicle
sasl_pwcheck_method: saslauthd
allowplaintext: yes
sasl_mech_list: plain
tls_ca_file: /sonicle/ssl/certs/acme/fullchain.pem
tls_cert_file: /sonicle/ssl/certs/acme/cert.pem
tls_key_file: /sonicle/ssl/certs/acme/key.pem
altnamespace: yes
virtdomains: userid
unixhierarchysep: 1
duplicatesuppression: 0
sendmail: /sonicle/sbin/sendmail
xlist-archive: Archives
xlist-drafts: Drafts
xlist-sent: Sent
xlist-spam: Spam
xlist-trash: Trash
If I try adding defaultdomain, what happens is that authentication works both with and without specifying domain, but then you're not on your normal inbox of user/name.lastname at domain.com but somewhere else strange, a SELECT INBOX returns mailbox not found.
?
What's wrong? How can I achieve my goal?
?
Thanks!
Gabriele
?
Sonicle S.r.l.?
:?
http://www.sonicle.com
Music:?
http://www.gabrielebulfon.com
Quantum Mechanics :?
http://www.cdbaby.com/cd/gabrielebulfon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190731/613c6ac6/attachment.html>

From Hagedorn at uni-koeln.de  Wed Jul 31 07:57:42 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Wed, 31 Jul 2019 13:57:42 +0200
Subject: is it safe to manually remove DELETED folders ?
In-Reply-To: <dcaa0a27-ef26-3e0e-9c2c-5d0a3d3752aa@yahoo.com>
References: <20854ccd-6f33-4b2c-c77d-c11ab0beebc0@yahoo.com>
 <590ae43d-88db-e6ad-22b8-3404bf5965d5@uni-koeln.de>
 <dcaa0a27-ef26-3e0e-9c2c-5d0a3d3752aa@yahoo.com>
Message-ID: <f35010c4-2899-94b0-ef2a-ef0482867298@uni-koeln.de>

To be honest, I'm not sure. But the output of "cyradm lm" depends on the
user you're logged in as, whereas ctl_mboxlist always dumps all
mailboxes. That's why I feel it is more reliable.

Am 30.07.19 um 16:05 Uhr schrieb Chentao Credungtao:
> Out of curiosity : does your answer imply that in some cases some
> mailboxes will show with ctl_mboxlist, but not with cyradm lm ? In which
> case ?
> 
> 
> On 07/30/2019 02:32 PM, Sebastian Hagedorn wrote:
>> You shpuld check the output of
>>
>> ctl_mboxlist -d
>>
>> Only if the DELETED mailbox(es) aren't included there is it safe to just
>> remove them.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0x197B06994D105B45.asc
Type: application/pgp-keys
Size: 17099 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190731/341eefe1/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Hagedorn.vcf
Type: text/x-vcard
Size: 333 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190731/341eefe1/attachment-0001.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5367 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190731/341eefe1/attachment-0001.p7s>

From chentaocredungtao at yahoo.com  Wed Jul 31 08:11:38 2019
From: chentaocredungtao at yahoo.com (Chentao Credungtao)
Date: Wed, 31 Jul 2019 13:11:38 +0100
Subject: is it safe to manually remove DELETED folders ?
In-Reply-To: <f35010c4-2899-94b0-ef2a-ef0482867298@uni-koeln.de>
References: <20854ccd-6f33-4b2c-c77d-c11ab0beebc0@yahoo.com>
 <590ae43d-88db-e6ad-22b8-3404bf5965d5@uni-koeln.de>
 <dcaa0a27-ef26-3e0e-9c2c-5d0a3d3752aa@yahoo.com>
 <f35010c4-2899-94b0-ef2a-ef0482867298@uni-koeln.de>
Message-ID: <9e4ee498-2cd4-c477-4e66-36e77ff70608@yahoo.com>

good point !

thanks


On 07/31/2019 12:57 PM, Sebastian Hagedorn wrote:
> To be honest, I'm not sure. But the output of "cyradm lm" depends on the
> user you're logged in as, whereas ctl_mboxlist always dumps all
> mailboxes. That's why I feel it is more reliable.
>
> Am 30.07.19 um 16:05 Uhr schrieb Chentao Credungtao:
>> Out of curiosity : does your answer imply that in some cases some
>> mailboxes will show with ctl_mboxlist, but not with cyradm lm ? In which
>> case ?
>>
>>
>> On 07/30/2019 02:32 PM, Sebastian Hagedorn wrote:
>>> You shpuld check the output of
>>>
>>> ctl_mboxlist -d
>>>
>>> Only if the DELETED mailbox(es) aren't included there is it safe to just
>>> remove them.


From vladislav.kurz at webstep.net  Wed Jul 31 08:44:23 2019
From: vladislav.kurz at webstep.net (Vladislav Kurz)
Date: Wed, 31 Jul 2019 14:44:23 +0200
Subject: is it safe to manually remove DELETED folders ?
In-Reply-To: <20854ccd-6f33-4b2c-c77d-c11ab0beebc0@yahoo.com>
References: <20854ccd-6f33-4b2c-c77d-c11ab0beebc0@yahoo.com>
Message-ID: <2e9b3492-436d-9bbe-c824-d4488b125556@webstep.net>

On 30/07/2019 15:22, Chentao Credungtao via Info-cyrus wrote:
> Hello
> 
> My main question : is it safe to manually remove (rm -rf) a folder
> like??
> /domain/e/example.net/*u/DELETED/user/someuser/Trash/INBOX/5BCA0EB9* ?
> 
> The folder doesn't show with cyradm ??? listmailboxes.

Hello,

I think this is the result of  imapd.conf:

delete_mode: delayed
deletedprefix: DELETED

So - it should disappear after some days set in cyrus.conf e.g:
deleteprune     cmd="/usr/sbin/cyrus expire -a -E 4 -D 28" at=0430


-- 
Best Regards
        Vladislav Kurz