<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Sep 25, 2012, at 11:57 AM, Deniss <<a href="mailto:cyrus@sad.lv">cyrus@sad.lv</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><br><br>On 25.09.2012 15:28, Eric Luyten wrote:<br><blockquote type="cite">On Tue, September 25, 2012 2:01 pm, Sebastian Hagedorn wrote:<br><blockquote type="cite">Hi,<br><br><br>about three weeks ago we upgraded our Cyrus installation from 2.3.x to 2.4.16.<br>We were aware of the reindexing issue, so we took precautionary<br>measures, but they didn't help a lot. We've got about 7 TB of mail data for<br>almost 200,000 mailboxes. We did the upgrade on a Sunday and had told our<br>users that mail access wouldn't be possible for the whole day. After the<br>actual software upgrade we ran distributed scripts that triggered the index<br>upgrades. We started with the largest mailboxes. The idea was that after those<br>that took the longest had been upgraded, the rest should be OK overnight and<br>early Monday. However, even though our storage infrastructure was kept at 99 %<br>I/O saturation, progress was much slower than anticipated.<br><br><br>Ultimately the server was virtually unuseable for the whole Monday and<br>parts of Tuesday. The last mailbox was finally upgraded on Thursday, although<br>on Wednesday most things were already working normally.<br><br>I realize that some of our problems were caused by infrastructure that's<br>not up to current standards, but nonetheless I would really urge you to never<br>again use an upgrade mechanism like that. Give admins a chance to upgrade<br>indexes in the background and over time.<br></blockquote><br><br>+1<br><br><br>Sebastian,<br><br><br>Thank you for sharing your experiences.<br><br>As a site willing/needing to upgrade from 2.3.16 to 2.4.X this fall, we<br>are interested in learning about your storage backend characteristics.<br><br>What read/write IOPS rates were you registering before/during/after your<br>upgrade process ?<br><br>I'd understand your reluctance to share this information in a public forum.<br>No offence taken whatsoever !<br><br><br>Kind regards,<br>Eric Luyten, Computing Centre VUB/ULB, <a href="mailto:Eric.Luyten@vub.ac.be">Eric.Luyten@vub.ac.be</a><br></blockquote><br><br>migration process from 2.3 to 2.4 took ~ one year for our installation. <br>we converted ~200Tb of users data.<br>first step we did - spread data on many nodes using cyrus replication.<br>next we started converting nodes one by one at weekends nights to <br>minimize IO load generated by users.<br>in fact cyrus read all data from disk to generate new indexes, so <br>convert is limited by disk IO mainly while CPU is pretty cheap nowadays.<br>we got around 500Gb in 8 hours rate for forced reindex with 100% disk load.<br>we started forced reindex with most active users meanwhile allowing <br>users to login and trigger reindex of their mailboxes<br><br><br></blockquote><br></div><div><br></div><div>Sorry for hi-jacking this thread, but I'm curious as to the preferred method of forcing a reindex on a mailbox? I know it triggers when a user logs in and accesses the mailbox. I would like to divide up users and perform the reindex in chunks. </div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">Thanks,<br>Bryan<br><br>---<br>Bryan D. Hill<br>UCSD Physics Computing Facility<br>CTBP Systems Support<br><br>9500 Gilman Dr. # 0319<br>La Jolla, CA 92093<br>+1-858-534-5538<br><a href="mailto:bhill@ucsd.edu">bhill@ucsd.edu</a><br>AIM: pozvibesd<br>Web: <a href="http://www.physics.ucsd.edu/pcf">http://www.physics.ucsd.edu/pcf</a></span>
</div>
<br></body></html>