# The recovery process seems to be stuck

**URL:** <https://www.singlestore.com/forum/t/the-recovery-process-seems-to-be-stuck/8246>\
**Category:** Help\
**Tags:** developer\
**Created:** [May 19, 2025, 8:25am UTC](https://www.singlestore.com/forum/t/the-recovery-process-seems-to-be-stuck/8246 "2025-05-19T08:25:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thao158](https://www.singlestore.com/forum/letter_avatar_proxy/v4/letter/t/b19c9b/32.png) [@Thao158](https://www.singlestore.com/forum/u/Thao158)\
**Post date:** [May 19, 2025, 8:25am UTC](https://www.singlestore.com/forum/t/the-recovery-process-seems-to-be-stuck/8246/1 "2025-05-19T08:25:24Z")

</div>

Hi team,  
Our singlestore cluster is deployed on OCP environment, using ICOS bucket as bottomless.  
Recently we have seen the following error:  
Leaf Error (node-memsql-cluster-leaf-ag2-9.svc-memsql-cluster:3306): Failed to synchronize database. SYNC POSITION timed out waiting for replay.

I have checked the recovery process on this leaf and this database and I noticed that the recovery process does not seem to change. For example, partition coedev\_106 has a replay % completion stuck at 14.9% and the ETA increases over time to more than 20 days.

DATABASE\_NAME,DATABASE\_ID,REPLAY\_ETA,REPLAY\_STATUS,REPLAYED\_LSN,STARTUP\_COMMITED\_LSN,RECOVERY\_TYPE  
coedev\_106,48311,“14.9% done, ETA 22 days 21 hours 52 minutes 35 seconds”,Replaying log files downloaded from master,45265414,27983872,Restart  
coedev\_131,48138,“16.0% done, ETA 21 days 2 hours 34 minutes 51 seconds”,Replaying log files downloaded from master,45334751,28077843,Restart  
coedev\_80,39340,“16.3% done, ETA 20 days 12 hours 28 minutes 57 seconds”,Replaying log files downloaded from master,45327153,28079714,Restart  
coedev\_120,48313,“16.3% done, ETA 20 days 13 hours 27 minutes 22 seconds”,Replaying log files downloaded from master,45320522,28040183,Restart  
coedev\_95,48312,“21.1% done, ETA 14 days 22 hours 27 minutes 45 seconds”,Replaying log files downloaded from master,45160754,27887766,Restart  
coedev\_92,39341,“21.9% done, ETA 14 days 6 hours 14 minutes 27 seconds”,Replaying log files downloaded from master,45369935,28098154,Restart  
coedev\_127,48137,“22.0% done, ETA 14 days 4 hours 34 minutes 20 seconds”,Replaying log files downloaded from master,45116522,27852800,Restart  
coedev\_101,48030,“23.8% done, ETA 18 days 4 hours 28 minutes 26 seconds”,Replaying log files downloaded from master,45171119,23033073,Restart

Can anyone explain and how we can push the recovery process to complete?

---

<div class="post-metadata">

**Author:** ![mmanthena](https://www.singlestore.com/forum/letter_avatar_proxy/v4/letter/m/cc9497/32.png) [@mmanthena](https://www.singlestore.com/forum/u/mmanthena)\
**Post date:** [June 3, 2025, 11:00pm UTC](https://www.singlestore.com/forum/t/the-recovery-process-seems-to-be-stuck/8246/2 "2025-06-03T23:00:02Z")

</div>

> [@Thao158](#):
>
> Failed to synchronize database. SYNC POSITION timed out waiting for replay.

Thank you for reaching out to us. The “Failed to synchronize database. SYNC POSITION timed out waiting for replay” error is typically caused by insufficient resources to run the replay process. But, we’ll need more information to diagnose the issue.

Could you please provide your `mv_cluster_status`/`mv_bt` or a cluster report so our engineering can investigate further?

One more thing you can try is to restarting the node. This can sometimes resolve ICOS-related slowdowns or object storage access issues, which may help unblock the process.
