Tasks status shows PENDING even after is was SUCCESS
I have noticed, that after I fitted the model and generated a synthetic dataset with the status SUCCESS or FAILURE, after some time I see the status as PENDING for all the datasets.
In more details:
I store the task_id in the model metadata just after we make a call on:
Then get the task status as:
tasks = meta_data.attributes.get('tasks') if tasks: task_id = tasks[dataset_store_name] task = om.runtime.celeryapp.AsyncResult(task_id) task_status = task.status
So, when I make a call to get status right after the fit-generate was requested I get a PENDING status.
Sometime later it shows the SUCCESS status, and everything seems to work fine. I even have a dataset that throws a FAILURE status (for test purposes).
But, after a while when I come back all the datasets show PENDING. Despite that I already have the model fitted, and synthetic dataset generated.
Currently, I provided a workaround for it, like:
if task_status == TaskStatus.PENDING and om.datasets.list(synthetic_dataset_store_name): task_status = TaskStatus.SUCCESS
But seems like this issue should be solved somehow, not avoided.
Could you, please, help me with finding the reason why this is happening and what can be done to avoid it?