Error: local variable 'omega' referenced before assignment

daria

Greetings!

In our project, we are using the CTGAN model to fit() and predict(). We have created a separate backend for it:

class CTGANBackend(BaseModelBackend):

...

@classmethod
def supports(cls, obj, name, **kwargs):
    return isinstance(obj, CTGANSynthesizer)

Previously we were using the ctgan package, and everything worked fine.

Recently we have replaced the ctgan package with a new package, called SDV which has a CTGAN model inside instead of the CTGANSynthesizer.

After we update the package use and replace the CTGANSynthesizer with CTGAN, we rebuild all the docker containers and restart the project. The updated backend also was installed.

After the rebuild the call to fit() returns the error:

Error: local variable 'omega' referenced before assignment

The whole traceback from the Celery events is:

 Traceback (most recent call last):

   File "/opt/conda/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task

    R = retval = fun(*args, **kwargs)

   File "/opt/conda/lib/python3.6/site-packages/omegaml/celery_util.py", line 97, in __call__

    return super().__call__(*args, **kwargs)

   File "/opt/conda/lib/python3.6/site-packages/celery/app/trace.py", line 641, in __protected_call__

    return self.run(*args, **kwargs)

   File "/opt/conda/lib/python3.6/site-packages/omegaml/notebook/tasks.py", line 80, in execute_scripts

    om = self.om

   File "/opt/conda/lib/python3.6/site-packages/omegaml/celery_util.py", line 80, in om

    self._om = get_omega_for_task(self)[bucket]

   File "/opt/conda/lib/python3.6/site-packages/omegaml/omega.py", line 195, in <lambda>

    get_omega_for_task = lambda task: _om.setup()

   File "/opt/conda/lib/python3.6/site-packages/omegaml/omega.py", line 162, in setup

    return omega

  UnboundLocalError: local variable 'omega' referenced before assignment

 As the error occurs while performing the Celery task inside the OmegaML it is rather complicated to test and debug the flow.

Do you have any suggestions concerning what may be wrong or where to take a closer look at to find the core of the issue?

Many thanks for your support!

Best regards,

Daria

support

Error: local variable 'omega' referenced before assignment

This should not happen. We are tracking this in https://github.com/omegaml/omegaml/issues/168

The root cause of the error however is likely with your omega-ml configuration. The error condition happens when the config.yml or the extension.yml cannot be loaded. For example, if the config.yml or extensions.yml reference a class to a non-existing or non-loadable backend. You may check this by removing any custom extensions and reloading omega-ml.

As the error occurs while performing the Celery task inside the OmegaML it is rather complicated to test and debug the flow.

To test the flow you may run your tasks locally by passing the --local parameter as follows

$ om runtime --local ... 

This will bypass sending the request to the remote omega-ml worker and run everything inside a single process on your local machine. This way all code can be debugged locally. It is also possible to debug celery remotely (see details in https://docs.celeryproject.org/en/stable/userguide/debugging.html). However please note we do not currently support this option in our hosted, cluster, mainly for security reasons.

daria

Thanks for the response!

Yes, I have to rebuild all the containers and delete all images and volumes.

After that - everything works fine!

Thanks!

Best regards,

Daria