OmegaML Celery app

daria · February 2021

Greetings!

Currently, inside the OmegaML there is already initialized Celery app with the range of tasks to perform:

om.runtime.model(model_store_name).predict(...)

In our project, we want to run some methods asynchronously also using Celery.

As for now, OmegaML worker is running in a separate container. We are planning to create a new Celery app inside our project and run it as a separate Docker container ('celery_worker'), connect the same MongoDB, etc

Are there any possibilities to connect to the OmegaML Celery app and create new tasks as the already existing in the OmegaML?

In order to not create a new structure above the existing one.

support · February 2021

we want to run some methods asynchronously also using Celery.

Are there any possibilities to connect to the OmegaML Celery app and create new tasks

It is of course possible to write your own celery tasks and run them inside a separate celery worker, although it is not possible to extend a running celery worker with your own tasks (as tasks are loaded on starting up the worker).

However in omegaml the same can be easily achieved by using a script (=pip installable package) or a jupyter notebook. E.g.

# assume "my notebook" contains your code
$ om runtime job "my notebook" --async

# or in python
result = om.runtime.job('my notebook').run()
...
result.get()

Effectively this runs the "my notebook" notebook as a celery task. The same is true of a script:

# assume "mypackage" contains your code
$ om runtime script "mypackage" --async

# or in python
result = om.runtime.script('mypackage').run()
...
result.get()

The advantage of doing it like this is that you get automated tracking in metadata, streams integration, access from the REST API as well as support for buckets (= multiple versions of the same package deployed in parallel). With "native" celery you would have to build this functionality separately.

If you decide that for some reason you need to run your own tasks, e.g. because you want to integrate some other package, it is always possible to deploy an extra celery worker with a separate queue and adding your own package (where `mypackage.tasks` contains your celery tasks).

$ celery -A omegaml.celeryapp worker -Q mytasks -I mypackage

Then you can send tasks e.g.

om.runtime.celeryapp.send_task('mypackage.tasks.some_task`, args=..., kwargs=..., queue='mytasks')

https://docs.celeryproject.org/en/stable/reference/celery.html#celery.Celery.send_task

OmegaML Celery app

Comments