Repository / ephemeral Jupyter server

ktx

Hi, is there a clear pattern how to deal with the ephemeral Jupyter server. They can be restarted and one has to retooling (e.g. installing editor/vim) and re-cloning repos (inclusive typing in credentials) again..

I know one could develop/edit locally but I prefer the Jupyter server sometimes.

Thanks!

support

the ephemeral Jupyter server

The Jupyter servers as such are ephemeral as they do not store any local data. However notebooks, models and datasets are stored in mongodb, and accessible via om.jobs , om.models , andom.datasets respectively. This is n-way replicated & managed storage.

There are two options to store local files:

• /app/pylib/user - mapped to a node-local disk, shared across all processes on that node (SSD-backed, fast, up to 5GB of capacity by default, ideal to install python packages, clone git repositories for easy re-use, etc. Data stored here will persist across Jupyter and worker restarts, still, should not be considered permanent as a node recycling/replacement will also reset any data stored here)
• /app/pylib/base - mapped to a NFS drive, shared across all Jupyter sessions and workers in the same account/namespace, across nodes (up to 100MB, NFS-backed, slow, meant to store small data only e.g. configuration files. Data stored here will persist across all Jupyter Sessions, node restarts etc.)

The full storage architecture is summarized in slide "Advanced: Understanding Storage"

The above concerns the managed service at https://omegaml.io. In the open source / docker version the /app/pylib directory is mapped to a local docker volume, shared across Jupyter and all workers. The storage architecture in docker is roughly equivalent, except that the managed storage is the docker host volume and there is no NFS.

`