For AI agents: the complete documentation index is available at https://docs.us.ovhcloud.com/en/llms.txt, the full documentation bundle is available at https://docs.us.ovhcloud.com/en/llms-full.txt, and this page is available as Markdown at https://docs.us.ovhcloud.com/en/public-cloud/ai-machine-learning/data-concepts-best-practices.md.
  • English

Data - Concept and best practices

View as Markdown

Definition

  • Data relates to any type of file (binary, text, code, images, etc.) that you want to use with OVHcloud AI tools.
  • Object Storage is a scalable, resilient, and secure storage place accessible from anywhere through HTTPS APIs. It is a perfect place to store static files for the long term.
  • Volumes are filesystem storage units mounted and used inside AI Training jobs and AI notebooks.

 

Best practices

OVHcloud Object Storage should be used to persist any data needed by AI Training jobs or AI notebooks.

There are two ways to manage your data:

 

How it works

AI Training jobs and AI Notebooks can read and write data from and to the OVHcloud Object Storage. Here is what is happening under the hood.

 

AI Training

  1. Before the job starts, data is synchronized from the object storage into an underlying filesystem volume. This synchronization is done during the INITIALIZING phase.
  2. When the job starts, the volume is attached to the desired directory. Data is then available inside the job as long as the RUNNING phase lasts.
  3. After the job stops, data is synchronized back from the underlying filesystem volume into the OVHcloud Object Storage. This synchronization is done during the FINALIZING phase.

PaaS AI general - step 1

 

AI Notebooks

  1. Before the notebook starts, data is synchronized from the object storage into an underlying filesystem volume. This synchronization is done during the STARTING phase.
  2. When the notebook starts, the volume is attached to the desired directory. The data is then available within the job as long as the RUNNING phase lasts.
  3. After the notebook stops, data is synchronized back from the underlying filesystem volume into the OVHcloud Object Storage. This synchronization is done during the STOPPING phase.

This applies to your internal /workspace and remote object storage volumes.

PaaS AI general - step 2

 

Capabilities

Access rights

Users can give three different access rights on Filesystem volumes attached to jobs and notebooks:

Read Only (RO): You are only able to read data in your job or notebook. It is forbidden to write, modify, or delete data in this container.

When to attach a volume in read-only RO...

The purpose of read-only permissions is to ensure that you do not modify your data by mistake. This permission is recommended for your input data: images, videos, sounds, text, CSV files, etc.

Warning

NOTE: Volumes that are in read-only mode are not synchronized with the Object Storage during the FINALIZING or STOPPING phase because there is no point in synchronizing data that has not changed during the life of a job or notebook.

Info

If you have directly added data from the Object Storage to the container connected in RO, you can request an intermediate synchronization. Otherwise, your data will be synchronized at the start of your next job or notebook.

Read Write (RW): You have full access to your data in your job or notebook. You can write, modify, or add data in this container.

When to attach a volume in read-write RW...

The purpose of the read-write permission is to modify the contents of the object container. This permission is recommended if your container contains code (e.g., a Python file), a notebook, or if you want to save a model or connection weights after your training.

Info

Volumes that are in read-write do not allow you to delete data from your Object Storage. If you delete data from a job or notebook, it will still be present in your object container, even after synchronization.

Read Write Delete (RWD): You have full access to your data in your job or notebook. You can write, modify, add, or delete data in this container.

When to attach a volume in read-write-delete RWD...

The purpose of the read-write-delete permission is to be able to modify or delete the object container contents. You can not only make changes to the different files in your container (Python code, requirements file, etc.) but also, especially, delete data from it. The RWD will allow you to permanently delete data you no longer need from your object container. This permission can be used if you want to delete old codes or connection weights, which can be heavy files.

Warning

NOTE: After the synchronization phase, your data deleted from your notebook or job will be permanently deleted in the Object Storage.

Info

For a volume in RW or RWD: As long as your job or notebook is in the FINALIZING or STOPPING state, this means that the upload is still in progress. Once the state changes to INTERRUPTED or STOPPED, it means all the data was uploaded to your Object Storage.

 

Volume caching and sharing

By default, filesystem volumes are created and deleted on the fly for each job as needed.

Users can enable a cache feature that allows jobs and notebooks to reuse available volumes instead of creating a new one each time. That feature serves two purposes:

  • Reducing synchronization time for data that has already been downloaded before.
  • Sharing the same volumes between several jobs or notebooks.
Warning

NOTE: Unused volume data is deleted after three days. Users should not rely on this cache feature for long-term persistence.

 

Go further

Browse all the AI & Machine Learning guides.

Browse the rest of the OVHcloud US documentation for guides on our other products and services.

Learn more about AI & Machine Learning on the OVHcloud US website.

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.