Persistent Data Stores for Ray Workloads - Teradata AI Studio

To support distributed training, fine-tuning, and checkpointing workloads running on Ray clusters, AI Studio provides an Amazon S3-based object storage solution. The storage is used for persisting intermediate checkpoints, training artifacts, and model outputs generated during Ray job execution.

Each deployment site is provisioned with a dedicated Amazon S3 bucket that is accessible to Ray workloads through IAM Roles for Service Accounts (IRSA), eliminating the need for static AWS credentials or additional client-side configuration.

For on-premises environments, you can access the S3 API compliant object storage information using the following code snippet and plug it into your Python workloads for persisting data. The Distributed Training with Ray Train example showcases how to use them correctly (See Distributed Training with Ray Train.)

ray.init("<ray_client_endpoint>", ignore_reinit_error=True)
@ray.remote
def get_bucket_property(name):
    import os
    return os.environ[name]

refs = [
    get_bucket_property.remote("S3_OBJECT_STORE_NAME"),
    get_bucket_property.remote("S3_OBJECT_STORE_ENDPOINT"),
    get_bucket_property.remote("S3_ACCESS_KEY_ID"),
    get_bucket_property.remote("S3_SECRET_ACCESS_KEY"),
]
print(ray.get(refs))

Also in on-premises environments, a shared filesystem is mounted on a common path on every node in the Ray cluster for storing workload artifacts. Each cluster’s mount path is unique and is located at /mnt/{cluster_name}. In the mounted example that follows, all files are saved to “/mnt/demo_cluster/experiment_name” for further processing.

from ray import train
from ray.train.torch import TorchTrainer

trainer = TorchTrainer(
    ...,
    run_config=train.RunConfig(
        storage_path="/mnt/demo_cluster ",
        name="experiment_name",
    )
)