OPF-Ray is a managed Ray offering on the AI Studio platform for running distributed Python, data engineering, machine learning, and model development workloads. It gives users self-service Ray clusters backed by CPU or GPU compute, a notebook-based development experience, access to Teradata data sources, and built-in observability through the Ray Dashboard.
With OPF-Ray, users can do all of the following in one environment:
- Create and delete Ray clusters without manually managing Kubernetes resources
- Run distributed Python tasks and data pipelines
- Train and tune models using Ray libraries such as Ray Train and Ray Tune
- Read data from Teradata using the standard SQL driver
- Persist checkpoints and artifacts to external storage
- Monitor cluster health, tasks, logs, and resource consumption
This guide helps users onboard to Open Python Framework on AI Studio, create a Ray cluster, connect from a notebook, access Teradata data, run distributed workloads, monitor execution, and clean up resources when finished.
For more information about Open Python Framework and Ray clusters see:
- Getting Started with Ray
- ray.data.read_sql
- Ray Dashboard
- Teradata AgentStack (See https://pypi.org/project/teradata-agentstack/)
Terminology used in this guide
- User:
- Administrator - Configures AI Studio, node pools, quotas, access, and platform settings.
- Data Scientist - Creates clusters, explores data, trains models, tunes experiments, and runs distributed notebooks.
- ML Engineer - Builds scalable ML workflows, packages workloads, and operationalizes model training or inference.
- Business Analyst - Runs approved notebooks and reviews outputs and results.
- Ray cluster: A cluster with one head node and one or more worker nodes.
- Node pool: The AI Studio compute pool where the Ray cluster is scheduled.
- Head node: The cluster coordinator that manages scheduling and cluster services.
- Worker node: A node that executes distributed tasks.