Additional Information and Tips
Note the following information about analytic GPUs:
- Each analytic GPU node has 8 AMPs. (Standard and analytic nodes have 24 AMPs.)
- At run time, each AMP is allocated to 1 container for a total of 8 containers.
- Similar to analytic clusters, each analytic GPU node can run two APPLY queries concurrently.
- Each analytic GPU compute cluster node has 4 GPUs. Each GPU is assigned two runtime containers that share the GPU.
- Each GPU has 16GB of memory. Since two containers share the GPU, it can only load up to 8GB per model. Use quantization and memory reduction techniques to load bigger models into GPU memory.The following references are from Hugging Face:
- Quantization - https://huggingface.co/docs/optimum/en/concept_guides/quantization
- Speed up inference - https://huggingface.co/docs/diffusers/en/optimization/fp16
- Reduce memory usage - https://huggingface.co/docs/diffusers/en/optimization/memory
When installing the PyTorch package with CUDA support in Python Conda user environments, you need to specify the packages pytorch::pytorch-cuda and nvidia::cuda-cudart in addition to pytorch::pytorch in requirements.txt to resolve the dependencies.
Limitations
- Only LLM/DL inference is supported
- Model (.zip) file size limit is 5GB.
- Only batch job inference use cases are supported, that is, read input data from a table and write the results to a table.
- For ML inference scripts, you must specify the GPU device in the Python script, else it will only run on the vCPU on the node.
- Use teradataml 20.00.00.02 or later to create Open Analytics Framework user environments, upload models, run Apply queries, and so on.