If the Enable Model Evaluation option is not selected for BYOM model, the Model Evaluation step will not be available on the Model Lifecycle page. See Approving or Rejecting a Model Version for the next lifecycle step.
Evaluate is the next step in the model lifecycle when you evaluate a trained model using the evaluation dataset. This dataset is used to see how well the model performs on historical data.
Model evaluation generates an evaluation report that displays a number of metrics to monitor the model performance. Once the model has been evaluated, you can compare models, mark a model as champion, and view/download artifacts. These tasks are discussed in the following sections:
- On the Model Version Lifecycle page, select Evaluate.
- In the Basic tab, set the following properties:
Property Description Dataset template Specify the required dataset template. For details, see Dataset Templates.
Dataset Specify the dataset to be used for evaluation job. For details, see Datasets.
- In the Advanced tab, set the following properties:
Property Description Engine Specify the engine to evaluate the model. Docker image Specify the docker image to be used to run evaluation script. Resource template Identify a predefined set of resources, including CPU and memory, which are the properties of the container created to run the task in. Select S Standard, M Medium, L Large, or Custom from the Template drop-down list. Set any of the following properties:- Memory: Specify the allocated RAM memory for the container. Must be an integer followed by the unit (m for megabytes or g for gigabytes).
- CPU: Specify the allocated CPU units (cores) for the container, can be an integer or decimal number. Accepts m as the unit for milicores.
- GPU (visible when Custom is selected): Specify the allocated GPU units for the container, can be an integer or decimal number. Accepts m as the unit for milicores.
- Select Evaluate model.The model version evaluation progress displays.
- Select to close the sheet when the evaluation progress completes.The Model Version Lifecycle page displays. The Evaluate step in the header is marked as completed and the model version status changes to Evaluated.
- Select to expand the Evaluation details section to see the details of the evaluation job.
The following details display:
Property Description Job ID Specifies the evaluation job ID. You can select View job details to see the event details of the job. For details, see Jobs.
Date Specifies the evaluation date. User Specifies the username who executed the evaluation job. Status Shows the status of the evaluation job as Completed Dataset ID Displays the testing dataset ID used to evaluate the job. You can select View Dataset Statistics and View Dataset to see the dataset details. For details, see Datasets.
Dataset name Displays the testing dataset name used to evaluate the job. Resources Specifies the resources utilized in the evaluation job, including CPU and Memory. Job progress Lists down all phases of the evaluation job. The job progress information includes: - Status: Status of each phase as Created, Scheduled, Running, Evaluated, Completed
- Start Date: Start date and time for each phase
- End Date: End date and time for each phase
- Duration: Duration of each phase
Evaluation artifacts Lets you view and download evaluation artifacts for auditability purposes, approve or reject models based on the charts, plots or statistics generated, or export the models. For details, see Viewing and Downloading Model Artifacts.