This procedure creates an auto-scalable cluster of Google Cloud instances with QueryGrid installed. When deploying QueryGrid to hosts that are members of a Google Cloud Instance Group, you can automate the tdqg-node installations and registration process by specifying a bootstrap script that runs on all newly added members of the Google Cloud Instance Group.
The required script, TDQG_DEPLOYMENT-scriptVersion.sh is packaged in the node package tdqg-node-version.tar.gz. The script can be either uploaded to Google Storage or its contents can be copied and pasted into the startup-script field.
- You have required privileges to create Instance Templates and Instance Groups.
- The cURL tool is installed on all nodes where you intend to install QueryGrid.If the node does not have Java installed, OpenJDK 11 is installed.
- Add a system in Viewpoint and download the tdqg-node.json token file that was generated by the QueryGrid Manager.For information about downloading tdqg-node.json, see Adding Nodes Manually.
- Download the node package, see Downloading Required Packages.
- Unzip the package using the tar -xzvf tdqg-node-version.tar.gz command.TDQG_DEPLOYMENT.sh is available in the qgdeployment/googlecloud path.
- In the Google Cloud Instance Template, select Create an Instance template and provide the following key value pairs to the Management Metadata section:
Required key value pair tdqg_node_json
The contents of the tdqg-node.json token file that was generated by QueryGrid Manager.For non-SLES Google Cloud instances, you can use the Google Cloud storage address for the file as the value using the following format:
Provide one of the following key value pairs startup-script
The contents of the TDQG_DEPLOYMENT-scriptVersion.sh file.
Path to the TDQG_DEPLOYMENT-scriptVersion.sh file on Google Storage.
Optional key value pair scale-in-enabled:trueThis enables automated scale-in performed by the QueryGrid software when a driver host has been idle for a certain amount of time. When combined with an Autoscale only out policy defined in the Google Cloud Instance Group, both scale in and scale out are automated.Because this task is performed by QueryGrid, if set to true, scale in occurs even if the Google Cloud Instance Group is set to Do Not Autoscale.
- In the Google Cloud Instance template, select Create Instance Group then select New managed instance group stateless after making your selections for name, region, and so on.For Instance Template, select the instance template you created for use with QueryGrid and select either Autoscale only out or Don't autoscale.Allowing Google Cloud to perform the autoscale-in rather than using the QueryGrid scale-in-enabled feature can cause queries to fail while the scale-in is being performed.Example: Combining the Google Cloud Autoscale only out policy with the QueryGrid automated scale-in-enabled feature.Configuration:
- BigQuery connector machine type: n2-standard-2 (2 cores, 8GB memory)
- Autoscaling policy: CPU utilization, 80%
- 4 concurrent users making 10 consecutive queries on an integer column in a table with 14 million rows: group size increases to 2 instances
- 8 concurrent users making 10 consecutive queries on an integer column in a table with 14 million rows: group size increases to 3 instances
- Group size reduces back to 1 instance after 15 minutes of idle time