Multiple instance support is available for write operations (writing to GCS). Using multiple instances of the DataConnector operator (each instance of the DataConnector operator will use a separate copy of the access module) will often improve performance of the object creation process. When multiple instances are used, the objects in GCS will have the following naming convention:
<base-object-name>-<instance number>
Thus, for example, with the Object=my_load_job specification in the initialization string, and using 3 instances of the DataConnector operator, the result will be 3 objects in GCS named:
- my_load_job-001
- my_load_job-002
- my_load_job-003
If you have an object name extension (for example, when trying to compress the object by specifying a .gz file extension, where Object=my_load_job.gz):
- my_load_job-001.gz
- my_load_job-002.gz
- my_load_job-003.gz