The Spark SQL initiator connector provides an easy-to-use, Java-based Foreign Server library. The library defines a "wrapper" Foreign Server class. The class contains a Foreign Server constructor to create one or more unique Foreign Server objects. Each Foreign Server object contains a catalog of permanent non-native tables that are backed up by a cache file on HDFS after a session ends. The Spark SQL initiator connector also contains static and instance methods to interact with the temporary and permanent non-native tables in each Foreign Server object and to retrieve metadata from remote tables.
The Spark SQL engine does not have objects similar to Foreign Servers (Teradata) or Catalogs (Presto). Therefore, the Spark SQL initiator connector operates at the table level by default.
Name | Parameter | Description |
---|---|---|
Constructors | ||
ForeignServer | String Link String Version String Name |
Construct a new Foreign Server object using a QueryGrid link. |
ForeignServer | String Name | Restore an existing Foreign Server. |
Static Methods | ||
showForeignServers | – | List all Foreign Server names. |
dropForeignServer | String Name | Drop a specified Foreign Server as well as all its non-native tables. |
sql | String SQL | Run a SQL statement (SELECT/INSERT/EXPLAIN only) and display all rows in result. |
getDatasetFromSql | String SQL | Run a SQL statement (SELECT/INSERT/EXPLAIN only) and return the Dataset<Row> object. |
Instance Methods | ||
showSchemas | – | List all target schemas. |
showTables | String Schema | List all tables under a target schema. |
showColumns | String Table | Display the schema of a target table specified by a fully-qualified name (such as, schema.table or catalog.schema.table). |
show | – | List each non-native table (temporary or permanent) under this Foreign Server and which target table it represents. |
show | String Name | Display the CREATE TABLE statement of a non-native table under this Foreign Server. This only works for permanent non-native tables, and the database must be part of the name.
|
create | String Table String Name |
Create a non-native table representing a target table (fully-qualified name) under this Foreign Server. In order to create a permanent non-native table, the database must be part of the name.
|
drop | String Name | Drop a non-native table under this Foreign Server. For permanent non-native tables, the database must be part of the name.
|
describe | String Name | Display the details of a non-native table under this Foreign Server. For permanent non-native tables, the database must be part of the name.
|
setSessionOverride | String Values | Set the connector properties to be overridden for the session. Properties are case-insensitive and non-overridable properties are ignored. Example: setSessionOverride("linkbuffersize=100000, numMappers=4") |