The paths to the foreign table objects include variables, so the CREATE FOREIGN TABLE statement can omit PATHPATTERN and PARTITION BY. Vantage generates PATHPATTERN and PARTITION BY, using the variables in the paths as partition column names and PATHPATTERN variable names.
Paths to Foreign Table Objects
Variables are bold.
/S3/s3.amazonaws.com/crimedata-vcol-pq/city=sf/year_x=2017/month_x=Oct/Day21.parquet /S3/s3.amazonaws.com/crimedata-vcol-pq/city=ny/year_x=2015/month_x=May/Day12.parquet /S3/s3.amazonaws.com/crimedata-vcol-pq/city=ny/year_x=2014month_x=Mar/Day8.parquet /S3/s3.amazonaws.com/crimedata-vcol-pq/city=sf/year_x=2019/month_x=Jan/Day14.parquet /S3/s3.amazonaws.com/crimedata-vcol-pq/city=sf/year_x=2017/month_x=Sep/Day21.parquet /S3/s3.amazonaws.com/crimedata-vcol-pq/city=ny/year_x=2014/month_x=Apr/Day6.parquet /S3/s3.amazonaws.com/crimedata-vcol-pq/city=ny/year_x=2013/month_x=Jan/Day4.parquet /S3/s3.amazonaws.com/crimedata-vcol-pq/city=ny/year_x=2013/month_x=Feb/Day3.parquet /S3/s3.amazonaws.com/crimedata-vcol-pq/city=ny/year_x=2015/month_x=Jun/Day12.parquet
CREATE FOREIGN TABLE Statement
CREATE MULTISET FOREIGN TABLE CRIMEDATA,
EXTERNAL SECURITY DEFINER TRUSTED S3
USING (
LOCATION ('/s3/s3.amazonaws.com/crimedata-vcol-pq/')
);
SHOW TABLE Output
CREATE MULTISET FOREIGN TABLE CRIMEDATA, FALLBACK,
EXTERNAL SECURITY DEFINER TRUSTED S3_AUTH,
MAP = TD_MAP1
(
Location VARCHAR(2048) CHARACTER SET UNICODE CASESPECIFIC,
"index" BIGINT,
ID_Number BIGINT,
Case_Number VARCHAR(8) CHARACTER SET UNICODE NOT CASESPECIFIC,
Year_x BIGINT,
Month_x VARCHAR(3) CHARACTER SET UNICODE NOT CASESPECIFIC,
Day_x BIGINT,
Arrest BYTEINT,
District FLOAT)
USING
(
LOCATION ('/s3/s3.amazonaws.com/crimedata-vcol-pq/')
MANIFEST ('FALSE')
PATHPATTERN ('$city/$year_x/$month_x/$var4')
STOREDAS ('PARQUET')
)
NO PRIMARY INDEX
PARTITION BY (COLUMN, city CHAR(2), Year_x BIGINT, Month_x VARCHAR(3));