Extending IPGeo - Aster Analytics

Teradata AsterĀ® Analytics Foundation User GuideUpdate 2

Product
Aster Analytics
Release Number
7.00.02
Published
September 2017
Language
English (United States)
Last Update
2018-04-17
dita:mapPath
uce1497542673292.ditamap
dita:ditavalPath
AA-notempfilter_pdf_output.ditaval
dita:id
B700-1022
lifecycle
previous
Product Category
Software

Because IPGeo cannot cover all the IP database providers for technical and license reasons, you can extend this function to support new database providers.

Only Maxmind GeoLite 1.2.8 ships with this function.

To extend IPGeo:

  1. Create a new class that implement the interface Converter. The interface Converter, which is defined in ipgeo.jar, ships with the analytics functions in this release. If you are using Aster Analytics Foundation version 5.10 or later, you can extract ipgeo.jar from ipgeo.zip. The interface Converter is defined as:
    package com.asterdata.sqlmr.analytics.location.ipgeo;
    public interface Converter
    {
    /**
    * initialize a Converter instance with corresponding resource
    * @param ipDatabasePath
    */
    void initialize(String ipDatabasePath);
    /**
    * release resources used by this instance before the SQL-MR function close
    */
    void finalize();
    /**
    * Lookup location information for the input ipv4 address and write the result to a IpLocation instance
    * @param ip
    *    input, IP address in ipv4 format
    * @param ipLocation
    *    output, to hold the location information
    */
    void findIpv4(String ip, IpLocation ipLocation);
    /**
    *
    * Lookup location information for the input ipv6 address and write the result to a IpLocation instance
    * @param ip
    *    input, IP address in ipv6 format
    * @param ipLocation
    *    output, to hold the location information
    */
    void findIpv6(String ip, IpLocation ipLocation);
    }

    Class IpLocation holds and emits location information (which is also in the ipgeo.jar file). The code has get and set functions for the following member variables, which correspond to the SQL-MapReduce function output:

    private String countryCode = null;
    private String countryName = null;
    private String state = null;
    private String city = null;
    private String postalCode = null;
    private float latitude = -1;
    private float longitude = -1;
    private String isp = null;
    private String organization = null;
    private String organizationType = null;
    private int areaCode = -1;
    private int metroCode = -1;
    private int dmaCode = -1;
    }

    The following implementation of MaxMindLite (MaxMindLite2)is an example converter for the MaxMind GeoLite2 database.

    package com.asterdata.sqlmr.analytics.location.ipgeo;
    import java.io.BufferedInputStream;
    import java.io.BufferedOutputStream;
    import java.io.File;
    import java.io.FileNotFoundException;
    import java.io.FileOutputStream;
    import java.io.IOException;
    import java.net.InetAddress;
    import java.net.UnknownHostException;
    import com.asterdata.ncluster.sqlmr.IllegalUsageException;
    import com.asterdata.ncluster.sqlmr.data.InstalledFile;
    import com.asterdata.sqlmr.analytics.location.ipgeo.Converter;
    import com.maxmind.geoip2.DatabaseReader;
    import com.maxmind.geoip2.exception.GeoIp2Exception;
    import com.maxmind.geoip2.model.CityResponse;
    /**
    * A Converter implementation for MaxMind GeoLite2 version
    */
    public class MaxMindLite2 implements Converter
    {
    private DatabaseReader reader = null;
    private static final String CITY_DATABASE = "GeoLite2-City.mmdb";
    private String tmpCityDatabase_ = null;
    //initialize the databaseReader
    public void initialize(String ipDatabasePath)
    {
    if(ipDatabasePath == null)
    {
    tmpCityDatabase_ = downloadFile(CITY_DATABASE);
    initializeDatabaseReader(tmpCityDatabase_);
    }
    else
    {
    String path = ipDatabasePath.endsWith("/") ?
    ipDatabasePath + CITY_DATABASE : ipDatabasePath + "/" + CITY_DATABASE ;
    initializeDatabaseReader(path);
    }
    }
    //find address according to ipv4 address
    public void findIpv4(String ip, IpLocation ipLocation)
    {
    CityResponse response = null;
    try
    {
    response = reader.city(InetAddress.getByName(ip));
    }
    catch (UnknownHostException e)
    {
    // do nothing
    }
    catch (IOException e)
    {
    // do nothing
    }
    catch (GeoIp2Exception e)
    {
    // do nothing
    }
    if (response != null) {
    setOutput(response, ipLocation);
    }
    }
    //find address according to ipv6 address
    public void findIpv6(String ip, IpLocation ipLocation)
    {
    findIpv4(ip, ipLocation);
    }
    //release resources
    public void finalize()
    {
    if(reader != null)
    {
    try
    {
    reader.close();
    }
    catch (IOException e)
    {
    // do nothing
    }
    }
    if(tmpCityDatabase_ != null)
    {
    new File(tmpCityDatabase_).delete();
    }
    }
    //Set the output to ipLocation from response
    private void setOutput(CityResponse response, IpLocation ipLocation)
    {
    ipLocation.setCountryCode(response.getCountry().getIsoCode());
    ipLocation.setCountryName(response.getCountry().getName());
    ipLocation.setState(response.getMostSpecificSubdivision().getName());
    ipLocation.setCity(response.getCity().getName());
    if(null != response.getLocation().getLatitude())
    {
    ipLocation.setLatitude(response.getLocation().getLatitude().floatValue());
    }
    if(null !=  response.getLocation().getLongitude())
    {
    ipLocation.setLongitude(response.getLocation().getLongitude().floatValue());
    }
    if(null != response.getLocation().getMetroCode())
    {
    ipLocation.setMetroCode(response.getLocation().getMetroCode().intValue());
    }
    ipLocation.setPostalCode(response.getPostal().getCode());
    ipLocation.setIsp(response.getTraits().getIsp());
    ipLocation.setOrganization(response.getTraits().getOrganization());
    }
    //save a file installed in Aster to file system
    private String downloadFile(String file)
    {
    BufferedInputStream in = null;
    try
    {
    in = new BufferedInputStream(InstalledFile.getFile(file).getStream());
    }
    catch (FileNotFoundException e1)
    {
    throw new IllegalUsageException("CAN'T found the default IP database," +
    " please check whether following file has been installed to Aster: " + file);
    }
    String tmpFile = "/tmp/" + file + System.currentTimeMillis();
    byte[] buffer = new byte[1024];
    BufferedOutputStream out = null;
    try
    {
    out = new BufferedOutputStream(new FileOutputStream(tmpFile));
    for(int len = in.read(buffer); len>-1; len=in.read(buffer))
    {
    out.write(buffer, 0, len);
    }
    }
    catch (FileNotFoundException e)
    {
    throw new IllegalUsageException("CAN'T create a tmp file: " + tmpFile
    + ". Please check the conflication in system");
    }
    catch (IOException e)
    {
    throw new IllegalUsageException("CAN'T write to tmp file: " + tmpFile
    + ". Please check if folder /tmp has more than 100M bytes free space.");
    }
    finally
    {
    try{
    if (in != null) in.close();
    if (out != null) out.close();
    }
    catch(IOException e)
    {
    //do nothing
    }
    }
    return tmpFile;
    }
    //Return a DatabaseReader for the specified file
    private void initializeDatabaseReader(String file)
    {
    // A File object pointing to your GeoIP2 or GeoLite2 database
    File database = new File(file);
    if(database.exists())
    {
    try {
    // This creates the DatabaseReader object, which should be reused across lookups.
    reader = new DatabaseReader.Builder(database).build();
    } catch (IOException e) {
    throw new IllegalUsageException("CAN'T initialize DatabaseReader. See details:" + e.getMessage());
    }
    }
    else
    {
    throw new IllegalUsageException("CAN'T find IP database: " + file);
    }
    }
    }
  2. Compile the new Converter class and package it in a JAR file with all the dependent libraries.

    For this example, you must package this class, and the classes in the following JAR files, to a new JAR file:

    • ipgeo.jar
    • MaxMind-DB-Reader (from MaxMind)
    • jackson-core-2.2.3.jar (this package is required by geolite2; version 2.2.3 is tested)
    • jackson-databind-2.2.3.jar
    • jackson-annotations-2.2.3.jar
  3. Install the JAR file on Aster Database.
  4. When calling the IPGeo function, set the and JAR filename and class name parameters of the Converter argument to the names of your JAR file and class.