Java API for Register File

Applies to: Kyvos Enterprise  Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace   Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

Overview

1. Create a RegisteredFile object and set the HDFS folder path to where the CSV file is stored.

2. Create a RegisteredFile object using the HCatalog data source.

3. Iterate the columns in RegisteredFile to format (formatting may include changing the caption, input format, data type, and so on).

4. Add the RegisteredFile object in the repository.

5. Retrieve existing RegisteredFile object from repository.

6. Update CSV properties of existing RegisteredFile object:

   1. path of CSV file

   2. line separator character

   3. field separator character

   4. escape character

   5. enclosed by character

   6. skip top lines

   7. encoding style

   8. if file contains header

   9. if need to ignore empty rows

   10. to include file name as a column

7. Refresh RegisteredFile columns.

8. Update RegisteredFile object in the repository.

9. Data profile for File with ONCE option

10. Data profile for File with NOW option

Example

1. Create the RegisteredFile object (This API fetches the Registered File object and sets column details in the Fetch step)

   /**
   * method to create Registered File object in Repository Manager.
   * @param folderPath: Full path of the datasource file.
   * @param userInfo: Object of type UserInfo containing information about logged in user.
   * @return RegisteredFile: Object of type RegisteredFile.
   */
   public RegisteredFile registerCSVFile(String folderPath, UserInfo userInfo);

2. Create the RegisteredFile object using HCatalogSource.

   /**
   * method to create Registered File object in Repository Manager.
   * @param hcatalog: Object of type HCatalogSource.
   * @param userInfo: Object of type UserInfo containing information about logged in user.
   * @return RegisteredFile: Object of type RegisteredFile.
   */
   public RegisteredFile registerHCatalogDataSource(HCatalogSource hcatalog, UserInfo userInfo);
   /**
   * method to create Registered File object using HCatalog database and table name.
   * @param dbName: Name of the hCatalog database.
   * @param tableName: Name of the hCatalog table.
   * @param userInfo: Object of type UserInfo containing information about logged in user
   * @return RegisteredFile: Object of type RegisteredFile.
   * @throws ReportObjectException.
   */
   public RegisteredFile registerHCatalogDataSource(String dbName, String tableName, UserInfo userInfo);

3. Fetch columns from the RegisteredFile object.

   /*
   * @return ArrayList<FetchColumn>: ArrayList of FetchColumn objects.
   */
   public ArrayList<FetchColumn> getColumnDetails();
   /*
    * On iterating this ArrayList, user will get a FetchColumn object for each column.
   */
   public void setName(String columnName);
   public void setDataType(String dataType);
   public void setFieldFormat(String fieldFormat);
   public void setColumnDetails(ArrayList <FetchColumn> columnList);

4. Add the RegisteredFile object to the repository.

   /**
   * method to add new Registered File in Repository Manager.
   * @param regFileObj: Object of type RegisteredFile.
   * @return ResponseStatus: Object of type ResponseStatus.
   */
   public ResponseStatus addRegisteredFileObject(RegisteredFile regFileObj);
   /**  
   * method to add new Registered File inside a folder in Repository Manager.
   * @param regFileObj: Object of type RegisteredFile.
   * @param folderName: Name of the folder in which RegisteredFile has to be added.
   * @return ResponseStatus: Object of type ResponseStatus.
   */
   public ResponseStatus addRegisteredFileObject(RegisteredFile regFileObj,String folderName);

5. Retrieve existing RegisteredFile object from repository.

   /**
   * method to get RegisteredFile object by Registered File name in Repository Manager.
   * @param registeredFileName: name of the Registered File.
   * @param userInfo: Object of type UserInfo containing information about logged in user.
   * @return regFileObj: Object of type RegisteredFile.
   */
   public RegisteredFile getRegisteredFileByName(String registeredFileName, UserInfo userInfo);
   /**
   * method to get RegisteredFile object by Registered File name and folder name in Repository Manager.
   * @param registeredFileName: Name of the Registered File.
   * @param folderName: Name of the folder that contains the Registered File.
   * @param userInfo: Object of type UserInfo containing information about logged in user.
   * @return regFileObj: Object of type RegisteredFile.
   */
   public RegisteredFile getRegisteredFileByName(String registeredFileName,String folderName,UserInfo userInfo);

6. Update CSV properties of existing RegisteredFile object.

   1. Update CSV file path

      /**
      * method to update CSV file path of existing Registered File.
      * @param csvPath: Full path of the CSV file.
      */
      public void updateCSVPath(String csvPath);

   2. Update line separator character.

      /**
      * method to update line separator of existing Registered File.
      * @param lineSeparator: Line separator used within the Registered File.
      */
      public void updateLineSeparator(String lineSeparator);

   3. Update field separator character.

      /**
      * method to update field separator of existing Registered File.
      * @param fieldSeparator: Field separator used within the Registered File.
      */
      public void updateFieldSeparator(String fieldSeparator);

   4. Update escape character

      /**
      * method to update Escape Character of existing Registered File.
      * @param escapeChar: Escape character used within the Registered File.
      */
      public void updateEscapeCharacter(String escapeChar);

   5. Update enclosed by character.

      /**
      * method to update enclosure of existing Registered File.
      * @param enclosedBy: Enclosed-by character used within the Registered File.
      */
      public void updateEnclosedBy(String enclosedBy);
      

   6. Update skip toplines.

      /**
      * method to update skip top lines of existing Registered File.
      * @param skipTopLine: Specify whether to skip top line of the Registered File or not. ‘1’: to skip, else ‘0’ to include top line.
      */
      public void updateSkipTopLines(int skipTopLine); 

   7. Update encoding style.

      /**
      * method to update encoding of existing Registered File.
      * @param encoding: Encoding style of the Registered File.
      */
      public void updateEncoding(String encoding);

   8. Update if file contains header.

      /**
      * method to update contains header of existing Registered File.
      * @param header: Specify whether to include header of the Registered File or not.
      */
      public void updateHeader(boolean header);

   9. Update if need to ignore empty rows

      /**
      * method to update ignoreEmptyRows of existing Registered File.
      * @param ignoreEmptyRows: Specify whether to ignore empty rows of the Registered File or not.
      */
      public void updateIgnoreEmptyRows(boolean ignoreEmptyRows);

   10. Update to include file name as column.

       /**
       * method to update nameAsColumn of existing Registered File.
       * @param nameAsColumn: Specify whether to include file name as column of the Registered File or not.
       */
       public void updateNameAsColumn(boolean nameAsColumn);

7. Refresh RegisteredFile columns upon updating CSV properties.

   /**
   * method to refresh Registered File Column.
   */
   public void refreshRegisteredFileColumn();

8. Update the RegisteredFile object in the repository.

   /**
   * method to update existing Registered File in Repository Manager.
   * @param regFileObj: Object of type RegisteredFile.
   */
   public void updateRegisteredFileObject(RegisteredFile regFileObj);
   

9. Preview on Register Files.

   /* method to preview on Register File 
   @return iResultSet: Type of object IResultSet 
   */ 
   public preview() code

10. Data profile for File with ONCE option.

    /**
    * method to add ONCE Data profile schedule for file object
    * @param qo: Object of type QueryObject containing file info
    * @param onceDateTime: date time in format mm/dd/yyyy hh:mm
    * @param profile: Object of type DataProfile containing field information
    * @param userInfo: Object of type UserInfo containing information about logged in user
    * @param buildConnectionName: name of build connection
    * @throws ClientException
    */
    public void profileFileOnce(QueryObject qo, String onceDateTime, DataProfile profile, UserInfo userInfo, String buildConnectionName) throws ClientException

11. Data profile for File with NOW option. 

    /**  
    * method to add NOW Data profile schedule for file object  
    * @param qo: Object of type Query Object containing file info
    * @param profile: Object of type DataProfile containing field information  
    * @param userInfo: Object of type UserInfo containing information about logged in user 
    * @param buildConnectionName: name of build connection  
    * @throws ClientException
    */
    public void profileFileNow(QueryObject qo, DataProfile profile, UserInfo userInfo, String buildConnectionName) throws ClientException

12.  Constructor to create a blank file containing a fetch step and a filter step linked together.

    /**
    * RegisteredFile's default constructor - to create RegisteredFile object with empty fetch and filter step linked to each other.
    */
    public RegisteredFile() throws Exception

13. Set an ID of the registered file.

    /** Method to add registered file id.
    * @param id The id to set.*/public void setId(String id)

14. Get a registered file id.   

    /** Method to get registered file id
    * @return Returns the id.
    */
    public String getId()

15.  Set a folder id of the registered file.   

    /**
    * This method to set folder id for File.
    * @param String id;
    */
    public void setFolderId(String folderId)

16. Set a folder name of the registered file.

    /**
    * This method to set folder name for File.
    * @param String name;
     */
    public void setFolderName(String folderName)

17. Set a connection name in the registered file to connect with the data source.

    /**
     * method to set connection name of existing Registered File
     * @param connectionName String
     */
    public void setConnectionName(String connectionName)

18.  Set input type in a file.

    /**
    * Method to create data source and update fetch step as per the provided input type.
    * @param RFInputTypes: Input Type for File.
    */
    public void setConnectionName(String connectionName)

19. Add a filter in the registered file. 

    /**
    * Method to add filters in the filter step.
    * @param fieldName
    * @param operation
    * @param firstValue
    * @param secondValue
    * @param fieldDataType
    */
    public void addFilter(String fieldName, String operation , String firstValue, String secondValue, String fieldDataType)

20. Add or update a user parameter. 

    /**
    * Method to add or update a parameter. if a parameter with given name already exists then it will update it, else will add a new parameter.
    * @param name
    * @param value
    * @param description
    */
    public void addOrUpdateQOPrameter(String name, String value, String description)

21. Remove a user parameter from the file. 

    /**
    * Method to remove a parameter by it's name.
    * @param name
    */
    public void removeParameter(String name)

22. Set column metadata into the file.

    /**
    * This method is used to column details into a Registered file object
    * @param ArrayList of FetchColumn
    */
    public void setColumnDetails(List columnDetails) 

23. Get the input type of the file.

    /**
    * Method to return inputType
    * @return RFInputTypes: Input Type for File.
    */
    public RFInputTypes getInputType()

24. Get FilterStep from the registered file.

    /**
    * method to get FilterStep from registered file
    * @return FilterStep object.
    */
    public FilterStep getFilterStep()

25. Set and get partition column details on the registered File.

    1. Get incremental identifier column name.

       /**
       * This method to get incremental identifier column name.
       * @return incremental identifier column name.
       */
       public void setPartitionColumnName(String val)

    2. Set partition table name.

       /**
       * This method to set partition table name.
       * @param String name.
       */
       public void setPartitionTableName(String val)

    3. Set the partition column's metadata mode. 

       /**
       * This method to set partition column's metadata mode
       * @param mode
       */
       public void setPartitionColumnMetadataMode(String mode)

    4. Set the partition column's minimum value.

       /**
       * This method to set partition column's min value
       * @param minvalue
       */
       public void setPartitionColumnMinValue(String minvalue)

    5. Set the partition column's maximum value.

       /**
       * This method to set partition column's max value
       * @param maxvalue
       */
       public void setPartitionColumnMaxValue(String max)

    6. Set the partition column's record count.

       /**
       * This method to set partition column's record count
       * @param recordCount
       */
       public void setPartitionTableRecordCount(String count)

    7. Set the number of partitions.

       /**
       * This method to set number of partitions
       * @param partitions
       */
       public void setNumberOfPartitions(String val)

26. Set incremental identifier column in the registered file.

    /**
    * This method to set incremental identifier column in registered file.
    * @param String identifier column name;
    */
    public void setIncrementalIDColumnName(String val)

Sample Code

/*Method to create RF based on CSV as Input*/ 
public static void addCSVRF(RepositoryManager repManager, UserInfo userInfo) { 
RegisteredFile rf; 
try { 
rf = repManager.registerCSVFile("CSV File Path", userInfo); 
rf.setName("RFName"); 
rf.getQueryObject().setConnectionName("ConnectionName"); 
rf.setUserInfo(userInfo); 
/*Folder to save RF must be present on Kyvos 
 *Refer to class CreateFolder to create folder via API*/ 
repManager.addRegisteredFileObject(rf, "FolderName"); 
} catch (Exception e) { 
e.printStackTrace(); 
} 
} 
/*Method to create RF based on Database and Table Name present on Kyvos Database*/ 
public static void addHCatalogRF(RepositoryManager repManager, UserInfo userInfo) { 
RegisteredFile rf; 
try { 
rf = repManager.registerHCatalogDataSource("DBName", "TableName", userInfo); 
rf.setName("RFName"); 
rf.getQueryObject().setConnectionName("ConnectionName"); 
rf.setUserInfo(userInfo); 
/*Folder to save RF must be present on Kyvos*/ 
repManager.addRegisteredFileObject(rf, "FolderName"); 
} catch (Exception e) { 
e.printStackTrace(); 
} 
       }