java.lang.Object
- com.c12e.cortex.profiles.module.job.IngestDataSourceJob

All Implemented Interfaces:: java.lang.Runnable

public class IngestDataSourceJob
extends java.lang.Object
implements java.lang.Runnable

Ingest a DataSource

Field Summary

Fields
Modifier and Type	Field	Description
`static java.util.function.Function<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>,org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>`	`DEFAULT_DATASOURCE_FORMATTER`
`java.util.function.Function<CortexDataSourceWriter,CortexDataSourceWriter>`	`defaultWriterOptions`	Default writer options, remove option by returning dsw
`static java.util.function.Function<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>,org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>`	`formatDatasetForDataSource`	transform on `DataSource` DataFrame to alter Timestamp and Date type columns to String Timestamp and Date type columns are not supported in Phoenix at this time
`java.util.function.Supplier<java.lang.Boolean>`	`performFeatureCatalogCalculations`	Perform `FeatureCatalog` calculations, is an expensive operation and may be set to false depending on the dataset
`java.util.function.Function<CortexConnectionReader,CortexConnectionReader>`	`readerOptions`	Expand on `CortexConnectionReader` reader options
`java.util.function.Function<CortexDataSourceWriter,CortexDataSourceWriter>`	`writerOptions`	Expand on `CortexDataSourceWriter` writer options

Constructor Summary

Constructors
Constructor	Description
`IngestDataSourceJob(java.lang.String project, java.lang.String sourceName, CortexContext cortexContext)`	Constructor

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type Method Description

void run()
- Methods inherited from class java.lang.Object
  clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_DATASOURCE_FORMATTER

public static final java.util.function.Function<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>,org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>> DEFAULT_DATASOURCE_FORMATTER

formatDatasetForDataSource

public static java.util.function.Function<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>,org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>> formatDatasetForDataSource

transform on DataSource DataFrame to alter Timestamp and Date type columns to String Timestamp and Date type columns are not supported in Phoenix at this time

readerOptions

public java.util.function.Function<CortexConnectionReader,CortexConnectionReader> readerOptions

Expand on CortexConnectionReader reader options

defaultWriterOptions

public java.util.function.Function<CortexDataSourceWriter,CortexDataSourceWriter> defaultWriterOptions

Default writer options, remove option by returning dsw

writerOptions

public java.util.function.Function<CortexDataSourceWriter,CortexDataSourceWriter> writerOptions

Expand on CortexDataSourceWriter writer options

performFeatureCatalogCalculations
```
public java.util.function.Supplier<java.lang.Boolean> performFeatureCatalogCalculations
```
Perform FeatureCatalog calculations, is an expensive operation and may be set to false depending on the dataset

Constructor Detail

IngestDataSourceJob

public IngestDataSourceJob(java.lang.String project,
                           java.lang.String sourceName,
                           CortexContext cortexContext)

Constructor

Parameters:: project - the project; sourceName - the DataSource name; cortexContext - the context

Method Detail
- run
```
public void run()
```
  Specified by:
  
  run in interface java.lang.Runnable

Class IngestDataSourceJob