Package com.c12e.cortex.profiles.module.datasource

Class DefaultCortexDataSourceStreamWriter.CortexWriteStreamParameters

java.lang.Object

com.c12e.cortex.profiles.module.datasource.DefaultCortexDataSourceStreamWriter.CortexWriteStreamParameters

All Implemented Interfaces:

CortexWriter<CortexDataSourceStreamWriter>, CortexDataSourceStreamWriter

Enclosing class:

DefaultCortexDataSourceStreamWriter

public class DefaultCortexDataSourceStreamWriter.CortexWriteStreamParameters
extends java.lang.Object
implements CortexDataSourceStreamWriter

Field Summary

Fields

Modifier and Type	Field	Description
CortexDataSourcePair	cortexDataSourcePair
java.lang.String	dataSourceName
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	ds
CortexDeltaMergeBuilder	mergeBuilder
org.apache.spark.sql.SaveMode	mode
java.lang.String[]	partitionColumns
java.lang.Boolean	performAggregation
java.lang.Boolean	performFeatureCatalogCalculations
java.lang.String	project
SparkOptionsHandler	sparkOptionsHandler

Constructor Summary

Constructors

Constructor	Description
CortexWriteStreamParameters()

Method Summary

Modifier and Type	Method	Description
CortexDataSourceStreamWriter	deltaMerge(CortexDeltaMergeBuilder mergeBuilder)	An instance of a builder to perform delta table merge.
CortexDataSourceStreamWriter	mode(org.apache.spark.sql.SaveMode saveMode)	Spark save mode
CortexDataSourceStreamWriter	option(java.lang.String key, java.lang.String value)	Overwrites value at key
CortexDataSourceStreamWriter	options(java.util.Map<java.lang.String,java.lang.String> writeOptions)	Merges new options with current options
CortexDataSourceStreamWriter	partitionBy(java.lang.String... columnNames)	Create partitions on column name(s)
CortexDataSourceStreamWriter	performAggregation(java.lang.Boolean performAggregation)	Perform groupBy aggregation on primary key, an expensive operation that should be turned off if the micro-batch dataset from the streaming data source always contains unique keys
CortexDataSourceStreamWriter	performFeatureCatalogCalculations(java.lang.Boolean performCalculations)	Perform feature catalog calculations FeatureCatalog
CortexDataSourceStreamWriter	project(java.lang.String project)	Sets the project
CortexDataSourceStreamWriter	resourceName(java.lang.String resourceName)	Sets the resource to write to
void	save()	Completes building and begins write process
CortexDataSourceStreamWriter	setDataset(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> ds)	Sets the dataset to write
CortexDataSourceStreamWriter	setDataSourcePair(CortexDataSourcePair cortexDataSourcePair)	Sets the stream reader return type

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

project

public java.lang.String project

dataSourceName

public java.lang.String dataSourceName

ds

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> ds

sparkOptionsHandler

public SparkOptionsHandler sparkOptionsHandler

mode

public org.apache.spark.sql.SaveMode mode

cortexDataSourcePair

public CortexDataSourcePair cortexDataSourcePair

partitionColumns

public java.lang.String[] partitionColumns

mergeBuilder

public CortexDeltaMergeBuilder mergeBuilder

performAggregation

public java.lang.Boolean performAggregation

performFeatureCatalogCalculations

public java.lang.Boolean performFeatureCatalogCalculations

Constructor Detail

CortexWriteStreamParameters

public CortexWriteStreamParameters()

Method Detail

project

public CortexDataSourceStreamWriter project(java.lang.String project)

Description copied from interface: CortexWriter

Sets the project

Specified by:

project in interface CortexWriter<CortexDataSourceStreamWriter>

Returns:

this instance

resourceName

public CortexDataSourceStreamWriter resourceName(java.lang.String resourceName)

Description copied from interface: CortexWriter

Sets the resource to write to

Specified by:

resourceName in interface CortexWriter<CortexDataSourceStreamWriter>

Returns:

this instance

setDataset

public CortexDataSourceStreamWriter setDataset(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> ds)

Description copied from interface: CortexWriter

Sets the dataset to write

Specified by:

setDataset in interface CortexWriter<CortexDataSourceStreamWriter>

Returns:

this instance

partitionBy

public CortexDataSourceStreamWriter partitionBy(java.lang.String... columnNames)

Description copied from interface: CortexWriter

Create partitions on column name(s)

Specified by:

partitionBy in interface CortexWriter<CortexDataSourceStreamWriter>

Returns:

this instance

deltaMerge

public CortexDataSourceStreamWriter deltaMerge(CortexDeltaMergeBuilder mergeBuilder)

Description copied from interface: CortexWriter

An instance of a builder to perform delta table merge. Only applicable if ContentType is delta

Specified by:

deltaMerge in interface CortexWriter<CortexDataSourceStreamWriter>

Returns:

this instance

options

public CortexDataSourceStreamWriter options(java.util.Map<java.lang.String,java.lang.String> writeOptions)

Description copied from interface: CortexWriter

Merges new options with current options

Specified by:

options in interface CortexWriter<CortexDataSourceStreamWriter>

Returns:

this instance

option

public CortexDataSourceStreamWriter option(java.lang.String key,
                                           java.lang.String value)

Description copied from interface: CortexWriter

Overwrites value at key

Specified by:

option in interface CortexWriter<CortexDataSourceStreamWriter>

Returns:

this instance

mode

public CortexDataSourceStreamWriter mode(org.apache.spark.sql.SaveMode saveMode)

Description copied from interface: CortexWriter

Spark save mode

Specified by:

mode in interface CortexWriter<CortexDataSourceStreamWriter>

Returns:

this instance

save

public void save()

Description copied from interface: CortexWriter

Completes building and begins write process

Specified by:

save in interface CortexWriter<CortexDataSourceStreamWriter>

setDataSourcePair

public CortexDataSourceStreamWriter setDataSourcePair(CortexDataSourcePair cortexDataSourcePair)

Description copied from interface: CortexDataSourceStreamWriter

Sets the stream reader return type

Specified by:

setDataSourcePair in interface CortexDataSourceStreamWriter

Returns:

this instance

performAggregation

public CortexDataSourceStreamWriter performAggregation(java.lang.Boolean performAggregation)

Description copied from interface: CortexDataSourceStreamWriter

Perform groupBy aggregation on primary key, an expensive operation that should be turned off if the micro-batch dataset from the streaming data source always contains unique keys

Specified by:

performAggregation in interface CortexDataSourceStreamWriter

Returns:

this instance

performFeatureCatalogCalculations

public CortexDataSourceStreamWriter performFeatureCatalogCalculations(java.lang.Boolean performCalculations)

Description copied from interface: CortexDataSourceStreamWriter

Perform feature catalog calculations FeatureCatalog

Specified by:

performFeatureCatalogCalculations in interface CortexDataSourceStreamWriter

Returns:

this instance