org.apache.spark.shuffle.api

Interface ShuffleExecutorComponents

@Private
public interface ShuffleExecutorComponents

:: Private :: An interface for building shuffle support for Executors.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
ShuffleMapOutputWriter	createMapOutputWriter(int shuffleId, long mapTaskId, int numPartitions) Called once per map task to create a writer that will be responsible for persisting all the partitioned bytes written by that map task.
default java.util.Optional<SingleSpillShuffleMapOutputWriter>	createSingleFileMapOutputWriter(int shuffleId, long mapId) An optional extension for creating a map output writer that can optimize the transfer of a single partition file, as the entire result of a map task, to the backing store.
void	initializeExecutor(String appId, String execId, java.util.Map<String,String> extraConfigs) Called once per executor to bootstrap this module with state that is specific to that executor, specifically the application ID and executor ID.

Method Detail

initializeExecutor

void initializeExecutor(String appId,
                        String execId,
                        java.util.Map<String,String> extraConfigs)

Called once per executor to bootstrap this module with state that is specific to that executor, specifically the application ID and executor ID.

Parameters:

appId - The Spark application id

execId - The unique identifier of the executor being initialized

extraConfigs - Extra configs that were returned by ShuffleDriverComponents.initializeApplication()

createMapOutputWriter

ShuffleMapOutputWriter createMapOutputWriter(int shuffleId,
                                             long mapTaskId,
                                             int numPartitions)
                                      throws java.io.IOException

Called once per map task to create a writer that will be responsible for persisting all the partitioned bytes written by that map task.

Parameters:

shuffleId - Unique identifier for the shuffle the map task is a part of

mapTaskId - An ID of the map task. The ID is unique within this Spark application.

numPartitions - The number of partitions that will be written by the map task. Some of these partitions may be empty.

Throws:

java.io.IOException

createSingleFileMapOutputWriter

default java.util.Optional<SingleSpillShuffleMapOutputWriter> createSingleFileMapOutputWriter(int shuffleId,
                                                                                              long mapId)
                                                                                       throws java.io.IOException

An optional extension for creating a map output writer that can optimize the transfer of a single partition file, as the entire result of a map task, to the backing store.

Most implementations should return the default Optional.empty() to indicate that they do not support this optimization. This primarily is for backwards-compatibility in preserving an optimization in the local disk shuffle storage implementation.

Parameters:

shuffleId - Unique identifier for the shuffle the map task is a part of

mapId - An ID of the map task. The ID is unique within this Spark application.

Throws:

java.io.IOException