org.apache.hadoop.mapred

Interface JTProtocols

All Superinterfaces:

AdminOperationsProtocol, GetUserMappingsProtocol, RefreshAuthorizationPolicyProtocol, RefreshUserMappingsProtocol, org.apache.hadoop.ipc.VersionedProtocol

All Known Implementing Classes:

JobTracker

public interface JTProtocols
extends RefreshUserMappingsProtocol, GetUserMappingsProtocol, RefreshAuthorizationPolicyProtocol, AdminOperationsProtocol

Field Summary

Fields

Modifier and Type	Field and Description
static int	TRACKERS_OK
static int	UNKNOWN_TASKTRACKER
static long	versionID version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549.
static long	versionID

Fields inherited from interface org.apache.hadoop.mr1security.RefreshUserMappingsProtocol

versionID

Fields inherited from interface org.apache.hadoop.mr1tools.GetUserMappingsProtocol

versionID

Fields inherited from interface org.apache.hadoop.mr1security.authorize.RefreshAuthorizationPolicyProtocol

versionID

Fields inherited from interface org.apache.hadoop.mapred.AdminOperationsProtocol

versionID

Method Summary

Methods

Modifier and Type	Method and Description
void	addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener) Registers a JobInProgressListener for updates from this TaskTrackerManager.
void	cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token) Cancel a delegation token.
void	failJob(JobInProgress job) Fail a job.
JobStatus[]	getAllJobs() Get all the jobs submitted.
java.lang.String	getBuildVersion() Returns the VersionInfo build version of the JobTracker
TaskReport[]	getCleanupTaskReports(JobID jobid) Grab a bunch of info on the cleanup tasks that make up the job
ClusterStatus	getClusterStatus()
ClusterStatus	getClusterStatus(boolean detailed) Get the current status of the cluster
org.apache.hadoop.security.token.Token<DelegationTokenIdentifier>	getDelegationToken(org.apache.hadoop.io.Text renewer) Get a new delegation token.
java.lang.String	getFilesystemName() The task tracker calls this once, to discern where it can find files referred to by the JobTracker
java.lang.String	getFilesystemName() A MapReduce system always operates on a single filesystem.
JobInProgress	getJob(JobID jobid) Obtain the job object identified by jobid
Counters	getJobCounters(JobID jobid) Grab the current job counters
JobProfile	getJobProfile(JobID jobid) Grab a handle to a job that is already known to the JobTracker.
JobStatus[]	getJobsFromQueue(java.lang.String queue) Gets all the jobs submitted to the particular Queue
JobStatus	getJobStatus(JobID jobid) Grab a handle to a job that is already known to the JobTracker.
TaskReport[]	getMapTaskReports(JobID jobid) Grab a bunch of info on the map tasks that make up the job
JobID	getNewJobId() Allocate a name for the job.
int	getNextHeartbeatInterval() Return the current heartbeat interval that's used by TaskTrackers.
int	getNumberOfUniqueHosts()
org.apache.hadoop.mapred.QueueAclsInfo[]	getQueueAclsForCurrentUser() Gets the Queue ACLs for current user
org.apache.hadoop.security.authorize.AccessControlList	getQueueAdmins(java.lang.String queueName) Get the administrators of the given job-queue.
JobQueueInfo	getQueueInfo(java.lang.String queue) Gets scheduling information associated with the particular Job queue
org.apache.hadoop.mapred.QueueManager	getQueueManager() Return the QueueManager which manages the queues in this TaskTrackerManager.
JobQueueInfo[]	getQueues() Gets set of Job Queues associated with the Job Tracker
TaskReport[]	getReduceTaskReports(JobID jobid) Grab a bunch of info on the reduce tasks that make up the job
TaskReport[]	getSetupTaskReports(JobID jobid) Grab a bunch of info on the setup tasks that make up the job
java.lang.String	getStagingAreaDir() Get a hint from the JobTracker where job-specific files are to be placed.
java.lang.String	getSystemDir() Grab the jobtracker system directory path where job-specific files are to be placed.
java.lang.String	getSystemDir() Grab the jobtracker system directory path where job-specific files are to be placed.
TaskCompletionEvent[]	getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents) Get task completion events for the jobid, starting from fromEventId.
TaskCompletionEvent[]	getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents) Get task completion events for the jobid, starting from fromEventId.
java.lang.String[]	getTaskDiagnostics(TaskAttemptID taskId) Get the diagnostics for a given task in a given job
java.lang.String	getVIVersion() Returns the VersionInfo version of the JobTracker
org.apache.hadoop.mapred.HeartbeatResponse	heartbeat(TaskTrackerStatus status, boolean restarted, boolean initialContact, boolean acceptNewTasks, short responseId) Called regularly by the TaskTracker to update the status of its tasks within the job tracker.
void	initJob(JobInProgress job) Initialize the Job
JobStatus[]	jobsToComplete() Get the jobs that are not completed and not failed
void	killJob(JobID jobid) Kill the indicated job
void	killJob(JobID jobid) Kill the job identified by jobid
boolean	killTask(TaskAttemptID taskId, boolean shouldFail) Kill indicated task attempt.
boolean	killTask(TaskAttemptID taskid, boolean shouldFail) Mark the task attempt identified by taskid to be killed
void	removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener) Unregisters a JobInProgressListener from this TaskTrackerManager.
long	renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token) Renew an existing delegation token
void	reportTaskTrackerError(java.lang.String taskTracker, java.lang.String errorClass, java.lang.String errorMessage) Report a problem to the job tracker.
void	setJobPriority(JobID jobid, java.lang.String priority) Set the priority of the specified job
JobStatus	submitJob(JobID jobName, java.lang.String jobSubmitDir, org.apache.hadoop.security.Credentials ts) Submit a Job for execution.
java.util.Collection<TaskTrackerStatus>	taskTrackers()

Methods inherited from interface org.apache.hadoop.mr1security.RefreshUserMappingsProtocol

refreshSuperUserGroupsConfiguration, refreshUserToGroupsMappings

Methods inherited from interface org.apache.hadoop.mr1tools.GetUserMappingsProtocol

getGroupsForUser

Methods inherited from interface org.apache.hadoop.mr1security.authorize.RefreshAuthorizationPolicyProtocol

refreshServiceAcl

Methods inherited from interface org.apache.hadoop.mapred.AdminOperationsProtocol

refreshNodes, refreshQueues

Methods inherited from interface org.apache.hadoop.ipc.VersionedProtocol

getProtocolSignature, getProtocolVersion

Field Detail

versionID

static final long versionID

version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549. version 5 introduced that removes locateMapOutputs and instead uses getTaskCompletionEvents to figure finished maps and fetch the outputs version 6 adds maxTasks to TaskTrackerStatus for HADOOP-1245 version 7 replaces maxTasks by maxMapTasks and maxReduceTasks in TaskTrackerStatus for HADOOP-1274 Version 8: HeartbeatResponse is added with the next heartbeat interval. version 9 changes the counter representation for HADOOP-2248 version 10 changes the TaskStatus representation for HADOOP-2208 version 11 changes string to JobID in getTaskCompletionEvents(). version 12 changes the counters representation for HADOOP-1915 version 13 added call getBuildVersion() for HADOOP-236 Version 14: replaced getFilesystemName with getSystemDir for HADOOP-3135 Version 15: Changed format of Task and TaskStatus for HADOOP-153 Version 16: adds ResourceStatus to TaskTrackerStatus for HADOOP-3759 Version 17: Changed format of Task and TaskStatus for HADOOP-3150 Version 18: Changed status message due to changes in TaskStatus Version 19: Changed heartbeat to piggyback JobTracker restart information so that the TaskTracker can synchronize itself. Version 20: Changed status message due to changes in TaskStatus (HADOOP-4232) Version 21: Changed information reported in TaskTrackerStatus' ResourceStatus and the corresponding accessor methods (HADOOP-4035) Version 22: Replaced parameter 'initialContact' with 'restarted' in heartbeat method (HADOOP-4305) Version 23: Added parameter 'initialContact' again in heartbeat method (HADOOP-4869) Version 24: Changed format of Task and TaskStatus for HADOOP-4759 Version 25: JobIDs are passed in response to JobTracker restart Version 26: Added numRequiredSlots to TaskStatus for MAPREDUCE-516 Version 27: Adding node health status to TaskStatus for MAPREDUCE-211 Version 28: Adding user name to the serialized Task for use by TT. Version 29: Adding available memory and CPU usage information on TT to TaskTrackerStatus for MAPREDUCE-1218 Version 30: Adding disk failure to TaskTrackerStatus for MAPREDUCE-3015 Version 31: Efficient serialization format for Framework counters (MAPREDUCE-901) Version 32: Adding version methods for HADOOP-8209

See Also:

Constant Field Values

TRACKERS_OK

static final int TRACKERS_OK

See Also:

Constant Field Values

UNKNOWN_TASKTRACKER

static final int UNKNOWN_TASKTRACKER

See Also:

Constant Field Values

versionID

static final long versionID

See Also:

Constant Field Values

Method Detail

heartbeat

org.apache.hadoop.mapred.HeartbeatResponse heartbeat(TaskTrackerStatus status,
                                                   boolean restarted,
                                                   boolean initialContact,
                                                   boolean acceptNewTasks,
                                                   short responseId)
                                                     throws java.io.IOException

Called regularly by the TaskTracker to update the status of its tasks within the job tracker. JobTracker responds with a HeartbeatResponse that directs the TaskTracker to undertake a series of 'actions' (see TaskTrackerAction.ActionType). TaskTracker must also indicate whether this is the first interaction (since state refresh) and acknowledge the last response it recieved from the JobTracker

Parameters:

status - the status update

restarted - true if the process has just started or restarted, false otherwise

initialContact - true if this is first interaction since 'refresh', false otherwise.

acceptNewTasks - true if the TaskTracker is ready to accept new tasks to run.

responseId - the last responseId successfully acted upon by the TaskTracker.

Returns:

a HeartbeatResponse with fresh instructions.

Throws:

java.io.IOException

getFilesystemName

java.lang.String getFilesystemName()
                                   throws java.io.IOException

The task tracker calls this once, to discern where it can find files referred to by the JobTracker

Throws:

java.io.IOException

reportTaskTrackerError

void reportTaskTrackerError(java.lang.String taskTracker,
                          java.lang.String errorClass,
                          java.lang.String errorMessage)
                            throws java.io.IOException

Report a problem to the job tracker.

Parameters:

taskTracker - the name of the task tracker

errorClass - the kind of error (eg. the class that was thrown)

errorMessage - the human readable error message

Throws:

java.io.IOException - if there was a problem in communication or on the remote side

getTaskCompletionEvents

TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                            int fromEventId,
                                            int maxEvents)
                                              throws java.io.IOException

Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

Parameters:

jobid - job id

fromEventId - event id to start from.

maxEvents - the max number of events we want to look at

Returns:

array of task completion events.

Throws:

java.io.IOException

getSystemDir

java.lang.String getSystemDir()

Grab the jobtracker system directory path where job-specific files are to be placed.

Returns:

the system directory where job-specific files are to be placed.

getBuildVersion

java.lang.String getBuildVersion()
                                 throws java.io.IOException

Returns the VersionInfo build version of the JobTracker

Throws:

java.io.IOException

getVIVersion

java.lang.String getVIVersion()
                              throws java.io.IOException

Returns the VersionInfo version of the JobTracker

Throws:

java.io.IOException

getNewJobId

JobID getNewJobId()
                  throws java.io.IOException

Allocate a name for the job.

Returns:

a unique job name for submitting jobs.

Throws:

java.io.IOException

submitJob

JobStatus submitJob(JobID jobName,
                  java.lang.String jobSubmitDir,
                  org.apache.hadoop.security.Credentials ts)
                    throws java.io.IOException

Submit a Job for execution. Returns the latest profile for that job. The job files should be submitted in jobSubmitDir.

Throws:

java.io.IOException

getClusterStatus

ClusterStatus getClusterStatus(boolean detailed)
                               throws java.io.IOException

Get the current status of the cluster

Parameters:

detailed - if true then report tracker names and memory usage

Returns:

summary of the state of the cluster

Throws:

java.io.IOException

getQueueAdmins

org.apache.hadoop.security.authorize.AccessControlList getQueueAdmins(java.lang.String queueName)
                                                                      throws java.io.IOException

Get the administrators of the given job-queue. This method is for hadoop internal use only.

Parameters:

queueName -

Returns:

Queue administrators ACL for the queue to which job is submitted to

Throws:

java.io.IOException

killJob

void killJob(JobID jobid)
             throws java.io.IOException

Kill the indicated job

Throws:

java.io.IOException

setJobPriority

void setJobPriority(JobID jobid,
                  java.lang.String priority)
                    throws java.io.IOException

Set the priority of the specified job

Parameters:

jobid - ID of the job

priority - Priority to be set for the job

Throws:

java.io.IOException

killTask

boolean killTask(TaskAttemptID taskId,
               boolean shouldFail)
                 throws java.io.IOException

Kill indicated task attempt.

Parameters:

taskId - the id of the task to kill.

shouldFail - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.

Throws:

java.io.IOException

getJobProfile

JobProfile getJobProfile(JobID jobid)
                         throws java.io.IOException

Grab a handle to a job that is already known to the JobTracker.

Returns:

Profile of the job, or null if not found.

Throws:

java.io.IOException

getJobStatus

JobStatus getJobStatus(JobID jobid)
                       throws java.io.IOException

Grab a handle to a job that is already known to the JobTracker.

Returns:

Status of the job, or null if not found.

Throws:

java.io.IOException

getJobCounters

Counters getJobCounters(JobID jobid)
                        throws java.io.IOException

Grab the current job counters

Throws:

java.io.IOException

getMapTaskReports

TaskReport[] getMapTaskReports(JobID jobid)
                               throws java.io.IOException

Grab a bunch of info on the map tasks that make up the job

Throws:

java.io.IOException

getReduceTaskReports

TaskReport[] getReduceTaskReports(JobID jobid)
                                  throws java.io.IOException

Grab a bunch of info on the reduce tasks that make up the job

Throws:

java.io.IOException

getCleanupTaskReports

TaskReport[] getCleanupTaskReports(JobID jobid)
                                   throws java.io.IOException

Grab a bunch of info on the cleanup tasks that make up the job

Throws:

java.io.IOException

getSetupTaskReports

TaskReport[] getSetupTaskReports(JobID jobid)
                                 throws java.io.IOException

Grab a bunch of info on the setup tasks that make up the job

Throws:

java.io.IOException

getFilesystemName

java.lang.String getFilesystemName()
                                   throws java.io.IOException

A MapReduce system always operates on a single filesystem. This function returns the fs name. ('local' if the localfs; 'addr:port' if dfs). The client can then copy files into the right locations prior to submitting the job.

Throws:

java.io.IOException

jobsToComplete

JobStatus[] jobsToComplete()
                           throws java.io.IOException

Get the jobs that are not completed and not failed

Returns:

array of JobStatus for the running/to-be-run jobs.

Throws:

java.io.IOException

getAllJobs

JobStatus[] getAllJobs()
                       throws java.io.IOException

Get all the jobs submitted.

Returns:

array of JobStatus for the submitted jobs

Throws:

java.io.IOException

getTaskCompletionEvents

TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                            int fromEventId,
                                            int maxEvents)
                                              throws java.io.IOException

Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

Parameters:

jobid - job id

fromEventId - event id to start from.

maxEvents - the max number of events we want to look at

Returns:

array of task completion events.

Throws:

java.io.IOException

getTaskDiagnostics

java.lang.String[] getTaskDiagnostics(TaskAttemptID taskId)
                                      throws java.io.IOException

Get the diagnostics for a given task in a given job

Parameters:

taskId - the id of the task

Returns:

an array of the diagnostic messages

Throws:

java.io.IOException

getSystemDir

java.lang.String getSystemDir()

Grab the jobtracker system directory path where job-specific files are to be placed.

Returns:

the system directory where job-specific files are to be placed.

getStagingAreaDir

java.lang.String getStagingAreaDir()
                                   throws java.io.IOException

Get a hint from the JobTracker where job-specific files are to be placed.

Returns:

the directory where job-specific files are to be placed.

Throws:

java.io.IOException

getQueues

JobQueueInfo[] getQueues()
                         throws java.io.IOException

Gets set of Job Queues associated with the Job Tracker

Returns:

Array of the Job Queue Information Object

Throws:

java.io.IOException

getQueueInfo

JobQueueInfo getQueueInfo(java.lang.String queue)
                          throws java.io.IOException

Gets scheduling information associated with the particular Job queue

Parameters:

queue - Queue Name

Returns:

Scheduling Information of the Queue

Throws:

java.io.IOException

getJobsFromQueue

JobStatus[] getJobsFromQueue(java.lang.String queue)
                             throws java.io.IOException

Gets all the jobs submitted to the particular Queue

Parameters:

queue - Queue name

Returns:

array of JobStatus for the submitted jobs

Throws:

java.io.IOException

getQueueAclsForCurrentUser

org.apache.hadoop.mapred.QueueAclsInfo[] getQueueAclsForCurrentUser()
                                                                    throws java.io.IOException

Gets the Queue ACLs for current user

Returns:

array of QueueAclsInfo object for current user.

Throws:

java.io.IOException

getDelegationToken

org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> getDelegationToken(org.apache.hadoop.io.Text renewer)
                                                                                     throws java.io.IOException,
                                                                                            java.lang.InterruptedException

Get a new delegation token.

Parameters:

renewer - the user other than the creator (if any) that can renew the token

Returns:

the new delegation token

Throws:

java.io.IOException

java.lang.InterruptedException

renewDelegationToken

long renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                          throws java.io.IOException,
                                 java.lang.InterruptedException

Renew an existing delegation token

Parameters:

token - the token to renew

Returns:

the new expiration time

Throws:

java.io.IOException

java.lang.InterruptedException

cancelDelegationToken

void cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                           throws java.io.IOException,
                                  java.lang.InterruptedException

Cancel a delegation token.

Parameters:

token - the token to cancel

Throws:

java.io.IOException

java.lang.InterruptedException

taskTrackers

java.util.Collection<TaskTrackerStatus> taskTrackers()

Returns:

A collection of the TaskTrackerStatus for the tasktrackers being managed.

getNumberOfUniqueHosts

int getNumberOfUniqueHosts()

Returns:

The number of unique hosts running tasktrackers.

getClusterStatus

ClusterStatus getClusterStatus()

Returns:

a summary of the cluster's status.

addJobInProgressListener

void addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)

Registers a JobInProgressListener for updates from this TaskTrackerManager.

Parameters:

jobInProgressListener - the JobInProgressListener to add

removeJobInProgressListener

void removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)

Unregisters a JobInProgressListener from this TaskTrackerManager.

Parameters:

jobInProgressListener - the JobInProgressListener to remove

getQueueManager

org.apache.hadoop.mapred.QueueManager getQueueManager()

Return the QueueManager which manages the queues in this TaskTrackerManager.

Returns:

the QueueManager

getNextHeartbeatInterval

int getNextHeartbeatInterval()

Return the current heartbeat interval that's used by TaskTrackers.

Returns:

the heartbeat interval used by TaskTrackers

killJob

void killJob(JobID jobid)
             throws java.io.IOException

Kill the job identified by jobid

Parameters:

jobid -

Throws:

java.io.IOException

getJob

JobInProgress getJob(JobID jobid)

Obtain the job object identified by jobid

Parameters:

jobid -

Returns:

jobInProgress object

initJob

void initJob(JobInProgress job)

Initialize the Job

Parameters:

job - JobInProgress object

failJob

void failJob(JobInProgress job)

Fail a job.

Parameters:

job - JobInProgress object

killTask

boolean killTask(TaskAttemptID taskid,
               boolean shouldFail)
                 throws java.io.IOException

Mark the task attempt identified by taskid to be killed

Parameters:

taskid - task to kill

shouldFail - whether to count the task as failed

Returns:

true if the task was found and successfully marked to kill

Throws:

java.io.IOException