org.apache.hadoop.mapred

Class LocalJobRunner

java.lang.Object

org.apache.hadoop.mapred.LocalJobRunner

public class LocalJobRunner
extends java.lang.Object

Implements MapReduce locally, in-process, for debugging.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	LOCAL_MAX_MAPS
static org.apache.commons.logging.Log	LOG
static long	versionID

Constructor Summary

Constructors

Constructor and Description
LocalJobRunner(JobConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token) Cancel a delegation token.
JobStatus[]	getAllJobs() Get all the jobs submitted.
TaskReport[]	getCleanupTaskReports(JobID id) Grab a bunch of info on the cleanup tasks that make up the job
ClusterStatus	getClusterStatus(boolean detailed) Get the current status of the cluster
org.apache.hadoop.security.token.Token<DelegationTokenIdentifier>	getDelegationToken(org.apache.hadoop.io.Text renewer) Get a new delegation token.
java.lang.String	getFilesystemName() A MapReduce system always operates on a single filesystem.
Counters	getJobCounters(JobID id) Grab the current job counters
JobProfile	getJobProfile(JobID id) Grab a handle to a job that is already known to the JobTracker.
JobStatus[]	getJobsFromQueue(java.lang.String queue) Gets all the jobs submitted to the particular Queue
JobStatus	getJobStatus(JobID id) Grab a handle to a job that is already known to the JobTracker.
static int	getLocalMaxRunningMaps(JobContext job)
TaskReport[]	getMapTaskReports(JobID id) Grab a bunch of info on the map tasks that make up the job
JobID	getNewJobId() Allocate a name for the job.
org.apache.hadoop.ipc.ProtocolSignature	getProtocolSignature(java.lang.String protocol, long clientVersion, int clientMethodsHash)
long	getProtocolVersion(java.lang.String protocol, long clientVersion)
org.apache.hadoop.mapred.QueueAclsInfo[]	getQueueAclsForCurrentUser() Gets the Queue ACLs for current user
org.apache.hadoop.security.authorize.AccessControlList	getQueueAdmins(java.lang.String queueName) Get the administrators of the given job-queue.
JobQueueInfo	getQueueInfo(java.lang.String queue) Gets scheduling information associated with the particular Job queue
JobQueueInfo[]	getQueues() Gets set of Job Queues associated with the Job Tracker
TaskReport[]	getReduceTaskReports(JobID id) Grab a bunch of info on the reduce tasks that make up the job
TaskReport[]	getSetupTaskReports(JobID id) Grab a bunch of info on the setup tasks that make up the job
java.lang.String	getStagingAreaDir() Get a hint from the JobTracker where job-specific files are to be placed.
java.lang.String	getSystemDir() Grab the jobtracker system directory path where job-specific files are to be placed.
TaskCompletionEvent[]	getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents) Get task completion events for the jobid, starting from fromEventId.
java.lang.String[]	getTaskDiagnostics(TaskAttemptID taskid) Returns the diagnostic information for a particular task in the given job.
JobStatus[]	jobsToComplete() Get the jobs that are not completed and not failed
void	killJob(JobID id) Kill the indicated job
boolean	killTask(TaskAttemptID taskId, boolean shouldFail) Throws UnsupportedOperationException
long	renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token) Renew an existing delegation token
void	setJobPriority(JobID id, java.lang.String jp) Set the priority of the specified job
static void	setLocalMaxRunningMaps(JobContext job, int maxMaps) Set the max number of map tasks to run concurrently in the LocalJobRunner.
JobStatus	submitJob(JobID jobid, java.lang.String jobSubmitDir, org.apache.hadoop.security.Credentials credentials) Submit a Job for execution.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

LOCAL_MAX_MAPS

public static final java.lang.String LOCAL_MAX_MAPS

See Also:

Constant Field Values

versionID

public static final long versionID

See Also:

Constant Field Values

Constructor Detail

LocalJobRunner

public LocalJobRunner(JobConf conf)
               throws java.io.IOException

Throws:

java.io.IOException

Method Detail

getProtocolVersion

public long getProtocolVersion(java.lang.String protocol,
                      long clientVersion)

getProtocolSignature

public org.apache.hadoop.ipc.ProtocolSignature getProtocolSignature(java.lang.String protocol,
                                                           long clientVersion,
                                                           int clientMethodsHash)
                                                             throws java.io.IOException

Throws:

java.io.IOException

getNewJobId

public JobID getNewJobId()

Allocate a name for the job.

Returns:

a unique job name for submitting jobs.

submitJob

public JobStatus submitJob(JobID jobid,
                  java.lang.String jobSubmitDir,
                  org.apache.hadoop.security.Credentials credentials)
                    throws java.io.IOException

Submit a Job for execution. Returns the latest profile for that job. The job files should be submitted in jobSubmitDir.

Throws:

java.io.IOException

killJob

public void killJob(JobID id)

Kill the indicated job

setJobPriority

public void setJobPriority(JobID id,
                  java.lang.String jp)
                    throws java.io.IOException

Set the priority of the specified job

Parameters:

id - ID of the job

jp - Priority to be set for the job

Throws:

java.io.IOException

killTask

public boolean killTask(TaskAttemptID taskId,
               boolean shouldFail)
                 throws java.io.IOException

Throws UnsupportedOperationException

Parameters:

taskId - the id of the task to kill.

shouldFail - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.

Throws:

java.io.IOException

getJobProfile

public JobProfile getJobProfile(JobID id)

Grab a handle to a job that is already known to the JobTracker.

Returns:

Profile of the job, or null if not found.

getMapTaskReports

public TaskReport[] getMapTaskReports(JobID id)

Grab a bunch of info on the map tasks that make up the job

getReduceTaskReports

public TaskReport[] getReduceTaskReports(JobID id)

Grab a bunch of info on the reduce tasks that make up the job

getCleanupTaskReports

public TaskReport[] getCleanupTaskReports(JobID id)

Grab a bunch of info on the cleanup tasks that make up the job

getSetupTaskReports

public TaskReport[] getSetupTaskReports(JobID id)

Grab a bunch of info on the setup tasks that make up the job

getJobStatus

public JobStatus getJobStatus(JobID id)

Grab a handle to a job that is already known to the JobTracker.

Returns:

Status of the job, or null if not found.

getJobCounters

public Counters getJobCounters(JobID id)

Grab the current job counters

getFilesystemName

public java.lang.String getFilesystemName()
                                   throws java.io.IOException

A MapReduce system always operates on a single filesystem. This function returns the fs name. ('local' if the localfs; 'addr:port' if dfs). The client can then copy files into the right locations prior to submitting the job.

Throws:

java.io.IOException

getClusterStatus

public ClusterStatus getClusterStatus(boolean detailed)

Get the current status of the cluster

Parameters:

detailed - if true then report tracker names and memory usage

Returns:

summary of the state of the cluster

jobsToComplete

public JobStatus[] jobsToComplete()

Get the jobs that are not completed and not failed

Returns:

array of JobStatus for the running/to-be-run jobs.

getTaskCompletionEvents

public TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                            int fromEventId,
                                            int maxEvents)
                                              throws java.io.IOException

Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

Parameters:

jobid - job id

fromEventId - event id to start from.

maxEvents - the max number of events we want to look at

Returns:

array of task completion events.

Throws:

java.io.IOException

getAllJobs

public JobStatus[] getAllJobs()

Get all the jobs submitted.

Returns:

array of JobStatus for the submitted jobs

getTaskDiagnostics

public java.lang.String[] getTaskDiagnostics(TaskAttemptID taskid)
                                      throws java.io.IOException

Returns the diagnostic information for a particular task in the given job. To be implemented

Parameters:

taskid - the id of the task

Returns:

an array of the diagnostic messages

Throws:

java.io.IOException

getSystemDir

public java.lang.String getSystemDir()

Grab the jobtracker system directory path where job-specific files are to be placed.

Returns:

the system directory where job-specific files are to be placed.

See Also:

JobSubmissionProtocol.getSystemDir()

getQueueAdmins

public org.apache.hadoop.security.authorize.AccessControlList getQueueAdmins(java.lang.String queueName)
                                                                      throws java.io.IOException

Get the administrators of the given job-queue. This method is for hadoop internal use only.

Returns:

Queue administrators ACL for the queue to which job is submitted to

Throws:

java.io.IOException

See Also:

org.apache.hadoop.mapred.JobSubmissionProtocol#getQueueAdmins()

getStagingAreaDir

public java.lang.String getStagingAreaDir()
                                   throws java.io.IOException

Get a hint from the JobTracker where job-specific files are to be placed.

Returns:

the directory where job-specific files are to be placed.

Throws:

java.io.IOException

See Also:

JobSubmissionProtocol.getStagingAreaDir()

getJobsFromQueue

public JobStatus[] getJobsFromQueue(java.lang.String queue)
                             throws java.io.IOException

Gets all the jobs submitted to the particular Queue

Parameters:

queue - Queue name

Returns:

array of JobStatus for the submitted jobs

Throws:

java.io.IOException

getQueues

public JobQueueInfo[] getQueues()
                         throws java.io.IOException

Gets set of Job Queues associated with the Job Tracker

Returns:

Array of the Job Queue Information Object

Throws:

java.io.IOException

getQueueInfo

public JobQueueInfo getQueueInfo(java.lang.String queue)
                          throws java.io.IOException

Gets scheduling information associated with the particular Job queue

Parameters:

queue - Queue Name

Returns:

Scheduling Information of the Queue

Throws:

java.io.IOException

getQueueAclsForCurrentUser

public org.apache.hadoop.mapred.QueueAclsInfo[] getQueueAclsForCurrentUser()
                                                                    throws java.io.IOException

Gets the Queue ACLs for current user

Returns:

array of QueueAclsInfo object for current user.

Throws:

java.io.IOException

setLocalMaxRunningMaps

public static void setLocalMaxRunningMaps(JobContext job,
                          int maxMaps)

Set the max number of map tasks to run concurrently in the LocalJobRunner.

Parameters:

job - the job to configure

maxMaps - the maximum number of map tasks to allow.

getLocalMaxRunningMaps

public static int getLocalMaxRunningMaps(JobContext job)

Returns:

the max number of map tasks to run concurrently in the LocalJobRunner.

cancelDelegationToken

public void cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                           throws java.io.IOException,
                                  java.lang.InterruptedException

Cancel a delegation token.

Parameters:

token - the token to cancel

Throws:

java.io.IOException

java.lang.InterruptedException

getDelegationToken

public org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> getDelegationToken(org.apache.hadoop.io.Text renewer)
                                                                                     throws java.io.IOException,
                                                                                            java.lang.InterruptedException

Get a new delegation token.

Parameters:

renewer - the user other than the creator (if any) that can renew the token

Returns:

the new delegation token

Throws:

java.io.IOException

java.lang.InterruptedException

renewDelegationToken

public long renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                          throws java.io.IOException,
                                 java.lang.InterruptedException

Renew an existing delegation token

Parameters:

token - the token to renew

Returns:

the new expiration time

Throws:

java.io.IOException

java.lang.InterruptedException