Performing Asynchronous Tasks

Last modified by Thomas Mortagne on 2020/04/10

Use Case

In this tutorial for the Job Module we implement a space-rename-function taking into account that:

  • a space can have many pages
  • each page can have many back-links
  • some pages can have large content and we want to update the relative links inside that content
  • hence this operation can take a lot of time so we need to display the progress which means we cannot block the HTTP request that triggers the operation; in other words the operation should be asynchronous.

API Design

Before we start doing the implementation we need to decide what the rename API would look like. There are two main ways to implement asynchronous tasks:

  1. push: you start the task and then you wait to be notified of the task progress, success or failure. In order to be notified you:
    • either pass a callback to the API
    • or the API returns a promise that you can use to register a callback
  2. pull: you start the task and then you ask for updates regularly until the task is done (with success or failure). In this case the API needs to provide some method to access the status of the task

The first option (push) is nice but it requires a two-way connection between the code that triggers the task and the code that executes the task. This is not the case with (standard) HTTP where the server (normally) doesn't push data to the client. It's the client who pulls the data from the server. So we're going to use the second option.

## Start the task.
#set ($taskId = $services.space.rename($spaceReference, $newSpaceName))
...
## Pull the task status.
#set ($taskStatus = $services.space.getRenameStatus($taskId))

Let's see how we can implement this API using the Job Module.

Request

The request represents the input for the task. It includes:

  • the data needed by the task (e.g. the space reference and the new space name)
  • context information (e.g. the user that triggered the task)
  • configuration options for the task. For instance:
    • whether to check access rights or not
    • whether the task is interactive (may require user input during the task execution) or not

Each request has an identifier that is used to access the task status.

public class RenameRequest extends org.xwiki.job.AbstractRequest
{
   private static final String PROPERTY_SPACE_REFERENCE = "spaceReference";

   private static final String PROPERTY_NEW_SPACE_NAME = "newSpaceName";

   private static final String PROPERTY_CHECK_RIGHTS = "checkrights";

   private static final String PROPERTY_USER_REFERENCE = "user.reference";

   public SpaceReference getSpaceReference()
   {
       return getProperty(PROPERTY_SPACE_REFERENCE);
   }

   public void setSpaceReference(SpaceReference spaceReference)
   {
        setProperty(PROPERTY_SPACE_REFERENCE, spaceReference);
   }

   public String getNewSpaceName()
   {
       return getProperty(PROPERTY_NEW_SPACE_NAME);
   }

   public void setNewSpaceName(String newSpaceName)
   {
        setProperty(PROPERTY_NEW_SPACE_NAME, newSpaceName);
   }

   public boolean isCheckRights()
   {
       return getProperty(PROPERTY_CHECK_RIGHTS, true);
   }

   public void setCheckRights(boolean checkRights)
   {
        setProperty(PROPERTY_CHECK_RIGHTS, checkRights);
   }

   public DocumentReference getUserReference()
   {
       return getProperty(PROPERTY_USER_REFERENCE);
   }

   public void setUserReference(DocumentReference userReference)
   {
        setProperty(PROPERTY_USER_REFERENCE, userReference);
   }
}

Questions

As we mentioned, jobs can be interactive by asking questions during the job execution. For instance, if there is already a space with the specified new name then we have to decide whether to:

  • stop the rename
  • or merge the two spaces

If we decide to merge the two spaces then there may be documents with the same name in both spaces, in which case we have to decide whether to overwrite the destination document or not.

To keep the example simple we're going to always merge the two spaces but we'll ask the user to confirm the overwrite.

public class OverwriteQuestion
{
   private final DocumentReference source;

   private final DocumentReference destination;

   private boolean overwrite = true;

   private boolean askAgain = true;

   public OverwriteQuestion(DocumentReference source, DocumentReference destination)
   {
       this.source = source;
       this.destination = destination;
   }

   public EntityReference getSource()
   {
       return source;
   }

   public EntityReference getDestination()
   {
       return destination;
   }

   public boolean isOverwrite()
   {
       return overwrite;
   }

   public void setOverwrite(boolean overwrite)
   {
       this.overwrite = overwrite;
   }

   public boolean isAskAgain()
   {
       return askAgain;
   }

   public void setAskAgain(boolean askAgain)
   {
       this.askAgain = askAgain;
   }
}

Job Status

The job status provides, by default, access to:

  • the job state (e.g. NONE, RUNNING, WAITING, FINISHED)
  • the job request
  • the job log ("INFO: Document X.Y has been renamed to A.B")
  • the job progress (72% completed)

Most of the time you don't need to extend the DefaultJobStatus provided by the Job Module, unless you want to store:

  • more progress information (e.g. the list of documents that have been renamed so far)
  • the task result / output

Note that both the request and the job status must be serializable so be careful with what kind of information your store in your custom job status. For instance, for the task output, it's probably better to store a reference, path or URL to the output instead of storing the output itself.

The job status is also your communication channel with the job:

  • if the job asks a question we
    • access the question from the job status
    • answer the question through the job status
  • if you want to cancel the job you have to do it through the job status
public class RenameJobStatus extends DefaultJobStatus<RenameRequest>
{
   private boolean canceled;

   private List<DocumentReference> renamedDocumentReferences = new ArrayList<>();

   public RenameJobStatus(RenameRequest request, ObservationManager observationManager,
        LoggerManager loggerManager, JobStatus parentJobStatus)
   {
       super(request, observationManager, loggerManager, parentJobStatus);
   }

   public void cancel()
   {
       this.canceled = true;
   }

   public boolean isCanceled()
   {
       return this.canceled;
   }

   public List<DocumentReference> getRenamedDocumentReferences()
   {
       return this.renamedDocumentReferences;
   }
}

Script Service

We now have everything we need to implement a ScriptService that will allow us to trigger the rename from Velocity and to get the rename status.

@Component
@Named(SpaceScriptService.ROLE_HINT)
@Singleton
public class SpaceScriptService implements ScriptService
{
   public static final String ROLE_HINT = "space";

   public static final String RENAME = "rename";

   @Inject
   private JobExecutor jobExecutor;

   @Inject
   private JobStatusStore jobStatusStore;

   @Inject
   private DocumentAccessBridge documentAccessBridge;

   public String rename(SpaceReference spaceReference, String newSpaceName)
   {
        setError(null);

        RenameRequest renameRequest = createRenameRequest(spaceReference, newSpaceName);

       try {
           this.jobExecutor.execute(RENAME, renameRequest);

            List<String> renameId = renameRequest.getId();
           return renameId.get(renameId.size() - 1);
       } catch (Exception e) {
            setError(e);
           return null;
       }
   }

   public RenameJobStatus getRenameStatus(String renameJobId)
   {
       return (RenameJobStatus) this.jobStatusStore.getJobStatus(getJobId(renameJobId));
   }

   private RenameRequest createRenameRequest(SpaceReference spaceReference, String newSpaceName)
   {
        RenameRequest renameRequest = new RenameRequest();
        renameRequest.setId(getNewJobId());
        renameRequest.setSpaceReference(spaceReference);
        renameRequest.setNewSpaceName(newSpaceName);
        renameRequest.setInteractive(true);
        renameRequest.setCheckRights(true);
        renameRequest.setUserReference(this.documentAccessBridge.getCurrentUserReference());
       return renameRequest;
   }

   private List<String> getNewJobId()
   {
       return getJobId(UUID.randomUUID().toString());
   }

   private List<String> getJobId(String suffix)
   {
       return Arrays.asList(ROLE_HINT, RENAME, suffix);
   }
}

Job Implementation

Jobs are components. Let's see how we can implement them.

@Component
@Named(SpaceScriptService.RENAME)
public class RenameJob extends AbstractJob<RenameRequest, RenameJobStatus> implements GroupedJob
{
   @Inject
   private AuthorizationManager authorization;

   @Inject
   private DocumentAccessBridge documentAccessBridge;

   private Boolean overwriteAll;

   @Override
   public String getType()
   {
       return SpaceScriptService.RENAME;
   }

   @Override
   public JobGroupPath getGroupPath()
   {
        String wiki = this.request.getSpaceReference().getWikiReference().getName();
       return new JobGroupPath(Arrays.asList(SpaceScriptService.RENAME, wiki));
   }

   @Override
   protected void runInternal() throws Exception
   {
        List<DocumentReference> documentReferences = getDocumentReferences(this.request.getSpaceReference());

       // Indicate that we start a process which a number of steps equals to documentReferences size
       this.progressManager.pushLevelProgress(documentReferences.size(), this);

       try {
           for (DocumentReference documentReference : documentReferences) {
               // Close the previous step and start a new one for each element
               this.progressManager.startStep(this);

               if (this.status.isCanceled()) {
                   break;
               }

               if (hasAccess(Right.DELETE, documentReference)) {
                    move(documentReference, this.request.getNewSpaceName());
                   this.status.getRenamedDocumentReferences().add(documentReference);
                   this.logger.info("Document [{}] has been moved to [{}].", documentReference,
                       this.request.getNewSpaceName());
               }
           }
       } finally {
           // Indicate that we finished all the steps of the process
           this.progressManager.popLevelProgress(this);
       }
   }

   private boolean hasAccess(Right right, EntityReference reference)
   {
       return !this.request.isCheckRights()
           || this.authorization.hasAccess(right, this.request.getUserReference(), reference);
   }

   private void move(DocumentReference documentReference, String newSpaceName)
   {
        SpaceReference newSpaceReference = new SpaceReference(newSpaceName, documentReference.getWikiReference());
        DocumentReference newDocumentReference =
            documentReference.replaceParent(documentReference.getLastSpaceReference(), newSpaceReference);
       if (!this.documentAccessBridge.exists(newDocumentReference)
           || confirmOverwrite(documentReference, newDocumentReference)) {
            move(documentReference, newDocumentReference);
       }
   }

   private boolean confirmOverwrite(DocumentReference source, DocumentReference destination)
   {
       if (this.overwriteAll == null) {
            OverwriteQuestion question = new OverwriteQuestion(source, destination);
           try {
               this.status.ask(question);
               if (!question.isAskAgain()) {
                   // Use the same answer for the following overwrite questions.
                   this.overwriteAll = question.isOverwrite();
               }
               return question.isOverwrite();
           } catch (InterruptedException e) {
               this.logger.warn("Overwrite question has been interrupted.");
               return false;
           }
       } else {
           return this.overwriteAll;
       }
   }
}

Server Controller

We need to be able to trigger the rename operation and to get status updates remotely, from JavaScript. This means the rename API should be accessible through some URLs:

  • ?action=rename -> redirects to ?data=jobStatus
  • ?data=jobStatus&jobId=xyz -> return the job status serialized as JSON
  • ?action=continue&jobId=xyz -> redirects to ?data=jobStatus
  • ?action=cancel&jobId=xyz -> redirects to ?data=jobStatus
{{velocity}}
#if ($request.action == 'rename')
  #set ($spaceReference = $services.model.resolveSpace($request.spaceReference))
  #set ($renameJobId = $services.space.rename($spaceReference, $request.newSpaceName))
  $response.sendRedirect($doc.getURL('get', $escapetool.url({
    'outputSyntax': 'plain',
    'jobId': $renameJobId
  })))
#elseif ($request.action == 'continue')
  #set ($renameJobStatus = $services.space.getRenameStatus($request.jobId))
  #set ($overwrite = $request.overwrite == 'true')
  #set ($discard = $renameJobStatus.question.setOverwrite($overwrite))
  #set ($discard = $renameJobStatus..answered())
#elseif ($request.action == 'cancel')
  #set ($renameJobStatus = $services.space.getRenameStatus($request.jobId))
  #set ($discard = $renameJobStatus.cancel())
  $response.sendRedirect($doc.getURL('get', $escapetool.url({
    'outputSyntax': 'plain',
    'jobId': $renameJobId
  })))
#elseif ($request.data == 'jobStatus')
  #set ($renameJobStatus = $services.space.getRenameStatus($request.jobId))
  #buildRenameStatusJSON($renameJobStatus)
  $response.setContentType('application/json')
  $jsontool.serialize($renameJobStatusJSON)
#end
{{/velocity}}

Client Controller

On the client side, the JavaScript code is responsible for:

  • triggering the task with an AJAX request to the server controller
  • retrieve task status updates regularly and update the displayed progress
  • pass the job questions to the user and pass the user answers to the server controller
var onStatusUpdate = function(status) {
  updateProgressBar(status);
 if (status.state == 'WAITING') {
   // Display the question to the user.
   displayQuestion(status);
  } else if (status.state != 'FINISHED') {
   // Pull task status update.
   setTimeout(function() {
      requestStatusUpdate(status.request.id).success(onStatusUpdate);
    }, 1000);
  }
};

// Trigger the rename task.
rename(parameters).success(onStatusUpdate);

// Continue the rename after the user answers the question.
continueRename(parameters).success(onStatusUpdate);

// Cancel the rename.
cancelRename(parameters).success(onStatusUpdate);

General Flow

jobFlow.png

Get Connected