In this blog i will describe how to use Azure Batch to run an arbitrary script that reads data from an Input Container (on azure blob storage) and writes data to an Output Container (also on Azure blob storage). The script itself is also stored in a Container named Resources on the same azure blob storage.

The first step is to create a storage account with 3 containers (Input/Resources/Output):

Then Create a simple bash script named somejob.sh with the following script in it:

cp $1 $2 to a resource container.

Next step is to create an input file input.txt  with some random content like “hello world” and store it to the Input-Container.

Now create an Azure Batch service and a pool with at least one compute node in the azure portal. (This part is straight forward, just fillo ut the forms in windows azure and.)

Next step is to create a node js app that submits a job to the new azure batch service. In this app you need to provide the location of the input file as well as the location of the script (each as a SAS-Blob-File-Token) and also the Output Container as a SAS-Token.

(I have not figured out how to generate the SAS-Tokens for containers in the Azure portal yet however there is a powershellscript on stackoverflow: https://stackoverflow.com/questions/44101670/azure-storage-generating-sas-token-portal-vs-powershell that can be used)

Generating the SAS Tokens for the input.txt and somejob.sh-script-files is quite simple in the azure portal:

)

Now create a new folder and run npm init –y

to submit jobs to azure batch you need the node.js azure-batch module so run: npm install 'azure-batch'

In the folder create a file app.js and insert the following and fillout the placeholders:

var batch = require('azure-batch');

console.log('starting...');

// Replace values below with Batch Account details

var accountName = 'azurebatchaccountname';

var accountKey = 'azurebatchkey';

var accountUrl = 'https://example.westeurope.batch.azure.com';

// Create Batch credentials object using account name and account key

var credentials = new batch.SharedKeyCredentials(accountName,accountKey);

// Create Batch service client

var batchClient = new batch.ServiceClient(credentials,accountUrl);

var now = new Date();

var poolId = 'blendergpu'

var poolInfo = {poolId:poolId};

// Job ID

var jobId = 'processcsvjob' + now.getFullYear() + now.getMonth() + now.getDay() + now.getHours() + now.getSeconds();

console.log('Creating job with ID : ' + jobId);

var jobConfig = {

id: jobId,

displayName: 'Simple Job',

poolInfo: {

poolId: 'blendergpu'

},

onAllTasksComplete: 'terminatejob'

}

var scriptURI = 'https://… your sas token to the somejob.sh file';

var inputtxtSASURI = 'https://… your sas token to the input.txt file';

var job = batchClient.job.add(jobConfig,function(error,result){

if(error !== null)

{

console.log("An error occured while creating the job...");

console.log(error);

}

else

{

var taskID = "Task1";

var taskConfig = {id:taskID,

displayName:'somejob',

commandLine:'sh somejob.sh input.txt output.txt',

resourceFiles:[{'blobSource':scriptURI,'filePath':'somejob.sh'},

{'blobSource':inputtxtSASURI,'filePath':'input.txt'}],

outputFiles: [

{

filePattern: "./output.txt",

destination: {

container: {

path: "output.txt",

containerUrl: https://example.blob.core.windows.net/fgrp-output?st=1999-01-01T09%3A06%3A28Z&se=1999-01-08T09%3A21%3A28Z&sp=w&sv=1999-07-29&sr=c&sig=asdfasdfasdfasdfasdfasdfasdfasdfasdf

}

},

uploadOptions: {

uploadCondition: "TaskCompletion"

}

}]

}

var task = batchClient.task.add(jobId,taskConfig,function(error,result){

if (error)

console.log(error);

else

{

console.log('task created!');

}

});

}

});

This will submit a job to your azure batch cluster that copies the input.txt-file to the output.txt-file on the clusternode. When the task completes it will  move the output.txt to the output Azure Blob Storage Container.