Get Even More Visitors To Your Blog, Upgrade To A Business Listing >>

SSH2.SFTP node.js backup script freezes with large files

SSH2.SFTP node.js backup script freezes with large files

Problem

I'm trying to create a script to transfer files to my NAS for backup purposes. I'm using the SSH2 module (https://github.com/mscdex/ssh2) to set up a SFTP connection. For smaller files all goes well, with larger files (>500MB) it often just freezes somewhere between 70% and 100% completion. The fastPut method, which i use for the transfers, suddenly just stops transfering, doesn't trigger any events anymore.

I've taken out the relevant piece of code:

var uploadUpdate=true;

sftp.fastPut(localPath+'/'+upload, receiver.path+'/'+newDir+'/'+upload, {step:function(totalTransferred, chunk, total){
    if(uploadUpdate){ //since this event will be triggered a lot I figured I should keep some time (5 sec) between the logs
        uploadUpdate=false;
        console.log("- Progress '"+upload+"' => "+(Math.floor(totalTransferred/total*10000)/100)+"%, "+totalTransferred+" of "+total+" bytes, chunk: "+chunk);
        setTimeout(function(){
            uploadUpdate=true;
        },5000);
    }
}}, function(err){
    if(err){
        if(!fileErrors[upload]) fileErrors[upload]=0;
        ++fileErrors[upload];

        if(fileErrors[upload]==5)
            console.log("'"+upload+"' failed 5 times... skipping now...");
        else{
            console.log("Upload of '"+upload+"' failed, re-add to queue.");
            uploadsToDo.push(upload);
        }
    }
    nextUpload();
});

Does anyone have any idea what might cause this problem?

Problem courtesy of: NullCrayfish

Solution

I've ended up splitting the files into smaller chunks and sending them, then whenever for a specific amount of time there had been no progress i restart the connection and start transfering where it died before.

Not very pretty but it gets the job done

Solution courtesy of: NullCrayfish

Discussion

View additional discussion.



This post first appeared on Node.js Recipes, please read the originial post: here

Share the post

SSH2.SFTP node.js backup script freezes with large files

×

Subscribe to Node.js Recipes

Get updates delivered right to your inbox!

Thank you for your subscription

×