Get Even More Visitors To Your Blog, Upgrade To A Business Listing >>

for-loop with async function runs out memory

for-loop with async function runs out memory

Problem

Before you start shouting "Do not use for-loops with async functions!" let me state that I do know how to program a loop with async functions calling the next iteration from within the callback.

But what, if I am not interested in the callback or in processing one iteration only if the predecessor has finished. In my case I am importing a big number of records from an xml-file and writing them into a mongoDB. The normal call to my "write"-function would be like this:

asyncWrite( record, function(err, result) {
    ...
});

Now as I said, im not interested in the callback (no errors will occur!) nor in a serial process. So I used a for-loop and left out the callback:

for( i=0; i

For small max this works perfectly. But with max getting bigger (> 100.000) I end up getting an 'out of memory' error. The only way to prevent this was turning back to a 'normal' async-loop (recursive function call from within the callback).

Any other idea?

Problem courtesy of: heinob

Solution

The async functions will not be executed until the for-loop has done its work. It has just collected a queue that is too beg. For big max you got out of memory error because the queue is full. You can see how your memory leaks by inserting console.log(process.memoryUsage()); inside of the loop. You need to work with chunks to process this task.

Solution courtesy of: Vadim Baryshev

Discussion

View additional discussion.



This post first appeared on Node.js Recipes, please read the originial post: here

Share the post

for-loop with async function runs out memory

×

Subscribe to Node.js Recipes

Get updates delivered right to your inbox!

Thank you for your subscription

×