for-loop with async function runs out memory
Problem
Before you start shouting "Do not use for-loops with async functions!" let me state that I do know how to program a loop with async functions calling the next iteration from within the callback.
But what, if I am not interested in the callback or in processing one iteration only if the predecessor has finished. In my case I am importing a big number of records from an xml-file and writing them into a mongoDB. The normal call to my "write"-function would be like this:
asyncWrite( record, function(err, result) {
...
});
Now as I said, im not interested in the callback (no errors will occur!) nor in a serial process. So I used a for-loop and left out the callback:
for( i=0; i
For small max
this works perfectly. But with max
getting bigger (> 100.000) I end up getting an 'out of memory' error. The only way to prevent this was turning back to a 'normal' async-loop (recursive function call from within the callback).
Any other idea?
Solution
The async functions will not be executed until the for-loop has done its work. It has just collected a queue that is too beg. For big max
you got out of memory
error because the queue is full. You can see how your memory leaks by inserting console.log(process.memoryUsage());
inside of the loop. You need to work with chunks to process this task.
Discussion
View additional discussion.