Hi I was running some test with Node.js vs Fulephp. I had a simple set up and I was trying to see what is faster. I had 10 thousand records in mogodb being pulled in to views. The set up was simple, no js, no css, with minimal html. I quickly noticed that php set up was twice as fast. At first I dismissed nodejs as being slower and moved on with my life. However I decided to try node without jade which I used as my templating engine, and by stroke of luck I came across a post on stackoverflow that philosophy behind jade is not so much speed but elegance. Then I decided to try node without any temp. engines. But I quickly ran into a problem since I realized that I have no idea how to pass data from database and node to client. I was in for a long night of horror and despair. At some point I came to a conclusion that I need socket.io's help. Though I was able to connect to socket.io eventually I still was not able to figure out how to pass the data. Then I decided to go back to using an temp. engine, but this time I decided to try ejs. Eventually I was able to render some data which had the following form [object Object], but it was not 10 thousand records, more like 25. I decided to do the right thing and post my question here. I would like to render view without templating engine to see if my assumptions are right. After all I am trying to do two things understand how to pass data to the client form node.js and see if it will improve my performance.

Here is my app.js with some comments:

/**
 * Mongo DB
 */

var mongous = require('mongous').Mongous,
    dbCollection = 'test.personnel';

/**
 * Module dependencies.
 */
var express = require('express'),
    app = module.exports = express.createServer(),
    pub = __dirname + '/public';

// Configuration
app.configure(function(){
    app.set('view options', {layout: false});
    //not sure if i need these here, but left it in case
    app.use(app.router);
    app.use(express.static(pub));
});

//Simple templating
//I took this example from stackoverflow,
//can't find that post anymore,
//though I can look if need be
app.register('.html', {
    compile: function(str, options){
        return function(locals){
            return str;
        }
    }
});

// Routes
//This is where data is right now, it need to end up on the client side
//This was working with jade and kinda worked with ejs (though I am not sure because I was getting [object Object])
//--------------------------------------------------
app.get('/', function(req, res){
    mongous(dbCollection).find(function(output){
        res.render('index.html',{
                //the data is here, it works, i tested it with console.log
                data: output.documents
        });
    });
});
//--------------------------------------------------
app.configure('production', function(){
    app.use(express.errorHandler());
});
app.listen(3000);
console.log('Express server listening on port %d in %s mode', app.address().port, app.settings.env);

and here is my view:

    Data needs to end up here. Somehow...

As you can see not much in there. Now I understand that I will most likely will need to use some sort of templating engine on the client side and once I have the data on client side, I will be able to work it out myself. Which maybe even slower in the end, but my primary goal is to understand how to pass data in node.js to client so that I could continue experimenting. Please help if you can, it will improve my understanding of node significantly. Thank you.

EDIT: With the help of all of you and especially josh3736 this is what I ended up with, if you are interested... http://pastie.org/private/z3fjjbjff8284pr2mafw

As you allude to in your answer, part of the problem is the speed of the templating engine itself; you've found out that Jade is not the fastest — in fact, it's one of the slowest.

My favorite engine is doT. In the performance test I linked to, doT can render the template 5.4 million times per second. Jade can render a similar template only 29,000 times per second. It's not even a contest.

However, templating engine speed is only a small part of the issue here. I believe your real problem is the Mongo driver you're using appears to be poorly designed for Node's asynchronous model. (Disclaimer: I've never actually used Mongous; I just spent a few minutes looking over the code.)

Node is meant to work with streams of data. In other words, you're supposed to operating on very small chunks of data at a time. In contrast, it looks like Mongous processes the entire dataset and returns it to your code as one JSON object.

This is convenient and fine for small datasets, but completely falls apart when working with large amounts of data like you are (10,000 records). Node will be completely locked up while parsing and handling that much data (which is very, very bad since it won't be able to handle any incoming connections), and the V8 memory management system isn't optimized for large heap allocations like that.

To work with large datasets properly, you have to use a Mongo driver that streams records to your code, like node-mongodb-native, or mongoskin, which makes the API a little easier to deal with.

maerics' answer was on the right track, but is wrong because it uses toArray, which creates the same problem you have under Mongous: the entire dataset is collated into an in-memory array. Instead, just use the cursor's each method, which asynchronously calls your callback function for each returned record as it comes in. This way, the entire resultset is never in memory; you only work with one at a time, allowing the records you've already processed to be discarded and garbage collected if necessary.

Now that we've established how to get your data out of the database, we need to figure out how to get it to the client.

The problem here is that an Express' view system expects you to have all your data available up-front so that the templating engine can render out a single string to be sent to the client. As we discussed above, this isn't such a good idea if you're dealing with thousands of records. The proper way to do this is to stream the data we get from Mongo directly to the client. Unfortunately, we can't really do this within an Express view — they're not designed to be asynchronous.

Instead, you're going to have to write a custom handler. You're already on that path with Hippo's answer and your own attempt, but what you really need to use is res.write(), not res.send. Like res.render, res.send expects you to have a complete response when you call it because it internally calls res.end, ending the HTTP response. In contrast, res.write simply sends data over the network, leaving the HTTP response open and ready to send more data — in other words, streaming your response. (Keep in mind that you have to set any HTTP headers before you start streaming. For example, res.contentType('text/html');)

Just because you're manually handling the response (foregoing the view rendering system) does not preclude you from taking advantage of a templating engine. You can use a template for the header and footer of your document and one for each record. Let's use doT to put everything together.

First, let's declare our templates. In real life, you might load these from files (or even hack Express to load them for you as views and get the template source), but we'll just declare them in our code.

var header = doT.template('');
var record = doT.template('
{{=it.fieldname}}, ...
');
var footer = doT.template('');

(doT.template returns a function which generates HTML from the template we gave above and an object you pass to that returned function when you invoke it. For example, now we can call header({title:'Test'});)

Now we do the real work in the request handler: (assumes we already have a collection from the Mongo driver)

app.get('/', function(req, res){
    res.contentType('text/html');
    res.write(header({title:'Test'}));

    collection.find({}).each(function(err, doc) {
        if (err) return res.end('error! ' + err); // bail if there's an error
        if (doc) {
            res.write(record(doc));
        } else { // `doc` will be null if we've reached the end of the resultset.
            res.end(footer());
        }
    });
});