A heavy query to mongodb hangs the website

Question

My website (using MEAN.JS) has 1 login page, and it has also multiple statistical pages (reporting pages built on data in db).

There is 1 heavy report which would draw ~100MB out of database, and take ~10s to complete. The problem is when running this report, most of all other web pages fail to load on browser as they make queries to MongoDB too. Eg., user can't login on the login page even the UI of the login page is already shown.

Would MongoDB be able to do multiple queries parallelly or just a query queue?

MongoDB server version is v3.2.3. The report uses 2 collections: users, trackinglocations. I'm building report of a month:

30 days * 100 users * 10000 gps locations/day/user
= 30 million locations

user: {
    username: (string),
    organization: (objectid)
}

trackinglocation: {
    username: (string),
    date: (date),
    locations: [
        {
            speed: (number),
            long: (number),
            lat: (number)
        }
    ]
}

I first query to find those users in the organization of the admin. Then query to make a report of travel distances of users in that organization between a date range.

Not only the login page, many other pages don't respond to browser (the UI part is still responded by ExpressJS but no chance for the data tables inside) for those 10 seconds when making the report.


Show source
| find   | node.js   | performance   | mongodb   | nosql   2016-12-28 10:12 2 Answers

Answers to A heavy query to mongodb hangs the website ( 2 )

  1. 2016-12-28 11:12

    The situation is not exclusive to mongodb. Yes, it can run queries in parallel, but even other database systems, with better parallelization, will grind to a halt if you launch a particularly large report.

    A common approach to mitigate this is to run heavy queries (reports, etc.) on a secondary/slave. This way, primary/master remains unaffected and keeps serving writes and lighter reads.

  2. 2016-12-29 06:12

    Another solution beside reading from secondary is to read from db multiple times, each time a short trunk of documents, then take a short rest after every read operation.

    //some lines below are pseudocode
    function getLocs(callback) {
        var idList  = ...;
        var index   = 0;
        var maxRead = 1000;
        var results = [];
    
        (function readATrunk(){
            TrackingLocation.find({
                id: {
                    $in: idList[index]...idList[min(index+maxRead,idList.length-1)]
                }
            }).
            exec(function(error,trunk){
                results = results.concat(trunk);
    
                if (index+maxRead<idList.length) {
                    index += maxRead;
    
                    //give a spare 1 second for other queries in 
                    //other web pages
                    setTimeout(readATrunk,1000); 
                }
                else {
                    callback(results);
                }
            });
        })();
    }
    

Leave a reply to - A heavy query to mongodb hangs the website

◀ Go back