EzDevInfo.com

Node.js worker memory spike Error R14 when using request & pkgcloud

This one has me totally baffled because it occurs seemingly at random. I have a Node.js app that streams tweets (using ntwitter), adding hundreds of rows per second to a mongodb. This part always works fine and memory use is steady. BUT recently I've started getting the URL of twitter avatars and saving them to our rackspace CDN (using "request" and "pkgcloud" modules). I cycle thru a max of 10 at a time (but usually way less) every 20 seconds and run this code:

request(idocs[ikey].tweet.user.profile_image_url.replace('_normal.', '_bigger.')).on('response', function(response) {
    console.log(response.request.path);
    avatars.push(path.basename(path.dirname(response.request.path)) + '/' + path.basename(response.request.path));
    var headers = {};
    if (response.headers && response.headers['content-type']) {
        headers['content-type'] = response.headers['content-type'];
    }
    if (response.headers && response.headers['content-length']) {
        headers['content-length'] = response.headers['content-length'];
    }
    response.headers = headers;
}).pipe(client.upload({
    container: process.env.RACKSPACE_CONTAINER,
    remote: path.basename(path.dirname(idocs[ikey].tweet.user.profile_image_url)) + '/'
        + path.basename(idocs[ikey].tweet.user.profile_image_url.replace('_normal.', '_bigger.'))
}));

Everything works okay for a while, but then I get the following madness in Nodetime:

And here's the output of "heroku logs -p worker -t" when things suddenly go haywire (notice how it's going along fine at around 600 MB then leaps to over 1GB after one particular image--and BTW all the images are standard twitter dimensions of 73x73, so it can't be a file size issue):

2014-03-10T00:27:28.425915+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.50 sample#load_avg_5m=0.24 sample#load_avg_15m=0.12
2014-03-10T00:27:28.426216+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=630.87MB sample#memory_rss=599.24MB sample#memory_cache=0.00MB sample#memory_swap=31.63MB sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33324565pages
2014-03-10T00:27:48.353902+00:00 app[worker.1]: /profile_images/434575414246539265/Noy_bE24_bigger.jpeg
2014-03-10T00:27:48.570496+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.36 sample#load_avg_5m=0.22 sample#load_avg_15m=0.12
2014-03-10T00:27:48.570793+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=638.78MB sample#memory_rss=607.07MB sample#memory_cache=0.00MB sample#memory_swap=31.71MB sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33403153pages
2014-03-10T00:28:08.451643+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.54 sample#load_avg_5m=0.27 sample#load_avg_15m=0.14
2014-03-10T00:28:08.452211+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=601.35MB sample#memory_rss=569.55MB sample#memory_cache=0.00MB sample#memory_swap=31.79MB sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33482782pages
2014-03-10T00:28:28.616784+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.67 sample#load_avg_5m=0.32 sample#load_avg_15m=0.16
2014-03-10T00:28:28.617032+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=605.14MB sample#memory_rss=573.27MB sample#memory_cache=0.00MB sample#memory_swap=31.87MB sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33560778pages
2014-03-10T00:28:48.332406+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.76 sample#load_avg_5m=0.36 sample#load_avg_15m=0.18
2014-03-10T00:28:48.332628+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=594.71MB sample#memory_rss=562.75MB sample#memory_cache=0.00MB sample#memory_swap=31.95MB sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33647657pages
2014-03-10T00:29:08.501121+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.83 sample#load_avg_5m=0.40 sample#load_avg_15m=0.20
2014-03-10T00:29:08.501351+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=599.86MB sample#memory_rss=567.83MB sample#memory_cache=0.00MB sample#memory_swap=32.03MB sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33730317pages
2014-03-10T00:29:28.606331+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.88 sample#load_avg_5m=0.44 sample#load_avg_15m=0.21
2014-03-10T00:29:28.606565+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=630.37MB sample#memory_rss=598.26MB sample#memory_cache=0.00MB sample#memory_swap=32.11MB sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33823519pages
2014-03-10T00:29:29.054831+00:00 app[worker.1]: /profile_images/1857857435/251396_10150278547536115_50300066114_9027856_3493590_n_bigger.jpg
2014-03-10T00:29:48.392172+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.91 sample#load_avg_5m=0.48 sample#load_avg_15m=0.23
2014-03-10T00:29:48.392490+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=1056.09MB sample#memory_rss=1023.97MB sample#memory_cache=0.00MB sample#memory_swap=32.13MB sample#memory_pgpgin=28008448pages sample#memory_pgpgout=33948237pages
2014-03-10T00:29:48.392958+00:00 heroku[worker.1]: Process running mem=1056M(103.1%)
2014-03-10T00:29:48.393236+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2014-03-10T00:30:08.453128+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.65 sample#load_avg_5m=0.45 sample#load_avg_15m=0.23
2014-03-10T00:30:08.453346+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=1055.90MB sample#memory_rss=1023.71MB sample#memory_cache=0.00MB sample#memory_swap=32.19MB sample#memory_pgpgin=177823744pages sample#memory_pgpgout=34011963pages
2014-03-10T00:30:08.453806+00:00 heroku[worker.1]: Process running mem=1055M(103.1%)
2014-03-10T00:30:08.454034+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)

So I ask you, ladies and gentlemen: WTF!!!

What "streams and pipe-capable" means in pkgcloud in NodeJS

My issue is to get image uploading to amazon working. I was looking for a solution that doesnt save the file on the server and then upload it to Amazon.

Googling I found pkgcloud and on the README.md it says:

Special attention has been paid so that methods are streams and pipe-capable.

Can someone explain what that means and if it is what I am looking for?

Skipper resize image before uploading using adapter

I have a similar question to this one. How can I have this upload to my RackSpace adapter as well?

Here's the code:

upload: function(req, res){
        var RackspaceAdapter = require('../adapters/rackspace/index'),
            receiver = RackspaceAdapter.receive({
                username: sails.config.rackspace.username,
                apiKey: sails.config.rackspace.apiKey,
                region: sails.config.rackspace.region,
                container: sails.config.rackspace.container
            });
        req.file('file').upload(function (err, uploadedFiles) {
          if (err) return res.send(500, err);
            for(u in uploadedFiles){
                gm(uploadedFiles[u]).resize('500','','^').gravity('Center').crop('500','500').stream().pipe(receiver);
            }
          return res.json({
            message: uploadedFiles.length + ' file(s) uploaded successfully!',
            files: uploadedFiles
          });
        });
    },

how to specify zone for aws in pkgcloud while createClient?

I'm using pkgcloud npm for multicloud access. and i want to list out all instances from all zone (like US East (N. Virginia),US West (Oregon),US West (N. California),EU (Ireland)) etc.

here is my code

var aws = require('pkgcloud').compute.createClient({
provider : 'amazon',
key:'Secret Access Key',
keyId:'Access Key ID'
});

one more thing i need to ask you is can i use pkgcloud in production ? I've also read apache libcloud so which is better to use for production ? pkgcloud or libcloud ?

Token authentication with openstack for many clients

My question is that is possible to create client only base on the token in pkgcloud library for nodejs.

Because what I see there is that straight-forward approach:

var rackspace = pkgcloud.storage.createClient({
    provider: 'openstack',
    username: 'your-user-name',
    password: 'your-password',
    authUrl: '...'
  });

which allow to create client by using username, password credentials. But let say that there is many clients and I would like to authenticate them once and after use only token to make operation on object store like allowing that OpenStack. I don't want to keep their credentials on my server side only their tokens. So from other side how could I make operations (like upload, list, create containers etc...) by using pkgcloud with only tokens for my authenticated clients ?

So scenario looks the following:

1. I authenticate user1 by username, password
2. I receive token1 from OpenStack for user1
3. I store that token1 in my database
4. I authenticate user2 by username, password
5. I receive token2 from OpenStack for user2
6. I store that token2 in my database
7. Now I want to use that token1 from user1 to make next operation (like list containers) on object store

Thank you very much for your answer !

Uploading files in Express v4 using Pkgcloud

I have a NodeJS application using Express 3.5 and using the npm module pkgcloud to upload files to Rackspace. I need to upgrade this application to Express 4. While I've had success uploading files in Express 4 using busboy and putting them in MongoDB with grid-fs, I can't get Express 4 to work with Pkgcloud and Rackspace. I've seen a few posts about the AWS S3 workflows, but none that have helped me with uploading to our Rackspace CDN. Anyone had success with this or know of another npm modules the work better with Express 4 and file uploads?

Unable to download a 68 mb file using Rest apis

We have uploaded a (scorm)zip file along with extract-archive=tar which explodes the zip file and stores the files correctly on object store.

The zip file contain some htm, js and a swf file.

When we make CURL and PKGCloud access for htm and js files we get the response correctly.

However while trying to download the swf file (64mb), we don't see any download happening. Please help us :(

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:05:00 --:--:-- 0

NodeJS + Multer + PkgCloud: resize image before sending it to cloud

I'm searching for a way to resize an image (uploaded by a standard html form submit) before sending it to cloud (openstack, s3 or else).

I'm using NodeJS and 3 plugins: Multer (upload management), lwip (image resizing) and PkgCloud (transfert to cloud).

I'm able to handle the upload with multer, and I can write a new resized file in the server with lwip.

var fs = require('fs'),
    multer  = require('multer'),
    upload = multer({ dest: 'uploads/medias/' }),
    pkgcloud = require('pkgcloud'),
    client = pkgcloud.storage.createClient({/* some config */});

// "router" is an instance of a route from Express
router.post('/media/:id/image', upload.single('file'), function (req) {
    var lwip = require('lwip'),
        type = 'png',
        npath = req.file.path + '_150x150';

    // the file exists and the log shows information
    console.log(req.file);

    // open image
    lwip.open(req.file.path, type, function (err, image) {
        image.batch()
            .resize(150, 150)
            .writeFile(npath, type, function () {
                // remove raw file
                fs.unlink(req.file.path, function () {
                    // how do I send the new resized file to the pkgcloud client?
                });
            });
    });
});

But I'm missing something, and I can't find a way to send this new file to the cloud. I discovered a plugin which manage transfert between multer and a configured pkgcloud instance (multer-storage-pkgcloud), but I can't figure out how to resize the file before sending it to cloud.

Does anyone have an idea how to handle that?

Thanks

how to get list of public images of aws using pkgcloud?

i am trying to launch instance on aws using pkgcloud so for that image option is must, so i used bellow code to get list of images.

aws = pkgcloud.compute.createClient({
        provider:'amazon',
        key:'',
        keyId:''
    });
aws.getImages(function (err, images) { 
        console.log(images);
    });

but above API returns empty array. so how do i get list of all public images available on aws?

any suggestion ?

pkgcloud and GCE (Google Compute Engine)

Is it possible to control GCE (Google Compute Engine) using pkgcloud for Node.js?

This page says "The GCE APIs are implemented as first-class citizens of OpenStack Compute": http://www.cloudscaling.com/blog/cloud-computing/why-google-compute-engine-for-openstack/

...and pkgcloud supports OpenStack. But I haven't seen anywhere where support is explicitly affirmed for GCE and if OAuth is handled by pkgcloud (speaking somewhat uninformedly here) and haven't seen any sample code specifically for GCE.

In Node.js how to send message to client side from a handler/callback method

I am a beginner in node.js. I am trying to create a webapp using node.js which will create servers in cloud and manage them. After doing some research I've decided to use node.js, socket.io, express and pkgcloud modules to create the webapp. Now, the problem I am facing is that I am unable to send messages to the front-end/client-side when a server is getting created. The server is getting created successfully but I am not getting how to send the server status to the browser screen.

Here's the code: index.js

var express = require('express'),
app = express();

app.use(express.static(__dirname + '/'));

var io = require('socket.io').listen(app.listen(1337));

io.sockets.on('connection', function(socket) {
    socket.on('mesg_to_server', function(data) {

       // This is for testing whether we're receiving msgs on client side
       io.sockets.emit("mesg_to_client",{ mesg: data["mesg"], mesg1: data["mesg1"] });

       var Rackspace = require('./ex_rackspace');
       var rackspace = new Rackspace();
       rackspace.create(data.mesg);
   });
}); 

ex_rackspace.js:

  var pkgcloud = require('pkgcloud'),
  _ = require('./node_modules/pkgcloud/node_modules/underscore');

  var client = pkgcloud.providers.rackspace.compute.createClient({
                 username: 'xxxx',
                 apiKey: 'xxxxx',
                 region: 'HKG'
              });

  // This function will handle our server creation,
  // as well as waiting for the server to come online after we've
  // created it.
  function handleServerResponse(err, server) {
     if (err) {
       console.dir(err);
       return;
     }

     console.log('SERVER CREATED: ' + server.name + ', waiting for active status');

     // Wait for status: ACTIVE on our server, and then callback
     server.setWait({ status: server.STATUS.running }, 5000, function (err) {
      if (err) {
        console.dir(err);
        return;
      }

      console.log('SERVER INFO');
      console.log(server.name);
      console.log(server.status);
      console.log(server.id);

      console.log('Make sure you DELETE server: ' + server.id +
  ' in order to not accrue billing charges');
  });
  }

  var Rackspace = function() {

  };  

  Rackspace.prototype.test = function (text) {
     console.log("provider: Rackspace: " + text);
  }       

  // first we're going to get our flavors
  Rackspace.prototype.create = function (server) {
     client.getFlavors(function (err, flavors) {
     if (err) {
       console.dir(err);
       return;
     }

     // then get our base images
     client.getImages(function (err, images) {
       if (err) {
         console.dir(err);
         return;
       }

       var flavor = _.findWhere(flavors, { name: '1 GB Performance' });

       var image = _.findWhere(images, { name: 'CentOS 5 (PV)' });

       // Create our first server
       client.createServer({
          name: server,
          image: image,
          flavor: flavor
       }, handleServerResponse);
     });
   });
  }

  module.exports = Rackspace;

page.html (client-side):

<!DOCTYPE html>
<html>
  <head>
    <script src="/socket.io/socket.io.js"></script>
    <script type="text/javascript">
        // our socket.io code goes here
        var socketio = io.connect("127.0.0.1:1337");

        socketio.on("mesg_to_client", function(data) {
            document.getElementById("chatlog").innerHTML = ("<hr/>" +
            data.mesg + ' ' + data.mesg1 + document.getElementById("chatlog").innerHTML);
        });

        socketio.on("test_output", function(data) {
            console.log(data.mesg); 
        });

        function sendMessage() {
            var msg = document.getElementById("mesg_input").value;
            socketio.emit("mesg_to_server", { mesg : msg, mesg1 : "here"});
        }

    </script>
 </head>
 <body>
    <input type="text" id="mesg_input"/>
    <button onclick="sendMessage()">Create</button>
    <div id="chatlog"></div>
 </body>
</html>

Getting Lists from callback in nodejs

I am trying to get list of "servers" for a particular "stack" in Rackspace. I have to do some processing of the Stack resource to get the server list as some of the servers are in a resource group(nested). I get each server through asynchronous call one by one. How to bundle them together and send at one go to the client? Below is the code:

   app.js

   function setClient(credentials)
   {
     var client = pkgcloud.orchestration.createClient({
       provider: 'rackspace',
       username: credentials.uname,
       apiKey: credentials.key,
       region: 'HKG'
     });
     return client;
   }

   app.io.route('ready', function(req) {
     var client = setClient(req.session.credentials);

     client.getStacks(function (err, stacks) {
        if (err) {
          req.io.emit("error", { error: err.result.error.message });
          return;
        }

        if (stacks.length > 0) {
          var stack = stacks[0].name;

          getSer(stack, client, req, 'init');
        }
        req.io.emit("showStacks", { stacks: stacks });
     });
   });

   app.io.route('create_stack', function(req) {
      var stack = req.data.stack_name;
      var tmpl = req.data.content;
      var client = setClient(req.session.credentials);

      client.createStack({
        name: stack,
        timeout: 60,
        template: tmpl // Heat template
       }, handle(req));
   });

   function handle(req) {
      return function handle(err, stack)
      {
         var client = setClient(req.session.credentials);

         if (err) {
            req.io.emit("error", { error: err.result.error.message });
            console.dir(err.result.error.message);
            return;
         }

         client.getStacks(function (err, stacks) {
            req.io.emit("showStacks", { stacks: stacks });
         });

         req.io.emit("status", {status: 'Stack created'});

         // Wait for status: CREATE_COMPLETE on our stack, and then callback
         stack.setWait({ status: 'CREATE_COMPLETE' }, 5000, function (err) {
            if (err) {
              req.io.emit("error", { error: err.result.error.message });
              console.dir(err);
              return;
            }

            client.getStacks(function (err, stacks) {
              req.io.emit("showStacks", { stacks: stacks });
            });

            client.getStack(stack, function (err, stack) {
               if (err) {
                  req.io.emit("error", { error: err.result.error.message             });
                  console.dir(err.result.error.message);
                  return;
                }
                req.io.emit("showStack", { stack: stack });
            });

            getSer(stack, client, req, 'create');
         });
      }
   }

   function getSer(stack, client, req, mode)
   {
      client.getResources(stack, function(err, resources) {
          if (err) {
             console.dir(err);
             req.io.emit("error", { error: err.result.error.message });
             return;
           }

           resources.forEach(function (resource) {
              var nestedlink = '';
              var nested = false;
              if (resource.links != undefined) {
                resource.links.forEach(function (link) {
                   if (link.rel == "nested") {
                      nestedlink = link.href;
                      nested = true;
                   }
                });
              }
              if (nested) {
                 var stack_name = nestedlink.substr(nestedlink.indexOf('stacks/')+7, nestedlink.lastIndexOf('/'));
                 getSer(stack_name, client, req, mode);
              } else {
                 var serId = resource.physicalResourceId;
                 var ser = serClient(req.session.credentials);
                 ser.getServer(serId, function (err, server) {
                   if (mode == 'init')
                     req.io.emit("showServer", { server: server});
                   else if (mode == 'create')
                     req.io.emit("stackCreate", { server: server});
                 });
              }
           });
      });
   }

Actually, the problem is pkgcloud has a method 'getResources' which given the stack name returns the resources. From which I get 'physical_resource_id' (server_id) of the server which is not in a group but for the servers which were created through 'OS::Heat::ResourceGroup' one's to do some extra work because the resource list has only a 'nested' link for it. So I take the stack_name from the nested 'link' and get its server-id var stack_name = nestedlink.substr(nestedlink.indexOf('stacks/')+7, nestedlink.lastIndexOf('/'));. Life would've been easy if pkgcloud had something like given a stack-name it'd return the servers list in it.

Custom Skipper adapter for Rackspace using pkgcloud

I've started a build of a custom adapter for Skipper in Sails.js using pkgcloud to upload to Rackspace cloud files, but I've gotten stuck. It doesn't appear that the file is actually getting uploaded and trying to capture the error doesn't seem to work. Maybe I'm missing something.

I've posted my code here: https://github.com/ccoppenbarger/skipper-rackspace

You would need to check it out in api/adapters/rackspace to use it.

I'm only working on the receiver part for now. The pkgcloud api is here: https://developer.rackspace.com/docs/cloud-files/getting-started/

My controller code is as follows:

upload : function(req, res){
               
        req.file('file').upload({
          adapter: require('../adapters/rackspace/index'),
          username: sails.config.rackspace.username,
          apiKey: sails.config.rackspace.apiKey,
          region: sails.config.rackspace.region,
          container: sails.config.rackspace.container
        }, function whenDone(err, uploadedFiles) {
          if (err) return res.negotiate(err);
          else return res.ok({
            files: uploadedFiles,
            textParams: req.params.all()
          });
        });
    },

Can someone take a look and see what I may be missing in the index.js?