EzDevInfo.com

needle

Nimble, streamable HTTP client for Node.js/IO.js. With proxy, iconv, cookie, deflate & multipart support.

How to curl/needle/request google.flight correctly?

I want to parse the page (https://www.google.com/flights/#search;f=JFK;t=SFO;d=2014-12-22;r=2014-12-30) on my backend using needleJs(https://github.com/tomas/needle) and Cheerio(https://github.com/cheeriojs/cheerio).

When I tried to request the google.flight page by the url, what I got is 

> <!DOCTYPE html> <html><head><meta http-equiv="content-type"
> content="text/html; charset=UTF-8"><meta name="flights::gwt:property"
> content="baseUrl=/flights/static/"><title>Flights - Google
> Search</title> <meta name="description" content="Choose your flight
> from a simple list of results, explore destinations on a map, and find
> travel dates with the lowest fare with Flight Search."><script
> language="javascript" type="text/javascript"> var __JS_ILT__ = new
> Date(); </script> <style
> type="text/css">#gbar,#guser{font-size:13px;padding-top:1px
> !important;}#gbar{height:22px}#guser{padding-bottom:7px
> !important;text-align:right}.gbh,.gbd{border-top:1px solid
> #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media
> all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline
> !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27
> !important}.gbf .gb4{color:#900 !important}</style><script
> language="javascript" type="text/javascript"> var __JS_INI__ =
> "[,[,[,[1275,1105,1266,1174,1144,1203,15,16,10]],[,\42USD\42,\42$\42,\42\\u00a4#,##0\42,\42\\u00a4#,##0.00\42,\42#,##0\42,1],[,2,1609.344,\42#,##0\42,\42#,##0.0\42],[,2,2.54,\42#,##0\42],\42google-travel\42,\42typeId:72275\42,\42ADS25WNJCodY_Q5E8phMiFpRv5Mjahv3VPknsbbucoHBj4CEkSDnkC31pFues4Idsta-tlsZHS3HRUa2nBAqRARRiSySmvbRZymx5Q\42,\42.com\42,\42en\42,\42US\42,1,,,,,,\42https://accounts.google.com/ServiceLogin?continue\\u003d_CONTINUE_\42],[,\42BOS\42,\42Boston\42,42.3644444,-71.005278,[\42BOS\42,\42PSM\42],[\42BOS\42],\42United
> States\42,\42MA\42,\42US\42,\42Massachusetts\42,,42.3584308,-71.0597732],[[,\42BOS\42,\42Boston
> Logan International\42,\42Boston\42,\42BOS\42,\42United
> States\42,42.3644444,-71.005278,\42MA\42,\42US\42]],[[,\42ONEWORLD\42,\42Oneworld\42],[,\42SKYTEAM\42,\42SkyTeam\42],[,\42STAR_ALLIANCE\42,\42Star
> Alliance\42]],,[[,\42_fli\42,\42Flights\42],[,\42_mor\42,\42More\42],[,\42_web\42,\42Web\42],[,\42#map\42,\42Maps\42],[,\42nws\42,\42News\42],[,\42shop\42,\42Shopping\42],[,\42isch\42,\42Images\42],[,\42vid\42,\42Videos\42],[,\42bks\42,\42Books\42],[,\42app\42,\42Apps\42]],1,,\42US
> Dollar\42,0]"; </script> <script>(function(){var
> gs=document.createElement('script');var wmm=window.matchMedia;var
> hires=!!wmm && !wmm('(-webkit-device-pixel-ratio:1.0)').matches &&
> !wmm('(-moz-device-pixel-ratio:1.0)').matches;gs.src=!hires?'static/D4C5482E0AEB161B751BBFC9C57F4C0D.cache.js':'static/7521F8CC51FAFDF1C1478664FF1CF4BD.cache.js';gs.type='text/javascript';gs.async=true;document.getElementsByTagName('head')[0].appendChild(gs);})();</script><script
> language="javascript" type="text/javascript"></script> <link
> rel="shortcut icon" rel='nofollow' href="/favicon.ico"/><meta name="google"
> value="notranslate"></head><body> <script language="javascript"
> type="text/javascript">(function() {  var script =
> document.createElement('script');  script.type = 'text/javascript'; 
> script.src = 'https://ssl.gstatic.com/feedback/api.js'; 
> document.body.appendChild(script);})();</script> <div id=gbar><nobr><a
> class=gb1 rel='nofollow' href="https://www.google.com/search">Search</a> <a class=gb1
> rel='nofollow' href="https://www.google.com/search?hl=en&tbm=isch&source=og">Images</a>
> <a class=gb1 rel='nofollow' href="https://maps.google.com/maps?hl=en">Maps</a> <a
> class=gb1 rel='nofollow' href="https://play.google.com/?hl=en">Play</a> <a class=gb1
> rel='nofollow' href="https://www.youtube.com/results">YouTube</a> <a class=gb1
> rel='nofollow' href="https://news.google.com/nwshp?hl=en">News</a> <a class=gb1
> rel='nofollow' href="https://mail.google.com/mail/">Gmail</a> <a class=gb1
> rel='nofollow' href="https://drive.google.com/">Drive</a> <a class=gb1
> style="text-decoration:none"
> rel='nofollow' href="http://www.google.com/intl/en/options/"><u>More</u>
> &raquo;</a></nobr></div><div class=gbh style=left:0></div><div
> class=gbh style=right:0></div><div id="_BrowserWarning_"
> style="border:1px solid
> #FFE475;background-color:#FEF7CB;padding:4px;text-align: center;color: #222;">Google Flight Search has not been optimized for your browser. For best results, please try Chrome, Firefox 3.5+, Internet Explorer
> 8+, Safari 4+.<a rel='nofollow' href="#" style="color: #222;"
> onclick="document.getElementById('_BrowserWarning_').style.display='none';
> return false;"> Close</a></div><div id="root"></div></body></html>

Which is not the page I want. For performance purpose, I guess the reason is that Google tries to dump a very small size of html when I request. And there are a few necessary javascript codes in the html which will be executed after the DOM loaded. And then those script will fetch the fares and render them. Correct me if I am wrong.

So what can I do in order to parse the right page by the url for the Google flight site?? Thanks.


Source: (StackOverflow)

How to get response headers when using needle.js in streaming mode?

I want to use the needle module for node.js in streaming mode, similar to this example from the needle docs:

var stream = needle.get('http://www.as35662.net/100.log');

stream.on('readable', function() {
  var chunk;
  while (chunk = this.read()) {
    console.log('got data: ', chunk);
  }
});

This allows me to read the response body from the stream.

How can I access the response headers?


Source: (StackOverflow)

Advertisements

needle : node.js returning 500 on POST

I have my own rest API, that internally calls an NLP API, for which I have to post some data on their URL. I am using needle to achieve this, but there's some error that I cannot catch, and my own api is returning 500 to the frontend.

Here is that part of my server.js code:

app.post('/api/get',function(req,res) {
  //console.log(req);
  console.log("here in post ");
  if(!req.body){
    return res.send(400);
  }
  //console.log(req.body.msg);
  var searchQuery =  req.body.msg;
  var options = { 'api-key' : '3080a0e0-1111-11e5-a409-7159d0ac8188' };

  needle.post('http://api.cortical.io:80/rest/text/keywords?retina_name=en_associative',searchQuery,options,function(err, resp){
    if(err){
      console.log('something went wrong :' + err);
    }
    console.log('Got :'+resp );
});

I reach here in post every time, but nothing after that. I am also curious that is this correct way to specify my api-key for the external API.

Thanks.


Source: (StackOverflow)

Multiple POST params in node.js POST, using needle

I am trying to POST some data to this API end point , Corresponding curl call is this :

curl -X POST \
    -d "apiKey=YOU_API_KEY" \
    -d "extractors=entities,entailments" \
    -d "text=Spain's stricken Bankia expects to sell off its vast portfolio of industrial holdings that includes a stake in the parent company of British Airways and Iberia." \
    https://api.textrazor.com/

If i directly try to use this with needle specifying API KEY , it connects to the API successfully but since it is missing the text param it gives me an error. So how do I specify the multiple POST params.

My code :

needle.post(TEXTRAZOR_URL,TEXTRAZOR_API_KEY,function(err, razorResponse){

      console.log(err || razorResponse.body);
  });

here TEXTRAZOR_URL is : https://api.textrazor.com/ and TEXTRAZOR_API_KEY is : apiKey=123123123

I know that i need to change the 2nd parameter in needle.post. How do I do this.


Source: (StackOverflow)

How to upload a binary body with a PUT request in Node.js using needle or request libraries

I'm trying to write a unit test on an amazon pre-signed upload url which takes a PUT request with a raw binary body.

Looking at examples for both the needle and request libraries, they all use form data examples. Can someone give me an example using either library that will send a local file up as a raw binary in the body of the request?

Request library https://github.com/request/request

Needle library https://github.com/tomas/needle

            var filename = 'bunny.jpg';
            var url = Amazon.getUploadUrl(filename);

            var data = {
                file: __dirname + '/' + filename,
                content_type: 'image/jpeg'
            };

            var file = fs.createReadStream(__dirname + '/' + filename);
            var request = require('needle');

            request
                .put(url, data, function(err, resp) {

                    console.log(resp.body.toString('utf-8'));
                    if (resp.statusCode !== 200) {
                        done(new Error(resp.statusMessage));

                    }
                    else
                        done(err);
                });

Source: (StackOverflow)