em-http-request
Asynchronous HTTP Client (EventMachine + Ruby)
I have an API with limited number of calls per sec (100 requests per 100 sec). How can I utilize em-http-request (ideally with Redis) to make such calls at the fastest rate possible (to not exceed api rate limit?).
Source: (StackOverflow)
How can I get only response headers in an em_http_request?
I tried to use this code:
EventMachine.run do
http = EventMachine::HttpRequest.new('my_url').get
http.headers do |headers|
Fiber.current.resume headers
end
end
but I don't want to receive the whole body. How I can stop the request's execution?
http.close doesn't work.
UPD
http.instance_variable_get(:'@conn').close helps me, but may be you know more interesting solution
Source: (StackOverflow)
A very simple case where I get the root Fiber error.
require 'em-synchrony'
require 'em-synchrony/em-http'
urls = %w{http://www.google.com http://www.google.com http://www.google.com http://www.google.com http://www.google.com http://www.google.com}
EM.synchrony do
EM::Synchrony::Iterator.new(urls, 2).each(
proc { |url, iter|
EM::HttpRequest.new(url).get
iter.next
}
)
end
I can use async here, but not a sync http request.
Source: (StackOverflow)
I have an EventMachine running that is listening for UDP packets. Every time a UDP packet is received, I would like to forward the data to some database using a REST call. I have created an EM::Connection subclass in which the receive_data method forwards the data through a REST call.
Since the frequency of data is quite high, I would like to reuse the connection between requests (using "keep-alive") and if possible also use pipelining. What would be a good way to share the connection between different calls?
Currently my UDPHandler looks something like the following:
module Udp
module Collector
class UDPHandler < EM::Connection
def receive_data(data)
http = EventMachine::HttpRequest.new('http://databaseurl.com/').post body: data
end
end
end
end
This class is called as follows:
EM.run do
EM.open_datagram_socket('0.0.0.0', 9000, Udp::Collector::UDPHandler)
end
I thought of making the request object a class variable but I don't think that is a good idea in the context of eventmachines. Or is it?
Source: (StackOverflow)
I've created a gist which shows exactly what happens.
https://gist.github.com/4418148
I've tested a version which used ruby's 'net/http' library and 'socksify/http' and it worked perfect but if the EventMachine version returns an unexpected result.
The response in Tor Browser is correct but using EventMachine is not!
It return a response but it's not the same as returned response when you send the request via browser, net/http with or without proxy.
For convenience, I will also paste it here.
require 'em-http-request'
DEL = '-'*40
@results = 0
def run_with_proxy
connection_opts = {:proxy => {:host => '127.0.0.1', :port => 9050, :type => :socks5}}
conn = EM::HttpRequest.new("http://www.apolista.de/tegernsee/kloster-apotheke", connection_opts)
http = conn.get
http.callback {
if http.response.include? "Oops"
puts "#{DEL}failed with proxy#{DEL}", http.response
else
puts "#{DEL}success with proxy#{DEL}", http.response
end
@results+=1
EM.stop_event_loop if @results == 2
}
end
def run_without_proxy
conn = EM::HttpRequest.new("http://www.apolista.de/tegernsee/kloster-apotheke")
http = conn.get
http.callback {
if http.response.include? "Oops"
puts "#{DEL}failed without proxy#{DEL}", http.response
else
puts "#{DEL}success without proxy#{DEL}", http.response
end
@results+=1
EM.stop_event_loop if @results == 2
}
end
EM.run do
run_with_proxy
run_without_proxy
end
Appreciate any clarification.
Source: (StackOverflow)
My first time dealing with sinatra and parallel em-http-request. And i dont know how to combine/merge into one results and when to EventMachine.stop? . Consider this:
get '/data/:query' do
content_type :json
EventMachine.run do
http1 = EventMachine::HttpRequest.new('v1/').get
http2 = EventMachine::HttpRequest.new('v2/').get
http1.errback { p 'Uh oh nooooooo'; EventMachine.stop }
http1.callback {
// do some operation http1.repsonse
Crack::XML.parse(http1.response).to_json
EventMachine.stop
}
http2.callback {
// do some operation http2.response
Crack::XML.parse(http2.response).to_json
EventMachine.stop
}
end
somehow merge
return merged_result
end
Source: (StackOverflow)
The following query works with requestmaker:
URI:
http://www.cleverbot.com/webservicemin/
Query:
start=y&icognoid=wsf&fno=0&sub=Say&islearning=1&cleanslate=false&stimulus=!!!%20there%20was%20an%20error%20!!!&icognocheck=af71393ce00d9126a247df2f53948e79
But it does not work with em-http-request:
require 'eventmachine'
require 'em-http-request'
uri = 'http://www.cleverbot.com/webservicemin/'
query = 'start=y&icognoid=wsf&fno=0&sub=Say&islearning=1&cleanslate=false&stimulus=!!!%20there%20was%20an%20error%20!!!&icognocheck=af71393ce00d9126a247df2f53948e79'
EM.run do
http = EM::HttpRequest.new(uri).post(query: query)
http.callback { puts http.response; EM.stop }
http.errback { puts 'There was an error'; EM.stop }
end
which prints There was an error. I feel stumped because this simple example works with any other method of sending a request and I've checked around to see if my usage was wrong but it doesn't seem to be.
Edit: Just for reference, this is not the correct way to use cleverbot. I made a second mistake by sending the data under :query. If you use http.post(body: query) it will work
Source: (StackOverflow)
I'm getting messages of a RabbitMQ queue and each message is a URL that I want to make a request to. Now I'm using the AMQP gem to subscribe to the queue and that uses EventMachine, so I'm using the the em-http-request library to make the http requests. According to the documentation here: https://github.com/igrigorik/em-http-request/wiki/Parallel-Requests
The following will issue asynchronous http-requests:
EventMachine.run {
http1 = EventMachine::HttpRequest.new('http://google.com/').get
http2 = EventMachine::HttpRequest.new('http://yahoo.com/').get
http1.callback { }
http2.callback { }
end
So when I subscribe to the RabbitMQ queue I have the following code:
x = 0
EventMachine.run do
connection = AMQP.connect(:host => '127.0.0.1')
channel = AMQP::Channel.new(connection)
channel.prefetch(50)
queue = channel.queue("http.requests")
exchange = channel.direct("")
queue.subscribe do |metadata, payload|
url = payload.inspect
eval "
@http#{x} = EventMachine::HttpRequest.new(url).get
@http#{x}.callback do
puts \"got a response\"
puts @http#{x}.response
end
x = x+1
"
end
end
This dynamically creates new variables and creates new http requests, similar to the way described in the em-http-request documentation. But is there a way to test whether the requests are actually being made asynchronously? Is it possible to write to the console every time a get request is fired off so I can see they are fired off one after the other without waiting for a response?
Source: (StackOverflow)
I want to use Ruby EventMachine and em-http-request to run parallel HTTP synchronous requests triggered from different threads.
The idea is to run a single reactor in its own thread and push HTTP requests to complete on its event queue thanks to EM.next_tick.
The pattern for every call could be
def complete_request(url, options)
Thread.new { EM.run } unless EM.reactor_running?
EM.run do
EM.next_tick do
con = EventMachine::HttpRequest.new(url, options[:connection_headers])
http = com.setup_request(verb, head: options[:headers], body: options[:body])
http.errback { }
http.callback { }
end
end
# wait for request completion (but how?)
...
end
reqs = []
responses = []
reqs << Thread.new { responses << complete_request('http://www.stackoverflow.com', verb: get) }
reqs << Thread.new { responses << complete_request('http://www.flickr.com', verb: get) }
reqs.each { |req| req.join }
To make the requests synchronous, I tried to use Fibers but unsuccessfully. Either the request fails to connect or it completes but never exits the event loop.
I don't want to call EM.stop in the callbacks because it would screw up other requests being executed in parallel I guess, and would also stop the reactor while I want it to run until I decide no more requests should be treated.
Does anyone already try to use EventMachine and em-http-request this way ? Can EventMachine support this use case ?
Source: (StackOverflow)
I'm new in Ruby. I need to fetch some info using post request. The number of requests can be till 1 000 000. So I want to perform it async, handling callbacks. I found some approaches that uses EventMachine, but there is no obvious way to perform multi http post request, so now I use following function:
def fetchFriends( )
EM.run do
1.upto(1000) do |x|
http = ( EM::HttpRequest.new('https://api.vk.com/method/friends.get', :connect_timeout => 10, :inactivity_timeout => 10 ).post :body => {:user_id => x } )
http.errback {
p "Oops";
EM.stop if counter == 1
counter-=1;
}
http.callback {
puts "succeed"
EM.stop if counter == 1
counter-=1;
}
end
end
end
But in this case only first request performs correctly, other requests returns 0 as response from server, and empty body, so it looks like some error in EM.
So, can anyone advice me how to write some ruby.async( "request", callback, errorback )?
Source: (StackOverflow)