EzDevInfo.com

What is the point/purpose of Ruby EventMachine, Python Twisted, or JavaScript Node.js?

I don't understand what problem these frameworks solve. Are they replacements for a HTTP server like Apache HTTPD, Tomcat, Mongrel, etc? Or are they more? Why might I use them... some real world examples? I've seen endless examples of chat rooms and broadcast services, but don't see how this is any different than, for instance, setting up a Java program to open sockets and dispatch a thread for each request.

I think I understand the non-blocking I/O, but I don't understand how that is any different than a multi-threaded web server. For Node.js I read that it only has a single thread, and that this may be more efficient than juggling multiple threads, but is that the only difference between these frameworks and a traditional web server?

How do EventMachine & Rails integrate?

I've found plenty of articles showing me what EventMachine is and how to set up endless "Hello World!" examples, but I'm still at a loss as to how this integrates with my Rails application.

As a example, I have an existing Rails app that has many concurrent users who may be editing the same row in my database simultaneously. I was thinking that I would allow the record to be loaded by two (or more) different people, but notify those users if the record was updated and force the latter users to reconcile any conflicting changes before saving it back to the database. I was thinking I could handle the notifications and reconciliations using Javascript on the client side and websockets (or flashsockets) to communicate with the browser (on the event that another user updates the record--like a push notification, or something).

The last part led me to EventMachine, but--as my lead-off question indicates--I'm at a loss about how to integrate this into my Rails app.

Can anyone give me some insight to this (like, a good macro-level viewpoint) or point me towards some good resources? (Apart from EventMachine's wiki, as I've already been there).

How do you spawn an EventMachine "inside" a Rails app?

I've got a Rails application, and am looking to add some sort of WebSocket support to it. From various googling, it appears that the best Ruby based WebSocket solution is em-websocket running on EventMachine.

I was wondering if there was a way to "integrate" an EventMachine reactor into Rails? Where do I put the initialization code? Is this the proper way to accomplish this?

I've seen this example that falls back on Sinatra to do an EventMachine GET request, but that isn't quite what I'm looking for.

Any help is appreciated.

Why is rails not fully supporting writing evented code out out the box

After Node.js came out, it was the one thing that popularized evented programming. But, Ruby does have EventMachine which supports writing evented code.

The requirements for supporting eventing in rails are:
1. Evented server (thin, rainbows) which runs a reactor
2. Fibers (Ruby 1.9.3) in order to make writing evented code easier, else we could have used threads.
3. All gems evented (example mysql2).

Nodejs showed the obvious benefits of evented programming. So why is the rails community NOT adopting eventmachine? I think one of the reasons rails is not fully portable to eventmachine is because of the dependency on underlying gems which may not be evented. But does anyone know if there is a plan to make a move in that direction?

Rails can do what Nodejs does, but Nodejs started out by advocating evented programming on all the library makers, so by convention most dependencies that you add to package.json in node, you know that it will be evented and will work with nodejs out of the box.

Eventmachine gem install fail

I am not able to install eventmachine

Ok I know this has been asked quite a lot, but no solutions I found has helped me so far.So I posted here as a last resort.

I use windows 8 x64 using ruby 2.0.0p195 (x64) (installed throught Ruby Installer for Windows with the apropriate Devkit)

So the Error is

    Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension
C:/Ruby200-x64/bin/ruby.exe extconf.rb
checking for main() in -lssl... no
checking for rb_trap_immediate in ruby.h,rubysig.h... no
checking for rb_thread_blocking_region()... yes
checking for inotify_init() in sys/inotify.h... no
checking for __NR_inotify_init in sys/syscall.h... no
checking for writev() in sys/uio.h... no
checking for rb_wait_for_single_fd()... yes
checking for rb_enable_interrupt()... no
checking for rb_time_new()... yes
checking for windows.h... yes
checking for winsock.h... yes
checking for main() in -lkernel32... yes
checking for main() in -lrpcrt4... yes
checking for main() in -lgdi32... yes
creating Makefile

make
generating rubyeventmachine-x64-mingw32.def
compiling binder.cpp
In file included from c:\users\btdy\downloads\devkit\mingw\bin\../lib/gcc/x86_64
-w64-mingw32/4.7.2/../../../../x86_64-w64-mingw32/include/process.h:12:0,
             from c:/Ruby200-x64/include/ruby-2.0.0/ruby/win32.h:60,
             from c:/Ruby200-x64/include/ruby-2.0.0/ruby/defines.h:153,
             from c:/Ruby200-x64/include/ruby-2.0.0/ruby/ruby.h:70,
             from c:/Ruby200-x64/include/ruby-2.0.0/ruby.h:33,
             from em.h:24,
             from project.h:150,
             from binder.cpp:20:
c:\users\btdy\downloads\devkit\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.2/../
../../../x86_64-w64-mingw32/include/sys/types.h:68:16: error: conflicting declar
ation 'typedef _pid_t pid_t'
In file included from binder.cpp:20:0:
project.h:97:13: error: 'pid_t' has a previous declaration as 'typedef int pid_t
'
In file included from project.h:151:0,
             from binder.cpp:20:
ed.h: In member function 'void EventableDescriptor::SetSocketInvalid()':
ed.h:43:40: warning: overflow in implicit constant conversion [-Woverflow]
make: *** [binder.o] Error 1      

Things I have tried putting gem "eventmachine", "~> 1.0.3" or gem "eventmachine", "~> 1.0.0.beta.4.1" in my gemfile and removed/modify eventmachine from gemfile.lock making a new testapp and instaling Eventmachine there (fail)

Things I suspect I tracked down dependencies for eventmachine(can be found on Rubygems) and I see that I cannot install bluecloth neither (shows even bigger log error) and neither can I install tidy-ext (bluecloth depencency) so I think it might have some relevance , also cannot install thin

For any info just ask

Thank you all in advance!!

using the postgresql gem async

I'm using Goliath (which is powered by eventmachine) and the postgres gem pg, currently I'm using the pg gem in a blocking way: conn.exec('SELECT * FROM products') (for example) and I'm wondering whether there is a better way to connect to a postgres database?

Does EventMachine issue close_notify on close_connection if tls_start was run?

I am trying to program a simple FTPS server based on EventMachine.

Control socket works properly. When using data socket for data transfer, it seems that everything works well (TLS handshake completes successfully and data is received), but then I get an error message. Here is what I get in Filezilla:

Command:    LIST
Response:   150 Opening ASCII mode data connection for file list
Trace:  CFtpControlSocket::TransferParseResponse()
Trace:    code = 1
Trace:    state = 4
Trace:  CFtpControlSocket::SendNextCommand()
Trace:  CFtpControlSocket::TransferSend()
Trace:    state = 5
Trace:  CTransferSocket::OnConnect
Trace:  CTlsSocket::Handshake()
Trace:  gnutls_session_get_data on primary socket failed: -51
Trace:  CTlsSocket::ContinueHandshake()
Trace:  CTlsSocket::OnSend()
Trace:  CTlsSocket::OnSend()
Trace:  CTlsSocket::OnRead()
Trace:  CTlsSocket::ContinueHandshake()
Trace:  CTlsSocket::OnRead()
Trace:  CTlsSocket::ContinueHandshake()
Trace:  Handshake successful
Trace:  Cipher: AES-256-CBC, MAC: SHA1
Trace:  CTlsSocket::OnRead()
Trace:  CTransferSocket::OnConnect
Trace:  CTransferSocket::OnReceive(), m_transferMode=0
Listing:    -rwxrwxrwx 1 owner  group         1011 Dec 12 23:05 a.tmp
Trace:  CTlsSocket::Failure(-9, 0)
Error:  GnuTLS error -9: A TLS packet with unexpected length was received.
Status: Server did not properly shut down TLS connection
Error:  Could not read from transfer socket: ECONNABORTED - Connection aborted
Trace:  CTransferSocket::TransferEnd(3)
Trace:  CFtpControlSocket::OnReceive()
Response:   226 Closing data connection, sent 70 bytes
Trace:  CFtpControlSocket::TransferParseResponse()
Trace:    code = 2
Trace:    state = 5
Trace:  CFtpControlSocket::SendNextCommand()
Trace:  CFtpControlSocket::TransferSend()
Trace:    state = 8
Trace:  CFtpControlSocket::TransferEnd()
Trace:  CFtpControlSocket::ResetOperation(2)
Trace:  CControlSocket::ResetOperation(2)
Trace:  CFtpControlSocket::ParseSubcommandResult(2)
Trace:  CFtpControlSocket::ListSubcommandResult()
Trace:    state = 3
Trace:  CFtpControlSocket::ResetOperation(2)
Trace:  CControlSocket::ResetOperation(2)
Error:  Failed to retrieve directory listing
Trace:  CFileZillaEnginePrivate::ResetOperation(2)

ftp-ssl also issues an error although somewhat different

--> LIST
150 Opening ASCII mode data connection for file list
===>START SSL connect on DATA
ftp: SSL_connect DATA error 0 - error:00000000:lib(0):func(0):reason(0)
226 Closing data connection, sent 70 bytes
421 Service not available, remote server has closed connection

From what I found on the web I suspect this behavior is because close_notify is not issued at the end of connection? If so, is there any known Ruby based solution for that (f.ex. using OpenSSL::SSL::SSLSocket)?

Error installing EventMachine gem on OSX Yosemite

Here is the full output of the error message:

    Fetching: eventmachine-1.0.3.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing eventmachine:
    ERROR: Failed to build gem native extension.

    /Users/Tyler/.rvm/rubies/ruby-2.2.0/bin/ruby -r ./siteconf20141229-47086-wfdm3m.rb extconf.rb
checking for rb_trap_immediate in ruby.h,rubysig.h... no
checking for rb_thread_blocking_region()... no
checking for inotify_init() in sys/inotify.h... no
checking for __NR_inotify_init in sys/syscall.h... no
checking for writev() in sys/uio.h... yes
checking for rb_wait_for_single_fd()... yes
checking for rb_enable_interrupt()... no
checking for rb_time_new()... yes
checking for sys/event.h... yes
checking for sys/queue.h... yes
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling binder.cpp
compiling cmain.cpp
compiling ed.cpp
compiling em.cpp
em.cpp:827:9: error: use of undeclared identifier 'rb_thread_select'; did you mean 'rb_thread_fd_select'?
        return EmSelect (maxsocket+1, &fdreads, &fdwrites, &fderrors, &tv);
               ^~~~~~~~
               rb_thread_fd_select
./em.h:25:20: note: expanded from macro 'EmSelect'
  #define EmSelect rb_thread_select
                   ^
/Users/Tyler/.rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby/intern.h:454:5: note: 'rb_thread_fd_select' declared here
int rb_thread_fd_select(int, rb_fdset_t *, rb_fdset_t *, rb_fdset_t *, struct timeval *);
    ^
em.cpp:827:32: error: cannot initialize a parameter of type 'rb_fdset_t *' with an rvalue of type 'fd_set *'
        return EmSelect (maxsocket+1, &fdreads, &fdwrites, &fderrors, &tv);
                                      ^~~~~~~~
/Users/Tyler/.rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby/intern.h:454:42: note: passing argument to parameter here
int rb_thread_fd_select(int, rb_fdset_t *, rb_fdset_t *, rb_fdset_t *, struct timeval *);
                                         ^
em.cpp:946:6: error: use of undeclared identifier 'rb_thread_select'; did you mean 'rb_thread_fd_select'?
                                        EmSelect (0, NULL, NULL, NULL, &tv);
                                        ^~~~~~~~
                                        rb_thread_fd_select
./em.h:25:20: note: expanded from macro 'EmSelect'
  #define EmSelect rb_thread_select
                   ^
/Users/Tyler/.rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby/intern.h:454:5: note: 'rb_thread_fd_select' declared here
int rb_thread_fd_select(int, rb_fdset_t *, rb_fdset_t *, rb_fdset_t *, struct timeval *);
    ^
3 errors generated.
make: *** [em.o] Error 1

make failed, exit code 2

Has anyone seen an error like this before? I have the command line tools - never ran into an error compiling this gem on my old laptop also running Yosemite.

Ruby concurrency: non-blocking I/O vs threads

I am playing around with concurrency in Ruby (1.9.3-p0), and have created a very simple, I/O-heavy proxy task. First, I tried the non-blocking approach:

require 'rack'
require 'rack/fiber_pool'
require 'em-http'
require 'em-synchrony'
require 'em-synchrony/em-http'

proxy = lambda {|*|
  result = EM::Synchrony.sync EventMachine::HttpRequest.new('http://google.com').get
  [200, {}, [result.response]]
}

use Rack::FiberPool, :size => 1000
run proxy

=begin
$ thin -p 3000 -e production -R rack-synchrony.ru start
>> Thin web server (v1.3.1 codename Triple Espresso)

$ ab -c100 -n100 http://localhost:3000/
Concurrency Level:      100
Time taken for tests:   5.602 seconds
HTML transferred:       21900 bytes
Requests per second:    17.85 [#/sec] (mean)
Time per request:       5602.174 [ms] (mean)
=end

Hmm, I thought I must be doing something wrong. An average request time of 5.6s for a task where we are mostly waiting for I/O? I tried another one:

require 'sinatra'
require 'sinatra/synchrony'
require 'em-synchrony/em-http'

get '/' do
  EM::HttpRequest.new("http://google.com").get.response
end

=begin
$ ruby sinatra-synchrony.rb -p 3000 -e production
== Sinatra/1.3.1 has taken the stage on 3000 for production with backup from Thin
>> Thin web server (v1.3.1 codename Triple Espresso)

$ ab -c100 -n100 http://localhost:3000/
Concurrency Level:      100
Time taken for tests:   5.476 seconds
HTML transferred:       21900 bytes
Requests per second:    18.26 [#/sec] (mean)
Time per request:       5475.756 [ms] (mean)
=end

Hmm, a little better, but not what I would call a success. Finally, I tried a threaded implementation:

require 'rack'
require 'excon'

proxy = lambda {|*|
  result = Excon.get('http://google.com')
  [200, {}, [result.body]]
}    
run proxy

=begin
$ thin -p 3000 -e production -R rack-threaded.ru --threaded --no-epoll start
>> Thin web server (v1.3.1 codename Triple Espresso)

$ ab -c100 -n100 http://localhost:3000/
Concurrency Level:      100
Time taken for tests:   2.014 seconds
HTML transferred:       21900 bytes
Requests per second:    49.65 [#/sec] (mean)
Time per request:       2014.005 [ms] (mean)
=end

That was really, really surprising. Am I missing something here? Why is EM performing so badly here? Is there some tuning I need to do? I tried various combinations (Unicorn, several Rainbows configurations, etc), but none of them came even close to the simple, old I/O-blocking threading.

Ideas, comments and - obviously - suggestions for better implementations are very welcome.

What is the best way to read files in an EventMachine-based app?

In order not to block the reactor I would like to read files asynchronously, but I've found no obvious way of doing it using EventMachine. I've tried a few different approaches, but none of them feels right:

• Just read the file, it'll block the reactor, but what the hell, it's not that slow (unless it's a big file, and then it definitely is).
• Open the file for reading and read a chunk on each tick (but how much to read? too much and it'll block the reactor, too little and reading will get slower than necessary).
• EM.popen('cat some/file', FileReader) feels really weird, but works better than the alternatives above. In combination with the LineAndTextProtocol it reads lines pretty swiftly.
• EM.attach, but I haven't found any examples of how to use it, and the only thing I've found on the mailing list is that it's deprecated in favour of…
• EM.watch, which I've found no examples of how to use for reading files.

How do you read files within a EventMachine reactor loop?

Rails app and goliath api and database/models sharing

I'm trying to create async api with Goliath framework. Service should write to mysql, add messages to RabbitMQ and receive responses back. There also should be a separate admin application built with Rails. I have several questions about that:

Is there a way to effectively share models between Rails and Goliath? Is there any problems to use Activerecord or any other orm with em? Are there any best practices, configuration (connection pool size, driver) or other options about that? What i have to use to receive messages from AMQP? Would it better to build a separate eventmachine daemon or i can use somehow Goliath's one for that stuff? Thanks for advance.

Thin EventMachine Sinatra vs. Rails

I have been looking into the possibility of backgrounding some jobs with EventMachine. In Sinatra this appears to work great but Rails 3 appears to execute all ticks before rendering a view.

When I run the following code under the thin webserver it behaves as expected. The first request returns immediately and the second request is waiting for the 3 second sleep call to finish. This is the expected behavior.

class EMSinatra < Sinatra::Base
  get "/" do
    EM.next_tick { sleep 3 }
    "Hello"
  end
end

Whereas in Rails 3 running I am trying to do the same thing: (running under thin)

class EmController < ApplicationController
  def index
    EM.next_tick {
      sleep(3)
    }
  end
end

In Rails the sleep call happens before rendering the view to the browser. The result is that I am waiting for 3 seconds for the initial page to render.

Does anybody know why this is happening? I am not looking for comments on wether this is a good practice or not. I am simply experimenting. Throwing small tasks into the reactor loop seems like an interesting thing to look into. Why should the client have to wait if I am going to make some non-blocking http-requests?

Why is EventMachine's defer slower than a Ruby Thread?

I have two scripts which use Mechanize to fetch a Google index page. I assumed EventMachine will be faster than a Ruby thread, but it's not.

EventMachine code costs: "0.24s user 0.08s system 2% cpu 12.682 total"

Ruby Thread code costs: "0.22s user 0.08s system 5% cpu 5.167 total "

Am I using EventMachine in the wrong way?

EventMachine:

require 'rubygems'
require 'mechanize'
require 'eventmachine'

trap("INT") {EM.stop}

EM.run do 
  num = 0
  operation = proc {
    agent = Mechanize.new
    sleep 1
    agent.get("http://google.com").body.to_s.size
  }
  callback = proc { |result|
    sleep 1
    puts result
    num+=1
    EM.stop if num == 9
  }

  10.times do 
    EventMachine.defer operation, callback
  end
end

Ruby Thread:

require 'rubygems'
require 'mechanize'


threads = []
10.times do 
  threads << Thread.new do 
    agent = Mechanize.new
    sleep 1
    puts agent.get("http://google.com").body.to_s.size
    sleep 1
  end
end


threads.each do |aThread| 
  aThread.join
end

How should I handle this use case using EventMachine?

I have an application that reacts to messages sent by clients. One message is reload_credentials, that the application receives any time a new client registers. This message will then connect to a PostgreSQL database, do a query for all the credentials, and then store them in a regular Ruby hash ( client_id => client_token ).

Some other messages that the application may receive are start,stop,pause which are used to keep track of some session times. My point is that I envision the application functioning in the following way:

• client sends a message
• message gets queued
• queue is being processed

However, for example, I don't want to block the reactor. Furthermore, let's imagine I have a reload_credentials message that's next in queue. I don't want any other message from the queue to be processed until the credentials are reloaded from the DB. Also, while I am processing a certain message ( like waiting for the credentials query to finish) , I want to allow other messages to be enqueued .

Could you please guide me towards solving such a problem? I'm thinking I may have to use em-synchrony, but I am not sure.

Integrating WebSockets with Rails using Rack and Event Machine

I have created an Asynchronous version of Rails 3 that I would like to integrate with a WebSocket implementation.

I am using EventMachine, Ruby 1.9, Fibers and various em-flavoured libraries as documented by the wickedly good Ilya Grigorik.

I have been looking at em-websocket as the handler for WebSocket connections but unsure of the best approach for hooking this into a Rails app.

Ideally, this would work in a similar fashion to node.js with Express and Socket.io - incoming connections should be detected and dispatched to the WebSocket handler or the regular rails stack as indicated by the HTTP headers & etc.

TL;DR

• WebSocket handler that plugs into an existing Rails application
• Transparently dispatch incoming WebSocket requests to endpoints in the app