twisted
Event-driven networking engine written in Python.
Twisted
an event-driven networking engine written in python and mit licensed.
I am trying to start learning twisted for socket servers creation. I want to add some useful features (like auth, and maybe some other). Maybe someone can point me to a good tutorial which will help me to start (+ maybe some other ideas)
Source: (StackOverflow)
I'm playing around with Twisted and documentation found on their homepage doesn't answer all my questions.
The topic I am most interested at the moment is Twisted Application Framework. Also some open source servers using twisted framework would provide nice material for studying how it's all tied up together in a bigger project.
So far I've checked out iCal and read documentation on twisted website (3x) and few good articles I found in google.
Source: (StackOverflow)
I tried to implement a protocol that can run TLS over TLS using twisted.protocols.tls, an interface to OpenSSL using a memory BIO.
I implemented this as a protocol wrapper that mostly looks like a regular TCP transport, but which has startTLS and stopTLS methods for adding and removing a layer of TLS respectively. This works fine for the first layer of TLS. It also works fine if I run it over a "native" Twisted TLS transport. However, if I try to add a second TLS layer using the startTLS method provided by this wrapper, there's immediately a handshake error and the connection ends up in some unknown unusable state.
The wrapper and the two helpers that let it work looks like this:
from twisted.python.components import proxyForInterface
from twisted.internet.error import ConnectionDone
from twisted.internet.interfaces import ITCPTransport, IProtocol
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
from twisted.protocols.policies import ProtocolWrapper, WrappingFactory
class TransportWithoutDisconnection(proxyForInterface(ITCPTransport)):
"""
A proxy for a normal transport that disables actually closing the connection.
This is necessary so that when TLSMemoryBIOProtocol notices the SSL EOF it
doesn't actually close the underlying connection.
All methods except loseConnection are proxied directly to the real transport.
"""
def loseConnection(self):
pass
class ProtocolWithoutConnectionLost(proxyForInterface(IProtocol)):
"""
A proxy for a normal protocol which captures clean connection shutdown
notification and sends it to the TLS stacking code instead of the protocol.
When TLS is shutdown cleanly, this notification will arrive. Instead of telling
the protocol that the entire connection is gone, the notification is used to
unstack the TLS code in OnionProtocol and hidden from the wrapped protocol. Any
other kind of connection shutdown (SSL handshake error, network hiccups, etc) are
treated as real problems and propagated to the wrapped protocol.
"""
def connectionLost(self, reason):
if reason.check(ConnectionDone):
self.onion._stopped()
else:
super(ProtocolWithoutConnectionLost, self).connectionLost(reason)
class OnionProtocol(ProtocolWrapper):
"""
OnionProtocol is both a transport and a protocol. As a protocol, it can run over
any other ITransport. As a transport, it implements stackable TLS. That is,
whatever application traffic is generated by the protocol running on top of
OnionProtocol can be encapsulated in a TLS conversation. Or, that TLS conversation
can be encapsulated in another TLS conversation. Or **that** TLS conversation can
be encapsulated in yet *another* TLS conversation.
Each layer of TLS can use different connection parameters, such as keys, ciphers,
certificate requirements, etc. At the remote end of this connection, each has to
be decrypted separately, starting at the outermost and working in. OnionProtocol
can do this itself, of course, just as it can encrypt each layer starting with the
innermost.
"""
def makeConnection(self, transport):
self._tlsStack = []
ProtocolWrapper.makeConnection(self, transport)
def startTLS(self, contextFactory, client, bytes=None):
"""
Add a layer of TLS, with SSL parameters defined by the given contextFactory.
If *client* is True, this side of the connection will be an SSL client.
Otherwise it will be an SSL server.
If extra bytes which may be (or almost certainly are) part of the SSL handshake
were received by the protocol running on top of OnionProtocol, they must be
passed here as the **bytes** parameter.
"""
# First, create a wrapper around the application-level protocol
# (wrappedProtocol) which can catch connectionLost and tell this OnionProtocol
# about it. This is necessary to pop from _tlsStack when the outermost TLS
# layer stops.
connLost = ProtocolWithoutConnectionLost(self.wrappedProtocol)
connLost.onion = self
# Construct a new TLS layer, delivering events and application data to the
# wrapper just created.
tlsProtocol = TLSMemoryBIOProtocol(None, connLost, False)
tlsProtocol.factory = TLSMemoryBIOFactory(contextFactory, client, None)
# Push the previous transport and protocol onto the stack so they can be
# retrieved when this new TLS layer stops.
self._tlsStack.append((self.transport, self.wrappedProtocol))
# Create a transport for the new TLS layer to talk to. This is a passthrough
# to the OnionProtocol's current transport, except for capturing loseConnection
# to avoid really closing the underlying connection.
transport = TransportWithoutDisconnection(self.transport)
# Make the new TLS layer the current protocol and transport.
self.wrappedProtocol = self.transport = tlsProtocol
# And connect the new TLS layer to the previous outermost transport.
self.transport.makeConnection(transport)
# If the application accidentally got some bytes from the TLS handshake, deliver
# them to the new TLS layer.
if bytes is not None:
self.wrappedProtocol.dataReceived(bytes)
def stopTLS(self):
"""
Remove a layer of TLS.
"""
# Just tell the current TLS layer to shut down. When it has done so, we'll get
# notification in *_stopped*.
self.transport.loseConnection()
def _stopped(self):
# A TLS layer has completely shut down. Throw it away and move back to the
# TLS layer it was wrapping (or possibly back to the original non-TLS
# transport).
self.transport, self.wrappedProtocol = self._tlsStack.pop()
I have simple client and server programs for exercising this, available from launchpad (bzr branch lp:~exarkun/+junk/onion). When I use it to call the startTLS method above twice, with no intervening call to stopTLS, this OpenSSL error comes up:
OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')]
Why do things go wrong?
Source: (StackOverflow)
Assuming a team of developers are equally comfortable with writing Javascript on the server side as they are with Python & Twisted, when is Node.js going to be more appropriate than Twisted (and vice versa)?
Source: (StackOverflow)
I'm increasingly hearing that Python's Twisted framework rocks and other frameworks pale in comparison.
Can anybody shed some light on this and possibly compare Twisted with other network programming frameworks.
Source: (StackOverflow)
I have two integers in my program; let's call them "a" and "b". I would like to add them together and get another integer as a result. These are regular Python int objects. I'm wondering; how do I add them together with Twisted? Is there a special performAsynchronousAddition function somewhere? Do I need a Deferred? What about the reactor? Is the reactor involved?
Source: (StackOverflow)
A (long) while ago I wrote a web-spider that I multithreaded to enable concurrent requests to occur at the same time. That was in my Python youth, in the days before I knew about the GIL and the associated woes it creates for multithreaded code (IE, most of the time stuff just ends up serialized!)...
I'd like to rework this code to make it more robust and perform better. There are basically two ways I could do this: I could use the new multiprocessing module in 2.6+ or I could go for a reactor / event-based model of some sort. I would rather do the later since it's far simpler and less error-prone.
So the question relates to what framework would be best suited to my needs. The following is a list of the options I know about so far:
- Twisted: The granddaddy of Python reactor frameworks: seems complex and a bit bloated however. Steep learning curve for a small task.
- Eventlet: From the guys at lindenlab. Greenlet based framework that's geared towards these kinds of tasks. I had a look at the code though and it's not too pretty: non-pep8 compliant, scattered with prints (why do people do this in a framework!?), API seems a little inconsistent.
- PyEv: Immature, doesn't seem to be anyone using it right now though it is based on libevent so it's got a solid backend.
- asyncore: From the stdlib: über low-level, seems like a lot of legwork involved just to get something off the ground.
- tornado: Though this is a server oriented product designed to server dynamic websites it does feature an async HTTP client and a simple ioloop. Looks like it could get the job done but not what it was intended for.
[edit: doesn't run on Windows unfortunately, which counts it out for me - its a requirement for me to support this lame platform]
Is there anything I have missed at all? Surely there must be a library out there that fits the sweet-spot of a simplified async networking library!
[edit: big thanks to intgr for his pointer to this page. If you scroll to the bottom you will see there is a really nice list of projects that aim to tackle this task in one way or another. It seems actually that things have indeed moved on since the inception of Twisted: people now seem to favour a co-routine based solution rather than a traditional reactor / callback oriented one. The benefits of this approach are clearer more direct code: I've certainly found in the past, especially when working with boost.asio in C++ that callback based code can lead to designs that can be hard-to-follow and are relatively obscure to the untrained eye. Using co-routines allows you to write code that looks a little more synchronous at least. I guess now my task is to work out which one of these many libraries I like the look of and give it a go! Glad I asked now...]
[edit: perhaps of interest to anyone who followed or stumbled on this this question or cares about this topic in any sense: I found a really great writeup of the current state of the available tools for this job]
Source: (StackOverflow)
Everything I've read and experienced ( Tornado based apps ) leads me to believe that ePoll is a natural replacement for Select and Poll based networking, especially with Twisted. Which makes me paranoid, its pretty rare for a better technique or methodology not to come with a price.
Reading a couple dozen comparisons between epoll and alternatives shows that epoll is clearly the champion for speed and scalability, specifically that it scales in a linear fashion which is fantastic. That said, what about processor and memory utilization, is epoll still the champ?
Source: (StackOverflow)
I have setup a run configuration in Eclipse and need to send SIGINT (Ctrl-C) to the program. There is cleanup code in the program that runs after SIGINT, so pressing Eclipse's "Terminate" buttons won't work (they send SIGKILL I think). Typing CTRL-C into the Console also doesn't work.
How do I send SIGINT to a process running inside an Eclipse Console?
(FWIW I am running a Twisted daemon and need Twisted to shutdown correctly, which only occurs on SIGINT)
Source: (StackOverflow)
I am running a django application on twisted using the django-on-twisted scripts from this site.
All requests are served by an nginx server which reverse proxies relevant requests to twisted. I have a url setup for an API, which basically just receives get requests and does some processing on the get parameters before sending a response. However, when a specific client is hitting the api, the twisted server just shuts down. Pasted below is the Nginx log:
the.ip.of.client - - [21/Apr/2012:11:30:36 -0400] "GET /api/url/?get=params&more=params HTTP/1.1" 499 0 "-" "Java/1.6.0_24"
The twisted logs show nothing but twisted stops working at this point. By the error code 499, i am assuming that the client closed the connection unexpectedly, which I have no problem with. Whether the client receives the response or not is not important to me. Here is the relevant django view:
def api_url(request):
if request.GET:
get_param = request.GET.get('get', [''])[0]
more_param = request.GET.get('more', [''])[0]
#some processing here based on the get params
return HttpResponse('OK')
else:
raise Http404
The request from the client is a valid request and does not affect the processing in an adverse way. I have tested it from the shell. When I tried it on the django development server, it crashed in the same way too without leaving any traces of receiving the request. Everything works perfectly well when testing it from the browser. Also, the twisted server works well for all the regular use cases. This is the first time I am facing an issue with it. Any help or pointers will be appreciated.
Source: (StackOverflow)
I'm having trouble developing a reverse proxy in Twisted. It works, but it seems overly complex and convoluted. So much of it feels like voodoo...
Are there any simple, solid examples of asynchronous program structure on the web or in books? A sort of best practices guide? When I complete my program I'd like to be able to still see the structure in some way, not be looking at a bowl of spaghetti.
Source: (StackOverflow)
Why do we need RabbitMQ when we have a more powerful network framework in Python called Twisted. I am trying to understand the reason why someone would want to use RabbitMQ.
Could you please provide a scenario or an example using RabbitMQ?
Also, where can I find a tutorial on how to use RabbitMQ?
Source: (StackOverflow)
If I send two TCP messages, do I need to handle the case where the latter arrives before the former? Or is it guaranteed to arrive in the order I send it? I assume that this is not a Twisted-specific example, because it should conform to the TCP standard, but if anyone familiar with Twisted could provide a Twisted-specific answer for my own peace of mind, that'd be appreciated :-)
Source: (StackOverflow)
I am about to get my hands dirty on building a application for one of the trading company entirely on python. I have experimented with django and flask. Here are few of the things I have come up.
- Django is far too easy to develop an application but I feel that the ORM is limiting me in several ways.
- I was very impressed with flask and how we can combine it with Jinja, SQLAlchemy and Twisted but it lacks a lots of documentation and support like Django has (Seriously Django documentation is totally awesome).
So I am kinda confused as what shall I do. Should I go with Django or Flask + Werzeug?
Source: (StackOverflow)
The Twisted Plugin System is the preferred way to write extensible twisted applications.
However, due to the way the plugin system is structured (plugins go into a twisted/plugins directory which should not be a Python package), writing a proper setup.py for installing those plugins appears to be non-trivial.
I've seen some attempts that add 'twisted.plugins' to the 'packages' key of the distutils setup command, but since it is not really a package, bad things happen (for example, an __init__.py is helpfully added by some tools).
Other attempts seem to use 'package_data' instead (eg, http://bazaar.launchpad.net/~glyph/divmod.org/trunk/view/head:/Epsilon/epsilon/setuphelper.py), but that can also fail in weird ways.
The question is: has anyone successfully written a setup.py for installing twisted plugins which works in all cases?
Source: (StackOverflow)