Twisted is a very general and powerful tool. It can power anything connected to a network, from your corporate message-broadcasting network to your desktop IRC client. This is great for integrating lots of different tools, but can make it very difficult to document and understand how the whole platform is supposed to work. A side effect of this is that it's hard to get started with a project using Twisted, because it's hard to find out where to start.
This guide is to help you understand the right way
to get started
working on a Twisted application. It probably won't answer your specific
questions about how to do things like schedule functions
to call in the future or listen on a socket;
there are other documents that address these concerns and you can read them
later. Although there are other ways for Twisted to call your code,
all Twisted projects should start as a plug-in of some kind.
If you're like most people that have asked me questions about this, you've probably come to Twisted thinking of it as a library of code to help you write an application. It can be, but it is much more useful to think of your code as the library. Twisted is a framework.
The difference between a framework and a library is that a developer's code will run a library's functions; a framework runs the developer's functions, instead. The difference is subtle, but significant; there are a range of resources which have to be allocated and managed regarding start-up and shut-down of an process, such as spawning of threads and handling events. You don't have to use Twisted this way. It is quite possible to write applications that use Twisted almost exclusively as a library. If you use it as a framework, though, Twisted will help you by managing these resources itself.
The central framework class that you will deal with, both as a Twisted
developer and administrator, is twisted.internet.app.Application
. There is one Application
instance per Twisted process, and it is the
top-level manager of resources and handler of events in the Twisted framework.
(Unlike some other frameworks, developers do not subclass Application
; rather than defining methods on it, you
register event handlers to be called by it.) To store configuration data, as
well as other information, Twisted serializes Application
instances, storing all event handlers that
have been registered with them. Since the whole Application
instance is serialized, Twisted
configuration
files are significantly more comprehensive than
those for other
systems. These files store everything related to a running Application
instance; in essence the full state of a
running process.
The central concept that a Twisted system administrator will work with are
files that contain
Application
instances serialized
in various formats optimized for different uses. .TAP
files are
optimized for speed of loading and saving, .TAX
files are
editable by administrators familiar
with XML syntax, and .TAS
files are generated Python source code,
most useful
for developers. The two command-line programs which work with these files are
mktap
and twistd
. The
mktap
utility create .TA*
files from
simple
command-line arguments, and the twistd
daemon will
load and run those files.
There are many ways in which your code will be called by various parts of
the Twisted framework by the time you're done. The initial one we're going to
focus on here is a plug-in for the mktap
utility.
mktap
produces complete, runnable
Application
instances, so no additional work is necessary to make your code work with
twistd
. First we will go through the process of
creating a
plug-in that Twisted can find, then we make it adhere to the mktap
interface. Finally we will load that plug-in with a
server.
Python makes it very easy to dynamically load and evaluate programs. The
plug-in system for Twisted, twisted.python.plugin
, is
a way to find (without loading) and then load plug-ins for particular
systems.
Unlike other plug-in
systems, like the well known ones associated
with The
Gimp, Photoshop, and Apache twisted.python.plugin
is
generic. Any one of the Twisted dot-products
can
define mechanisms for extensibility using plug-ins. Two Twisted dot-products
already load such plug-ins. The
twisted.tap
package loads Twisted Application
builder modules (TAP plug-ins) and the
twisted.coil
package loads configuration
modules (COIL plug-ins).
Twisted finds its plug-ins by using pre-existing Python concepts; the load
path, and packages. Every top-level Python package (that is, a directory whose parent is on sys.path and which
contains an __init__.py
) can potentially contain
some number of plug-ins. Packages which contain plug-ins are called
drop-ins
, because you drop
them into your
sys.path
. The only
difference
between a package and a drop-in is the existence of a file named plugins.tml
(TML for Twisted Module List) that contains
some special Python expressions to identify the location of sub-packages or
modules which can be loaded.
If you look at twisted/plugins.tml
, you will
notice that Twisted is a drop-in for itself! You can browse through it for
lots of examples of plug-ins being registered.
The most prevalent kind of plug-in is the TAP
(Twisted
Application builder) type. These are relatively simple to get started with.
Let's look at an excerpt from Twisted's own plugins.tml for an example of
registering one:
# ... register("Twisted Web Automated TAP builder", "twisted.tap.web", description=""" Builds a Twisted Application instance that contains a general-purpose web server, which can serve from a filesystem or application resource. """, type="tap", tapname="web") # ...
plugins.tml
will be a list of calls to one function:
register(name, module, type=plugin_type, description=user_description [, **plugin_specific_data])
name
is a free-form string, to be
displayed to the user in presentation contexts (like a web page, or a
list-box in a GUI).module
is a string which must
be the fully-qualified name of a Python module.type
is the name of the system you are
plugging in to. Be sure to spell this right, or Twisted won't find your
plug-in at all!**plugin_specific_data
is a dictionary of
information associated with the plug-in, specific to the type
of plug-in it is. Note that some plug-in
types may require a specific bit of data in order to work.Note the tapname
parameter given in
the example above. This parameter is an example of **plugin_specific_data
. The parameter tapname
is only used by "tap"
-type modules. It indicates what name to use
on the mktap
command line. In English, this
particular call to register
means When the user
types
.
mktap web
, it selects the module twisted.tap.web
to handle the rest of the arguments
Now that you understand how to register a plug-in, let's move along to writing your first one.
As an example, we are going to work on a Quote of the Day application,
TwistedQuotes
.
Aspects of this application will be explored in more depth
throughout in the Twisted documentation.
TwistedQuotes
is a very simple plugin which is a great
demonstration of
Twisted's power. It will export a small kernel of functionality -- Quote of
the Day -- which can be accessed through every interface that Twisted supports:
web pages, e-mail, instant messaging, a specific Quote of the Day protocol, and
more.
First, make a directory, TwistedQuotes
, where you're going to
keep
your code. If you installed Twisted from source, the path of least resistance
is probably just to make a
directory inside your Twisted-X.X.X
directory, which will already be in your sys.path
. If you want to put it elsewhere, make
sure that your TwistedQuotes
directory is a package on your python
path.
The directory you add to your PYTHONPATH
needs to be the directory containing your package's
directory! For example, if your TwistedQuotes directory is
/my/stuff/TwistedQuotes, you can export
PYTHONPATH=/my/stuff:$PYTHONPATH
in UNIX, or edit the PYTHONPATH
environment variable to add /my/stuff;
at the beginning through the System
Properties dialog on Windows.
You will then need to add an __init__.py
to this
directory, to mark it as a package. (For more information on exactly how
Python packages work, read this section of the Python tutorial.) In order to test that everything is
working, start up the Python interactive interpreter, or your favorite IDE, and
verify that the package imports properly.
Python 2.1.3 (#1, Apr 20 2002, 22:45:31) [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 Type "copyright", "credits" or "license" for more information. >>> import TwistedQuotes >>> # No traceback means you're fine.
(You'll need to put this code into a file called quoters.py
in your TwistedQuotes
directory.)
This code listing shows us what the Twisted Quotes system is all about. The
code doesn't have any way of talking to the outside world, but it provides a
library which is a clear and uncluttered abstraction: give me the quote of
the day
.
Note that this module does not import any Twisted functionality at all! The
reason for doing things this way is integration. If your business
objects
are not stuck to your user interface, you can make a module that
can integrate those objects with different protocols, GUIs, and file formats.
Having such classes provides a way to decouple your components from each other,
by allowing each to be used independently.
In this manner, Twisted itself has minimal impact on the logic of your
program. Although the Twisted dot products
are highly interoperable,
they
also follow this approach. You can use them independently because they are not
stuck to each other. They communicate in well-defined ways, and only when that
communication provides some additional feature. Thus, you can use twisted.web
with twisted.enterprise
, but neither requires the other, because
they are integrated around the concept of Deferreds.
(Don't worry we'll get to each of those features in later documentation.)
Your Twisted applications should follow this style as much as possible. Have (at least) one module which implements your specific functionality, independant of any user-interface code.
Next, we're going to need to associate this abstract logic with some way of displaying it to the user. We'll do this by writing a Twisted server protocol, which will respond to the clients that connect to it by sending a quote to the client and then closing the connection. Note: don't get too focused on the details of this -- different ways to interface with the user are 90% of what Twisted does, and there are lots of documents describing the different ways to do it.
(You'll need to put this code into a file called quoteproto.py
in your TwistedQuotes
directory.)
This is a very straightforward Protocol
implementation, and the
pattern described above is repeated here. The Protocol contains essentially no
logic of its own, just enough to tie together an object which can generate
quotes (a Quoter
) and an object which can relay
bytes to a TCP connection (a Transport
). When a
client connects to this server, a QOTD
instance is
created, and its connectionMade
method is called.
The QOTDFactory
's role is to specify to the
Twisted framework how to create a Protocol
instance
that will handle the connection. Twisted will not instantiate a QOTDFactory
; you will do that yourself later, in the
mktap
plug-in below.
Note: you can read more specifics of Protocol
and
Factory
in the Writing
Servers HOWTO.
Once we have an abstraction -- a Quoter
-- and we have a
mechanism to connect it to the network -- the QOTD
protocol -- the
next thing to do is to put the last link in the chain of functionality between
abstraction and user. This last link will allow a user to choose a
Quoter
and configure the protocol.
Practically speaking, this link is an interface for a savvy user who will
run the server. (In this case, you; when you have more users, a system
administrator.) For the purposes of this example we will first implement a
mktap
interface. Like most system administrator
tools, this is command-line oriented. (It is possible to implement a graphical
front-end to mktap, using the same plug-in structure, but this has not been
done yet.)
Creating the extension to mktap
is done through
implementing a
module that follows the mktap
plug-in interface,
and then
registering it to be found and loaded by twisted.python.plugin
.
As described above, registration is done by adding a call to
register
in the file
TwistedQuotes/plugins.tml
(You'll need to put this code into a file called quotetap.py
in your TwistedQuotes
directory.)
This module has to conform to a fairly simple interface. It must have a
class called Options
which is a subclass of twisted.python.usage.Options
. It must also have a function
updateApplication(app, config)
, which will be
passed an instance of a twisted.internet.app.Application
and an instance of the
Options
class defined in the module itself, TwistedQuotes.quotetap.Options
. Command-line options
given on the mktap
command line fill in the values
in Options
and are used in updateApplication
to make the actual connections between
objects.
A more detailed discussion of twisted.python.usage.Options
can be found in the document Using usage.Options
.
Now that we've implemented all the necessary pieces, we can finish putting
them together by writing a TML file which allows the mktap
utility to find our protocol module.
Now the QOTD server is ready to be instantiated! Let's start up a server and get a quote from it.
% mktap qotd Saving qotd application to qotd.tap... Saved. % twistd -f qotd.tap % nc localhost 8007 An apple a day keeps the doctor away. % kill `cat twistd.pid`
Let's walk through the above example. First, we run mktap
specifying the Application type
(qotd
) to create.
mktap
reads in our plugins.tml
file, instantiates an Application
object, fills in the appropriate data, and
serializes it out to a qotd.tap
file. Next, we
launch the server using the twistd daemon, passing qotd.tap
as a command line option. The server launches,
listens on the default port from quotetap.py
. Next,
we run nc
to connect to the running server. In this
step, the QOTDFactory
creates a Quoter
instance, which responds to our network connection
by sending a quote string (in this case, the default quote) over our
connection, and then closes the connection. Finally, we shutdown the server by
killing it via a saved out process id file.
(nc
is the netcat
utility, which no UNIX system should be without.)
So we just saw Twisted in action as a framework. With relatively little code, we've got a server that can respond to a request over a network, with two potential alternative back-ends (fortune files and static text).
After reading this (and following along with your own example, of course), you should be familiar with the process of getting your own Twisted code with unique functionality in it running inside of a server. You should be familiar with the concept of a drop-in and a plug-in, and understand both how to create them and how to install them from other people on your system.
By following the rules set out at the beginning of this HOWTO, we have accidentally implemented another piece of useful functionality.
% mktap Usage: mktap [options] <command> [command options] Options: -x, --xml DEPRECATED: same as --type=xml -s, --source DEPRECATED: same as --type=source -e, --encrypted Encrypt file before writing -p, --progress Show progress of plugin loading -d, --debug Show debug information for plugin loading -u, --uid= [default: 1000] -g, --gid= [default: 1000] -a, --append= An existing .tap file to append the plugin to, rather than creating a new one. -t, --type= The output format to use; this can be 'pickle', 'xml', or 'source'. [default: pickle] --help display this message Commands: coil A web-based configuration manager. ftp An FTP server. im A multi-protocol chat client. inetd issues Bug reporting/tracking service. mail An email service. manhole An interactive remote debugger service. news News Server parent Parent service. pinger Zoot Pinger TAP builder module ponger Zoot Ponger TAP builder module portforward A simple port-forwarder. qotd Example of a TAP builder module. sister Sister service. socks A SOCKSv4 proxy service. ssh telnet A simple, telnet-based remote debugging service. toc An AIM TOC service. web A general-purpose web server which can serve from a filesystem or application resource. words A chat service. zoot Zoot TAP builder module
Not only does our Options
class get
instantiated by mktap
directly, the user can query mktap for interactive
help! This is just one small benefit to using Twisted as it was designed. As
more tools that use the
style of
plug-in, more useful functionality will become available from Twisted Quotes.
For example, a graphical tool could provide not just help messages at the
command line, but a listing of all available TAP types and forms for each, for
the user to enter information.tap
It is this kind of power that results from using a dynamic, powerful framework like Twisted. I hope that you take your newfound knowledge and discover all kinds of cool things like this that you get for free just by using it!
The plug-in system is a relatively new part of Twisted, and not as many things use it as they should yet. Watch this space for new developments regarding plug-ins, other systems that you can plug your code into, and more documentation for people wanting to write systems that can be plugged in to!