Introduction to Perspective Broker

Introduction

Suppose you find yourself in control of both ends of the wire: you have two programs that need to talk to each other, and you get to use any protocol you want. If you can think of your problem in terms of objects that need to make method calls on each other, then chances are good that you can use twisted's Perspective Broker protocol rather than trying to shoehorn your needs into something like HTTP, or implementing yet another RPC mechanismMost of Twisted is like this. Hell, most of unix is like this: if you think it would be useful, someone else has probably thought that way in the past, and acted on it, and you can take advantage of the tool they created to solve the same problem you're facing now..

The Perspective Broker system (abbreviated PB, spawning numerous sandwich-related puns) is based upon a few central concepts:

• serialization: taking fairly arbitrary objects and types, turning them into a chunk of bytes, sending them over a wire, then reconstituting them on the other end. By keeping careful track of object ids, the serialized objects can contain references to other objects and the remote copy will still be useful.
• remote method calls: doing something to a local object and causing a method to get run on a distant one. The local object is called a RemoteReference, and you do something by running its .callRemote method.

This document will contain several examples that will (hopefully) appear redundant and verbose once you've figured out what's going on. To begin with, much of the code will just be labelled magic: don't worry about how these parts work yet. It will be explained more fully later.

Class Roadmap

To start with, here are the major classes involved in PB, with links to the file where they are defined (all of which are under twisted/, of course). Don't worry about understanding what they all do yet: it's easier to figure them out through their interaction than explaining them one at a time.

• Application : internet/app.py
• Service : spread/pb.py, subclassed from Service in cred/service.py
• MultiService : internet/app.py
• Factory : internet/protocol.py
• BrokerFactory : spread/pb.py
• Broker : spread/pb.py
• AuthRoot : spread/pb.py

Other classes that are involved at some point:

• RemoteReference : spread/pb.py
• pb.Root : spread/pb.py, actually defined as Root in spread/flavors.py
• pb.Referenceable : spread/pb.py, actually defined as Referenceable in spread/flavors.py

Classes that get involved when you start to care about authorization and security:

• Authorizer : cred/authorizer.py
• Identity : cred/identity.py
• Perspective : spread/pb.py, subclassed from Perspective in cred/perspective.py

Subclassing

Technically you can subclass anything you want, but techically you could also write a whole new framework, which would just waste a lot of time. Knowing which classes are useful to change (by making subclasses) is one of the bits of knowledge you pick up after using Twisted for a few weeks. Here are some hints to get started:

• Protocol: subclass this if you need to implement a new protocol on the wire, like HTTP or SMTP (except that almost all of the standard ones are already implemented). You might also subclass one of the standard implementations if you want to change its back-end behavior: make an SMTP server which actually stores the messages in files instead of mailing them, or a Finger server that returns random messages instead of current login status.
• pb.Root, pb.Referenceable: you'll subclass these to make remotely-referenceable objects using PB. You don't need to change any of the existing behavior, just inherit all of it and add the remotely-accessible methods that you want to export.
• pb.Perspective, pb.Service: you'll probably end up subclassing these when you get into PB programming (with authorization). There are a few methods you'll change, especially with regards to creating new Perspectives.
• Authorizer: subclass this if you want to get users from /etc/passwd, or a database, or LDAP, or other list of usernames and passwords.

XXX: add lists of useful-to-override methods here

Things you can Call Remotely

At this writing, there are three flavors of objects that can be accessed remotely through RemoteReference objects. Each of these flavors has a rule for how the callRemote message is transformed into a local method call on the server. In order to use one of these flavors, subclass them and name your published methods with the appropriate prefix.

• twisted.spread.pb.Perspective

  This is the first class we dealt with. Perspectives are slightly special because they are the root object that a given user can access from a service. A user should only receive a reference to their own Perspective. PB works hard to verify, as best it can, that any method that can be called on a perspective directly is being called on behalf of the user who is represented by that perspective. (Services with unusual requirements for on behalf of, such as simulations with the ability to posess another player's avatar, are accomplished by providing indirected access to another user's Perspective.)

  Perspectives are not usually serialized as remote references, so do not return a perspective directly.

  Remotely accessible methods on Perspectives are named with the perspective_ prefix.

• twisted.spread.flavors.Referenceable

  Referenceable objects are the simplest kind of PB object. You can call methods on them and return them from methods to provide access to other objects' methods.

  However, when a method is called on a Referenceable, it's not possible to tell who called it.

  Remotely accessible methods on Referenceables are named with the remote_ prefix.

• twisted.spread.flavors.Viewable

  Viewable objects are remotely referenceable objects which have the additional requirement that it must be possible to tell who is calling them. The argument list to a Viewable's remote methods is modified in order to include the Perspective representing the calling user.

  Remotely accessible methods on Viewables are named with the view_ prefix.

Things you can Copy Remotely

In addition to returning objects that you can call remote methods on, you can return structured copies of local objects.

There are 2 basic flavors that allow for copying objects remotely. Again, you can use these by subclassing them. In order to specify what state you want to have copied when these are serialized, you can either use the Python default __getstate__ or specialized method calls for that flavor.

• twisted.spread.flavors.Copyable

  This is the simpler kind of object that can be copied. Every time this object is returned from a method or passed as an argument, it is serialized and unserialized.

  Copyable provides a method you can override, getStateToCopyFor(perspective), which allows you to decide what an object will look like for the user who is requesting it. The perspective argument will be an instance of the Perspective subclass for your service, the one which is either pasing an argument or returning a result an instance of your Copyable class.

  For security reasons, in order to allow a particular Copyable class to actually be copied, you must declare a RemoteCopy handler for that Copyable subclass. The easiest way to do this is to declare both in the same module, like so:

  from twisted.spread import flavors
  class Foo(flavors.Copyable):
      pass
  class RemoteFoo(flavors.RemoteCopy):
      pass
  flavors.setCopierForClass(str(Foo), RemoteFoo)
    

  In this case, each time a Foo is copied between peers, a RemoteFoo will be instantiated and populated with the Foo's state. If you do not do this, PB will complain that there have been security violations, and it may close the connection.

• twisted.spread.flavors.Cacheable

  Let me preface this with a warning: Cacheable may be hard to understand. The motivation for it may be unclear if you don't have some experience with real-world applications that use remote method calling of some kind. Once you understand why you need it, what it does will likely seem simple and obvious, but if you get confused by this, forget about it and come back later. It's possible to use PB without understanding Cacheable at all.

  Cacheable is a flavor which is designed to be copied only when necessary, and updated on the fly as changes are made to it. When passed as an argument or a return value, if a Cacheable exists on the side of the connection it is being copied to, it will be referred to by ID and not copied.

  Cacheable is designed to minimize errors involved in replicating an object between multiple servers, especially those related to having stale information. In order to do this, Cacheable automatically registers observers and queries state atomically, together. You can override the method getStateToCacheAndObserveFor(self, perspective, observer) in order to specify how your observers will be stored and updated.

  Similar to getStateToCopyFor, getStateToCacheAndObserveFor passes a Perspective instance from your service. It also passes an observer, which is a remote reference to a secret fourth referenceable flavor: RemoteCache.

  A RemoteCache is simply the object that represents your Cacheable on the other side of the connection. It is registered using the same method as RemoteCopy, above. RemoteCache is different, however, in that it will be referenced by its peer. It acts as a Referenceable, where all methods prefixed with observe_ will be callable remotely. It is recommended that your object maintain a list (note: library support for this is forthcoming!) of observers, and update them using callRemote when the Cacheable changes in a way that should be noticeable to its clients.

  Finally, when all references to a Cacheable from a given Perspective are lost, stoppedObserving(perspective, observer) will be called on the Cacheable, with the same perspective/observer pair that getStateToCacheAndObserveFor was originally called with. Any cleanup remote calls can be made there, as well as removing the observer object from any lists which it was previously in. Any further calls to this observer object will be invalid.