1 """Generic socket server classes.

     2 

     3 This module tries to capture the various aspects of defining a server:

     4 

     5 For socket-based servers:

     6 

     7 - address family:

     8         - AF_INET{,6}: IP (Internet Protocol) sockets (default)

     9         - AF_UNIX: Unix domain sockets

    10         - others, e.g. AF_DECNET are conceivable (see <socket.h>

    11 - socket type:

    12         - SOCK_STREAM (reliable stream, e.g. TCP)

    13         - SOCK_DGRAM (datagrams, e.g. UDP)

    14 

    15 For request-based servers (including socket-based):

    16 

    17 - client address verification before further looking at the request

    18         (This is actually a hook for any processing that needs to look

    19          at the request before anything else, e.g. logging)

    20 - how to handle multiple requests:

    21         - synchronous (one request is handled at a time)

    22         - forking (each request is handled by a new process)

    23         - threading (each request is handled by a new thread)

    24 

    25 The classes in this module favor the server type that is simplest to

    26 write: a synchronous TCP/IP server.  This is bad class design, but

    27 save some typing.  (There's also the issue that a deep class hierarchy

    28 slows down method lookups.)

    29 

    30 There are five classes in an inheritance diagram, four of which represent

    31 synchronous servers of four types:

    32 

    33         +------------+

    34         | BaseServer |

    35         +------------+

    36               |

    37               v

    38         +-----------+        +------------------+

    39         | TCPServer |------->| UnixStreamServer |

    40         +-----------+        +------------------+

    41               |

    42               v

    43         +-----------+        +--------------------+

    44         | UDPServer |------->| UnixDatagramServer |

    45         +-----------+        +--------------------+

    46 

    47 Note that UnixDatagramServer derives from UDPServer, not from

    48 UnixStreamServer -- the only difference between an IP and a Unix

    49 stream server is the address family, which is simply repeated in both

    50 unix server classes.

    51 

    52 Forking and threading versions of each type of server can be created

    53 using the ForkingMixIn and ThreadingMixIn mix-in classes.  For

    54 instance, a threading UDP server class is created as follows:

    55 

    56         class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass

    57 

    58 The Mix-in class must come first, since it overrides a method defined

    59 in UDPServer! Setting the various member variables also changes

    60 the behavior of the underlying server mechanism.

    61 

    62 To implement a service, you must derive a class from

    63 BaseRequestHandler and redefine its handle() method.  You can then run

    64 various versions of the service by combining one of the server classes

    65 with your request handler class.

    66 

    67 The request handler class must be different for datagram or stream

    68 services.  This can be hidden by using the request handler

    69 subclasses StreamRequestHandler or DatagramRequestHandler.

    70 

    71 Of course, you still have to use your head!

    72 

    73 For instance, it makes no sense to use a forking server if the service

    74 contains state in memory that can be modified by requests (since the

    75 modifications in the child process would never reach the initial state

    76 kept in the parent process and passed to each child).  In this case,

    77 you can use a threading server, but you will probably have to use

    78 locks to avoid two requests that come in nearly simultaneous to apply

    79 conflicting changes to the server state.

    80 

    81 On the other hand, if you are building e.g. an HTTP server, where all

    82 data is stored externally (e.g. in the file system), a synchronous

    83 class will essentially render the service "deaf" while one request is

    84 being handled -- which may be for a very long time if a client is slow

    85 to reqd all the data it has requested.  Here a threading or forking

    86 server is appropriate.

    87 

    88 In some cases, it may be appropriate to process part of a request

    89 synchronously, but to finish processing in a forked child depending on

    90 the request data.  This can be implemented by using a synchronous

    91 server and doing an explicit fork in the request handler class

    92 handle() method.

    93 

    94 Another approach to handling multiple simultaneous requests in an

    95 environment that supports neither threads nor fork (or where these are

    96 too expensive or inappropriate for the service) is to maintain an

    97 explicit table of partially finished requests and to use select() to

    98 decide which request to work on next (or whether to handle a new

    99 incoming request).  This is particularly important for stream services

   100 where each client can potentially be connected for a long time (if

   101 threads or subprocesses cannot be used).

   102 

   103 Future work:

   104 - Standard classes for Sun RPC (which uses either UDP or TCP)

   105 - Standard mix-in classes to implement various authentication

   106   and encryption schemes

   107 - Standard framework for select-based multiplexing

   108 

   109 XXX Open problems:

   110 - What to do with out-of-band data?

   111 

   112 BaseServer:

   113 - split generic "request" functionality out into BaseServer class.

   114   Copyright (C) 2000  Luke Kenneth Casson Leighton <lkcl@samba.org>

   115 

   116   example: read entries from a SQL database (requires overriding

   117   get_request() to return a table entry from the database).

   118   entry is processed by a RequestHandlerClass.

   119 

   120 """

   121 

   122 # Author of the BaseServer patch: Luke Kenneth Casson Leighton

   123 

   124 # XXX Warning!

   125 # There is a test suite for this module, but it cannot be run by the

   126 # standard regression test.

   127 # To run it manually, run Lib/test/test_socketserver.py.

   128 

   129 __version__ = "0.4"

   130 

   131 

   132 import socket

   133 import sys

   134 import os

   135 

   136 __all__ = ["TCPServer","UDPServer","ForkingUDPServer","ForkingTCPServer",

   137            "ThreadingUDPServer","ThreadingTCPServer","BaseRequestHandler",

   138            "StreamRequestHandler","DatagramRequestHandler",

   139            "ThreadingMixIn", "ForkingMixIn"]

   140 if hasattr(socket, "AF_UNIX"):

   141     __all__.extend(["UnixStreamServer","UnixDatagramServer",

   142                     "ThreadingUnixStreamServer",

   143                     "ThreadingUnixDatagramServer"])

   144 

   145 class BaseServer:

   146 

   147     """Base class for server classes.

   148 

   149     Methods for the caller:

   150 

   151     - __init__(server_address, RequestHandlerClass)

   152     - serve_forever()

   153     - handle_request()  # if you do not use serve_forever()

   154     - fileno() -> int   # for select()

   155 

   156     Methods that may be overridden:

   157 

   158     - server_bind()

   159     - server_activate()

   160     - get_request() -> request, client_address

   161     - verify_request(request, client_address)

   162     - server_close()

   163     - process_request(request, client_address)

   164     - close_request(request)

   165     - handle_error()

   166 

   167     Methods for derived classes:

   168 

   169     - finish_request(request, client_address)

   170 

   171     Class variables that may be overridden by derived classes or

   172     instances:

   173 

   174     - address_family

   175     - socket_type

   176     - allow_reuse_address

   177 

   178     Instance variables:

   179 

   180     - RequestHandlerClass

   181     - socket

   182 

   183     """

   184 

   185     def __init__(self, server_address, RequestHandlerClass):

   186         """Constructor.  May be extended, do not override."""

   187         self.server_address = server_address

   188         self.RequestHandlerClass = RequestHandlerClass

   189 

   190     def server_activate(self):

   191         """Called by constructor to activate the server.

   192 

   193         May be overridden.

   194 

   195         """

   196         pass

   197 

   198     def serve_forever(self):

   199         """Handle one request at a time until doomsday."""

   200         while 1:

   201             self.handle_request()

   202 

   203     # The distinction between handling, getting, processing and

   204     # finishing a request is fairly arbitrary.  Remember:

   205     #

   206     # - handle_request() is the top-level call.  It calls

   207     #   get_request(), verify_request() and process_request()

   208     # - get_request() is different for stream or datagram sockets

   209     # - process_request() is the place that may fork a new process

   210     #   or create a new thread to finish the request

   211     # - finish_request() instantiates the request handler class;

   212     #   this constructor will handle the request all by itself

   213 

   214     def handle_request(self):

   215         """Handle one request, possibly blocking."""

   216         try:

   217             request, client_address = self.get_request()

   218         except socket.error:

   219             return

   220         if self.verify_request(request, client_address):

   221             try:

   222                 self.process_request(request, client_address)

   223             except:

   224                 self.handle_error(request, client_address)

   225                 self.close_request(request)

   226 

   227     def verify_request(self, request, client_address):

   228         """Verify the request.  May be overridden.

   229 

   230         Return True if we should proceed with this request.

   231 

   232         """

   233         return True

   234 

   235     def process_request(self, request, client_address):

   236         """Call finish_request.

   237 

   238         Overridden by ForkingMixIn and ThreadingMixIn.

   239 

   240         """

   241         self.finish_request(request, client_address)

   242         self.close_request(request)

   243 

   244     def server_close(self):

   245         """Called to clean-up the server.

   246 

   247         May be overridden.

   248 

   249         """

   250         pass

   251 

   252     def finish_request(self, request, client_address):

   253         """Finish one request by instantiating RequestHandlerClass."""

   254         self.RequestHandlerClass(request, client_address, self)

   255 

   256     def close_request(self, request):

   257         """Called to clean up an individual request."""

   258         pass

   259 

   260     def handle_error(self, request, client_address):

   261         """Handle an error gracefully.  May be overridden.

   262 

   263         The default is to print a traceback and continue.

   264 

   265         """

   266         print '-'*40

   267         print 'Exception happened during processing of request from',

   268         print client_address

   269         import traceback

   270         traceback.print_exc() # XXX But this goes to stderr!

   271         print '-'*40

   272 

   273 

   274 class TCPServer(BaseServer):

   275 

   276     """Base class for various socket-based server classes.

   277 

   278     Defaults to synchronous IP stream (i.e., TCP).

   279 

   280     Methods for the caller:

   281 

   282     - __init__(server_address, RequestHandlerClass)

   283     - serve_forever()

   284     - handle_request()  # if you don't use serve_forever()

   285     - fileno() -> int   # for select()

   286 

   287     Methods that may be overridden:

   288 

   289     - server_bind()

   290     - server_activate()

   291     - get_request() -> request, client_address

   292     - verify_request(request, client_address)

   293     - process_request(request, client_address)

   294     - close_request(request)

   295     - handle_error()

   296 

   297     Methods for derived classes:

   298 

   299     - finish_request(request, client_address)

   300 

   301     Class variables that may be overridden by derived classes or

   302     instances:

   303 

   304     - address_family

   305     - socket_type

   306     - request_queue_size (only for stream sockets)

   307     - allow_reuse_address

   308 

   309     Instance variables:

   310 

   311     - server_address

   312     - RequestHandlerClass

   313     - socket

   314 

   315     """

   316 

   317     address_family = socket.AF_INET

   318 

   319     socket_type = socket.SOCK_STREAM

   320 

   321     request_queue_size = 5

   322 

   323     allow_reuse_address = False

   324 

   325     def __init__(self, server_address, RequestHandlerClass):

   326         """Constructor.  May be extended, do not override."""

   327         BaseServer.__init__(self, server_address, RequestHandlerClass)

   328         self.socket = socket.socket(self.address_family,

   329                                     self.socket_type)

   330         self.server_bind()

   331         self.server_activate()

   332 

   333     def server_bind(self):

   334         """Called by constructor to bind the socket.

   335 

   336         May be overridden.

   337 

   338         """

   339         if self.allow_reuse_address:

   340             self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

   341         self.socket.bind(self.server_address)

   342         self.server_address = self.socket.getsockname()

   343 

   344     def server_activate(self):

   345         """Called by constructor to activate the server.

   346 

   347         May be overridden.

   348 

   349         """

   350         self.socket.listen(self.request_queue_size)

   351 

   352     def server_close(self):

   353         """Called to clean-up the server.

   354 

   355         May be overridden.

   356 

   357         """

   358         self.socket.close()

   359 

   360     def fileno(self):

   361         """Return socket file number.

   362 

   363         Interface required by select().

   364 

   365         """

   366         return self.socket.fileno()

   367 

   368     def get_request(self):

   369         """Get the request and client address from the socket.

   370 

   371         May be overridden.

   372 

   373         """

   374         return self.socket.accept()

   375 

   376     def close_request(self, request):

   377         """Called to clean up an individual request."""

   378         request.close()

   379 

   380 

   381 class UDPServer(TCPServer):

   382 

   383     """UDP server class."""

   384 

   385     allow_reuse_address = False

   386 

   387     socket_type = socket.SOCK_DGRAM

   388 

   389     max_packet_size = 8192

   390 

   391     def get_request(self):

   392         data, client_addr = self.socket.recvfrom(self.max_packet_size)

   393         return (data, self.socket), client_addr

   394 

   395     def server_activate(self):

   396         # No need to call listen() for UDP.

   397         pass

   398 

   399     def close_request(self, request):

   400         # No need to close anything.

   401         pass

   402 

   403 class ForkingMixIn:

   404 

   405     """Mix-in class to handle each request in a new process."""

   406 

   407     active_children = None

   408     max_children = 40

   409 

   410     def collect_children(self):

   411         """Internal routine to wait for died children."""

   412         while self.active_children:

   413             if len(self.active_children) < self.max_children:

   414                 options = os.WNOHANG

   415             else:

   416                 # If the maximum number of children are already

   417                 # running, block while waiting for a child to exit

   418                 options = 0

   419             try:

   420                 pid, status = os.waitpid(0, options)

   421             except os.error:

   422                 pid = None

   423             if not pid: break

   424             self.active_children.remove(pid)

   425 

   426     def process_request(self, request, client_address):

   427         """Fork a new subprocess to process the request."""

   428         self.collect_children()

   429         pid = os.fork()

   430         if pid:

   431             # Parent process

   432             if self.active_children is None:

   433                 self.active_children = []

   434             self.active_children.append(pid)

   435             self.close_request(request)

   436             return

   437         else:

   438             # Child process.

   439             # This must never return, hence os._exit()!

   440             try:

   441                 self.finish_request(request, client_address)

   442                 os._exit(0)

   443             except:

   444                 try:

   445                     self.handle_error(request, client_address)

   446                 finally:

   447                     os._exit(1)

   448 

   449 

   450 class ThreadingMixIn:

   451     """Mix-in class to handle each request in a new thread."""

   452 

   453     # Decides how threads will act upon termination of the

   454     # main process

   455     daemon_threads = False

   456 

   457     def process_request_thread(self, request, client_address):

   458         """Same as in BaseServer but as a thread.

   459 

   460         In addition, exception handling is done here.

   461 

   462         """

   463         try:

   464             self.finish_request(request, client_address)

   465             self.close_request(request)

   466         except:

   467             self.handle_error(request, client_address)

   468             self.close_request(request)

   469 

   470     def process_request(self, request, client_address):

   471         """Start a new thread to process the request."""

   472         import threading

   473         t = threading.Thread(target = self.process_request_thread,

   474                              args = (request, client_address))

   475         if self.daemon_threads:

   476             t.setDaemon (1)

   477         t.start()

   478 

   479 

   480 class ForkingUDPServer(ForkingMixIn, UDPServer): pass

   481 class ForkingTCPServer(ForkingMixIn, TCPServer): pass

   482 

   483 class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass

   484 class ThreadingTCPServer(ThreadingMixIn, TCPServer): pass

   485 

   486 if hasattr(socket, 'AF_UNIX'):

   487 

   488     class UnixStreamServer(TCPServer):

   489         address_family = socket.AF_UNIX

   490 

   491     class UnixDatagramServer(UDPServer):

   492         address_family = socket.AF_UNIX

   493 

   494     class ThreadingUnixStreamServer(ThreadingMixIn, UnixStreamServer): pass

   495 

   496     class ThreadingUnixDatagramServer(ThreadingMixIn, UnixDatagramServer): pass

   497 

   498 class BaseRequestHandler:

   499 

   500     """Base class for request handler classes.

   501 

   502     This class is instantiated for each request to be handled.  The

   503     constructor sets the instance variables request, client_address

   504     and server, and then calls the handle() method.  To implement a

   505     specific service, all you need to do is to derive a class which

   506     defines a handle() method.

   507 

   508     The handle() method can find the request as self.request, the

   509     client address as self.client_address, and the server (in case it

   510     needs access to per-server information) as self.server.  Since a

   511     separate instance is created for each request, the handle() method

   512     can define arbitrary other instance variariables.

   513 

   514     """

   515 

   516     def __init__(self, request, client_address, server):

   517         self.request = request

   518         self.client_address = client_address

   519         self.server = server

   520         try:

   521             self.setup()

   522             self.handle()

   523             self.finish()

   524         finally:

   525             sys.exc_traceback = None    # Help garbage collection

   526 

   527     def setup(self):

   528         pass

   529 

   530     def handle(self):

   531         pass

   532 

   533     def finish(self):

   534         pass

   535 

   536 

   537 # The following two classes make it possible to use the same service

   538 # class for stream or datagram servers.

   539 # Each class sets up these instance variables:

   540 # - rfile: a file object from which receives the request is read

   541 # - wfile: a file object to which the reply is written

   542 # When the handle() method returns, wfile is flushed properly

   543 

   544 

   545 class StreamRequestHandler(BaseRequestHandler):

   546 

   547     """Define self.rfile and self.wfile for stream sockets."""

   548 

   549     # Default buffer sizes for rfile, wfile.

   550     # We default rfile to buffered because otherwise it could be

   551     # really slow for large data (a getc() call per byte); we make

   552     # wfile unbuffered because (a) often after a write() we want to

   553     # read and we need to flush the line; (b) big writes to unbuffered

   554     # files are typically optimized by stdio even when big reads

   555     # aren't.

   556     rbufsize = -1

   557     wbufsize = 0

   558 

   559     def setup(self):

   560         self.connection = self.request

   561         self.rfile = self.connection.makefile('rb', self.rbufsize)

   562         self.wfile = self.connection.makefile('wb', self.wbufsize)

   563 

   564     def finish(self):

   565         if not self.wfile.closed:

   566             self.wfile.flush()

   567         self.wfile.close()

   568         self.rfile.close()

   569 

   570 

   571 class DatagramRequestHandler(BaseRequestHandler):

   572 

   573     # XXX Regrettably, I cannot get this working on Linux;

   574     # s.recvfrom() doesn't return a meaningful client address.

   575 

   576     """Define self.rfile and self.wfile for datagram sockets."""

   577 

   578     def setup(self):

   579         try:

   580             from cStringIO import StringIO

   581         except ImportError:

   582             from StringIO import StringIO

   583         self.packet, self.socket = self.request

   584         self.rfile = StringIO(self.packet)

   585         self.wfile = StringIO()

   586 

   587     def finish(self):

   588         self.socket.sendto(self.wfile.getvalue(), self.client_address)