THIS FILE IS NO LONGER USED THIS FILE IS NO LONGER USED THIS FILE IS NO LONGER USED We now use the Bugzilla installation at Lysator to handle the bugs and feature requests of lyskomd. See the following: http://bugzilla.lysator.liu.se/ The Bugzilla URL:s below point out the relevant entries. The ideas were copied to Bugzilla during October 2001. THIS FILE IS NO LONGER USED THIS FILE IS NO LONGER USED THIS FILE IS NO LONGER USED * Open Secret Conferences (mirar) Bugzilla: It would be nice to be able to set permissions on a conference so that it is considered open to all with write permissions and closed or secret to all others. * Better Text Garbing (nisse@lysator.liu.se) Bugzilla: Keep texts that have comments newer than N days, where N is a parameter of the conference. The parameter N has been implemented (keep_commented). The garb has to be changed (this is also in the TODO file.) It might be desirable to look even deeper into the comment chain so we keep *chains* that have additions newer than N days. * Chop up texts before sending (byers@lysator.liu.se) Bugzilla: We aren't too good at handling large texts. The problem is that we start off by reading the entire thing into memory from disk, then shove it all onto an ISC queue. For big texts that means we sit and wait for I/O of the text, then we sit and wait while we queue it all up, and while we're doing that, we actually wait for the server to write the garbage to the pipe! It would be nice to be able to spread the work around. I think the trick would be to not read everything from disk at once. We want to read a bit, send it, read some more, and send that. We add a function to ISC that lets us push an ISC packet onto the ISC queue that contains no data, just an instruction to call a callback. That callback reads some more data from the disk and queues that in ISC too. When we want to output a text, we start by enqueueing a callback to read the second bit of the text. Then we read the first bit and enqueue that. When we call isc_flush, isc will start by calling the callback. It will enqueue a callback to read the third bit from disk, then read the second bit from disk and enqueue that. Then we return to ISC and let ISC write the first bit to the socket. The result is that the ISC queue will contain block N of the text, then a callback to get N+2, then block N+1. The reason for doing it this way is that if we can get the callback to do the reads asynchronously, we can have ISC writing a bit of data to the socket while we are reading the next bit from disk. If we can do async reads, when ISC gets to the callback, the queue will contain a callback for block N+1, then block N of the text. The callback will enqueue a callback for block N+2, then initiate an async read for block N+1 and return immediately. The server will write block N to the socket concurrently with the read of block N+1. When the async read returns, its handler (callback, signal handler, whatever) enqueues the data in the ISC queue. This should result in the server spending much less time in IO wait. If we can't do async reads, then we might as well just put the callback for block N+1 after the data for block N. Async reads are available under Solaris, but not in Linux 2.0. Regrettably the interface is probably vendor-dependent. A potential problem is that we need to make *sure* that the text file is not cleaned when a text is deleted. That is not a problem in the current server, but could be in the future. If we have a pending request for text N, and someone deletes it before we can send the whole thing, we have to be able to continue sending it. Changes need to be made to ISC (typed queue elements, call the callback); prot_a_output and mux.c (need to output texts in a special way); plus we have to write the callbacks. This should be totally doable. NEWS: The aio_ interface is not supported by Linux, at least not yet. It is, however, a Posix 4 extension, not just a Solaris thing. On Linux we'd have to use poll or select to do asynchronous IO. Is it possible to use ISC for this as well? note: it is impossible to use select to do asynchronous I/O against files. select always considered data from a file to be immediately availabe. /ceder ceder says: adding support for aio_* to ISC is probably a good idea. It should be possible to do it in a way that is useful even on platforms that don't have aio_*. The idea then is to make sure that we never read more than one block from the file until we have written it to the client. Having a 8 KiB buffer where chunks are copied from file to client, from file to client, is a lot better than reading 1 MiB from file, chopping it up in small blocks, and storing them i an ISC queue. * Sync the database without blocking requests (byers@lysator.liu.se) Bugzilla: Currently the database is synced in three phases. Phase one creates a snapshot of all changed objects by traversing the internal arrays and moving the data pointer to the snapshot pointer in the cache nodes. The second phase writes objects to disk. The third phase frees the snapshot data from memory. The first and last phase block the server completely. Rewrite the sync algorithm like this. Assume that all objects in the database can be ordered, regardless of type. The server can be in one of two states: syncing or not syncing. When the server starts syncing, open the target data file and write the database header to the file. Write kom_info to the database. Set next_to_sync to zero. Periodically, sync a few objects. Examine object next_to_sync. If it has been saved to disk, do nothing. If it has a snapshot, write the snapshot to disk. If it is in memory, write it to disk from memory. If it is on disk, copy it from the old data file. Clear the snapshot. Clear the dirty flag. Clear the saved-to-disk flag. Increase next_to_sync and repeat this step until a fixed number of objects have been saved. When all objects are done, set the state of the server to not syncing, set the current data file to the one just synced. While the server is syncing, access to objects needs to be done carefully. If an object is requested that has not been synced, create a snapshot of it before returning it, or write it to disk immediately. If an object that has been synced but is not in memory is requested, then it must be read from the new data file. Tricky stuff: if something is added while we are syncing, it should not be written to disk. If something is deleted while we are syncing is should be written to disk anyway. The basic idea here is to make sure that the database file contains a copy of the database as it was when the sync was started, no matter what happens with it afterwards. We have like an index; everything before the index has been synced and can be treated as if we are not syncing at all. Everything beyond the frontier may be unsynced, and we need to make sure that it is retained unchanged until it gets written to disk. * Hierarchical Conferences (byers@lysator.liu.se) Bugzilla: LysKOM conferences should be organized hierarchically. Give each conference a parent conference pointer and a children conference list. Internally, prepend the parent ID to each conference name, so that we can have two conferences with the same name contained below different parent conferences. The prepending should not be visible to the client. Permissions in a conference are based on the permissions in the conference closest in child-parent order to the requested conference in which the person is a member (whew!) Some examples: conference A is rd_prot and conference B is not. Persons who are members of A can see anything in A and B. Persons who are members in B can see anything in B, but not in A. To persons who are members of neither, B behaves as if it were rd_prot, since its parent conference is. It has to be possible to do name lookup on a subtree of the conference tree. The small_conf_stat structure should probably mirror the tree structure and contain permission information about conferences. Don't forget that we want to be able to do sequential traversal of all elements in a subtree. This should require surprisingly small changes in the server. Mainly small_conf_stat, name lookup and storage, and permissions checks. * Garbage Collecting Conferences (byers@lysator.liu.se) Bugzilla: Conferences have an expire field. Conferences that have not contained texts for that many days should be removed automatically by the garbage collector or by dbck. * Mark Types (byers@lysator.liu.se) Bugzilla: Marks should be fixed. Marks should have flags and stuff. Texts should know who has marked them. There's a text about this somewhere in LysKOM. * Multibyte character compatibility (byers@lysator.liu.se) Bugzilla: The server should be capable of operating in multi-byte environments. We'll have to be able to specify a coding system and charset for conference and person names. We'll have to make sure the regexp engine can deal with multibyte characters. We'll need a new method of separating the subject line from the text body. The name matching will have to be more flexible. For eight-bit characters we'll need a way of specifying a collate table. For multibyte characters, maybe we'll have to be able to specify a procedure to determine equivalence of two characters. Local variables: mode: outline paragraph-separate: "[ \t ]*$" outline-regexp: "[* ]+" end: