A brief description of Multi-Pass and how it works
There is general inefficiency on the Internet for trading files so I’m proposing a new standard to replace the impromtu warez ftp sites, the web pages full of links that never work, and bots on IRC that have a minimum speed requirement or are always busy serving up everybody else’s requests but never yours, and they can never resume their transfer on a 50MB download.
Here’s the low down. I’ll describe the client side first, and then the server side.
“Trader” Client Side
A client is known as a Trader. Trader’s have a client side program that connects to a central WhereHouse Server, similar to IRC, the WhereHouse Server handles a bulletin board system of messages and location data. The bulletin board allows people to post requests, answer requests, and have general threaded conversations, but it is not intended as a chat network. When the Trader Client first connects it registers the Trader’s name. Other info is also sent to the WhereHouse Server to let it know what kind of connection the client is using, and the resources available for the WhereHouse Server to use.
The client receives a list of Trades from the WhereHouse Server that specify what other people are making available.
Trades & Offers
A Trader can tell the WhereHouse Server that they have Offers available, a Trader may have more than one Offer available. An Offer may be made up of multiple files totalling many hundreds of megabytes. The Trader client automatically breaks each offer in to a number of fixed size blocks, calculates a 128-bit MD5 for each block and sends this list of MD5’s, along with a brief description of the Offer, a small bitmap for any group affiliation and a file_id.diz from the Offer (if one is available) to the WhereHouse Server. The WhereHouse Server then distributes this information to all other Trader’s currently connected.
Making an Offer
To be filled in
Requesting a Trade
A Trader can retrieve an Offer and sends a request to the WhereHouse Server asking for a block from the request. A Trader’s request may only contain a request for a single block, but multiple requests can be made immediately after each other. Though it is advised that the Trader client does not send more than 20 or so simultaneous requests to prevent burdening the WhereHouse Server. As a request is responded to another should be sent to the WhereHouse Server immediately. A request may not be responded to immediately. It is placed in to a list for that Offer, and when an available slot opens the WhereHouse Server informs the Trader client that it may proceed to download. When a Request Acknowledgement is received it contains an MD5 crc for the block requested, an IP address that the block may be retrieved from, and a Request Location ID.
The Trader client, upon receipt of a Request Location ID should open up a TCP/IP connection to the Request Location IP, wait for a “Please send Request Location ID” command from that IP, and then send it’s Request Location ID. This Request Location ID is good for the transfer of a single block of data, usually about 4KB. Once this block is retrieved the Trader client should close the TCP/IP connection, calculate the MD5 crc for the retrieved block, and inform the WhereHouse Server if it was the Request MD5 matched the calculated CRC, or whether it failed. If the CRC check failed the Trader client should re-issue the request for that block again, otherwise if should issue a request for the next block, if any more are remaining to be retrieved. Once an Offer has been completely retrieved it must be “shuffled” in to it’s correct order as requested blocks of data will be received out of order. This shuffling may take place during transfer of the entire Offer, but the data must be shuffled in to it’s correct order.
When a requested block of data is retrieved, it is also placed into a cache, this cache is user definable in size, but the more the Trader caches, the more generous the WhereHouse Server will be in answering requests for data.
Once a block is cached, the Trader client must inform the WhereHouse Server that the block is being cached.
Visitors & Visitor Centre
A Trader client can receive a command of “Incoming Cache Request” from the WhereHouse Server, this command will contain an IP address, a Request Location ID, and a Cache Block ID, this cache request will be from another Trader, the Trader should listen for an incoming connection, when it connects it should send a “Please send Request Location ID” command to the Trader. When the Trader responds, it will send a Request Location, and a Cache Block ID. The IP of theTrader, the Request Location ID, and the Cache Block ID should all match those that the WhereHouse Server sent. If they do not then the connection is refused, and the WhereHouse Server is informed.
If a Trader client ever receives a “Cache Request Denied” from another Trader it should inform the WhereHouse Server that it’s request was denied. If a Trader is ever unable to connect to another Trader for a Cache Request it should inform the WhereHouse Server that it failed to connect. If a Trader’s connection times out while downloading from another Trader it should inform the WhereHouse Server.
A Trader can also withdraw their Offers from the WhereHouse Server, these Offers then become unavailable, and all cached blocks for that Offer being held by other Traders also become invalid.
When a Trader connects to the WhereHouse Server it informs the WhereHouse Server how much bandwidth it is offering, and how much cache space it is offering. This influences how quickly the WhereHouse Server will respond to Requests. A Trader may not offer more bandwidth than they are actually capable of handling. The WhereHouse Server keeps statistics on each Trader and it is only after a Trader has been connected and handled several Requests for Cached Blocks of data that the WhereHouse Server will “trust” the Trader client enough to give it a Request status equal to it’s “announced” bandwidth.
The cache is broken up in to 64KB chunks, each 64KB represents a single Data block, all transfers of an Offer takes place in 64KB chunks. There may be several 64KB transfers of a single Offer going on at once for a single Trader. A Trader may also have several Offers downloading at once. Each Block Request runs in its own thread, a thread handles each Offer, and a central thread handles all Offer transfers and coordinates the shuffling of data.
Every Trader that connects to request a cached data block receives a single thread of their own, and a central thread for that handles all incoming connections and disconnections.
A Trader can offer “pre-cache” ability. This indicates that they are willing to request and cache data blocks for parts of Offers that they are not personally requesting. This allows the WhereHouse Server to intelligently manage cached blocks and relieve the burden on both very popular Offers and also not so popular Offers. A Trader client’s cache is managed on the “least requested, least accessed” principal, when the cache becomes full and room must be made for newer, or more popular blocks, the list is sorted in to the least accessed blocks first, and then sub-sorted in to oldest first, then the blocks at the top of the list are removed. Cached blocks expire after a period of time that the WhereHouse Server sets, the WhereHouse Server will inform the Trader client when to expire blocks, either because the Offer was withdrawn, the Trader making the Offer has disconnected, or the block is considered invalid because it is “old” and it may not be actually valid anymore.
When a Trader client sends a command to the WhereHouse Server it generates a unique 128 bit Command ID, the WhereHouse Server will use this Command ID when replying, so that the Trader client knows what the reply is referring to. As the commands and replies happen asynchronously there may be multiple outstanding commands that the Trader has sent to the WhereHouse Server that are awaiting a response, and similarly the WhereHouse Server may have sent multiple commands to the client which may be waiting for attention, or further input. A command expires after 120 seconds, if either the Trader client or the WhereHouse Server fails to respond to a command in this time, it expires and the command goes unanswered. Depending on the particular command will depend on the action taken. There is one exception to this, a Request Data Block command to the WhereHouse Server from a Trader client. This command has a time-out that is set by the WhereHouse Server but will usually be anything up to half-an-hour, and usually never less than ten minutes. For a popular Offer there may not be enough Cached Data available on all the Traders for a response to come back immediately.
A Trader Client will not acknowledge a Expect Connection command from the WhereHouse Server until it is sure it can handle the transfer. If current outbound throughput for all transfers to other Traders is greater than 90% of maximum bandwidth then the command goes unanswered. Should the command sit idle for more than 60 seconds then the Trader Client informs the WhereHouse Server that the command is “refused due to insufficient bandwidth”.
When a Trader disconnects from the WhereHouse Server and then reconnects at a later time, it’s cache is invalid, it needs to validate the cache with the WhereHouse Server and purge any blocks that are invalid. The Trader sends commands to the WhereHouse Server for each Cache Block it has, along with it’s unique Cache Block ID, the WhereHouse Server will return a valid/invalid response and the Cache Block can be either removed or kept. It is assumed that if the Cache Block ID is valid according to the WhereHouse, that the Trader will be keeping the Cache Block.
WhereHouse Server
The WhereHouse Server listens for incoming connections from Trader Clients, when a connection arrives it hands off to a separate thread that will handle that Trader exclusively (this may not be that efficient and should be looked in to). The WhereHouse Server sends a command to the Trader Client requesting that they send their preferences and resource availability. The WhereHouse Server maintains a list of Traders that are connected, Offers that those Traders are making available, a list of Cache Blocks pertaining to each Offer, where those Cache Blocks are located, their MD5 CRC numbers, and statistics on each Offer, including a brief description, a file_id.diz, and a bitmap for a group affiliation logo. A Trader can make an Offer and the WhereHouse Server will receive a list of CRC numbers for each block that comprises that Offer, including some other information.
When a Trader makes a Request Data Block, the WhereHouse Server will place this in a list of Pending Requests, sorted on a priority base that results from the resources that the Trader is offering. The more resources a Trader offers, the higher the priority that they are given, and therefore, close to the front of the queue when a request is made. A Trader offering a full T3 and 10GB of cache will be right at the front of a queue compared to someone offering a 28.8Kb modem with 1MB of cache.
If a Trader has announced that they will pre-cache data they are sent commands by the WhereHouse Server to request particular blocks from other Traders.
Should a Trader disconnect from the WhereHouse Server the list of blocks that they were caching is invalidated, and removed from the WhereHouse Server’s list of known blocks. Any Offers that they made available is also removed, all connected Trader’s are informed that these Offers should be removed from their lists, and the Server will also send commands to any Traders that were caching those Data Blocks that related to the Offers to invalidate them and remove them from the cache.
The WhereHouse Server keeps a list of Pending Requests, it also keeps a list of Available Resources, Resources are basically an amount of bandwidth that each Trader is making available. When a Request is in the queue that matches a cached block, and that cached block has resource available for transfer, ie, there is a Trader that can handle the connection, the WhereHouse Server sends an Expect Connectionb command to the Trader. This command will have an attached IP address of the Trader who will be connecting, a Request Data Block ID, and a Cache Block ID. When the Trader responds to this command, the Server then sends a command to the Trader making the Request with the IP address of where the data block is cached, the Request Data Block ID, and a Cache Block ID, and an MD5 CRC for that data block. The response to this command will be received once the data block has been transferred by the Trader Client. The response may be “OK” or “FAILED” and a reason for failure, such as incorrect CRC match. The Trader Client will also send the WhereHouse Server the statistics for the transfer, ie, the length of time it took to transfer the actual data. This way the Server can keep track of bandwidth usage and also whether a Trader Client that announces their bandwidth capability actually has what they say they have.
Once several transfers from a particular Trader have taken place, the WhereHouse Server will upgrade their status, or downgrade it, depending on network performance. If network performance falls below announced levels for that Trader then their status is downgraded. Announced Performance is equivalent to 100% efficiency from that Trader and they should be trusted, anything below will mean that the trader loses status in their list to handle requests. Also the server will need this information for knowing when that Trader client can handle another request.
Should a Trader Client disconnect the WhereHouse Server needs to remove from it’s cache data block list any blocks that were held by that Trader Client.
Visitor Centre
Trader
MD5
WhereHouse Server
Trade
Offer
Request
Command
Acknowledge