Justin Lloyd's Thinkering

Somebody needs to think about this stuff...

Archives for 1998

How much of you is you?

by

What part of you is you?

If you could transfer your thoughts, memories and other logical faculties to a virtual world or virtual reality or a robot body, would you do it?

Why not?

What’s the difference?

How much of you must remain for you to identify you are still you?

Studying – Painting the sea in oils

by

This month I am studying “Painting the sea in oils”

Oil painting.

Because we all want to be Bob Ross — or Nancy if you are British and grew up in the 80s.

This is four separate half-day, in-person workshops.

Update: Oil painting is freaking messy.

Update: I think I am done with oil painting. I might return to it in the future as a hobby but it’s not something that interests me right now.

Log update: 20 hours spent in class, and another 7 hours for a total 27 hours of study and practice.

Dangerous acquiescence

by

I have been told I was dangerous because I didn’t ask permission to do something.

“Did it succeed?”
Yes.

“Did it help the bottom line?”
Yes.

“Was it in line with company goals?”
Yes.

“Did it move our project forward?”
Yes.

“Is everybody happy?”
Yes.

“Then what’s the problem?”

“You didn’t ask permission. Initiative is good, but you should ask permission first.”

“I don’t think that word means what you think it means.”

“I could easily add trouble-maker to your HR notes.”

“I think the word you are looking for is irreverent.” I looked over her shoulder at her notes. “That’s two Rs.”

You can never have too much bread

by

So my left wrist hurts, quite severely, where the pain is debilitating whenever I twist it in a particular way.

My health nut friend says “That’s because you eat too much bread. The gluten causes your joints to hurt.”

I reply “No, my wrist joint hurts because years ago I snapped my wrist and didn’t realise until weeks later, when the pain got bad enough, and I went to the doctor, and they did X-Rays, that I found out I snapped my wrist, but by then it was too late to do anything.”

My health nut friend says “You wouldn’t have snapped your wrist if you didn’t eat so much gluten.”

I reply “I don’t think eating gluten had anything to do with the fact that my wrist saved me from snapping my neck when the b-lay line gave out leaving me to slide down the cliff face I was climbing.”

So my health nut friend says “You would have been strong enough to hold on if you had a proper diet that didn’t include so much gluten.”

And they were dead fucking serious.

Some people are so wrapped up in their dogma and beliefs that they no longer sweat when they exercise.

They just exude bullshit.

Change is not a competitive sport

by

You don’t win prizes when you make things that matter.

There is no Pulitzer Prize for writing that makes a difference.

There is no award for art that changes the world.

There is no competition that you can win for creating a charity that improves the lives of millions.

Winning a prize for something just shows that whatever it was you did was popular with the “right” people rather than something that made a difference.

Taking liberties

by

The problem with the human race is that we eventually take everything for granted.

And we have all lived on Earth for so long we take our planet for granted.

Studying – Fundamentals of oil painting

by

This month I am studying “Fundamentals of oil painting”

I enjoyed the oil painting class last month enough I’d like to try doing some more. Not sure I am going to make a hobby out of it though.

A series of four, one day (six hours a day if you don’t count coffee and lunch) workshops.

Update: 24 hours of class time, 3 hours of extra practice.

Sticks & Stones

by

Close to three years in America and I have come to realise something.

The Right hates you because you are physically different.

The Left hates you because you think different.

And then both resort to name clalling as they try to stigmatize whatever it is they don’t like about you.

Emotional equipment

by

Applause seems to be an American affectation that is used everywhere and especially when the audience doesn’t know how to react.

Perhaps it is a reaction, in the same way that shock or laughter is, because the audience is not emotionally equipped to deal with the situation?

Has anybody ever studied this?

Putting children to work

by

Taking your kids to work is a great way to combine the two most annoying things in somebody else’s life.

Promoted into harm’s way

by

Managing expectations are important and if we have some idea of what to expect, we can manage it.

I have always felt that people should be promoted rapidly so that we can get some idea of the scale of fuck up they can achieve.

Paper – Dynamic Non-Bayesian Decision Making

by

Today I read a paper titled “Dynamic Non-Bayesian Decision Making”

The abstract is:
The model of a non-Bayesian agent who faces a repeated game with incomplete information against Nature is an appropriate tool for modeling general agent-environment interactions.

In such a model the environment state (controlled by Nature) may change arbitrarily, and the feedback/reward function is initially unknown.

The agent is not Bayesian, that is he does not form a prior probability neither on the state selection strategy of Nature, nor on his reward function.

A policy for the agent is a function which assigns an action to every history of observations and actions.

Two basic feedback structures are considered.

In one of them — the perfect monitoring case — the agent is able to observe the previous environment state as part of his feedback, while in the other — the imperfect monitoring case — all that is available to the agent is the reward obtained.

Both of these settings refer to partially observable processes, where the current environment state is unknown.

Our main result refers to the competitive ratio criterion in the perfect monitoring case.

We prove the existence of an efficient stochastic policy that ensures that the competitive ratio is obtained at almost all stages with an arbitrarily high probability, where efficiency is measured in terms of rate of convergence.

It is further shown that such an optimal policy does not exist in the imperfect monitoring case.

Moreover, it is proved that in the perfect monitoring case there does not exist a deterministic policy that satisfies our long run optimality criterion.

In addition, we discuss the maxmin criterion and prove that a deterministic efficient optimal strategy does exist in the imperfect monitoring case under this criterion.

Finally we show that our approach to long-run optimality can be viewed as qualitative, which distinguishes it from previous work in this area.

Multi-Pass Trader Client

by

Trader Client Description

The Trader Client works on several layers. At the top layer is the User Interface, this is a separate thread that handles everything that the Trader wants to do such as downloading files, making Offers, etc. At the lowest level is the Command layer, handling all input from other Traders, the WhereHouse Server, etc. Anything below this level is considered network layer, and beyond the reach of the Trader Client.

Making a Request

When a Trader clicks “Download Request” the User Interface creates a Download object and sets it running. This Download object inserts Download Data Block commands in to a Download Stream, sends Commands to the WhereHouse Server to request Data Blocks. This Download Stream is intelligently managed by the Trader Client so that it does not flood the WhereHouse Server with requests. The Download Stream has a thread that monitors the number of active requests at the Server and the amount of available bandwidth, as bandwidth becomes available so the Download Stream will send instructions to the Command layer to retrieve data.

Downloading an Offer requires that the Trader Client look in its database of Offers, retrieve the number of blocks that make up that Offer, and place an appropriate number of Request Data Block commands in to the Download Stream. An entry is added to the User Interface to signify that the Trade has been queued, this Trade may be removed from the queue at any time. To do this the Trader Client must clean up the Download Stream, close any active Transfers for that Trade and inform the WhereHouse Server to purge all Request Data Block commands from it’s own queue. The Trader Client expects an “OK” response from this purge command.

When the WhereHouse Server responds to a Request Data Block command it will send the Trader Client a Request Data Block ID, an Trader IP Address, a Cache Block ID and an MD5 checksum. To retrieve the Data Block the Trader Client must connect to the supplied IP address, send a Request Cache Block command along with the Request Data Block ID and the Cache Block ID. It will receive a Response Code of one of the following:
a) OK – the Response Code also contains a single Data Block (4KB of data) as a parameter
b) Request Data Block ID Invalid/Expired – the request is invalid or expired
c) Cache Block ID Invalid – a Cache Block was asked for that does not exist
d) Trader IP Invalid – the incoming trader’s IP address does not match the expected IP address

Once the transfer of data is complete the Trader Client should disconnect from the other Trader.

Once the Data Block is transferred the Trader Client should disconnect from the other Trader and the connection closed. The Trader Client should calculate a 128-bit MD5 checksum for the 4KB of data to ensure it’s integrity. This calculated checksum is compared against that supplied by the WhereHouse Server when the Request Data Block command was responded too. If they match then the data is good and can be placed on the hard drive for later shuffling. If the checksum is bad then the Data Block must be thrown away and a Request Data Block command for that Data Block placed in to the Download Stream again.

The Trader Client at the end of the MD5 calculation should send a Complete Data Block command to the WhereHouse Server that indicates whether the MD5 matches. If the Trader Client failed to receive all of the data, or if the connection was prematurely terminated, or if the Trader’s request for a Data Block was refused for any reason, or was unable to connect, the Trader Client should inform the WhereHouse Server with appropriate parameters. Three of the parameters that are sent to the WhereHouse Server with this Complete Data Block command are the Request Data Block ID, the Trader IP Address, and the Cache Block ID that the Trader Client originally received. The Server will use these items to monitor the network. Another parameter is how long the transfer took from when the Request Cache Block command was sent to the other Trader to when the connection was closed, in milliseconds. This is used by the Server to handle network statistics and Fair Trading parameters for each Trader.

The reason to use this mechanism is to prevent Traders just making a single request to the WhereHouse Server for a data block and then constantly connecting to another Trader to retrieve all Data Blocks from that location.

Request Data Block command (to server)
Request Cache Block command (to another Trader)
Request Data Block ID
Trader IP Address
Cache Block ID
Data Block (item)
Cache Block (item)

MultiPass Protocol

by

A brief description of Multi-Pass and how it works

There is general inefficiency on the Internet for trading files so I’m proposing a new standard to replace the impromtu warez ftp sites, the web pages full of links that never work, and bots on IRC that have a minimum speed requirement or are always busy serving up everybody else’s requests but never yours, and they can never resume their transfer on a 50MB download.

Here’s the low down. I’ll describe the client side first, and then the server side.

“Trader” Client Side
A client is known as a Trader. Trader’s have a client side program that connects to a central WhereHouse Server, similar to IRC, the WhereHouse Server handles a bulletin board system of messages and location data. The bulletin board allows people to post requests, answer requests, and have general threaded conversations, but it is not intended as a chat network. When the Trader Client first connects it registers the Trader’s name. Other info is also sent to the WhereHouse Server to let it know what kind of connection the client is using, and the resources available for the WhereHouse Server to use.

The client receives a list of Trades from the WhereHouse Server that specify what other people are making available.

Trades & Offers
A Trader can tell the WhereHouse Server that they have Offers available, a Trader may have more than one Offer available. An Offer may be made up of multiple files totalling many hundreds of megabytes. The Trader client automatically breaks each offer in to a number of fixed size blocks, calculates a 128-bit MD5 for each block and sends this list of MD5’s, along with a brief description of the Offer, a small bitmap for any group affiliation and a file_id.diz from the Offer (if one is available) to the WhereHouse Server. The WhereHouse Server then distributes this information to all other Trader’s currently connected.

Making an Offer

To be filled in

Requesting a Trade
A Trader can retrieve an Offer and sends a request to the WhereHouse Server asking for a block from the request. A Trader’s request may only contain a request for a single block, but multiple requests can be made immediately after each other. Though it is advised that the Trader client does not send more than 20 or so simultaneous requests to prevent burdening the WhereHouse Server. As a request is responded to another should be sent to the WhereHouse Server immediately. A request may not be responded to immediately. It is placed in to a list for that Offer, and when an available slot opens the WhereHouse Server informs the Trader client that it may proceed to download. When a Request Acknowledgement is received it contains an MD5 crc for the block requested, an IP address that the block may be retrieved from, and a Request Location ID.

The Trader client, upon receipt of a Request Location ID should open up a TCP/IP connection to the Request Location IP, wait for a “Please send Request Location ID” command from that IP, and then send it’s Request Location ID. This Request Location ID is good for the transfer of a single block of data, usually about 4KB. Once this block is retrieved the Trader client should close the TCP/IP connection, calculate the MD5 crc for the retrieved block, and inform the WhereHouse Server if it was the Request MD5 matched the calculated CRC, or whether it failed. If the CRC check failed the Trader client should re-issue the request for that block again, otherwise if should issue a request for the next block, if any more are remaining to be retrieved. Once an Offer has been completely retrieved it must be “shuffled” in to it’s correct order as requested blocks of data will be received out of order. This shuffling may take place during transfer of the entire Offer, but the data must be shuffled in to it’s correct order.

When a requested block of data is retrieved, it is also placed into a cache, this cache is user definable in size, but the more the Trader caches, the more generous the WhereHouse Server will be in answering requests for data.

Once a block is cached, the Trader client must inform the WhereHouse Server that the block is being cached.

Visitors & Visitor Centre
A Trader client can receive a command of “Incoming Cache Request” from the WhereHouse Server, this command will contain an IP address, a Request Location ID, and a Cache Block ID, this cache request will be from another Trader, the Trader should listen for an incoming connection, when it connects it should send a “Please send Request Location ID” command to the Trader. When the Trader responds, it will send a Request Location, and a Cache Block ID. The IP of theTrader, the Request Location ID, and the Cache Block ID should all match those that the WhereHouse Server sent. If they do not then the connection is refused, and the WhereHouse Server is informed.

If a Trader client ever receives a “Cache Request Denied” from another Trader it should inform the WhereHouse Server that it’s request was denied. If a Trader is ever unable to connect to another Trader for a Cache Request it should inform the WhereHouse Server that it failed to connect. If a Trader’s connection times out while downloading from another Trader it should inform the WhereHouse Server.

A Trader can also withdraw their Offers from the WhereHouse Server, these Offers then become unavailable, and all cached blocks for that Offer being held by other Traders also become invalid.

When a Trader connects to the WhereHouse Server it informs the WhereHouse Server how much bandwidth it is offering, and how much cache space it is offering. This influences how quickly the WhereHouse Server will respond to Requests. A Trader may not offer more bandwidth than they are actually capable of handling. The WhereHouse Server keeps statistics on each Trader and it is only after a Trader has been connected and handled several Requests for Cached Blocks of data that the WhereHouse Server will “trust” the Trader client enough to give it a Request status equal to it’s “announced” bandwidth.

The cache is broken up in to 64KB chunks, each 64KB represents a single Data block, all transfers of an Offer takes place in 64KB chunks. There may be several 64KB transfers of a single Offer going on at once for a single Trader. A Trader may also have several Offers downloading at once. Each Block Request runs in its own thread, a thread handles each Offer, and a central thread handles all Offer transfers and coordinates the shuffling of data.

Every Trader that connects to request a cached data block receives a single thread of their own, and a central thread for that handles all incoming connections and disconnections.

A Trader can offer “pre-cache” ability. This indicates that they are willing to request and cache data blocks for parts of Offers that they are not personally requesting. This allows the WhereHouse Server to intelligently manage cached blocks and relieve the burden on both very popular Offers and also not so popular Offers. A Trader client’s cache is managed on the “least requested, least accessed” principal, when the cache becomes full and room must be made for newer, or more popular blocks, the list is sorted in to the least accessed blocks first, and then sub-sorted in to oldest first, then the blocks at the top of the list are removed. Cached blocks expire after a period of time that the WhereHouse Server sets, the WhereHouse Server will inform the Trader client when to expire blocks, either because the Offer was withdrawn, the Trader making the Offer has disconnected, or the block is considered invalid because it is “old” and it may not be actually valid anymore.

When a Trader client sends a command to the WhereHouse Server it generates a unique 128 bit Command ID, the WhereHouse Server will use this Command ID when replying, so that the Trader client knows what the reply is referring to. As the commands and replies happen asynchronously there may be multiple outstanding commands that the Trader has sent to the WhereHouse Server that are awaiting a response, and similarly the WhereHouse Server may have sent multiple commands to the client which may be waiting for attention, or further input. A command expires after 120 seconds, if either the Trader client or the WhereHouse Server fails to respond to a command in this time, it expires and the command goes unanswered. Depending on the particular command will depend on the action taken. There is one exception to this, a Request Data Block command to the WhereHouse Server from a Trader client. This command has a time-out that is set by the WhereHouse Server but will usually be anything up to half-an-hour, and usually never less than ten minutes. For a popular Offer there may not be enough Cached Data available on all the Traders for a response to come back immediately.

A Trader Client will not acknowledge a Expect Connection command from the WhereHouse Server until it is sure it can handle the transfer. If current outbound throughput for all transfers to other Traders is greater than 90% of maximum bandwidth then the command goes unanswered. Should the command sit idle for more than 60 seconds then the Trader Client informs the WhereHouse Server that the command is “refused due to insufficient bandwidth”.

When a Trader disconnects from the WhereHouse Server and then reconnects at a later time, it’s cache is invalid, it needs to validate the cache with the WhereHouse Server and purge any blocks that are invalid. The Trader sends commands to the WhereHouse Server for each Cache Block it has, along with it’s unique Cache Block ID, the WhereHouse Server will return a valid/invalid response and the Cache Block can be either removed or kept. It is assumed that if the Cache Block ID is valid according to the WhereHouse, that the Trader will be keeping the Cache Block.

WhereHouse Server
The WhereHouse Server listens for incoming connections from Trader Clients, when a connection arrives it hands off to a separate thread that will handle that Trader exclusively (this may not be that efficient and should be looked in to). The WhereHouse Server sends a command to the Trader Client requesting that they send their preferences and resource availability. The WhereHouse Server maintains a list of Traders that are connected, Offers that those Traders are making available, a list of Cache Blocks pertaining to each Offer, where those Cache Blocks are located, their MD5 CRC numbers, and statistics on each Offer, including a brief description, a file_id.diz, and a bitmap for a group affiliation logo. A Trader can make an Offer and the WhereHouse Server will receive a list of CRC numbers for each block that comprises that Offer, including some other information.

When a Trader makes a Request Data Block, the WhereHouse Server will place this in a list of Pending Requests, sorted on a priority base that results from the resources that the Trader is offering. The more resources a Trader offers, the higher the priority that they are given, and therefore, close to the front of the queue when a request is made. A Trader offering a full T3 and 10GB of cache will be right at the front of a queue compared to someone offering a 28.8Kb modem with 1MB of cache.

If a Trader has announced that they will pre-cache data they are sent commands by the WhereHouse Server to request particular blocks from other Traders.

Should a Trader disconnect from the WhereHouse Server the list of blocks that they were caching is invalidated, and removed from the WhereHouse Server’s list of known blocks. Any Offers that they made available is also removed, all connected Trader’s are informed that these Offers should be removed from their lists, and the Server will also send commands to any Traders that were caching those Data Blocks that related to the Offers to invalidate them and remove them from the cache.

The WhereHouse Server keeps a list of Pending Requests, it also keeps a list of Available Resources, Resources are basically an amount of bandwidth that each Trader is making available. When a Request is in the queue that matches a cached block, and that cached block has resource available for transfer, ie, there is a Trader that can handle the connection, the WhereHouse Server sends an Expect Connectionb command to the Trader. This command will have an attached IP address of the Trader who will be connecting, a Request Data Block ID, and a Cache Block ID. When the Trader responds to this command, the Server then sends a command to the Trader making the Request with the IP address of where the data block is cached, the Request Data Block ID, and a Cache Block ID, and an MD5 CRC for that data block. The response to this command will be received once the data block has been transferred by the Trader Client. The response may be “OK” or “FAILED” and a reason for failure, such as incorrect CRC match. The Trader Client will also send the WhereHouse Server the statistics for the transfer, ie, the length of time it took to transfer the actual data. This way the Server can keep track of bandwidth usage and also whether a Trader Client that announces their bandwidth capability actually has what they say they have.

Once several transfers from a particular Trader have taken place, the WhereHouse Server will upgrade their status, or downgrade it, depending on network performance. If network performance falls below announced levels for that Trader then their status is downgraded. Announced Performance is equivalent to 100% efficiency from that Trader and they should be trusted, anything below will mean that the trader loses status in their list to handle requests. Also the server will need this information for knowing when that Trader client can handle another request.

Should a Trader Client disconnect the WhereHouse Server needs to remove from it’s cache data block list any blocks that were held by that Trader Client.

Visitor Centre
Trader
MD5
WhereHouse Server
Trade
Offer
Request
Command
Acknowledge

Pin It on Pinterest