Seems that someone is developing another p2p client for I2P. I think the following information is usefull for everyone, thus I decided to borrow the whole story from the
Here is the basic documentation for a working proof-of-concept P2P application I developed the past weekend.
Comments are very welcome.
The sources are available at link:
http://tld.digitalcurse.com/clunk.zip if anyone cares. Rough instructions are included in the zip file.
It's C/C++ in the worst possible form. Not for the faint of heart.
Compiles on NetBSD with gcc 3.3.3, don't know about any other platform. Most probably doesn't compile on windows.
CLUNK
=====
0. Contents
-----------
0. Contents
1. Introduction
2. User experience
3. Behind the scenes
4. The protocol
5. The future
6. And beyond
1. Introduction
---------------
Clunk is a peer2peer application for sharing and downloading of files.
clunk is accessible via i2p and exposes a basic html interface (you can also connect via the native protocol if you dare).
The html interface offers the possibility of browsing another user's clunk public directory and downloading the
files he offers. It also offers the possibility of inspecting the files you're currently downloading in the
background.
The files you share to the network and those you downloaded are in different directories. You have full control
over which files are shown to the other users.
2. User experience
------------------
NAME SPACE:
clunk generates a global namespace everyone can access.
Every instance of clunk has its own subspace which mirrors the files you have in your public directory. By
running clunk you are given the address to your public directory. You can access another user's directory
exclusively by knowing its address, and they can access yours only if (and when) they know yours.
All accesses to this global directroy structure are readonly.
There is also a special subdirectory (namely /_/) which allows access to the download manager. With this
subdirectory you can access an html interface for managing your downloads. This directory is private to your
instance of clunk.
[The current implementation is extremely susceptible to trailing slashes. Make sure you put them in place,
expecially when accessing other users' directories]
SHARING:
The files you share are in the pub/ subdirectory.
Every file or directory placed under that directory is shared.
Symbolic links are followed, so you can avoid replicating files needlessly.
Filesystem access to the pub/ directory is the only way to write into your own namespace.
The files you have successfully downloaded are placed in the incoming/ subdirectory. Their content is shared to
the network, but other users won't be able to see the file when they browse your directories.
BROWSING:
The html interface lets you access to the globally available filesystem.
While there is no way of knowing available public directories, you can access the subdirectory and browse it as
long as you can reach (on i2p) the other client.
As long as the other peer is connected and the clunk application is running, the share is available. The
instance dies, the namespace becomes unavailable. Content isn't resilient.
Browsing a user's directory is similar to browsing an FTP site with a web browser: you see directories and
files in the usual representation.
DOWNLOADING:
Once you have put your eyes on a file you want to download, you left click on it in your browser, as you are
probably used to.
The program will inform you of the download progress (by accessing the /_/ directory).
Once the file is downloaded, you'll find it into the incoming/ directory on your filesystem. [perhaps in the
future the file will be served in the html interface too]
SEARCHING:
No way of searching. Perhaps in the future.
Note however that since there is an html interface, a web crawler can be used and its results indexed.
3. Behind the scenes
--------------------
GETTING:
clunk uses a proprietary protocol for peer2peer communication.
Directory browsing and file download requests happen between your instance and the target user's client.
A download request will transfer a digest of the file's content (a list of <block_size> <block_sha1>) ; your
client will then start a background transfer for those blocks.
Those you already have don't get downloaded (saving bandwidth).
For the other blocks, duplicates removed, your client enqueues the download from that remote client.
Your client will also query the network for alternate sources for the block you seek [right now there's a
central server; hopefully soon it will be a distributed network]. Querying is done periodically, depending on
how many sources you have available for each block.
The transfer happens in parallel, block by block. Each peer on the network gets no more than one block request
at a time. Each block is downloaded from at most one peer at a time. [Right now, block size is at most 8k]
PUTTING:
Periodically, your client will inform the network about which blocks he has available for download.
Note that the blocks' published information is only a reference to size and SHA1 hash, with no information to
link them to the file which they belong.
The information about block ownership decays with time. There is no need (and no way) for a client to inform
the network that the block is no more available. [Right now the decay time is 10 minutes.]
As soon as other peers share at least one block being requested by a downloader, the original block provider
will see a reduced load (transfer requests are redirected to the other peers)
4. The protocol
---------------
Below is a quick documentation of the peer2peer communication protocol.
This protocol is probably subject to change (cleanup, rewriting or extensions).
The protocol is datagram-based. The datagram have no upper size limit. Datagram reading and writing is an
atomic operation.
In the followind, I use the following terms:
"a list of" is a series of entries not separated in any way (no blanks either)
"nothing is returned" is a severed connection between the peers [or possibly another less drastic way
of informing the other peer that something went awfully wrong]
"newline" is the '\n' C character, ASCII 13. No \r (ASCII 10) is allowed.
OFFER <sha1><newline><number><newline><sources>
where
<sha1> is a base16 sha1 string [case sensitive, must be uppercase]
<sources> is a list of <pubkey><newline>
<pubkey> is a peer's public key
CHUNK <data256>
where
<data256> is a series of binary values in full 8bits
The sender has sent some data, presumably because of a request. This is the natural response to a FETCH
request.
SERVE <name>
The sender is requesting a description of the entry from the public namespace, file or directory alike. In the
current implementation, it is the element with the same name based in pub/ directory.
Note that subdirectory elements are allowed, so "img/photos/ferret0005.jpg" (without "es) is a valid request.
If the request maps to a file, a HEREF is sent back describing the file. If the request maps to a directory, a
HERED is sent back describing a directory. If it maps to nothing (valid) nothing is returned.
HERED <content>
where
<content> is a list of <file_or_dir><entry><newline>
<file_or_dir> is the character F for a file and D for a directory
<entry> an entry name in the directory
The list describes a directory. It is a response to a SERVE message.
Example:
HERED D.
D..
FHelloWorld.txt
FThesis.doc
HEREF <hashlist>
where
<hashlist> is a list of <size> <sha1><newline>
<size> is the block size in bytes, ASCII base 10
<sha1> is a base16 sha1 string [case sensitive, must be uppercase]
This message describes a file. It is a response to a SERVE message.
Example:
HEREF 8192 D798786808D5A8095AB898A637D8E4563EA98E59
23 A8095AB898A637D8E4563EA98E59D798786808D5
FETCH <sha1>
<sha1> is a base16 sha1 string [case sensitive, must be uppercase]
The sending peer is asking for the block which has sha1 digest <sha1>
IHAVE <sha1>
<sha1> is a base16 sha1 string [case sensitive, must be uppercase]
The sending peer has the block having hash <sha1>. This message is sent to the appropriate peer in the network
[distributed network; right now it's a central server the receiver of this message]
ISEEK <sha1>
<sha1> is a base16 sha1 string [case sensitive, must be uppercase]
The sending peer is looking for sources to the block indicated as parameter. The sending peer expects a OFFER
response (possibly empty).
5. The future
-------------
The current (and only) implementation has the following characteristics:
+ Multisource concurrent transfer
+ Partial downloads are published as soon as you complete a block
+ Unique names (like freenet's SSK, but far longer than freenet's)
+ Keeps downloading even if the server and the other starting client dies
- Fixed block size
- Chunk publishing and source retrieval is server-based
- No way to publish your public key
- Addresses change every time you restart clunk
- Files get rehashed every time you restart clunk
- Pending downloads are lost on clunk shutdown or disconnect
- No search functionality
- No check for filesystem changes while running (so if you remove, replace or add a file you better pray)
- Does not return the downloaded file to the requester's browser session
- Uncompressed communication
- No input validation (BEWARE!)
Future versions will have to change the "-" points.
Possible further extensions include:
+ Shorter names for the public keys (petnames and such).
+ In-clunk symbolic links (relative to the global namespace and not to the local filesystem)
+ Binary messages: plaintext is cute to read but i2p is low-bandwidth. Base16 is 2x, base64 is 4/3x.
6. And beyond
-------------
Thanks for the attention.
Have fun.
I have put a server up. Its address is:
XUtP2w2jaVdGcTKWZhg1mRwNfx9S2wVIgTTgeKfauFu6uhoBL~l0z0btFxBuXOiV4v1Yor8vsjcmfeQXaW74PJ~IZL9HFAD6-QBY3u2iljkNhUpcf9UJ~DvOxhikQWTtNSkxHBFwGtmogyRNli47-gdf4gQpY2AnxkwuUWOUrpARpASBZAEdDPsXFNp27YDFVOc7rnCS0BEv~gGyKMAk6uZPUUDsPKRUbumzDRYiXlfXzqfdT3HrfnOlNbZWxXKdcCRqEt6F7aeVRGF7bpIQzHLhfU~TeSKDcHkjk37NCLyx8WT~GU3R27CAt6nZMnD2ck6~vVa2IKbOibPQ~1XcJHWv3Zir~m7ZU1ueADE2AogZBunEG92o4aRFWW9RnDSPKkYMTpAKreGFoJvimIinmEZCmJ3N2Tavid4L7Tm44S50w1dCpl1IT2zns5MQr9b9nITmRAJHN5ELHlzL6dShbytoQh70rPojWDf7UbVDmFzJF~SEgHtqGX8NTF-pZrQNAAAA
until it dies (which will happen far too soon, I fear).
