Author Topic: Developments in Off  (Read 11493 times)

0 Members and 1 Guest are viewing this topic.

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Developments in Off
« on: January 01, 2008, 06:38:28 PM »
As requested by Hummel, this topic will contain current developments in the off project.
Get the news while it's hot :)


« Last Edit: January 02, 2008, 04:17:27 PM by rb2k »

Offline hummel

  • Elite
  • *****
  • Posts: 429
  • Karma: +12/-0
    • View Profile
Re: Developments in Off
« Reply #1 on: January 01, 2008, 06:40:48 PM »
Thanks, I'm really curious now!

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Developments in Off
« Reply #2 on: January 01, 2008, 06:43:56 PM »
Ok, a little bit of gossip. Sorry that I have to keep it short, but I can't seem to remember that much:

- The main coder on OFF has created a working ?solomon-reed? based "self healing inserts" routine (I can't remember it using something else e.g. Raptor Codes... could be possible though). So basically some added parity that will allow users to grab a file even if some blocks are missing because a peer went offline. It will take some time to receive release status though :)

UPDATE: "he studied reed-solomon and then developed his own off-centric thing"



- A solution to the "where can I get Links to Off Files from" thingy is currently in the making (maybe Off-Meistes cares to elaborate :D)

« Last Edit: January 02, 2008, 04:07:50 PM by rb2k »

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Developments in Off
« Reply #3 on: January 06, 2008, 08:10:08 PM »
Someone has been working :D
Now in CVS, installers should be online after a short test... :D


-----------------------------------------------------------------

0.18.01

Responses to nodelist requests no longer need to make a copy of
the main nodelist.

Fixed bug where the settings for Disable Cache Trimming and
Show Node Speeds would get confused with each other of config read.

Moved the read and write threads out of the OFFCnxnPool Class.

Pool close msgs, sent on shutdown, are now sent to all pool nodes
simultaneously.

Implemented a list+updates method for the download block request list
and removed the unneeded mutex lock from this list since it is now only
accessed from the ACH thread.

Implemented Novice, Expert and Developer GUI modes. These can
be switched from the new Autopilot menu. We will probably end up
moving stuff between these modes over the next few versions as
and when the majority of users express their preferences. Also
most changes can still be made independently of mode from the
specific display options. For now the following options have been
changed:

 Specific store types in the toolbar menu : only shown in Dev mode.

 Local URLs, Map submenu                  : not shown in Novice mode.
 Local URLs, Tag options                  : not shown in Novice mode.
 Local URLs, Repair options               : not shown in Novice mode.
 Local URLs, Keys submenu                 : not shown in Novice mode.

 Known Nodes, Authorise/Score options     : not shown in Novice mode.
 Known Nodes, Ping, Ask Nodelist          : not shown in Novice mode.
 Known Nodes, Change Port                 : not shown in Novice mode.

 Advanced Security tab                    : only shown in Dev mode.

 File Menu, specific store types          : only shown in Dev mode.
 File Menu, Check Cache                   : not shown in Novice mode.
 File Menu, Import Blocks                 : not shown in Novice mode.
 File Menu, Configure Cache Locations     : only shown in Dev mode.
 File Menu, View Relayed Searches         : not shown in Novice mode.
 File Menu, View Local Blocks             : only shown in Dev mode.
 File Menu, Import/Export Nodelist        : not shown in Novice mode.

 View Menu, tooltip options               : not shown in Novice mode.
 View Menu, Import/Export Colour Scheme   : not shown in Novice mode.
 View Menu, Toggle ListCtrl Lines         : not shown in Novice mode.
 View Menu, histogram print options       : only shown in Dev mode.

 Panel Menu : panel lock options          : not shown in Novice mode.

NOTE: the default setting is Novice Mode, so if your favourite obscure
option is missing on upgrade please try changing the GUI mode. :)
Further suggestions of which option should be shown in which mode are
welcome.

Added an option to minimise timed gui updates, under the AutoPilot
menu. Enabling this option will:
   i) turn off display of node speeds.
  ii) turn off auto sorting of lists (note: re-enabling will not
turn auto sorting back on - because it is still such a frigging pain)
 iii) turn off display of node current and total uptimes.
  iv) turn off ages and ETAs of downloads
   v) turn off URL ages
  iv) turn off the display of used upload and download slots
Other updates will probably be included in this option later. The purpose
of this option is for nodes reported to use too much CPU while idling
on some platforms, if this is the case try enabling this option and
check if CPU use is reduced.

Fixed memory leak when downloads fail to load due to an error in the
save file.

Implemented Reed-Solomon codes for creating redundancy blocks. The order
of the file blocks when the RS calculation is performed is as follows:
blocks are divided into 256 lists and are added in the order in which
they appear in the insert to the head of the list corresponding to the
first byte in the block name. If a block is already in the list it is
not added. The lists are then concatenated sequentially 00-ff to get
the list of blocks used in the RS calculation. This somewhat complex
procedure is motivated by the fact that we cannot really allow repeated
blocks in the RS system because when such blocks go missing they will appear
to be more than one erasure and the system may appear to be insoluble
when in fact it is not. The RS system should thus contain only the
unique blocks in the insert.

Placed a ceiling of 500 on the number of Reed-Solomon blocks no matter
the size of the insert, usually this number will be the number of
128k sections of the stored file divided by 5.

Reed-Solomon blocks are inserted into the cache just as the file
datablocks are. The tuples of these inserts are written into the
descriptor after the hashmap tuple - which may involve additional
daisy-chaining if the data tuples finish near the end of a descriptor block.

The number of Reed-Solomon blocks in an insert is stored in the field
before the version in the URL of that insert. The reason for this positioning
is so that previous client versions will not notice the extra field
and be emotionally distressed by it.

Reed-Solomon encoding is only done for inserts with 5 file tuples or more
(files >512k) to reduce the chance of deterministic blocks resulting from
the RS block insertions. Larger tuples may be used for smaller files
later, but for now RS is not done for such files.

Listing inserts in Local URLs, now shows the RS blocks.

Distributing a URL now also distributes the RS blocks.

Any missing Reed-Solomon blocks can now be recovered given a completed
download. (Actually recovering file blocks given the requisite number
of RS blocks is not coded yet.)

NOTE: A number of unsolved problems remain in the Reede-Solomon implementation
so it will not be available for a few versions. Some of the above entries
may become outdated as development continues.

Fixed double mutex datarace when pool cnxns are removed, this could cause
deadlocks and possibly crashes.

Fixed crasher when large inserts were listed. Seemed to be a problem
with posting too many gui events to quickly (as each log message posts
an event) and the fix was to add log msgs as normal but only post an
event to display them on the last block of each tuple. This fixed the
crash locally, but some testing in the wild is recommended.

Fixed overrun crasher for mime types longer than 40 bytes, mime types
longer than 128 bytes will now be truncated. (This crasher was discovered
when inserting Open Office files - thanks go to the planetpeer user who
reported it.)

Added a few more tool tips - thanks go to the kindly contributor who
wrote them

Fixed GTK critical on targetted store, wxW did not seem to like the file
selector dialog using the target window as a parent.

Fixed memory leaks of store file paths and arg structs on store.

Fixed memory leak on import blocks.

« Last Edit: January 06, 2008, 08:32:09 PM by rb2k »

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Developments in Off
« Reply #4 on: January 12, 2008, 01:23:58 PM »
There are some ruby files available that do block operations
(read the cache in a Database, enter URLs, retrive the content to a file or to mplayer, ...)
It's just a bit of toying arround though. Maybe more stuff will follow, maybe not :)


cvs -z3 -d:pserver:anonymous@offsystem.cvs.sourceforge.net:/cvsroot/offsystem co -P BlocksDB

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Developments in Off
« Reply #5 on: January 12, 2008, 01:26:03 PM »
Some GUI changes seem to result in low CPU usage :)

I quote:
Quote
holy crap, with these gui changes, a release build of OFF is averaging 4-5% CPU.
occasional peaks to 10%
Very occasional peaks to 20.
spends most of its time arouns 1-3%
1200MHz CPU
checking minimal gui updates chops that in half at least.
god it's even spending some time at zero percent now!

Offline ankern

  • Elite
  • *****
  • Posts: 343
  • Karma: +4/-0
    • View Profile
Re: Developments in Off
« Reply #6 on: January 13, 2008, 12:24:43 PM »
One minor thing :
You may want to add Planet Peer to the DontReadMe, like you've already done on the website.

Offline ankern

  • Elite
  • *****
  • Posts: 343
  • Karma: +4/-0
    • View Profile
Re: Developments in Off
« Reply #7 on: January 13, 2008, 05:49:15 PM »
0.18.02 really performs well (might also be due to the 50+ online nodes that are a regular sight now).
But when I first started it, I immediately got "some fucker" messages en masse,
and stuff like

Quote
OFFFile Open(): failed to open file: block file ( Block::Load())
xx:36:31:            path: x:\OFFSystem\blockcache/94/94a68d4147403a0fa44e456bf43de0811cc945b1.ofb
xx:36:31:           flags: rb
xx:36:31:    strlen(path): 71
xx:36:31:     error no:
xx:36:31:        Bad block size 0 (131072) name: ...................................43de0811cc945b1
xx:36:31:         Moving corrupt block: ./blockcache/94/..................................cc945b1.ofb


OFFFile Open(): failed to open file: block file ( Block::Load())
  path: ..........................ofb
  flags: rb
  strlen(path): 71
  error no:
  Bad block size 0 (131072) name: ..............................................................61f
  Moving corrupt block: ./blockcache/ea/.........................................ofb
  ListDescriptor(): failed to get hash map.
  Aborting.

I've never had this many error messages before.
But still - the speed is good :) .
« Last Edit: January 13, 2008, 05:52:55 PM by ankern »

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Developments in Off
« Reply #8 on: January 15, 2008, 09:54:58 PM »
0.18.03

Fixed bug in block cache statistics where the bucket radius could
be assigned to a value at which there were no blocks.

Reduced CPU use of calculating bucket stats on cache load.

Fixed bug where cmd line builds would shut down immediately
on the very first startup.

PoolCnxn objects now contain an OFFReadBuffer object. This is the same
object that was created for the ping (incoming or outgoing) from which
the pool cnxn was established. This fixes a bug that could cause network
msgs to occasionally be lost or corrupted and significantly reduces
the number of OFFReadBuffer objects created and destroyed during normal
operation.

Fixed bug where dl speeds would occasionally freeze on a finite value
when no blocks were being obtained.

Fixed memory corruption bug (crasher) in OFFSocket read/write buffers.

Fixed memory leak of download blk req updates on shutdown.


Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Developments in Off
« Reply #9 on: January 22, 2008, 07:19:06 PM »
Another kind of "development": there are >66 nodes in the network :)

Offline Siedlerchr

  • Advanced
  • ***
  • Posts: 160
  • Karma: +0/-0
    • View Profile
    • Siedlerchr.jeak.de
Re: Developments in Off
« Reply #10 on: January 23, 2008, 05:39:01 PM »
Another kind of "development": there are >66 nodes in the network :)

Hm. Do you mean online Nodes ?

My OFF says there are about 180 nodes in the list ( most 1/3 is on or sometimes more)

Offline apo-

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Developments in Off
« Reply #11 on: January 23, 2008, 05:41:24 PM »
on the bottom left there is a number xx/xxx (e.g. 64/424 @ my node) the left number tells u the user amount online atm

Offline apo-

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Developments in Off
« Reply #12 on: March 27, 2008, 02:52:52 PM »
« Last Edit: March 27, 2008, 02:54:39 PM by apo- »

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Developments in Off
« Reply #13 on: April 23, 2008, 08:51:55 PM »
Quote from the developer:
Quote
well, I've got the disperse queue display correct, and the disperse resume working.
so I'm going to call it a night.


:D

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Developments in Off
« Reply #14 on: May 01, 2008, 07:51:53 PM »

Quote
[22:47] da1l6: rb2k, i just checked in new BlocksDB (aka "the ruby port", even though its not a port) source, if you are interrested... testers are welcome :D
--> cvs -z3 -d:pserver:anonymous@offsystem.cvs.sourceforge.net:/cvsroot/offsystem co -P BlocksDB

At the moment doesn't do any network stuff, but it can watch the BlockCache / receive files from BlockCache / ...