Author Topic: Newest OFF Version (here only OFF Version publish)  (Read 8278 times)

0 Members and 1 Guest are viewing this topic.

Offline ContractSlayer

  • Elite
  • *****
  • Posts: 605
  • Karma: +4/-0
    • View Profile
Newest OFF Version (here only OFF Version publish)
« on: September 01, 2008, 11:08:53 PM »
What is OFF?
More information & download:



New console commands:

 "getdlspeed", "getulspeed" - get the download and upload speed limits.
 "showurl <n>"              - print the n'th local URL.

Command line help reorganised.

Implemented a whitelist for the loopback server (which listens on port
23402). IPs on the whitelist will be allowed to send messages to that
port meaning they can send URLs (regardless of whether the node is configured
for remote browser access) and access the web interface (such as it is).
The web interface will also be returned to a whitelisted host if index.html
is request on the nodes main listen port. From cmd line the whitelist can
be configured with:

 "addwhitelistip <IP addr>"     - add an address
 "removewhitelistip <n>"        - remove the n'th address
 "showipwhitelist"              - print the list

Added a dialog to configure the whitelist from the gui which can be opened
by a button in the Network tab.

URLs can now be copied to the clipboard from the disperse tab.


Shaved another 4 bytes from the memory use of a local block,
by removing the little-used 'known meanings' field.

Converted the blockcache database from 256 lists to a 256x256 array of
lists this speeds up block lookups and reduces page faults at the expense
of some RAM.

Converted each of the block lists from a double linked list to an
std::vector containing pointers to block_info. This saves a lot of RAM
since the block_info class no longer need the list pointers and now has
a size of 24 bytes, but a large number of vector objects is used so there
is a slight RAM increase for small caches. There is also some overhead as
the capacity of a vector must increase exponentially, if this becomes
significant we will consider writing a custom vector class which does not
have this property.

OK, got bored and wrote the custom vector for cache blocks. This contains
block objects rather than ptrs and handles their allocation. This
saves more RAM from the old list ptr and the allocation ptr for lists
with more than one block. It also reduces page faults since blocks in each
list are contiguous in memory. In tests the cache changes resulted in a
reduction in RAM usage of >25% from the 19.27 release for a 75G cache. YMMV.


Fixed bug where the loopback server would stop after rejecting a msg
which was not from localhost.

Broken use of '!' to access system commands fixed.

Fixed crasher if there is an error creating the socket object for block
requests. Haven't found the source of the original error.

Fixed bug where the disperse queue was used unlocked in a few places.


Block requests now hold the available nodes in the main list of TempNodeList
and the busy nodes in the aux list - previously this was vice versa.
« Last Edit: September 07, 2008, 09:10:05 PM by ContractSlayer »

Offline ContractSlayer

  • Elite
  • *****
  • Posts: 605
  • Karma: +4/-0
    • View Profile
Re: Newest OFF Version (here only OFF Version publish)
« Reply #1 on: September 07, 2008, 09:09:13 PM »


Implement a hard close of a pool sockets in the event of certain network
errors, at present a hard close in done in the following cases:
 -socket error on read or write
 -timeout on read or write
 -node manually blacklisted or removed
Other cases may be added later.

Implemented a bandwidth scheduler. This can be configured from a button in
the network tab and allows a different bandwidth limit to be set for each
hour of the day over a one week period for both upload and download. When
the scheduler is enabled the bandwidth limit boxes are disabled and show
the current limits. In the config dialog, hourly periods can be edited
individually or set in groups using the "Set Selected Cells" button.

Under cmd line the scheduler can be enabled with "toggle_bw_schedule",
but the only way to set the schedule is to edit config/bandwidth_schedule.txt"
and then load it with "load_bw_schedule". We'll fix this when someone
complains about it!

"Retrieve and Execute" is now tentatively working under Linux.


Got rid of the old bodge where the number of requests received for a
relayed home block would be faked to trigger activation.


When a single complete OFFDir is selected in the download tab, the
"Retrieve and Execute" option is replaced with "Open OFF Dir" and selecting
this option opens the offdir window with URLs from the selected dir insert.

The offdir window now has a checkbox which to automatically download any
URLs that are added from the dir. By default, this box is checked if the
window is opened from Downloads and unchecked if it is opened from Local URLs.


Removed the 40 sec delay between home block requests which was a hangover
from before requests for a given block were constrained to one node at a time.

Fixed bug where the controls in the bandwidth config dialog would not
return to the unmodified colours when the apply button was pressed.


Some tidying of cnxn pool code and relay block request code.

Fixed a lot of outdated comments referring to threads that no longer exist.

Relayed block requests that have expired are no longer discarded when the
relay count reaches the maximum and are remembered for several hours. This
is to prevent the same few unobtainable blocks from clogging the active slots
and being reactivated as soon as they expire due to being close to the node.
A maximum of 5000 expired requests is held at which point they are discarded
by age as before.

Relayed block requests no longer activated until the initial pings are
done or there are more than 20 nodes online. This stops some blocks
with larger hamdists from being activated due to there being very few nodes
to compare them with (this may still happen for new nodes which have small
initial nodelists or if the nodelist file is deleted for some reason).

Incoming srchs now return a fail msg until the initial pings are done or
there are more than 20 nodes online, relayed srchs are still started meaning
that when we begin to return results there should be a significant number
of relayed URLs in the results.

When a request for a home block from a current download expires (after asking
all known nodes), it cannot activate again for 180 iterations of the main
worker thread which should work out to be 5-10 minutes.

Offline ContractSlayer

  • Elite
  • *****
  • Posts: 605
  • Karma: +4/-0
    • View Profile
Re: Newest OFF Version (here only OFF Version publish)
« Reply #2 on: September 19, 2008, 07:15:19 AM »



Removed the "Open OFF Dir" entry from the Local URLs context menu. When
a complete offdir is selected the "Retrieve and Execute" entry will change
to "Open OFF Dir" as in Downloads.

Removed mutex lock from internal AsyncCnxnOut data as all acces is now via
the ACH thread (and has been for ages).

Removed the option to authorise nodes for upstream pushes, old and pointless
code which predates the block DHT.


Reduced the RAM use of push_block objects.

Cancelled disperses now set a cancelled flag in all current disperse pushes
which then die after any currently active push and are ignored by the
disperse manager. The next disperse can therefore start immediately instead
of having to wait for pushes from the cancelled one to finish.

Implemented a set of binary flags for the most common version checks,
these are set when a node version is assigned, then we can just check
the flags instead of having to compare version numbers every time we
perform a version dependent task.

When a disperse completes, the counts of any blocks that could only be
pushed to 2, 1, and 0 nodes are shown in the log,


Fixed bug where some push fails were not treated as such.

Fixed bug where block requests in 2way comms did not respect the single node
rule introduced a couple of versions back.

Fixed bug where disperses sometimes showed less than 100% when complete
because of repeated blocks in the insert being skipped.

Fixed bug where the disperse queue would stall if a url was submitted which
did not have all the descriptors present.
Fixed crasher on download repair due to implicit = operator of block_name
class. This may or may not have had memory corruption effects elsewhere.


Added a "standby" message to block pushes, if a push is already being
received and a new push offer is made for the same block the standby
message is returned to the second push and the node will try again later.
This will get more sophisticated in future versions and will include
blocks returned to requests, some comparison of node upload speeds, etc.

Implemented a new push protocol which allows goups of blocks to be pushes
sequentially, similar to to new block request protocol back around 19.10.
It request confirmation messages every few blocks (currently 5 but may be
dynamic later) and supports cancellation of scheduled blocks which have
arrived by other means since the push connection started (although this
is not used yet). As before, nodes from this version onwards will reply to
new pushes, but not use them. Once the wrinkles are ironed out the new
protocol will be enabled for general use.


Included some experimental UDP code linking to UDT. The code expects udt4
to be in the same dir as src and requires UDP_ON to be set. Initial tests
have not been promising, so this code may disappear with little to no notice.

The estimate of the pool upload speed is flagged as unknown if less than
4k has been written, this should stop overly large chunks due to only
the data from a few small msgs being available.

Encapsulated 8 binary flags in a small class. Inheriting this class makes
a few of the block classes a bit tidier.

Finalise actions for pings, srches and block pushes are now performed in the
specific typed handler instead of the AsyncCnxnOut class.

Organised data-hiding in the TNLNode class.

Push blocks waiting on busy target nodes are now stored in a separate map.

New src files added:

Offline ContractSlayer

  • Elite
  • *****
  • Posts: 605
  • Karma: +4/-0
    • View Profile
Re: Newest OFF Version (here only OFF Version publish)
« Reply #3 on: September 30, 2008, 08:16:40 PM »

CRITICAL: Fixed bug where trim pushes in the busy wait state were not
included in the count of trim push blocks. This caused the displayed count
to vary wildly, but more seriously, could cause the next trim to start
prematurely which would allow preserved blocks to be trimmed.

CRITICAL: Fixed bug where blocks were only pushed within TBR, and not
within actual bucket radii. This caused many pushes to fail which should not
have done. (This crept in last version when the pushes were refactored for
the new protocol.)

CRITICAL: Fixed bug where a blank username or a username containing only spaces
would cause a nodelist to be rejected as mal-formed, which interrupted
block request cnxns.


Removed the facility to preserve new URLs for a number of sessions - was
just confusing and hasn't worked properly for ages anyway. Local URLs are
now either preserved or unpreserved and a checkbox for the default setting now
replaces the old spinctrl. (URLs added as downloads or disperses are always


Did some optimisation of URL parsing.

Changed the format of local_blocks.txt to store less redundant info and
read faster. Added a version header line to the file to make future changes

Optimised writing of local_blocks.txt to speed up the shutdown process.


Organised data-hiding in the URL class and saved a bit of RAM.

Refactored error checking in URLBase class to save the RAM use of redundant
check flags.

Offline ContractSlayer

  • Elite
  • *****
  • Posts: 605
  • Karma: +4/-0
    • View Profile
Re: Newest OFF Version (here only OFF Version publish)
« Reply #4 on: October 03, 2008, 06:31:36 PM »


Implemented a web interface shutdown command for when OFF is running in
gui mode: "http://localhost:23402/index.htm?shutdown" should close the client.

Added a command line argument to set the working dir of the program.
use: "offsystem --workdir=<path>"


Added a map of local URLs keyed by the filename. (Yes we can spend a bit
of the RAM we've saved recently.) This speeds up filename based checks
such as used in the local filter or the autostore.

If a descriptor or hashmap tuple is found to have lost some blocks
when a download is checked for completion, that tuple is resubmitted
automatically as block requests. Note: this has been disabled for testing
and will be finished in a future version

If gui items are queued for processing a new wxEvent is only posted once
(or if there are 1000 in the queue for some reason), which should ease
the pressure on the wxW event handling system when many events get scheduled.
(When an event is processed, all queued items are picked up.

Combined the various queues for gui updates into a single queue of
inherited types.

The download progress is now updated per block by passing the tuple locations
of that block to the gui, instead of sending the entire tuple map periodically.
This means more updates, but less data-passing and also fixes a bug where
the final block in a download would sometimes not be shown in the gui until
after completion.

Repair functions will now regenerate a single missing block from a
repairable tuple. This means that if only file blocks are missing and
only one from each incomplete tuple, then the URL will be unchanged
and the formerly missing blocks added to the cache.

The config tabs are now rebuilt when the language is changed. So they should
now pick up the translations.

Did some optimisation of HTTP header reading and parsing.


Fixed replacing of blocks to handle daisychained hashmaps.

Fixed false error on parsing headers from very old nodes which predate
the bucket radii fields after the node id.

Fixed broken detection of corrupt hashmapblocks in downloads.

Fixed bug where complete downloads would retrieve again on load if the
global "retrieve on complete" option was set.

Enforced the cache>bucket and bucket<cache rules for cmd line - OFF will
refuse to set an illegal value (the gui will cache or bucket if necessary
to allow the new value.)

Fixed bug on importing blocks from external drives under linux, apparently
rename() tries to create a link internally and fails with EXDEV. We do
a manual copy and remove if we detect that error.

Fixed bug where encryption pads withlost files whould not be deleted.

Fixed bug where disperses of concatenated inserts with offsets larger than
one block would be assigned incorrect progress values on resume because
the part of the data stream before the insert was erroneously included.


If "List Selected" is done on urls totalling more than 5MB then a dialog
opens asking if you want to list to files instead. Since that many blocks
won't fit in the log.

Organised data-hiding in the SrchResult class.

RetrieveData now inherits Insert instead of having an Insert member object.

Replaced the ubiquitous arrays of block_name objects with a small class
representing the hashes in a tuple.

When a download is complete, the "Cancel and Remove URL" menu entry
now reads "Clear and Remove URL" and does the obvious.

Auto-retrieve of downloads now happens in the retrieve thread instead of
one of the download manager threads.

Refactored block submission and completion checking of downloads to be
asynchronous and done in the ACH thread. The 3 threads assigned to
the download manager have been removed.

Download block requests can now be added directly to the waiting list
instead of using a mutexed update list.

Descriptor result blocks of downloads are now saved in the download directory
as desc_00.bin, desc_01.bin, etc This saves us having to re-xor the descriptor
blocks every time we want to check a tuple or submit more blocks. In future,
these files could be used to recover any descriptor blocks that have become
lost through some tragic happenstance - as long as only one block per
descriptor tuple gets lost. It is important to note that these files do not
form part of the blockcache and are never transferred between nodes. They are
purely a local convenience.

When a downloaded block completes a tuple in a current download, that tuple
is now verified against the hashmap immediately. This means that it is
no longer necessary to call a completion checking function on downloads
every hour or so to look for and verify complete tuples. It also reduces
the amount of disk access needed to verify a download as we usually have
the block in hand and only need to load the other blocks in the tuple.

The characters '/', '\' and ':' are no longer allowed in URL filenames as
the cause problems retrieving on some or all platforms. Filenames are also
not allowed to start with a '.' character. This currently only enforced
when urls and downloads are added or when a new filename is supplied by the
user - with luck this will propagate and we can avoid the expense of checking
the filenames of srch results. A user input filename is rejected if illegal,
but if OFF finds illegal characters internally they are replaced with '-'s.

If a relayed block request is inside the bucket but not flagged as home,
say due to the TBR, and finds itself closer than the closest active relayed
relayed request then it is converted to a home block if a request for it
is received.

Push standby msgs are now remembered for 60 secs during which time we will
not try to push the block to the node that sent the standby msg.

New src files added:

Offline ContractSlayer

  • Elite
  • *****
  • Posts: 605
  • Karma: +4/-0
    • View Profile
Re: Newest OFF Version (here only OFF Version publish)
« Reply #5 on: October 08, 2008, 09:50:20 PM »

CRITICAL: Fixed bug where the read timings of block requests would not be
reset on receiving a "processing" message. This caused false timeouts and some
lost messages.


The new block push protocol has been enabled for general use.


Disperses now keep a bitfield map of which blocks have been pushed and
whether those pushes succeeded or failed. On resume, any blocks which
have not yet been successfully pushed will be resubmitted, including any
that failed. This also fixes a bug that made it possible blocks to be
missed by disperse on resume as only the insert position was saved but
blocks were not necessarily pushed in order, Note: for this reason any
unfinished disperses will restart from the beginning on upgrade to this
version - once they have saved their maps they will resume as before.

Disperses now do several passes through the insert. The first pass is as
before, subsequent passes will only disperse any failed blocks. The number
of passes can be set between 1 and 10 in the disperse tab. The disperse will
finish if there are no more failed blocks.


Fixed bug where the internal value of a spinctrl was not changed if an
out-of-range value was entered.

Fixed bug from last version where downloads of small inserts would not
correctly show progress on restart.


The queued_disperse class now inherits URLBase instead of containing a
text url.

Block pushes now favour nodes with more than a week of uptime when pushing
to closest or fastest nodes. Except for the last two pushes of an ONWARD
push type (which tries to make sure the 4 closest nodes have the block).

Block requests for non-home blocks now ask all bucket nodes for the block
every 5 cycles.

Offline ContractSlayer

  • Elite
  • *****
  • Posts: 605
  • Karma: +4/-0
    • View Profile
Re: Newest OFF Version (here only OFF Version publish)
« Reply #6 on: October 28, 2008, 10:39:01 PM »

CRITICAL: Fixed bug that could cause corrupt inserts if mis-sized block files
were chosen as randomisers. These blocks are now checked and discarded just as
they are during retrieve.

CRITICAL: Fixed bug that resulted in a corrupt insert if regenerating a block
failed when repairing an insert from the source file.


The web interface shutdown is now disabled by default (as we assume most people
will not use it) It can be enabled with the checkbox in Options->Network or
with "toggle_web_shutdown" under command line. OFF will also return a page
to the browser saying that the client is closing, or that web shutdown is

The Local URLs context menu now contains an option to "Disperse Descriptors Only"
which disperses the descriptors and any hashmap blocks, but not the file blocks.

Added an option to slow the filling of the bucket for newer nodes, the idea
is to reduce the amount of bandwidth used when the bucket radius is large.
This is enabled by a checkbox in the Behaviour tab. The only current effect is
to advertise the TBR instead of the actual bucket radius if it is the smaller
of the two. Other effects will be added if this turns out to be insufficient,
but it drastically reduces the number of block pushes received. The option
defaults to off for existing nodes but is turned on when the startup wizard
is run for new nodes.


Implemented a manually buffered file class for reading ascii config file,
to reduce disk accesses when reading large files. This does not make much
difference on a fast disk, but seemed to speed up reading on slower or
external drives. Only currently used for the blockcache database file.

Nodes are now written to the known_nodes.txt file in order of online time,
this means that on startup the nodes that have been online the longest will
get checked first (on the assumption that these are the ones most likely
to still be contactable). This speeds up connecting to nodes on startup
especially for large nodelists.


Fixed bug where disperse pushes were miscounted when a new disperse push
upgraded an existing push block that was in the active or busy-wait queues.

Fixed bug where a node would not be added to the contactable nodes list if it
came online, did not ping back (i.e. the first node was walled) and no pool
cnxn was added.

Fixed bug where nodes with no pool cnxn would not be removed from the
contactable nodes list on some connection fails.

Fixed bug where a node would become a "zombie" if it was firewalled, had
no pool cnxn and came online with a connection that was not a ping. In this
case it would sit in the nodelist with status "online - firewall unchecked"
and not every go offline. The fix was to schedule a pingback for any connection
when the firewall status was unchecked, previously this was done only for ping

Fixed bug in 2way comms that caused "time out on non pool cnxn" errors.

Fixed bug in 2way srches that could occasionally cause bogus results to
appear in the srch tab when no srches had been started by the user.


The max allowed active requests for a download is now 10% of the total number
of blocks needed instead of the remaining blocks needed.

If the "Trim Now" button is pressed, the max cache size is no longer checked
and the cache will be trimmed even if it is within the limit (but not if the
bucket is not full).

Increased the limit at which pushes, block reqs, etc start from 20 nodes to
40 nodes. 20 was a lot when the limit was set.

« Last Edit: October 28, 2008, 10:40:35 PM by ContractSlayer »