Author Topic: [Feature Request] redundancy in inserted files with FEC  (Read 4459 times)

0 Members and 1 Guest are viewing this topic.

Offline Nemo

  • Global Moderator
  • Elite
  • *****
  • Posts: 1303
  • Karma: +27/-0
    • View Profile
[Feature Request] redundancy in inserted files with FEC
« on: July 16, 2008, 05:21:40 PM »
At the moment I'm trying OFF and it works great. Now I'm missing a feature that I know from Freenet: FEC (Forward Error Correction). It works like the good old PAR-files, but it's transparent for the user: the inserter generates extra-blocks for the original data and insert them; a downloader could regenerate missing original-data-blocks by using this extra FEC-blocks.

A week ago I inserted many downloads as test. At the moment I have two downloads which hang for many days waiting for the last (obviously missing) OFF-block.  :( I know that FEC doesn't solve the problem of disappeared data on OFF (no file is forever in such a network), but it would increase the opportunity of a successful download. I know that OFF has some block-overhead and FEC would increase this, but I think the usability would much increase.
A nice solution as implemented on Freenet: FEC works transparently for the user and any downloader "heals" inserted files (regenerating all extra FEC-blocks after a successful download and reinserting them with all missing original fileparts).


Some test downloads are still at 0.00% to 0.7% after one week. I assume this are very old files and they aren't complete on OFF. In this case FEC doesn't solve the problem because there are too many lost blocks...  >:(  I have no idea how to implement an availability-feedback on an anonymous P2P network as realised in eMule or on BitTorrent-networks...  ::)

Greetings,
Nemo.

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: [Feature Request] redundancy in inserted files with FEC
« Reply #1 on: July 16, 2008, 09:38:35 PM »
<snip>
To solve this issue, there first was the idea of recovery blocks using erasure encoding, this how-
ever could lead to a problem for anonymity of data according to the developers.
The developer said this about the erasure coding situation:
annoyingly, it came down to this:
EITHER, the RS blocks were trivially identifiable as resulting from a given insert,
and thus subject to legal attack, OR, the insert+RSblock system was not ful ly recov-
erable from a completed download, and thus subject to cumulative failure. neither
situation was tolerable, and none of us could think of a solution. so I cal led a halt
to dev on the RS system, pending some brainwave or other.
<snap>

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: [Feature Request] redundancy in inserted files with FEC
« Reply #2 on: July 17, 2008, 12:36:48 AM »
the RS blocks were trivially identifiable as resulting from a given insert,
and thus subject to legal attack, OR, the insert+RSblock system was not ful ly recov-
erable from a completed download, and thus subject to cumulative failure.
The legal attack part doesn't make sense, you can send out a whole big file but a few blocks of FEC are a problem?

The FEC (or whatever) at the end of the download may be a problem, but I don't think a download always goes in block order (I haven't checked this yet in the code, hope not), so when you are waiting forever for that last block, it could be one in the middle. FEC data would be +filesize, meaning blocks that go past the file size, so you know those are the FEC data, extract them and finish the file.

The simple way, and one you could do now would be to include two or more extra files with each upload, named as "sha1filehash.fec1", "sha1filehash.fec2", "sha1filehash.fec3". OFF could also download one of those automatically (your choice and a user option), each FEC file gets larger because it has more FEC data.

Now if you are having big trouble getting those last blocks, just download the largest FEC file and see if that works. A external program could also try correction using two ore more FEC files.

Isn't FEC data something that helps you use the data you own? What if I have a scratched up, dog eaten DVD but was able to recover some data? Couldn't FEC help me to recover my bought and paid for DVD? Is it then possible to post FEC files on a regular website?


Offline Eroli

  • Elite
  • *****
  • Posts: 2435
  • Karma: +22/-0
  • StealthNet Developer
    • View Profile
Re: [Feature Request] redundancy in inserted files with FEC
« Reply #3 on: July 17, 2008, 04:30:07 AM »
Quote
What if I have a scratched up, dog eaten DVD but was able to recover some data? Couldn't FEC help me to recover my bought and paid for DVD?

Quote
he inserter generates extra-blocks for the original data

You see, it does not fit ;)
Your bought DVD has no extra blocks.

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: [Feature Request] redundancy in inserted files with FEC
« Reply #4 on: July 17, 2008, 03:05:08 PM »
tl;dr

The main point was that the blocks in the off system don't have any detectable meaning at the moment, they all look pretty mich like random data. The recovery blocks would be identifyable as far as I got the explenation.
To get more information: come to the off IRC Channel (#thebighack on irc.p2pchat.net) and ask the devs for yourself :D

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: [Feature Request] redundancy in inserted files with FEC
« Reply #5 on: July 18, 2008, 01:16:04 AM »
So if you stored the extra FEC file as I explained above, it would be made of random blocks just like the original content. No worries.

If the FEC blocks were just direct blocks of the FEC data, then yes, you can probably identify them as FEC blocks, and if you see someone pushing them out you might say that they came from that node, but who knows if he got it from someone else?

The idea above solves that and a little bit more code could be added to OFF so it can automatically find the FEC data, if the user wants that.

You could start doing this today by including FEC data files with every "store" of a file (if there was a easy, available program that did this).


Offline Nemo

  • Global Moderator
  • Elite
  • *****
  • Posts: 1303
  • Karma: +27/-0
    • View Profile
Re: [Feature Request] redundancy in inserted files with FEC
« Reply #6 on: September 10, 2008, 07:17:26 AM »
At the moment I have two downloads which hang for many days waiting for the last (obviously missing) OFF-block.  :( I know that FEC doesn't solve the problem of disappeared data on OFF (no file is forever in such a network), but it would increase the opportunity of a successful download. I know that OFF has some block-overhead and FEC would increase this, but I think the usability would much increase.
The current statistics (sept. 10th 2008) on a more or less 24/7 OFF node:

15 files (each has 700MB size) are over 97%, requested 48 days (!) ago, only a few blocks missing. I think I have to cancel these downloads because there's no progress...  >:(

I inserted about 100 testdownloads 48 days ago, mostly files with 700MB size. I think I could download about one half of them, which is not the best result (comparable to older Freenet content). 15 downloads are still active (see above) while I cancelled the others (some didn't get over 5%, most stuck at 50-90%).

I started more testdownloads with files between 50-300MB size one week ago: many succeeded, I cancelled many between 0-20%, now 50 of these downloads are still active (46 of them are over 90%).

I think FEC would massively increase the rate of successful downloads! It would help downloading older content for longer time.


The cache size is incredible huge, block cache is about 170GB (setting is 130GB...), bucket cache is 10GB (setting is 10GB). Is there a reason why OFF doesn't respect my setting? My current version is 0.19.29. I think 3 x sum(current downloads file sizes) is the worst case for needed space, but it seems OFF don't want to throw out blocks from succeeded downloads?!?  ::) ???


Just my two cents.

Greetings,
Nemo.