Author Topic: Super fast downloads via Wifi !  (Read 12938 times)

0 Members and 1 Guest are viewing this topic.

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Super fast downloads via Wifi !
« on: July 15, 2008, 06:09:47 PM »
Why would you want to do this? Solid, sustained, un-throttled 1 to 50Mb/s block transfers.

"But I already have 3Mb/s now". Not really. Everyone out there has a slower upload speed. People aren't going to take up all their bandwidth with OFF, they want to browse etc. People don't want their ISP banning them, or other reasons so they are throttling their connections.

Yea, this would whoop ass over anything you have now.

There are many ways you can do this. Broadcast, direct and SMB.

Direct - This one is simple and can be done now, find some friends that are within about 16 miles of each other (see info below) set up a "adhoc" network, run OFF and connect to each other.

It's possible to run 2 OFF clients at a time on the same computer, one on Wifi and one on the internet, some simple modifications may have to be made to the Wifi side client so it only makes one connection to the other internal client and stays off the internet. Now the outside OFF network can benefit from your Wifi group.

Why not just share files instead of using OFF? Blocks are simple random data, Wifi is open, enough said.

SMB - This one can be done now too, SMB is also called "Workgroups" on windows.

Unknown to most people, whenever your wonderful windows machine is on, it broadcasts packets everywhere telling everyone it's there, who it is, and that it's sharing folders (if you have some shared).

Any broadcasts on your local network are also broadcast over your Wifi router. Why? Because people want to get to their main folders on their main computer from a laptop on the patio.

Same thing as above, find friends and "adhoc" your blockcache folder, others "import blocks" every once and a while from your blockcache.

Broadcast - This is a little more difficult and would require some software additions to OFF. In short, UDP or ethernet broadcast packets on your local network are also broadcast over your Wifi router.

Random blocks sent constantly. Any blocks requested via the internet side are also added to the outgoing broadcast queue with a priority.

OFF would watch for any broadcast packets going by on your ethernet connection and put the blocks into your blockcache. More fancy, have OFF make requests via UDP.

Standard off the shelf Wifi routers can be used for any of the above.

Want a Wifi connection to your friend 16 miles away? I know someone who did with a pair of antennas like this. Don't worry, the "beam" width is pretty wide, you will probably be able to hit a lot of other people in that general direction. Wifi repeaters are easy too.
http://www.radiolabs.com/products/antennas/2.4gig/2.4-aluminum-parabolic.php

(more details see IEEE 802.11b multicast protocols)

Edit: I just added a HOWTO for filesharing over Wifi
http://board.planetpeer.de/index.php/topic,5037.0.html

« Last Edit: July 25, 2008, 02:15:26 AM by scripter »

Offline Nightwalker_z

  • ---+ Captain Cool +---
  • Elite
  • *****
  • Posts: 484
  • Karma: +8/-4
  • Netzzwerg
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #1 on: July 16, 2008, 05:15:30 AM »
(more details see IEEE 802.11b multicast protocols)

Please allow me the question, what an IEEE 802.11b multicast protocol is..... The 802.11 standard doesn't know anything about multicasting. 802.11 handles the first two layers of the OSI protocol.

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #2 on: July 16, 2008, 06:32:28 AM »
(more details see IEEE 802.11b multicast protocols)

Please allow me the question, what an IEEE 802.11b multicast protocol is..... The 802.11 standard doesn't know anything about multicasting. 802.11 handles the first two layers of the OSI protocol.

Multicasts on the Mac Layer (using only FF:FF:FF:... as the Mac Address)?!
But I think the idea is unusable... because my guess would be that no OFF User actually lives within WLAN Range of another OFF User atm ^^

Offline Nemo

  • Global Moderator
  • Elite
  • *****
  • Posts: 1303
  • Karma: +27/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #3 on: July 16, 2008, 04:53:28 PM »
OFF would watch for any broadcast packets going by on your ethernet connection and put the blocks into your blockcache. More fancy, have OFF make requests via UDP.

Standard off the shelf Wifi routers can be used for any of the above.
Using UDP would be cool. I live in a town with a Freifunk wireless Mesh network, running on Linksys WRT54GL routers. My friend is over an unstable wireless link about 9 to 11 wireless hops away (depends on the chosen route) and I tried to copy data from his FTP-server. After a week of re-trying my script downloaded 130MByte data, but the datatransfer was very unstable. TCP doesn't like random latency so once the connection was established I got about 0.5-1kByte/s....

UDP works much better over lossy networks like a Mesh! I'm still searching for the perfect application in our mesh network: UDP-based, anonym, datastore, easy of use, ressource-friendly (works perhaps on small NAS-devices like Linksys NSLU2... ::)

Quote
Want a Wifi connection to your friend 16 miles away? I know someone who did with a pair of antennas like this. Don't worry, the "beam" width is pretty wide, you will probably be able to hit a lot of other people in that general direction. Wifi repeaters are easy too.
Wireless works best with Line Of Sight connections; can you see your friends house? If not then you have to build a relay station or building a Mesh network... Directed wireless links are nice backbones for a wireless mesh network. :-)

Greetings,
Nemo.

Offline Nightwalker_z

  • ---+ Captain Cool +---
  • Elite
  • *****
  • Posts: 484
  • Karma: +8/-4
  • Netzzwerg
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #4 on: July 16, 2008, 04:57:24 PM »
Multicasts on the Mac Layer (using only FF:FF:FF:... as the Mac Address)?!
I guess that there is a MAC-address pendant for each Multicast IP. But you are right, that Multicasts are handled like broadcasts in a layer-2 domain.
My question was, what 802.11b multicasts protocols are.....

Offline Nightwalker_z

  • ---+ Captain Cool +---
  • Elite
  • *****
  • Posts: 484
  • Karma: +8/-4
  • Netzzwerg
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #5 on: July 16, 2008, 05:00:07 PM »
UDP works much better over lossy networks like a Mesh! I'm still searching for the perfect application in our mesh network: UDP-based, anonym, datastore, easy of use, ressource-friendly (works perhaps on small NAS-devices like Linksys NSLU2... ::)
Absolutely, when having latencies greater than 50ms, you'll have problems utilizing a 10MBit/s link with just one connection (TCP-Slow-Start).
UDP would be much more efficient in such situations.

Btw... that would be my feature request for RShare :-)

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #6 on: July 16, 2008, 11:45:50 PM »
(more details see IEEE 802.11b multicast protocols)

Please allow me the question, what an IEEE 802.11b multicast protocol is..... The 802.11 standard doesn't know anything about multicasting. 802.11 handles the first two layers of the OSI protocol.

Multicasts on the Mac Layer (using only FF:FF:FF:... as the Mac Address)?!
But I think the idea is unusable... because my guess would be that no OFF User actually lives within WLAN Range of another OFF User atm ^^
Yea, the Mac layer is what I called "ethernet broadcast packets", I was being pretty general and short about this because there's lots you can do with broadcast but for most people a adhoc setup would be good enough and can be done now.

Nightwalker is correct. In general, Wifi creates a wireless ethernet extension of your wired network. So when you go "adhoc" you extend your network over to other people's wired networks. Thus broadcast packets fly everywhere, being "multicast". There's a number of papers out there where people refer to this as "802.11b multicast protocol" which isn't 100% technically correct, but gets the point across.

But now you have to write special software to pick these packets up. For that to get popular it's going to take a while. People like off the shelf, turn it on and it goes stuff.

Nemo, let's work on getting you and your friend connected with OFF over your mesh. The first question would be, would your mesh network "group" like it when you start taking a lot of bandwidth?

The next question would be, do you think you could convince your group to start using OFF? Maybe enough to create the hops you need?

I was thinking more in the situation where all the people on the adhoc (or maybe mesh) network would use OFF so no one would care about bandwidth use. And no one would have a reason to throttle either :)

As for UDP, you have a good idea. Searching google for "reliable UDP linux" gives a lot of info, there might be a script or program that lets you sort of proxy over a UDP connection or something.

In my mind, creating a simple messaging system that just retries any screwed up packets would probably work since you have the room for some overhead. Since OFF checks blocks anyway, if one is messed up you could either ask for a retry or just drop it because another one is coming in 2us anyway :)

On a P2P network like this a lot of protocol messages get dropped for a lot of reasons, incoming buffer limits, throttling, nodes going offline, etc.. So OFF already retries messages.

This guy wrote some code just for fun.
http://andrewtrusty.com/2006/11/22/reliable-udp-file-transfer-2/

I'm just throwing out ideas. Since the OFF source already has code written to create sockets, it wouldn't be a great stretch to say that it could do a UDP thing of some sort. It might be as easy as telling it to just use UDP for it's sockets since it retries anyway.

TCP doesn't handle network noise very well. You can argue this all day long but I have direct personal experience with this. TCP isn't the end all of reliable packet transmission. It's good at assembling packets in proper order, but only if your network is "static" free (and that's in a general sense, thus the quotes). It could be that complacency has creeped into the TCP code stack, because we have such reliable IP level transmissions now. We do know that if your latency is a bit too much or you do TCP over TCP (as in tunnel) you can have problems where connections just stop. We also know that a one bit error in a TCP header can cause a RST, as in oops! your ISP flipped that bit :)

Edit: To get two nodes to run on the same machine under Linux, one way to get around the OFF lockfile would be to create a new user (same group so you can share files), ssh into that user so you can use your same GUI, and start OFF. It's not as hard as it sounds. Just open a terminal and...

ssh -2 -Xc blowfish other-user@127.0.0.1

They would fight over ports so you probably have to set them up different. Then have OFF connect to the other user's client. Haven't tried this yet but I know it works to run firefox as two users at the same time, it's just a thought.

« Last Edit: July 17, 2008, 12:02:47 AM by scripter »

Offline Hunnter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #7 on: July 17, 2008, 10:28:01 AM »
I like this idea very much.

Another thing you could do as well is auto-redirect any new MACs to a page that explains why the wireless mesh exists and what it aims to do.
Inform them of OFF, how to get it setup, how to be secure and so on.

And, if you live in a place without a mesh, you could maybe even get together with a bunch of people around your town and put money together to build mini-wireless relay stations and dot them around the place.  (just as Nemo mentioned above with the Linksys NSLU2s, just make sure no fingerprints or DNA gets onto them, if you live in a strict area....)

And to go even further, use dish-relays to connect other places together. 

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #8 on: July 17, 2008, 01:17:04 PM »
If someone wants to try this, here's the steps.

Two computers, two off the shelf Wifi routers, read the manual so you know how to get to it's configuration.

When you boot up the router gives you a IP like 192.168.1.3, if both computers get the same IP you will have to change one of the computer's IP to be like 192.168.1.5, meaning set a static IP for that computer (leave the rest the same).

It's probably important for proper routing that they both have a 192.168.1 as the first three numbers. So try to use the same type of Wifi boxes, at least for this test.

Set the network name or "SSID" the same on each box, call it "off" and set the box's wireless mode to "adhoc".

Now run OFF, make sure it's port is set to 23403.

Go to the "nodes" screen in OFF, right click and select "Add Node", put the IP of the other computer in the box and it should connect.

If it connects then it works! Now do a search and try to download something the other node has all the blocks for, something the other node is able to "retrieve" locally.

If you have problems connecting make sure your firewall has incoming port 23403 open, and try going to each router's config and port forward 23403 to the computer.

You could try the same thing with two laptops that have Wifi cards just for fun. Try three!

Please post your results.
« Last Edit: July 18, 2008, 12:59:36 AM by scripter »

Offline rb2k

  • Advanced
  • ***
  • Posts: 297
  • Karma: +3/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #9 on: July 17, 2008, 03:08:33 PM »
Go to the "nodes" screen in OFF, right click and select "Add Node", put the IP of the other computer in the box and it should connect.

If it connects then it works! Now do a search and try to download something the other node has all the blocks for, something the other node is able to "retrieve" locally.

Most of the blocks aren't even in the block radius of the "serving" node, so it should deny requests for those (at least in theory, not sure about the current code)

Offline Nemo

  • Global Moderator
  • Elite
  • *****
  • Posts: 1303
  • Karma: +27/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #10 on: July 17, 2008, 06:30:09 PM »
Another thing you could do as well is auto-redirect any new MACs to a page that explains why the wireless mesh exists and what it aims to do.
Inform them of OFF, how to get it setup, how to be secure and so on.
We use DHCPSplash (it's a captive portal on every Mesh node WIFI router): a notebook in the range of a node connects to the ad-hoc wireless network, gets an IP via DHCP and the first website they must accept is an agreement about the openness of this network (unencrypted, data security, fair use, only legitimate traffic, ...) and then they see a welcome page. This welcome page could show the things you describe.  :)

Quote
And, if you live in a place without a mesh, you could maybe even get together with a bunch of people around your town and put money together to build mini-wireless relay stations and dot them around the place.  (just as Nemo mentioned above with the Linksys NSLU2s, just make sure no fingerprints or DNA gets onto them, if you live in a strict area....)
I read on the english wikipedia that Linksys has discontinued the NSLU2...  :(

Today I bought an Acer mini PC tower for about SFr 900.-, it has a wireless card, 2GB RAM, 320GB harddisk, CPU about 2GHz... I try the usability as 24h/7d Linux server, it's very small and I hope it doesn't make much noise. It would be overkill as Mesh node, but it's a nice OFF/Freenet/whatever node in my living room.  ;D Just a bit expensive if everybody must have one in the neighbourhood for a citywide OFF-network...  ::)
But I think it has some advantages against small NAS-devices like the NSLU2 (200MHz ARM-processor, 32MB RAM): more power, more RAM, harddisk embedded into the case, no problem for compiling/running software because it's i386-architecture and runs (I hope without troubles) usual desktop-Linux.

The NSLU2 has advantages when it comes to size, cooling (has no moving parts) and autonomous operation without grid power supply (just put it together with a large lead battery and a voltage regulator and it should run some days on battery, together with a wireless router you get your own little meshed mobile(?) data center for camping or anonymous usage or whatever  ;D).

Just my two cents (or just my crazy ideas  ;)).

Greetings,
Nemo.

Offline Hunnter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #11 on: July 17, 2008, 09:23:09 PM »
^ OH NO  ???
I can't believe they discontinued such a fantastic little device, that is pretty saddening to say the least... oh Linksys.
And i wanted to buy one of those too. :(

Could attempt using those Linksys WRTSL54GSs and a USB drive connected, then pop in some custom firmware, add some casing and that could possibly work. (that is if these allow custom firmware, don't think it specifically mentioned that)
Or even better, for no moving parts or noise, if it could support a USB hub being connected to it, pop a 4 USB hub on there and use a bunch of flash drives.
Only downside with flash is the limited writes....
Although, the above could be checked for by regularly scanning the size of the drive, if it was to shrink, then there must be some dead parts and it could send an alert.
But even then, it is writing loads of little blocks frequently, that kills the drives quite fast... i know from personal experience, never use a flash drive for swap.   ::)

Don't know of any other methods sadly
edit: lies
Just spotted this http://en.wikipedia.org/wiki/Asus_Eee_Box
Could be interesting... but the price is a bit sharp for it sadly.
« Last Edit: July 17, 2008, 09:41:12 PM by Hunnter »

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #12 on: July 18, 2008, 12:56:07 AM »
Nemo, from what you said it is possible to reach a IP address across several hops of your mesh network, is that right? You mentioned doing FTP to a friend.

If FTP worked, you should try to see if you can get two OFF nodes to connect. I know you said the TCP connection was bad, but give it a try and let me know.

You may be able to leave other OFF internet nodes connected too, if you have problems you can try removing the "known_nodes.txt" file in the config folder before starting OFF so it doesn't connect to anything for the test. Then just manually connect by IP.

If you have internet connected nodes you can get blocks your friends don't have, but when they have those blocks those come in super fast. The more people running OFF on your mesh the better. It may be a good idea to only have one or two OFF nodes that are near the "internet gateway" mesh node to be connecting to internet OFF nodes (did that make sense?).

I assume that the mesh network you have there is pretty much like having a long ethernet cable to everyone's house. Meaning it's the same as a "adhoc" network, a large ethernet network except that it can hop ethernet packets through several routers.

I suggest throttling so that you don't take up the whole mesh's bandwidth for now. Remember that long distance links go to 1Mb/s, closer ones can go a lot faster. That's why I pointed to the cool antenna, more signal, more speed. Routers adjust automatically as the signal quality increases.

rb2k, as for "block radius of the "serving" node", I don't think that's going to be a problem. A block request goes to a directly connected node and that node just sends it back, same with searches as far as I can tell with the stuff I have done so far. If you have more info on this, please post.

Hunnter, WRT54G and a lot of other routers can run linux, see http://www.dd-wrt.com/

I was trying to keep it simple for most people, a "adhoc" network on standard routers would be easy for just about anyone as long as everyone can "see" everyone else.

On a lot of the newer routers you run out of flash space pretty fast, so a few scripts are OK. RAM is a bit better, but whatever it's doing is gone after reboot.

It may be possible to cut down the OFF code and make a block transfer OFF client that runs, but you have no storage. It would have to be a store and forward thing using RAM. But mesh or adhoc would do it easier for everyone.

You have to remember that not everyone wanting to do this in your town is going to want to mess with router stuff.

Just getting them to buy a router to dedicate to this and put some sort of antenna up is going to be a challenge.

It will be fun to bypass those nasty ISPs that are throttling and trying to tell you what you can do with your bandwidth!


Offline Hunnter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #13 on: July 18, 2008, 08:28:13 AM »
Too true.
Just getting people to go through that hassle is a challenge as it is, wiring hard-drives / flash drives to it... maybe a little bit of a far stretch.
But if you had a dedicated group of computer people, could be possible.  (i only know about 5 personally though... woo)

Only reason i suggested adding a hard drive was for what you just mentioned, blocks vanishing after a reboot.
But i guess if everything was done under the RAM, it could be a lot faster, so it might slightly make up for there being less space.. just slightly.

Me and a friend had been working towards getting our town away from the nasty ISPs recently, but the project was halted for awhile because it would require a little effort from some people around the town, of course, this was until a few days ago when i never realised how great (and easy) it would have been to use dishes to increase the signal ranges.
This gives slightly more hope to get it going again.  ;D
We are just trying to raise a little bit of money just now to help fund the projects. 
Hopefully spread the message to other towns and get them to create networks too.

Yay, my router is on that list, saves getting a new one.  :D

edit:
Hmm, well, i just got an idea.
Just now i have one of those Netgear NAS, the SC101s.
Well, straight to the point, taking off the casing of a router and sticking it into one of the drive slots, the other one could be used for a drive, perfect little enclosure.
Altjhough, it does complicate things because there are now 3 devices.... (not including any extenders, such as a dish)
« Last Edit: July 18, 2008, 10:14:05 AM by Hunnter »

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #14 on: July 18, 2008, 08:18:14 PM »
I just thought of how to do the UDP packet thing.

You always know exactly where the end of a UDP packet is, just step back two bytes and read the CRC.

Meaning, append two CRC bytes to every packet. It's easy to read, it's always two bytes back, and it's easy to add, just do a CRC (16 bits) on the data and add the byte to the outgoing buffer.

If the CRC is wrong, discard the packet. OFF should retry naturally, and if some parts aren't set up that way right now, make it that way.

Hunnter, it would be cool if you could get a whole computer club to set stuff up. I find that when you get to the point that people have to mount stuff on the roof or drill holes the idea sort of slides.

If you could find a professional installer for things like TV antennas or alarms, that guy would be good to have around. If you have a location that you really, really need, offer the guy free installation.

Another thing you could do is set dd-wrt up in "relay" or "client bridge" mode and it should relay packets for adhoc or other situations. Then take that and put it in a waterproof box, mount it to the back of the antenna (or mount the antenna to it depending on the antenna), and all you need is a power cable up to the thing. Make sure it's remote accessible and has the watchdog timer on.

The WRT54G's are great for this, they are very dependable and have the antenna connectors right on the back. People have modified them for power over ethernet so you can just run one ethernet cable up to the roof and be done.

It's best to have the router as close to the antenna as possible because of RF cable loss.

With dd-wrt you can adjust the power up to 200mW, if I remember right. If you're close enough and someone has a two story house, just putting it up against a upper story window would work.

Make sure anything you put up on the roof is well grounded, lightning protected.

On development and chat, chat is disruptive. I don't know too many programmer type people who like getting interrupted while working on some code. You have to concentrate. That's why I prefer to use a forum or possibly a e-mail list. I can answer when I'm taking a break, on my own schedule, and so can the other party.

Another problem people don't understand is if you write a lot of big changes to someone else's code, you could spend a lot of hours working and have that wasted because the guy feels that it's "not invented here", not "clean enough" and up to his tidy standards, he's got a better idea, or who knows what. Then there's the bother the main programmer has to go through to read your code, understand it and then change it to his liking before it's included.

That's why it's hard to get a group of developers to work on something.

Not that I won't provide a small patch here and there.