Author Topic: Super fast downloads via Wifi !  (Read 12824 times)

0 Members and 1 Guest are viewing this topic.

Offline Nemo

  • Global Moderator
  • Elite
  • *****
  • Posts: 1303
  • Karma: +27/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #15 on: July 19, 2008, 01:11:13 PM »
Nemo, from what you said it is possible to reach a IP address across several hops of your mesh network, is that right? You mentioned doing FTP to a friend.

If FTP worked, you should try to see if you can get two OFF nodes to connect. I know you said the TCP connection was bad, but give it a try and let me know.

You may be able to leave other OFF internet nodes connected too, if you have problems you can try removing the "known_nodes.txt" file in the config folder before starting OFF so it doesn't connect to anything for the test. Then just manually connect by IP.

If you have internet connected nodes you can get blocks your friends don't have, but when they have those blocks those come in super fast. The more people running OFF on your mesh the better. It may be a good idea to only have one or two OFF nodes that are near the "internet gateway" mesh node to be connecting to internet OFF nodes (did that make sense?).

I assume that the mesh network you have there is pretty much like having a long ethernet cable to everyone's house. Meaning it's the same as a "adhoc" network, a large ethernet network except that it can hop ethernet packets through several routers.
Perhaps my friend says yes to an OFF-test over the mesh network. It would work, but with bad performance due to TCP (a colleague said he tunnels TCP inside OpenVPN and gets better performance over his WLAN... ::)).

The inner working of our mesh network:
Every node (Linksys WRT54GL WLAN router with Freifunk firmware) operates on the same WLAN channel, same WLAN network name, in ad-hoc mode, with wifi-IP in the same IP-range. A router has a layer2 connection to all nodes in the WLAN range (Ethernet mechanism like ARP between Mesh neighbours).
Every node runs a routing daemon called OLSR which sends UDP packets to the broadcast IP address of the Mesh IP-range. These broadcasted UDP packets don't go far, only to the neighbour nodes (remember: broadcast packets don't get routed!). The OLSR routing protocol checks the link quality to the neighbour nodes and spreads this local topology information over the whole Mesh network: now every Mesh node knows the topology of the whole Mesh network. The OLSR daemon goes through it's database and searches for the best route to every other Mesh node and simply adds a host route for every destination node with a neighbour node as gateway.

You see, our Mesh network does routing, so every software based on TCP/IP unicast and anycast-based network traffic should work with the limitation of the lossy nature of a wireless mesh network; at the moment our Mesh is only used for sharing Internet access. Software which relies on broadcasts or multicasts don't work because this type of traffic don't work over simple routers.

A new experimental mesh routing protocol is B.A.T.M.A.N. (Better Approach To Mobile Adhoc Networking), it works different from OLSR but it also creates a mesh network with routing. The newest absolutely experimental thing is B.A.T.M.A.N. Advanced: creating a city-wide wireless switch, which does B.A.T.M.A.N. routing with MAC-addresses. Simply attach any ethernet device and IPX/DHCP/IPv4/IPv6/whatever should work out of the box over the wireless mesh network... Nice idea!  :)

If you want to know why not using WDS (Wireless Distribution System) as city wide wireless network: I don't know the details, but WDS doesn't work well or completely doesn't work with some dozens or some hundreds of nodes.

Greetings,
Nemo.

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #16 on: July 19, 2008, 04:53:53 PM »
Thanks for the description on the mesh you have there. That sounds like a good setup.

Can you monitor the bandwidth being used on the network? I would be interested to know how much is left over when people are using the internet over it.

Is there a internet gateway near each local node or is there just one for the whole group?

I still see the best set up as a group of people who are interested in file sharing over wifi so that you don't get complaints about slow web page loading. Meaning most people in the group have their own internet access and just want the extra speed for sharing.

I'm thinking about ways to do the UDP connection stuff and testing it here on a local network. The best way would be to have the connection code wait for a connection on the port like it does now, and if it's UDP then that connection is considered UDP for the rest of the session, just add the CRC bytes at the end of the packets etc...

Or the other one that may be a simple way to get this going quickly on a testing basis would be to just change the TCP socket code to UDP, probably just a few lines change, I don't know yet. But then you would have to be able to compile OFF and also make a version for your friend.

Nemo, are you and your friend running Linux? That would make compiling easier.

It wouldn't be too hard to add a config file option to turn OFF into a total UDP version, if I get that far and my code is accepted and put into a release version there would be windows and mac binaries available.

You should be happy the ethernet broadcast packets aren't meshed all over, windows sends them constantly when you turn it on and the mesh would be flooded.

People there may already be sharing within their local node range just by making folders public via SMB, have you seen this happening?

So what you are saying is on your mesh and probably most sane setups a UDP brodcast packet with block data wouldn't go far. So people should just set up a adhoc network and maybe use standard "range extenders" to do any long hops or get around line of sight issues. I think those should pass UDP broadcasts along.


Offline Nemo

  • Global Moderator
  • Elite
  • *****
  • Posts: 1303
  • Karma: +27/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #17 on: July 19, 2008, 08:33:22 PM »
Can you monitor the bandwidth being used on the network? I would be interested to know how much is left over when people are using the internet over it.

Is there a internet gateway near each local node or is there just one for the whole group?
We have three sponsored antenna locations (with some WRT54GL routers on the roofs), these locations are interconnected by glass fibre and goes to the sponsored 10MBit/s main Internet gateway. As you can see on the traffic statistic for the official Internet gateway we never reach a full pipe, we assume bad TCP performance and there are too much collisions in the radio layer around the mentioned antenna locations, because these locations are on high buildings and they have many neighbour nodes.
As you can see on our mesh map we have many private Internet gateway providers which share their Internet access decentralised (this is the original idea behind Freifunk in Berlin). Thanks to these nice people we have usable free Internet access where it wouldn't because too lossy or too many hops to the Internet or simply not enough neighbour nodes for a reliable route to the Internet.

Quote
Nemo, are you and your friend running Linux? That would make compiling easier.
Yes we do.

Quote
People there may already be sharing within their local node range just by making folders public via SMB, have you seen this happening?
Not yet. Our default usage case for normal users: connecting the computer via ethernet cable to LAN-ports of their WRT54GL mesh node, then their computer is hidden by a NAT (Network Address Translation) behind their mesh router. Notebooks in the wireless range get an IP address, but they're also behind a NAT. This is the default scenario for the Freifunk firmware from Berlin; other Freifunk networks like Weimarnetz in Weimar (Germany) don't have NAT inside their mesh network and use it also for filesharing with DirectConnect.
Short answer: There's no filesharing inside the mesh network because the computers aren't direct accessible (NAT) as default and most of our users don't know each other.
I know from Freifunk Leipzig that selfish filesharing over others Internet gateway is a huge problem in their mesh because it makes the Internet access for many other surfers slow and unreliable. The gateway providers fight with port blocking, iptables-p2p-filters and MAC-address-blocking against such users, but only layer 8 (encouraging people in community, information) can solve this problem forever (P2P-programs and their users are creative in bypassing firewalls...).
Encouraging people for using filesharing on the mesh network could decrease p2p-traffic into the Internet because content is on mesh. Or it could increase Internet traffic because people wants to get fresh material for sharing or because the used programs connects to meshed peers and peers on the Internet at the same time... ::)


Greetings,
Nemo.

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #18 on: July 20, 2008, 07:50:51 PM »
Nemo, if everyone is behind a NAT, how did you FTP to your friend? Are some people able to port forward, or can anyone?

To do it without disrupting the current mesh of people wanting "normal" internet is going to be difficult. The more direct adhoc network would be the easiest and could be set up on a different channel to avoid conflicts.

The other way would be to have a majority decide they want file sharing more than free internet and they go get internet from a wired ISP.

The group would agree that one or two nodes would be the internet OFF "gateway", running two clients, one on the internet and the other on mesh so you have fresh content. No direct mesh connection to the internet would be set up and everyone would agree to use OFF as file sharing for the group.

"whats in it for me?" is always the motivator, it would take some convincing and proof that this works before people would want to switch off of a free internet connection.

It's probably easier to start with a group of people who understand what you are trying to do and let the others catch on as you go.

I found the source code that creates the sockets and changed it to be a UDP connection, it tries to connect out on the net and seems to be working correctly, I see UDP packets going out, but I haven't set up yet to try it with two local LAN nodes.

You can try it if you want to set up and mess around, it's a quick edit.

In "offcnxn.cxx" at about line 824 in OpenListenPort() and at about line 342 in NonBlockingConnect() change "SOCK_STREAM" to "SOCK_DGRAM". It will start sending UDP packets to the nodes on your list (with no confirmed connections of course). A UDP listening port is opened on 23403.

I found out the "port space" for UDP is different than the TCP ports, so it's not trying to send packets to the open TCP port 23403 on the other end and being rejected. It would be sent to a open UDP port with that same number if there was one out there.

Meaning that OFF could easily open a TCP *and* UDP socket both listening on port 23403 and wait for connections just like it does now.

This simple change I did above should work for connection messages and pings, any small messages, so a connection on the GUI list should show everything is normal and packets should start to fly. I'm not sure how it will handle block transfers yet because I'm not sure if it creates "chunk" messages for block transfers, breaking up the block into smaller messages to send over the socket. For local connections on my LAN I think it should work.

It may be that when using a UDP connection (a flag bit can be set for UDP), OFF always "chunks" the block in small parts to make transport and retries easier. It may have to confirm chunk receipt if it doesn't already do that. You need a flag bit anyway to do the CRC at the end thing if we get that far along.

There's more work to be done, but if this thing will simply connect via UDP then that's a big part of it.

Chunks of blocks in the real internet world can come in out of order so that may be the biggest change needed to support UDP sockets. It may work as is by ignoring out of order block chunks and then just wait for them to be resent, not the best way but it would probably work well over adhoc Wifi because the path is more direct and packets probably arrive in order most of the time.

And just to get an idea of the speed of this, at 1Mb/s (the lowest Wifi speed) it takes about 1 second to transfer a 128K block. That's in a perfect situation of course.


Offline Nemo

  • Global Moderator
  • Elite
  • *****
  • Posts: 1303
  • Karma: +27/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #19 on: July 21, 2008, 09:25:48 AM »
Nemo, if everyone is behind a NAT, how did you FTP to your friend? Are some people able to port forward, or can anyone?
I have a special setup on my side: a small IP-subnet inside the mesh-IP-range without NAT between; I configured OLSR (the mesh routing protocol) to announce my wireless mesh routers as gateway to my small IP-subnet. Other mesh nodes receive this network announcement and insert the best route to this network into their routing table. So I don't have NAT-troubles in providing services on the mesh.  :)

Every owner of a wireless mesh node (Linksys WRT54GL) has the root password for his device (and no other has it). If the owner is tech-savvy and has network knowledge, then he can setup portforwarding via the webinterface; if he knows Linux and iptables he could write the firewall rule directly into a start script. But the whole procedure isn't simple for everybody...  :P
(The NAT world in our mesh should protect 0815 users inside the LAN behind the mesh router, but it also disables direct network access to services...)

Quote
I found the source code that creates the sockets and changed it to be a UDP connection, it tries to connect out on the net and seems to be working correctly, I see UDP packets going out, but I haven't set up yet to try it with two local LAN nodes.

You can try it if you want to set up and mess around, it's a quick edit.
Well, I don't want to break my OFF node, so excuse for not messing around in the source code...

Quote
This simple change I did above should work for connection messages and pings, any small messages, so a connection on the GUI list should show everything is normal and packets should start to fly. I'm not sure how it will handle block transfers yet because I'm not sure if it creates "chunk" messages for block transfers, breaking up the block into smaller messages to send over the socket. For local connections on my LAN I think it should work.

It may be that when using a UDP connection (a flag bit can be set for UDP), OFF always "chunks" the block in small parts to make transport and retries easier. It may have to confirm chunk receipt if it doesn't already do that. You need a flag bit anyway to do the CRC at the end thing if we get that far along.
TCP transfers are like putting data into a pipe and it comes out in the correct order on the other side. With UDP all this things have to be done manually: order of the data, blocksize/MTU, state of the "connection", .... I don't think that all OFF messages (e.g. block transfers) are smaller than MTU, so fragmentation of the OFF message has to be done.
Perhaps there is already a cool library out there with functions like reliable connections over lossy network where TCP dies (like Airhook) or for UDP hole punching. Depending on the situation, UDP transfer could be the better solution.

Greetings,
Nemo.

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #20 on: July 21, 2008, 08:22:29 PM »
Well, I don't want to break my OFF node, so excuse for not messing around in the source code...
No problem, make a new dir, download source into it, unpack OFF, make, cp offsystem ../../ (copy the executable back to your new dir), copy the themes dir into the new dir and ./offsystem (I posted instructions for Ubuntu here if that's what you are using, but it's the same for most other distros).

Now you have a copy you can mess with and the one you were using is left alone. Just don't run them both at the same time, OFF creates a lock file at start up.

TCP transfers are like putting data into a pipe and it comes out in the correct order on the other side. With UDP all this things have to be done manually: order of the data, blocksize/MTU, state of the "connection", .... I don't think that all OFF messages (e.g. block transfers) are smaller than MTU, so fragmentation of the OFF message has to be done.
That's why I like talking to you. I wasn't even thinking about the MTU, thanks for the reminder. It's normally 1500, even on wifi.

Google found this, "UDP leaves the breaking up of the output to the IP layer. If necessary, IP fragments the output into pieces that comply with the MTU, so that no outgoing packet exceeds the MTU limit."

"The sending UDP builds the checksum and the receiving UDP checks it. If the check fails, the packet is dropped."

Checksum isn't the best, but OK for now, if it's there and works, let's use it. We can add the CRC later.

"Must fit in one buffer. This means that the buffer pools on both sides of UDP must have buffer sizes that are adequate for the applications' requirements. The maximum size of a UDP packet is 64 KB."

Lucky for us, after reading more code, it looks like OFF sends blocks in 32K "chunks" using the "Range:" HTTP header. If that varies we just add a "if using UDP" statement to lock that at 32K when using UDP.

As for packet order, on a local LAN or most Wifi situations everything should go in order, there are no other paths to take which would make packets go at different speeds in different directions to the other end, so everything should show up in that order, except for packet loss, in that case we can do a retry of the packet or whole block (losing 1 second). We will have to see how that works out in the real world with multi hops, for adhoc it shouldn't be a problem.

Since OFF sends messages after connect, maybe when using UDP you just use a activity timeout. OFF will not know if a connection is "closed" since there is no real "connection", but I do know OFF sends a message to the other end when it quits, to tell the other nodes goodbye.

Now you can see why I just wanted to try this and see if it works, then fix the few things that go wrong. We can study this forever, it's faster to just try it.


Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #21 on: July 22, 2008, 05:19:34 AM »
LAN testing, IT WORKS! This means wifi "adhoc" should work also. I used 192.168.0.X addresses on a wired ethernet LAN. It connects TCP and transfers blocks OK.

If nodes on the wifi adhoc network stick to 192.168.X.X addresses and never connect to internet nodes then the node list should remain only on the adhoc network. You could use a firewall setting to enforce that, but it would be nice if OFF let you restrict connections to a range of IPs for this purpose. Another feature for this would be to have OFF always try to keep connected to a list of IPs you can define, even if they drop connection or timeout.

Nemo, sorry but the quick mod for UDP didn't work, same LAN setup. It's probably going to take more work to get UDP to talk. The other end doesn't respond at all, I confirmed that UDP packets were sent out. I think the other end is waiting for a TCP connection "signal". If I somehow take out that line of code and bypass it that may work but I was starting to think how can it handle multiple connections to other nodes?

I wonder how other P2P programs handle UDP? Maybe you have to open a lot of UDP ports so each one is connected to another node, the clients would have to try a range of ports until one reports that it's open. It was a interesting test.

UDP broadcasts of blocks is still a possibility, those are just blasted out. Now we know it's possible to open a UDP socket.

You may want to try that TCP tunnel you talked about, but in my experience, TCP over TCP on a not so good connection causes TCP to sort of lock up. Unless that tunnel uses UDP packets, that may be the way to go on your mesh network.


Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #22 on: July 25, 2008, 02:13:22 AM »
I just added a HOWTO for filesharing over Wifi
http://board.planetpeer.de/index.php/topic,5037.0.html


Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #23 on: July 28, 2008, 06:52:08 AM »
A little more research info for the developers if they want to do UDP.

Writing UDP packets out is easy, it works as described above even though you should use "sendto", it still seemed to work.

You can open a UDP receive socket the same way as above, that works OK.

The structure to read from a UDP socket is going to have to be a little different. As packets come in via UDP you can tell where they come from by the IP address that's returned with each read.

You no longer use "accept" because it's connectionless, and the "fd" is only the "sockfd" now, so that may be a problem when you call "CreateNewSvrMsgReadSkt(newsockfd,hostIP.c_str());" in handle_server_connection() (after doing the Accept()).

What's going to have to happen is that every time a packet comes in, it's going to have to be passed to the correct read thread that handles that IP address / connection.

I've been trying to think of ways to trick the code into thinking we have connections, possibly some "sub code" that works on it's own to do the trick but that may be more code than doing something just for UDP. I'm probably not going to go any further with this.

Also note that the call to read would look something like this

struct sockaddr_in cli_addr;
unsigned int clilen = sizeof(cli_addr);

long n=recvfrom(sockfd,buffer,nbytes,0,(struct sockaddr *) &cli_addr,(socklen_t*) &clilen);

(see accept() for how to get the IP address, same thing)

Sending would look something like this
long n=sendto(sockfd,buffer,nbytes,0,(struct sockaddr *) &cli_addr,(socklen_t*) &clilen);

The good news is that it should be possible to accept all traffic on one UDP port and just split it as it comes in by IP address.

The other good news is that the Wifi protocol breaks up packets longer than the MTU automatically and it adds a checksum or CRC (I don't remember which) to each packet and re-transmits them if noise becomes a problem. So it may be possible to just change the code to handle UDP sockets and not check the packets because the other layer does it for you.

OFF messages should have a ACK and re-transmit anyway, so UDP shouldn't cause any changes in that area.

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #24 on: July 30, 2008, 10:17:41 AM »
Nemo, have you considered I2P? It uses UDP and encryption. That might be a good test to see if UDP will help speed your wifi transfers.

In your test you should somehow set your firewall to allow only UDP from the I2P program. That may be as simple as only allowing incoming connections to one of the test nodes, probably yours, and only allow UDP to that incoming port.

The reason is that I2P started using a new TCP protocol and if that doesn't work then it uses the UDP one. You can read some of the technical details at
http://www.i2p2.de/ntcp.html

They also tried to improve TCP problems with their "NTCP", so you might want to see which one works best in your situation by disabling UDP and turn on TCP as a further test.

I2P has bandwidth limiters you can set so that you don't flood your mesh network when doing a transfer.

The only problem is I don't see a way to enter a IP to connect to, even in the config files. It may be hard to keep it from connecting to the internet. Maybe someone else knows?

It looks like you could use it to tunnel a FTP connection as a quick test. There's a feature you turn on called a "Eepsite", where you create your own web site on I2P, that might also be a quick way to test this.

Don't forget about tcpdump, it really helps diagnose problems.

Another one that says it uses UDP is GNUnet.


Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #25 on: August 02, 2008, 10:33:43 AM »
I did a little more work on UDP and got to the point it talks to the other end and the other end sends back (via UDP) a request for encryption keys, that's 3 packets (key req too) one way and one the other way, it sends the nodeIDs and name (one side) and shows up on the list at both ends. At that point the socket is closed for a unknown reason and that's where I'm stuck.

I tried disabling anything that times out sockets with no luck, and it's hard to find what is closing the socket because it closes in the socket destructor, which also means it gets rid of the pointer to the thread, I think.

Nemo, if you want to try this code you are welcome to it, but it will only confirm that you can send and receive UDP packets over your mesh to the other end. It prints some debug messages so you know what it's doing. Oh, and it only handles one connection from one IP at this point.

I'm starting to think it would be easier to make a new set of code that simulates a TCP connection and just redirect the socket calls to the new code when using UDP. At first I wouldn't have to do anything too fancy but later the code could be improved to do a better TCP over UDP like the I2P guys did with "NTCP" (that's in Java, sorry). I think they solved a lot of the timeout issues TCP has over flakey networks.

Tech details....

I found that "MSG_PEEK" lets you look at the data waiting in the queue without removing it from the queue. So with that I can get the IP address of the incoming packet for the original TCP "Accept" routine and then the data is picked up later in the read section. Code for recvfrom is below.

I also made the "listen" loop stop looping after the first detection of a packet, thus it can only handle one connection (for the test of course).

recvfrom(sockfd,buffer,0,MSG_PEEK,(struct sockaddr *) &cli_addr,(socklen_t*) &clilen);

Tracking down what calls what is a real pain in this code, I'm a little upset at small subroutines that are called only by one routine, and sometimes it's in another file.

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #26 on: August 09, 2008, 11:33:03 AM »
Did I say UDP was easy? :)

Just a little update, I have UDP working, it connects and OFF is pushing blocks to another local test node that needs blocks. The other node responds back and OFF acts like it has a normal TCP connection.

I re-wrote the UDP code so that it's more like a TCP replacement, but I have a few issues with code sections (thread level) that are making direct low level socket calls using "sockfd". I'm trying to trick my way around that so I don't disturb the existing code too much (I tie in using a few "if" statements near each TCP socket call).

Right now it doesn't do any error checking or low level packet retries, that can be put in later.

This will be experimental test code and a user selectable option in the config file if everything works out and if the code is accepted into the project.

Offline scripter

  • Regular
  • **
  • Posts: 88
  • Karma: +2/-0
    • View Profile
Re: Super fast downloads via Wifi !
« Reply #27 on: August 29, 2008, 07:49:21 AM »
More reasons to go Wifi

It's Official, Comcast to Have 250GB Data Cap Starting Oct 1st (Aug 2008)
http://www.zeropaid.com/news/9722/It%27s+Official%2C+Comcast+to+Have+250GB+Data+Cap+Starting+Oct+1st

ISU Begins Blocking P2P, Launches 'BirdTrax' (Aug 2008)
http://www.zeropaid.com/news/9720/ISU+Begins+Blocking+P2P%2C+Launches+%27BirdTrax%27