[00:04:03] *** n215 has quit IRC
[01:46:45] *** deltab has quit IRC
[01:47:34] *** deltab has joined #bittorrent
[01:55:14] *** deltab has quit IRC
[01:58:05] *** deltab has joined #bittorrent
[02:03:15] *** Andrius[] has quit IRC
[02:11:01] *** hlindhe_ has quit IRC
[02:11:46] *** hlindhe_ has joined #bittorrent
[02:56:09] *** TheSHAD0W has quit IRC
[02:57:18] *** kwinz2 has joined #bittorrent
[03:04:44] *** kjetilho has quit IRC
[03:04:44] *** softwareelves has quit IRC
[03:04:44] *** HandheldPenguin` has quit IRC
[03:07:38] *** kjetilho has joined #bittorrent
[03:07:38] *** HandheldPenguin` has joined #bittorrent
[03:07:38] *** softwareelves has joined #bittorrent
[03:54:44] *** bittwist has joined #BitTorrent
[03:58:06] *** burris has quit IRC
[03:58:26] *** The_8472 has quit IRC
[03:58:32] *** wadim has joined #bittorrent
[03:58:33] *** wadim is now known as The_8472
[03:58:36] *** burris has joined #bittorrent
[04:06:20] *** init0_ has joined #bittorrent
[04:10:42] *** ProperNoun has joined #bittorrent
[04:12:13] *** bt42 has quit IRC
[04:19:05] *** init0 has quit IRC
[04:43:13] *** TheSHAD0W has joined #bittorrent
[05:08:24] *** goussx has quit IRC
[05:10:36] <choykloun> finally managed to compile HASP! next challenge: TSO.. when will there be a BT client for OS/900 o such? :P
[05:10:47] <choykloun> http://knark.net/~user/os360ipl.png http://knark.net/~user/os360hasp.png http://knark.net/~user/os360haspjobs.png
[05:17:59] *** goussx has joined #bittorrent
[05:29:50] <alus> when more than 2 people use it
[05:30:06] <alus> so.. never :P
[05:30:47] <choykloun> its modern descendants are in use
[05:31:07] <choykloun> chances are you indirectly interact with them everyday when you do any electronic banking
[05:31:25] <TheSHAD0W> choykloun: Get Python working on it, and I'll happily get BitTornado running.  ^__^
[05:31:38] <choykloun> is COBOL close enough? :)
[05:31:45] <choykloun> JCL? :)
[05:32:12] <choykloun> jesus christ it must have taken me 12-18h in total to get this far
[05:32:21] <choykloun> everything is 100% different from anything else
[05:32:28] <choykloun> even VMS is sane compared to this
[05:32:35] <TheSHAD0W> Is there even an IP stack for it?
[05:36:04] <choykloun> yes ofcourse
[05:36:20] <choykloun> and web servers etc
[05:36:25] <choykloun> java
[05:37:02] <choykloun> http://www-03.ibm.com/systems/z/os/zos/
[05:37:18] <choykloun> its widely used within certain sectors such as banking
[05:37:20] <TheSHAD0W> If it has java, it might run vuze.
[05:37:40] <choykloun> i only think it has java servlets already made
[05:37:41] <TheSHAD0W> Or snark.
[05:37:45] <choykloun> but obviously it would have a jvm
[05:37:51] <choykloun> but it wouldnt have sockets like *ix
[05:40:16] <choykloun> http://www-01.ibm.com/software/os/systemz/en_US/ certainly not a dead OS :)
[05:42:04] <choykloun> never heard of ibm websphere btw?
[05:42:06] <choykloun> its for z/os
[05:42:12] <choykloun> quite common in e-commerce
[05:56:38] *** rrr has joined #bittorrent
[05:57:01] *** n215 has joined #bittorrent
[06:01:10] <alus> itym ECOMMERCE
[06:04:03] *** n215_ has quit IRC
[06:19:58] *** Gottaname has quit IRC
[06:42:27] *** Gottaname has joined #bittorrent
[07:12:58] *** The_8472 has quit IRC
[07:13:19] *** wadim has joined #bittorrent
[07:13:21] *** wadim is now known as The_8472
[07:20:39] *** ryanprior has quit IRC
[07:56:37] *** GTHK has quit IRC
[08:26:34] *** bt42 has joined #BitTorrent
[08:34:01] *** bittwist has quit IRC
[08:36:45] *** chelz has joined #bittorrent
[08:54:42] *** chelz has quit IRC
[08:58:49] *** _rafi_ has joined #bittorrent
[09:30:48] <choykloun> alus: might be because in swedish "ekommers" looks and is pronounced very odd, but "e-kommers" works just fine :)
[09:41:09] <choykloun> btw for general protection against what we discussed yesterday (and i tested on my dear lab rat :)) and future unforeseen issues, what about only giving out 'nodes', never 'values'
[09:41:54] <choykloun> if a node is in the swarm he/she is free to tell the original requestor so
[09:44:39] <choykloun> and of course you'd make sure the udp packet rations involved blah blah arent on the side of the attacker
[09:45:50] <choykloun> ^^ please tear apart for me with your experience :)
[09:49:30] <choykloun> that way the worst that could be done with any get_peers response voodoo is that the target gets a slow stream of small dht ping's or whatever
[09:52:58] *** Andrius has joined #bittorrent
[09:57:19] <choykloun> s/rations/ratios/
[11:01:37] *** razvand has joined #bittorrent
[12:10:05] *** burris has quit IRC
[12:12:15] *** burris has joined #bittorrent
[12:12:41] <choykloun> hey when do you guys wake up.. i wanna discuss solutions to certain get_peers issues :p
[12:13:09] <choykloun> i have an idea that's simple and won't break backwards compat but there's probably some fine point that makes it a disaster in a real dht net
[12:15:03] <choykloun> we need to find longterm solutions to issues like these
[12:15:08] <choykloun> and not just fix those that are known
[12:15:35] <choykloun> thats why i keep rambling about protocol hygiene :)
[13:38:21] *** bittwist has joined #BitTorrent
[13:46:26] *** bt42 has quit IRC
[13:53:53] *** The_8472 has quit IRC
[13:54:09] *** Switeck has joined #bittorrent
[13:54:14] *** wadim has joined #bittorrent
[13:54:16] *** wadim is now known as The_8472
[14:34:20] *** razvand has quit IRC
[14:37:51] <The_8472> <choykloun> btw for general protection against what we discussed yesterday (and i tested on my dear lab rat :)) and future unforeseen issues, what about only giving out 'nodes', never 'values' <- uhm
[14:37:56] <The_8472> values is the whole point of get_peers
[14:38:04] <The_8472> well, that and tokens
[14:54:22] <choykloun> my idea was that the only 'values' ever being returned is the node itself
[14:55:01] <choykloun> i can't see any obvious ramifications for routing by doing that
[14:55:44] <choykloun> you understand what i mean?
[14:55:58] <choykloun> would increase traffic esp for torrents with many peers though
[14:56:27] <choykloun> but that can be dealt with using future protocol extensions... this way wont break backwards compatibility
[14:59:04] <choykloun> btw basically what my dos code does is that even though it does do routing (it actually did something perfectly legal in addition to testing the DoS part yesterday!)
[14:59:27] <choykloun> but i keep track of, query and when possible answer all nodes it ever sees
[14:59:41] <choykloun> statistics kinda take care of the rest
[15:01:02] <choykloun> main issue was my laptop running out of memory and my router iptables conntrack filling up :P
[15:01:30] <The_8472> <choykloun> my idea was that the only 'values' ever being returned is the node itself <- values returns peer addresses, not node addresses
[15:01:37] <choykloun> yes exactly
[15:01:59] <choykloun> but peer ip addr == node ip addr anyway
[15:02:32] <choykloun> so just have it return one entry, its ip addr and the port the BT client uses
[15:02:34] <The_8472> still, it doesn't make sense what you're saying
[15:02:58] <The_8472> the node which is answering a get_peers query is not participating in the torrent swarm itself
[15:03:09] <choykloun> nope
[15:03:11] <The_8472> thus returning its peer address is useless to the one who queries
[15:03:27] <choykloun> but if future clients dont accept any other values than ones pointing bak to the node answering the query ..
[15:03:45] <The_8472> then you'll never find any peers
[15:03:51] <choykloun> yes you will
[15:03:57] <choykloun> because you will get the nodes
[15:04:07] <The_8472> uhm, no.
[15:04:14] <choykloun> which in turn will give you the 'values'
[15:05:00] *** rrr has quit IRC
[15:05:01] <choykloun> or hm ya there is a routing problem there
[15:05:14] <choykloun> since the delta will practically be random
[15:05:31] <choykloun> possible to work around maybe ?
[15:05:58] <The_8472> Node A has ID A000, the torrent has the infohash 0300, thus Node B looks up ID 0300 and findes Nodes C1-C8, all near the ID 0300, none of which participate in the swarm that A000 participates in
[15:06:16] <choykloun> ya thats appx what i just said :)
[15:06:28] <choykloun> i have this problem of either thinking too little or too much before typing..
[15:06:48] <choykloun> esp after an all-nighter
[15:07:50] <choykloun> there _should_ be some clever way to atleast maximize the cost of abusing it though
[15:10:09] <choykloun> and/or minimizing impact
[15:11:23] <choykloun> its a shame its not standardized that udp port should equal tcp port
[15:11:33] <choykloun> then the querying node could easily verify it with a small udp packet
[15:12:41] <The_8472> hrrm, i thought of the same. but yeah, differing ports.
[15:12:48] <choykloun> i think you guys are much better suited for building workarounds than me
[15:13:04] <choykloun> so try to think of something while i think of future security extensions
[15:13:15] <choykloun> doing web of trust with dht would be really fucking awesome :)
[15:13:16] <The_8472> what i can think of is defining some max. distance from the target where we accept values
[15:13:38] <choykloun> ya
[15:14:32] <choykloun> i have logs from last night that might be useful
[15:16:58] <The_8472> but then again a node could just adjust the ID in the response to be close to the target key he's answering to
[15:17:25] <choykloun> in a small sample the get_peers requests are coming from almost uniformely distributed nodeid's!
[15:17:53] <choykloun> but that's related to the fact that i picked ipl (seed) nodes to achieve a certain distribution
[15:18:02] <choykloun> ya but thats detectable
[15:18:34] <The_8472> the issue with the mainline DHT is that infohash = node ID instead of SHA1(infohash) = node ID, otherwise we could encrypt/MAC values to make sure that people can't return bogus values without knowing the infohash
[15:18:57] <choykloun> my code even has support for listening to multiple interfaces and pretending to be different nodeIDs for each...
[15:20:01] <choykloun> and i have plenty of free ip addrs to play with
[15:20:53] <choykloun> properly done that would be a HUGE attack
[15:21:45] <choykloun> with 5 targets (different ipaddr:ports) i got around 50 connections/sec to the one i monitored with my simple hack
[15:22:28] <choykloun> and could easily leave an attack like that running for months from any box
[15:22:37] <choykloun> noone would find the source
[15:23:16] <choykloun> normal ddos attacks almost always dies down quickly since ppl notice their interwebz are slow
[15:24:05] <choykloun> something resembling an exponential loss during the first 3 days
[15:24:59] <choykloun> luckily people havent figured out how to do decentralized ddos coordination yet so that the target would get hit with precisely the amount of traffic needed to keep it down and nothing more
[15:26:09] <choykloun> most ddos bots are pure crap anyway
[15:26:38] <choykloun> when ppl ddos'ed prq i had a habit of hacking their botnet and destroying it
[15:26:51] <choykloun> and when possible put it into permanently ddosing the culprit
[15:29:34] <choykloun> btw, just checked and deltas for nodes sending get_peers are evenly distributed except for the lowest 1/5 or so
[15:30:32] *** rrr_ has joined #bittorrent
[15:30:56] <choykloun> small sample size though, messed up most of my get_peers logs :(
[15:34:29] <choykloun> dht with asym crypto web of trust would be quiiite cool by the way :>
[15:37:17] <The_8472> yes, but it would probably be impractical due to more packets needed and/or increased packet sizes due to all the signatures
[15:37:29] <choykloun> let me prove you wrong :-)
[15:40:25] <choykloun> if i could attempt to initiate KEX with 1.7 million nodes a lot seems posible :)
[15:54:15] <Astro>
[15:59:36] <choykloun> Press Ctrl-P for entering Monitor. 3 2 1 [...] Mysec: The appliance is in PRODUCTION mode. [...] MC 55 Alarm transmitter
[15:59:40] <choykloun> Mysec Sweden AB
[15:59:42] <choykloun> mc55dev login:
[15:59:46] <choykloun> should get a bunch of those set up for bt testing
[16:00:03] <choykloun> but it'd be kinda embarassing if i screwed up and they shipped with a pre-installed bt client heh
[16:00:09] <choykloun> or even traces of it on the flash memory
[16:05:35] *** The_8472 has quit IRC
[16:05:35] *** bittwist has quit IRC
[16:05:35] *** Andrius has quit IRC
[16:05:36] *** _rafi_ has quit IRC
[16:05:36] *** n215 has quit IRC
[16:05:36] *** goussx has quit IRC
[16:05:36] *** BentMyWookie has quit IRC
[16:05:37] *** charles has quit IRC
[16:07:53] *** charles has joined #bittorrent
[16:09:28] *** _rafi_ has joined #bittorrent
[16:10:22] *** bittwist has joined #BitTorrent
[16:11:46] *** n215 has joined #bittorrent
[16:13:23] *** The_8472 has joined #bittorrent
[16:14:25] *** Andrius has joined #bittorrent
[16:14:25] *** goussx has joined #bittorrent
[16:14:25] *** BentMyWookie has joined #bittorrent
[16:24:04] *** _rafi2_ has joined #bittorrent
[16:24:09] *** rrr_ has quit IRC
[16:30:20] *** _rafi2_ has quit IRC
[16:30:21] *** BentMyWookie has quit IRC
[16:30:21] *** Andrius has quit IRC
[16:30:21] *** goussx has quit IRC
[16:31:20] *** MassaRoddel has joined #bittorrent
[16:31:55] *** rrr has joined #bittorrent
[16:34:23] *** Snoopotic has joined #bittorrent
[16:35:31] *** _rafi2_ has joined #bittorrent
[16:35:31] *** Andrius has joined #bittorrent
[16:35:31] *** goussx has joined #bittorrent
[16:35:31] *** BentMyWookie has joined #bittorrent
[16:39:31] *** _rafi_ has quit IRC
[17:01:01] *** GTHK has joined #bittorrent
[17:07:33] *** deltab has quit IRC
[17:08:08] *** deltab has joined #bittorrent
[17:41:19] <Astro> I've got 5 MB/s incoming DHT traffic
[17:41:42] <Switeck> say what?!
[17:41:55] <Switeck> 5 MB/sec from DHT traffic or 5 MB/sec from peers/seeds learned from DHT traffic?
[17:42:58] <Switeck> And would that be 5 MegaBYTES/second or 5 megabits/second? XD
[17:43:23] <_rafi2_> after some exlpanations by alus yesterday about tcp_rate_control, I have tried again V2.0 with it on&off, and I still think it's buggy with DL limit set to 0 (the default!) ...  I'll post some screenshots in the forum for reference
[17:43:26] <Astro> I have implemented udp-based krpc only
[17:43:57] <choykloun> hm none of my dht experiments are running so atleast its not my fault :)
[17:44:08] <choykloun> but how the hell do you manage
[17:44:34] <choykloun> ive queried 1.7M nodes and never gotten above 50KB/sec or so at the highest peaks
[17:45:18] <Astro> ? internet shitstorm reasearcher ;)
[17:45:57] <choykloun> did you see about my DoS experiments with dht yesterday?
[17:46:06] <Astro> no?
[17:46:23] <Astro> I did a talk a couple of hours ago: http://events.ccc.de/congress/2009/wiki/File:26c3lt-bittorrent-dht.pdf
[17:46:33] <Astro> 4 minute lightning talk
[17:46:35] <choykloun> turns into a tcp connection flood
[17:46:48] <The_8472> for reference... 0.5 to 3KB/s is what a node should normally see
[17:46:55] <choykloun> brought down 5 servers at a webhosting company i dislike
[17:47:25] <choykloun> (and i got around 50 conns/sec to the server of my own i included for measurement)
[17:48:04] <choykloun> and could easily be done a couple orders of a magnitude worse
[17:48:14] <choykloun> so we have some work to do heh
[17:48:53] <The_8472> hurr, that representation of the DHT keyspace is somewhat misleading in your pdef. Kademlia is not Chord, it's not circular xD
[17:49:36] <choykloun> uhm you should see my attempting to explain the mesh networking shit i designed and implemented
[17:49:45] <choykloun> i dont think anyone else ever understood how it worked :p
[18:01:09] <TheSHAD0W> If it were easy to understand, people would already be using to do DDoS attacks.
[18:01:11] <TheSHAD0W> Oh, wait...
[18:02:40] <choykloun> DHT is very painless and straightforward to make a functional implementation of
[18:02:45] <choykloun> except for the bencoe :P
[18:03:02] <Switeck> who is bencoe?
[18:03:14] <TheSHAD0W> 9_9
[18:03:55] <Switeck> or did he mean bencode?
[18:04:03] * TheSHAD0W nods
[18:04:18] <Switeck> XD
[18:05:41] <TheSHAD0W> A quick fix would be to disallow peers from connecting to port 80 from data received via DHT.
[18:06:24] <choykloun> weeeell
[18:06:28] <choykloun> ratelimit based on port maybe
[18:06:30] <TheSHAD0W> That would still leave attacks on other ports open though.  Port 23 would be an obvious alternative, and a surprising number of people run their client on port 23.
[18:06:40] <choykloun> often common service ports are the only ones open in fw's
[18:06:53] <Switeck> I've seen a couple recommendations to put DHT on port 50
[18:07:33] <The_8472> <TheSHAD0W> A quick fix would be to disallow peers from connecting to port 80 from data received via DHT. <- most clients already have the option to ignore certain ports
[18:07:41] <The_8472> could just add 80 to the default settings
[18:07:54] <choykloun> ratelimits based on ports could be very useful
[18:08:15] <choykloun> <1024 certainly could be much stricter limited than >=1024
[18:08:23] <The_8472> mhh, not with many implementations/people running on 6881
[18:08:37] <choykloun> also what about actually ceasing trying to connect if it isn't a BT client at the other end :P
[18:09:24] <choykloun> although with Ne+6 nodes there's still a lot of possibilities even with limits etc
[18:09:52] <choykloun> maybe we could do some passive monitoring to detect various dht abuses including this
[18:11:13] <TheSHAD0W> One other thing...
[18:11:56] <TheSHAD0W> The_8472: What does your client do if it receives a DHT payload containing multiple copies of the same IP?
[18:12:10] <choykloun> i never did that though
[18:12:17] <TheSHAD0W> Checking for redundancies like that should be easy.
[18:12:20] <The_8472> we throw it into a Set, with equals() only checking for IPs, not Ports
[18:12:30] <TheSHAD0W> Good.
[18:12:30] <choykloun> but if it does multiply then someone should get shot :)
[18:14:42] <The_8472> but if you target multiple things at once that doesn't matter. you just have each node pick different targets at once instead of 1 target N times
[18:14:54] <choykloun> ya
[18:15:07] <choykloun> when i tested it yesterday i targeted the 5 separate servers
[18:15:19] <choykloun> not even same port on all
[18:16:55] <The_8472> all defenses i can think of either break compatibility or only require a constant amount (2-5) malicious, cooperating nodes to circumvent them
[18:17:10] <The_8472> and setting up a handful of malicious nodes is trivial
[18:17:18] <choykloun> ya
[18:17:24] <choykloun> my code even supports doing it \o o/
[18:17:51] <choykloun> what about passive detection then
[18:17:51] <The_8472> for example:
[18:18:11] <choykloun> get "IDS" nodes in strategic locations in global dht
[18:18:18] <The_8472> idea: check the node ID of responses against the nodes lists from previous find_node replies
[18:18:56] <The_8472> circumvention: have 1 "proper" node point to an ID-spoofing node with the spoofed ID already in the nodes-list
[18:19:07] <The_8472> only needs 2 nodes to circumvent that
[18:19:10] <choykloun> 'values' should have included the nodeID of the peers :(
[18:19:31] <choykloun> also would've saved me from implementing 'l' benc :)
[18:19:49] <The_8472> would use up too much space
[18:20:07] <The_8472> max. packet size should be around 1200 bytes
[18:20:39] <Switeck> is that with or without TCP overheads?
[18:20:53] <Switeck> (TCP/IP overheads I mean, since it probably uses UDP)
[18:21:30] <choykloun> well ok first of all make sure that all BT client impls stop connecting to ports that dont speak BT
[18:21:39] <choykloun> esp if they are detected as being a well-known protocol
[18:22:07] <choykloun> i can provide signatures for the services most likely to be attacked and most harmful to attack
[18:22:46] <choykloun> HTTP is the really obvious one
[18:23:04] <choykloun> SSL, to a lesser degree
[18:23:12] *** _rafi2_ is now known as _rafi_
[18:24:57] <choykloun> MySQL, SSH, FTP, all esay to detect..
[18:25:33] <The_8472> yeah right... so build protocol detection for a gazillion protocols into all clients
[18:25:38] <The_8472> that solution lacks... neatness
[18:25:39] <choykloun> nah
[18:25:50] <choykloun> 99% of all ddos attacks are against a few protocols
[18:25:54] <choykloun> ya
[18:26:18] <choykloun> but it also provides detection if we do some passive monitoring of dht
[18:26:27] <choykloun> which i think it will need anyways sooner or later
[18:26:29] <choykloun> think of usenet
[18:28:09] <choykloun> and you get a nice shitlist of attackers :)
[18:29:36] <choykloun> dht needs an immune defense anyways
[18:31:27] <choykloun> and uhm
[18:32:15] <choykloun> like 100k random hosts doing full tcp connections to your httpd is possible one of the most difficult things to filter
[18:32:55] <choykloun> even worse with other protocols, with http you could offload connection handling until you see its a http request
[18:33:07] <choykloun> kinda like i did with my very experimental tracker
[18:44:43] <choykloun> the_8472: i can provide a one-liner for detecting the most relevant protocols if you want :)
[18:45:15] <The_8472> i could get those from l7filter too ^^
[18:45:34] <The_8472> not the point though, it's defense at the wrong level... sevel layers of abstraction away from the DHT
[18:45:36] <choykloun> we dont exactly need to do DPI
[18:45:38] <choykloun> ya
[18:46:14] <choykloun> atleast i feel like monitoring nodes should be setup to see if stuff like this is already being done
[18:46:26] <The_8472> i can think of a handful defenses that would make all those attacks impossible. but they'd require some trivial but incompatible protocol changes
[18:46:45] <choykloun> ya same same
[18:47:35] <choykloun> need a combintion of approaches
[18:48:14] <choykloun> and improve security in the future so currently unknown attacks will be less likely/harmful
[18:48:47] <choykloun> for now, what about this
[18:48:59] <choykloun> we set up monitoring nodes as widely distributed as possible
[18:49:10] <choykloun> useful for a lot of stuff
[18:49:54] <choykloun> like i said i do think that dht will need an immune defense sooner or later
[18:50:21] <The_8472> wouldn't hurt i gues
[18:51:25] <choykloun> i could get at least a /22 or so of addrs worth
[18:51:38] <choykloun> and my impl can do precisely this
[19:13:18] <alus> _rafi_: buggy is the wrong word. doesn't work like you expect
[19:14:33] <_rafi_> well, that maybe... I'll post a link to the porum in a sec. People (& devs...) can judge for themselves... :)
[19:14:40] <_rafi_> *forum
[19:15:34] <_rafi_> I think that might "convince" you to build another test-version... ;)
[19:15:41] <_rafi_> alus...
[19:18:04] <alus> include ping times to google or something running at the same time
[19:18:26] <_rafi_> k
[19:18:51] <_rafi_> +  ping time to the local
[19:18:57] *** kwinz2 has quit IRC
[19:19:01] <_rafi_> ..ISP cache peer ?
[19:20:12] <_rafi_> (posting in 5min...)
[19:20:28] <choykloun> icmp echo is NOT a good way to measure latency
[19:20:41] <choykloun> since its often rate limited and/or downprioritzed and/or blocked
[19:22:42] <The_8472> use tcp pings then ^^
[19:22:54] <choykloun> SYN can be rate limited etc etc etc
[19:23:08] * The_8472 throws nmap at choykloun
[19:23:11] <The_8472> or hping
[19:24:01] <choykloun> ya but SYN roundtrip times can be just as affected as icmp echo
[19:24:23] <choykloun> or even the other way around
[19:24:26] <The_8472> there are other ways than syn
[19:24:28] <choykloun> if a host is highly loaded but uses syncookies
[19:24:31] <choykloun> ya
[19:25:12] <choykloun> just wanted to make clear that it's non-trivial :)
[19:31:44] <alus> I know it's non-trivial, but I'm too lazy to give _rafi_ a build which prints real latency measurements
[19:34:23] <_rafi_> alus: http://forum.utorrent.com/viewtopic.php?pid=443331#p443331
[19:46:03] <choykloun> 'time wget' ? :)
[19:48:26] <_rafi_> why should I care about latency, if the speed drops 50-70% when the DL limit is 0... :P
[19:49:06] <choykloun> i dont know the context of this but tcp uses latency to adjust its bandwidth usage
[19:49:17] <choykloun> to the current load
[19:50:23] <_rafi_> oh, and let me know if it is  a  problem for uT that I leave far away from you guys... ;)
[19:51:39] <_rafi_> and alus, you have all the flags, you can calculate distences and deduct latencies... :P  no need for extra printouts..
[19:53:29] *** _rafi2_ has joined #bittorrent
[19:57:27] *** bbear has joined #bittorrent
[19:57:52] *** bt42 has joined #BitTorrent
[19:59:01] *** GTHK has quit IRC
[19:59:12] <alus> choykloun: tcp does not use latency
[19:59:20] <choykloun> yes it oes
[19:59:21] <choykloun> does
[19:59:35] <alus> at least not Reno
[19:59:37] <The_8472> depends on the congestion controller
[20:00:14] <choykloun> well yeah obviously its not a REQUIREMENT
[20:00:26] <choykloun> but its part of the algorithm almost everyone uses
[20:00:32] <alus> I don't think so.
[20:01:13] <choykloun> it is
[20:01:17] <choykloun> trust me on this, please :-)
[20:01:40] <alus> choykloun: I'd rather not.
[20:01:51] <alus> choykloun: to my knowledge Windows uses Reno, which operates on loss
[20:02:23] <choykloun> ya packet loss is the most important factor
[20:02:49] <choykloun> havent you ever been on a high latency link? like satellite
[20:02:52] <alus> choykloun: if you'd like to prove otherwise with some sort of documentation from Microsoft, that's fine
[20:03:06] <choykloun> i dont know what the latest windows version uses, no :)
[20:03:14] <choykloun> and i dont feel like REing the windows tcp code either :)
[20:03:33] <alus> I'll take documentation on any Windows version since Win95
[20:03:57] <choykloun> http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Congestion_control
[20:04:21] <bbear> Hello
[20:04:49] <bbear> Do you know how to share a file from one peer to another peer with Bittorrent ?
[20:04:52] <choykloun> simple way to find out: next time im on satellite i'll see if it behaves like expected with high latency high speed connections :>
[20:04:54] *** bittwist has quit IRC
[20:05:37] <alus> are you talking about the RTO calculation?
[20:05:40] <bbear> It's a bidirectionnal link I want set, with an easy mean to replug the connection later, etc.
[20:05:56] <choykloun> and uhm
[20:05:58] <choykloun> tcp window scaling
[20:06:05] <choykloun> windows certainly does that..
[20:06:15] <bbear> Mmhh ... Bittorrent seems perfect to exchange files between two users.
[20:06:18] <alus> choykloun: proof please
[20:06:23] <choykloun> but yes im not familiar with exactly what windows does
[20:06:33] <choykloun> proof that windows does window scaling?
[20:06:50] <alus> based on latency
[20:06:53] <choykloun> i dont have a win box but dump some traffic ?
[20:07:20] <choykloun> nah its not strictly related to congestion control
[20:08:20] <_rafi2_> ehm... alus,  so how are my screenshots/"proof" ? helpful for you guys ?
[20:08:44] <Switeck> bbear, if you're unfirewalled it shouldn't be too hard to send a premade .torrent to someone
[20:08:58] <choykloun> http://en.wikipedia.org/wiki/TCP_congestion_avoidance_algorithm
[20:08:59] <choykloun> anyway
[20:09:01] <alus> _rafi2_: my guess is the drop in your download speed is based on uTP's reaction to network effects.
[20:09:03] <choykloun> we are both correct
[20:09:05] <Switeck> best to use a public tracker instead of an internal tracker if the info isn't purely private
[20:09:42] <_rafi2_> but... but.. is does not happen with "uTP only"  setup
[20:10:58] <_rafi2_> (though the cache here will not operate under wuth this setup:( )
[20:12:45] <alus> _rafi2_: yes, in pure uTP mode, there are no TCP connections being dumb
[20:13:38] *** _rafi_ has quit IRC
[20:15:27] <choykloun> alus: btw you should know that there are special products for satellite providers to solve issues caused by the latency :P
[20:17:32] <choykloun> anyway
[20:17:37] <choykloun> lets agree that we are both correct
[20:18:23] <_rafi2_> alus: the point is - the *default*  setup is not working well at all (DL limit=0, tcp_rate_cont =true, transp_disposition=15)
[20:18:46] <choykloun> should implement optimistic ACKing in BT clients.. haha
[20:19:01] <Switeck> optimistic how?
[20:19:04] <choykloun> it'd be perfect actually since BT allows you to request a specific part
[20:19:09] *** _rafi2_ is now known as _rafi_
[20:19:23] <choykloun> switeck: its a way to grab all available bandwidth on the slowest link between you and the other end
[20:19:42] <choykloun> google it
[20:19:42] <Switeck> without losses?
[20:19:51] <choykloun> well
[20:19:57] <choykloun> the classic way is to do it with http
[20:19:58] *** kwinz2 has joined #bittorrent
[20:20:00] <choykloun> and then use Range:
[20:20:14] <choykloun> to get the missing parts
[20:21:21] <Switeck> "Optimistic ACKing - because TCP congestion window growth is a function of RTT, receiver can fake shorter RTTs by ACKing packets it has not received, forcing sender to increase cwind. If sender gets ACKs for packets not sent it ignores, allowing receiver to be arbitrarily aggressive, concealing losses and leading the sender to overwhelm network."
[20:21:31] <Switeck> um...no
[20:21:39] <choykloun> um...ive done it
[20:21:59] <choykloun> was a couple of years ago though
[20:22:13] <choykloun> one school, one 100Mbps connection, one laptop per student, go figure :)
[20:22:44] <_rafi_> alus: you might consider fixing it (or 'tuning' if you don't like the word "buggy"...) or not making TCP_rate_control=true  the default (at least, yet...) .
[20:25:07] <Switeck> The goal of BitTorrent is reliability even over speed
[20:25:22] <Switeck> optimistic ACKs seems to be an aggressive way to cause link losses
[20:25:27] <choykloun> haha yes
[20:25:30] <choykloun> its blatant abuse
[20:25:35] <choykloun> i was being a bit sarcastic
[20:25:46] <choykloun> but its good if you share your connection with others and hate them :p
[20:26:00] <Switeck> I've heard too many "modest proposals" lately to not discount even the outlandish ones.
[20:26:13] <Switeck> ...because I'm not sure of sarcasm levels
[20:26:46] <choykloun> babies are an excellent nutritional source.. lots of high-energy fat
[20:27:04] <choykloun> :)
[20:27:40] * TheSHAD0W recommends a honey-glaze recipe
[20:27:51] <choykloun> swift suggested soup iirc
[20:28:40] <TheSHAD0W> No, you soup the older ones; with babies you don't want to abuse the tender, tender flesh...
[20:28:42] * TheSHAD0W drools
[20:30:00] <choykloun> though here something of like 1/3 of the kids are malnourished so you need quite a lot for sustenance
[20:32:25] <TheSHAD0W> That's what skewers are for.
[20:32:37] <TheSHAD0W> You just line 'em up and roast several at a time.
[20:33:10] <choykloun> btw, some of my favorite snacks: http://www.estoykh.com/pics/kateatoh.jpg
[20:33:52] * TheSHAD0W starts singing "la cucaracha"...
[20:34:46] <choykloun> la cucaracha la cucaracha, ya no puede caminar, porque no tiene, porque le falta, marijuana que fumar!
[20:35:08] <TheSHAD0W> 9_9
[20:35:42] <Switeck> That doesn't look like ordinary roaches...
[20:35:48] <choykloun> its not cockroaches
[20:35:50] <Switeck> what exact critter is that?!
[20:35:53] <choykloun> hm
[20:35:56] <choykloun> dont know english name
[20:36:01] <choykloun> they live in/close to water
[20:36:01] <TheSHAD0W> They're roaches.
[20:36:03] <Switeck> I don't know if it even has one
[20:36:09] <choykloun> attack their pray like spiders do
[20:36:22] <choykloun> inject them with venom that dissolves the insides and then sucks it out
[20:36:22] <TheSHAD0W> One variety of 'em.
[20:36:45] * TheSHAD0W prefers grasshoppers
[20:37:00] <choykloun> ya grasshoppers are tasty
[20:37:13] <TheSHAD0W> In the news...
[20:37:14] <choykloun> kateatoh are more like a small meal
[20:37:23] <choykloun> i mean they measure about 10cm across
[20:37:29] <choykloun> you eat them like shellfish
[20:37:32] <TheSHAD0W> A flight from Amsterdam to Detroit was disrupted by a Nigerian man.
[20:37:32] <choykloun> peel off the wings and shell
[20:37:34] <choykloun> white meat inside
[20:37:36] <TheSHAD0W> AGAIN.
[20:37:41] <choykloun> tastes slightly shellfish-like
[20:39:03] <Switeck> How'd they know he was from Nigeria?
[20:39:23] <TheSHAD0W> Probably offered to transfer $18 million to their bank accounts...
[20:39:45] <choykloun> uhm
[20:40:00] <choykloun> does he really claim he's educated by al qaeda and got assigned to blow up an airplane?
[20:40:13] <TheSHAD0W> No idea about the 2nd one.
[20:40:20] <Switeck> maybe after questioning he does...
[20:40:22] *** burris has quit IRC
[20:40:28] <choykloun> al qaeda certainly knows that airplanes are among the worst things to target nowadays..
[20:40:34] <choykloun> or well, professional terrorists that is
[20:40:39] *** GTHK has joined #bittorrent
[20:41:20] <TheSHAD0W> Maybe, but they make the biggest splashes when they go down.
[20:41:52] <choykloun> 9-11 was successful only because of the element of surprise
[20:42:20] <Switeck> a cargo ship full of munitions going off in a large harbor works well though
[20:42:27] <Switeck> France has done it a few times
[20:42:36] <choykloun> just do a timothy!
[20:42:54] <choykloun> a ton or so of AN/FO or even better AN/NM
[20:43:21] *** bbear has quit IRC
[20:43:30] <choykloun> blow a car/trailer full of that ANYWHERE in a populate area and you're gonna cause national panic
[20:43:48] *** burris has joined #bittorrent
[20:45:19] *** bbear has joined #bittorrent
[20:55:30] *** burris has quit IRC
[20:58:22] *** burris has joined #bittorrent
[21:02:10] *** The_8472 has quit IRC
[21:03:00] *** The_8472 has joined #bittorrent
[21:26:08] *** Elrohir has joined #bittorrent
[21:32:57] *** n215_ has joined #bittorrent
[21:39:48] *** n215 has quit IRC
[21:52:03] *** kwinz2 has quit IRC
[21:54:52] <The_8472> alus, http://forum.bittorrent.org/viewtopic.php?id=132
[21:54:59] <The_8472> any chances of supporting something like that?
[21:55:14] <The_8472> more specifically http://forum.bittorrent.org/viewtopic.php?pid=888#p888
[21:59:54] <TheSHAD0W> If DHT needs a revamp, any chance of moving it to randpeer?
[22:00:03] <TheSHAD0W> Among other things, they already have a stats system.
[22:00:43] <The_8472> randpeer... i have talked to mole2 like 6 years ago about it and he said "i only need some more simulations, then i'll publish it"
[22:00:52] <The_8472> yeeahh... here we are, 6 years later ^^
[22:01:00] <TheSHAD0W> Yup, and it's been stalled for a couple of years.
[22:01:16] <TheSHAD0W> But he apparently has a library, and I bet if he were pushed...
[22:01:36] <TheSHAD0W> He still shows up in freenode.
[22:01:47] <The_8472> well, a library would be useless for most people here
[22:01:55] <The_8472> wrong programming language and all ^^
[22:02:10] <TheSHAD0W> What language?
[22:02:31] <TheSHAD0W> C?
[22:02:34] * TheSHAD0W shrugs
[22:02:38] <TheSHAD0W> Roll out libraries.
[22:02:54] <The_8472> idk what he's writing it in, but BT is written in C, C++, Java, Python and what not
[22:03:35] <TheSHAD0W> It's stupid to have incompatible DHTs for the major clients, and this is a good excuse to merge the clouds.
[22:04:33] <The_8472> well, my overhaul-proposal would be slightly less ambitious....
[22:04:47] <The_8472> keep the packet format, most of the logic and just do a few incompatible changes
[22:06:32] <The_8472> could even do the transition in several steps to keep everything from breaking
[22:06:49] <Switeck> join the DHT clouds together so we can all DDoS ourselves!
[22:07:57] <TheSHAD0W> Well.
[22:08:23] <TheSHAD0W> Hashcash is one good way to reduce DDoS potential, and I'd recommend it be examined.
[22:08:54] <The_8472> yep, fixing the node IDs would be one good step towards a solution
[22:09:40] <TheSHAD0W> What about torrent-specific DoS?  Are there any good fixes for that?
[22:09:49] <TheSHAD0W> Keep people from taking over keyspace.
[22:10:33] <The_8472> well, node ID = sha1(ip address)
[22:10:44] <The_8472> but that has the drawback that you have to change your node ID when the IP changes
[22:12:25] <The_8472> extending node IDs to 256 bits would be interesting too, then we could use public keys of that ECC curve as node IDs.
[22:12:53] <The_8472> but that's not really necesasry to prevent dDoS attacks with the DHT
[22:13:00] <TheSHAD0W> How do you handle when peers can't easily see their external IPs?
[22:13:16] <The_8472> your-external-ip field in the ping responses
[22:13:22] <TheSHAD0W> Gotcha.
[22:13:47] <TheSHAD0W> One last question before I run out...
[22:14:05] <TheSHAD0W> How are we going to handle it when gov'ts outlaw the DHT and shut down the startup nodes?
[22:14:14] <TheSHAD0W> :-P
[22:14:32] <The_8472> that's easy. join #DHT and someone for their IP/Port to join
[22:14:39] <The_8472> *and ask
[22:14:39] <TheSHAD0W> Heh.
[22:14:46] <TheSHAD0W> Okie.
[22:14:47] <TheSHAD0W> l8.
[22:14:51] <The_8472> *waves*
[22:22:46] <choykloun> yeah its so fucking great
[22:22:51] <choykloun> ONE node and you're in
[22:23:08] <choykloun> a 32+16 bit number
[22:23:52] <choykloun> if they tried to prevent dissemination of 48 bit integers it would shock even me..
[22:29:48] <choykloun> hah, the company i develop alarm systems for had a 'small disaster' today.. the UPS blew up and fried everything in the test setup
[22:31:12] *** deltab_ has joined #bittorrent
[22:33:34] <choykloun> Sun_Dec_27_04:08:29_2009 -67 1261883309 415f9168 1 1 44 alshu0 triggered internal hostdown relayed 'alshu0' '' '' 'Monitored peer is DOWN (iALSHU POLL)' '' '' ''
[22:33:38] <choykloun> Sun_Dec_27_22:01:51_2009 -67 1261947711 03e6913d 1 2 45 alshu0 !triggered internal !hostdown relayed 'alshu0' '' '' 'Monitored peer is UP (iALSHU POLL)' '' '' ''
[22:36:53] <choykloun> ^^ an alarm transmitter reporting that it lost and regained contact with the central system :)
[22:36:54] *** bittwist has joined #BitTorrent
[22:41:16] <choykloun> oh well, its actually great that this happened... good test of worst case scenarios
[22:41:32] <choykloun> the real systems will be doubled of course..
[22:43:53] *** bbear has left #bittorrent
[22:44:16] *** bt42 has quit IRC
[22:50:28] <The_8472> http://forum.bittorrent.org/viewtopic.php?pid=892#p892 <- a few of my ideas what to change
[22:56:22] *** silver_hook has joined #bittorrent
[22:56:46] <silver_hook> Hullo. Can anyone point me to a .torrent parsing Python module?
[23:01:01] <The_8472> well, just rip the one from bittornado
[23:01:53] <silver_hook> The_8472: That's an idea...
[23:02:46] <silver_hook> The problem is that I need it only to get the torrent location from a .torrent file :P
[23:03:09] <The_8472> torrent location?
[23:03:21] *** Waldorf has joined #bittorrent
[23:04:49] <silver_hook> Yup, like this:
[23:04:50] <silver_hook> http://imgjam.com/torrents/album/371/1371/1371-ogg3.torrent/Vate - Aeronautica y otras tecnologias -- Jamendo - OGG Vorbis q7 - 2006.03.07 [www.jamendo.com].torrent
[23:06:20] <silver_hook> I get that info if I mouse over a torrent in KTorrent and even TorrentParse seems to find it ? see:  http://torrentparse.berlios.de/screenshots/0.2.Linux.Skinned.Parsed.png
[23:07:19] <silver_hook> KTorrent has its own API and DBus calls, but by using those I still can't access that URI/location/filename data ...although KTorrent obviously has it stored somewhere.
[23:09:07] <The_8472> what info? oO
[23:10:15] <silver_hook> See screenshot ? that what's called there "Torrent Filename"
[23:10:45] <The_8472> that's not a location
[23:10:50] <The_8472> it's a filename
[23:11:04] <The_8472> or the torrent name, depends
[23:11:04] <silver_hook> KTorrent (the GUI) calls it "Location"
[23:11:27] <silver_hook> Not name ...I need the original filename/location/whatever.
[23:11:53] <The_8472> i don't see how "dsl.0.7.9.iso" is a location
[23:11:55] <silver_hook> It bothers me that KTorrent offers me even to copy it via a RMB menu ...but I can't get it via a script if it kills me :P
[23:12:41] <silver_hook> The_8472: But, http://[...]/torrents/dsl-0.7.3.iso.torrent *IS* a location
[23:12:42] <The_8472> i guess it's the name field in the torrent's root dictionary in the case of multifile torrents or the filename in the info dictionary in the case of single-file torrents
[23:12:49] <silver_hook> and *that* is what I need
[23:12:57] <The_8472> that kind of location isn't stored in torrents by default
[23:13:03] <silver_hook> Nope, name is something else.
[23:13:21] <The_8472> some clients might add a "downloaded from" field or whatever, but it's not part of the torrent standard
[23:13:32] <silver_hook> Hmmm
[23:13:33] <silver_hook> Crap
[23:13:46] <The_8472> and if you look at your own screenshot then that kind of data isn't in there ^^
[23:13:48] <silver_hook> That's what I need actually...
[23:13:58] <The_8472> oh wait
[23:14:01] <The_8472> torrent filename oO
[23:14:28] <The_8472> not sure where that comes from, maybe by opening the file from an URL?
[23:14:34] <silver_hook> See ? http://imagebin.ca/view/JbO3MVa.html
[23:14:47] <silver_hook> KTorrent stores the info *somewhere* but I can't find it :\
[23:14:56] <silver_hook> Could be...
[23:15:49] <The_8472> yep
[23:15:55] <The_8472> that info is not in the .torrent
[23:16:11] <The_8472> i gues KTorrent just saved the URL where you downloaded it from if you open a URL within ktorrent
[23:17:12] <silver_hook> Hmmm, OK.
[23:17:38] <silver_hook> At least now I know that looking at a .torrent parser is not the solution ? that helps ;)
[23:17:40] <silver_hook> Thanks :]
[23:20:07] <Switeck> I'm sure most anyone who cares has already seen this by now:  http://blog.bittorrent.com/2009/12/24/point-click-wait-or-point-click-watch%E2%80%A6-will-streaming-break-bittorrent/
[23:20:55] <Switeck> uTorrent v2.1's Streaming feature will almost require YouTube settings to work reliably.
[23:25:00] <The_8472> i'd like less assurances and more details ~~
[23:27:46] *** _rafi_ has quit IRC
[23:34:57] <Switeck> The criteria for when a torrent is ALLOWED to Stream has to be almost ridiculous.
[23:36:18] <Switeck> The seed/peer values reported by tracker, DHT, and PEX are often exaggerated...counting them multiple times, plus counting dead and hopelessly firewalled peers/seeds. Even if you have a lot of seeds...what if they're bloody SLOW?
[23:38:07] <Switeck> A marginal torrent might appear to have lots of seeds...so uTorrent v2.1 would allow it to be streamed, creating a very grating user experience.
[23:38:49] <Switeck> The one sure way to ensure the seeds both exist and are fast is to connect to them...LOTS of them!
[23:39:32] <choykloun> the_8472: your suggestions look good
[23:39:42] <choykloun> ill look into doing more asymmetric stuff
[23:39:58] <Switeck> Then and only then can a torrent be verified as being reliable enough to be streamed.
[23:41:18] <choykloun> by the way where can i find the specs for PEX
[23:41:36] <Switeck> you mean which BEP?
[23:42:06] <The_8472> there is no BEP for pex. You basically have to speak LTEP or AZMP and understand the pex messages, which are added/dropped lists for v4 and v6
[23:42:31] <The_8472> you send added when you get new connections, dropped when the connection is closed.
[23:43:09] <choykloun> gah
[23:43:13] <choykloun> ok
[23:43:18] <choykloun> so what about some protocol dumps :p
[23:43:19] <The_8472> oh, and the messages are bencoded
[23:43:38] <choykloun> just gimme a couple of sample exchanges
[23:43:56] <The_8472> no wireshark on this box ^^
[23:44:37] <The_8472> http://bittorrent.org/beps/bep_0010.html <- you need to implement this first
[23:45:50] *** silver_hook has quit IRC
[23:46:05] <Switeck> How/why is there no BEP for PEX?
[23:46:19] <The_8472> nobody wrote one
[23:46:32] <choykloun> well i have to implement much of the actual bt protocol too..
[23:47:07] <Switeck> How in the world is the varying PEX implementations even vaguely compatible at this time? :P
[23:47:09] <choykloun> basically i 1) want to get really involved in a certain big site again and 2) get a break from the fucking code i encounter at my dayjob :)
[23:47:29] <The_8472> Switeck, transmission's was horribly broken until 1.8
[23:47:52] <Switeck> uTorrent's has been horribly broken in some versions as well.
[23:47:59] <choykloun> and 3) dht has a lot of potential for liberty reasons
[23:48:08] <choykloun> same reason im involved in wikileaks, tor, etc
[23:49:04] <choykloun> btw regarding an older discussion, i'll design and implement a standard for encrypting dht packets and we'll see if you guys like it or not
[23:49:14] <choykloun> not much work really since ive done most of it before
[23:50:20] <Switeck> PEX not being documented sounds exactly like what Microsoft would do.
[23:50:27] <Switeck> (undocumented standards)
[23:50:30] <The_8472> we'd need tight integration into the DHT for encryption to really work without imparing efficiency, right now i can see little interest in that... so...
[23:50:48] <The_8472> Switeck, i'm really just too lazy to document it and so are others
[23:50:57] <The_8472> if anybody asks i can explain it and provide code samples ^^
[23:51:02] <choykloun> i just want to be able to do some tests
[23:51:21] <The_8472> well, for local tests you can do whatever you want
[23:51:25] <The_8472> we don't have to agree on that ^^
[23:51:32] <choykloun> haha
[23:52:15] <choykloun> i also want to know that my tests arent in some entirely different hash table (more fitting than universe here i think) :PP
[23:54:08] <choykloun> i have this dream of dht with asymmetric crypto and web of trust systems etc that i just cant let go of :)
[23:55:19] <The_8472> webs of trust are almost impossible in an automated environment since there is no source of trust
[23:55:40] <choykloun> can still be built among people who well .. trust each other
[23:55:49] <The_8472> yes, but that's not automated
[23:55:57] <The_8472> it requires humans
[23:55:57] <choykloun> yup
[23:56:12] <Switeck> "friends' list"?
[23:56:14] <choykloun> also im using the term in a wider sense than pgp's
[23:56:19] <Switeck> um...no thanks XD
[23:56:35] <The_8472> Switeck, it makes sense for darknets, but not for something like a DHT
[23:56:46] <choykloun> certain things make sense
[23:57:03] <The_8472> essentially you don't need trust on the DHT
[23:57:04] <Switeck> The_8472, I don't think I'm in disagreement with you on that XD
[23:57:12] <choykloun> and some need to be done BEFORE [insert any of the potential ones here] start abusing dht
[23:57:39] <The_8472> yes, but a web of trust probably is not necessary to do that
[23:57:49] <choykloun> as i said, im using the term loosely
[23:58:43] *** Elrohir has quit IRC
[23:59:28] <choykloun> and i love to experiment!





top