[01:17:21] *** Andrius has quit IRC
[02:30:51] *** GTHK has quit IRC
[02:43:04] *** bakerb has joined #bittorrent
[02:48:59] *** bakerb has left #bittorrent
[03:26:32] *** Switeck has quit IRC
[03:29:48] *** bbelt16ag has quit IRC
[03:30:52] *** init0_ has joined #bittorrent
[03:43:35] *** init0 has quit IRC
[03:45:19] <uriel> K`Tetch: you might want to post a TF article about this: http://www.eff.org/press/archives/2009/11/13
[03:50:35] *** KyleK_ has quit IRC
[03:58:57] *** The_8472 has quit IRC
[03:59:17] *** wadim has joined #bittorrent
[03:59:19] *** wadim is now known as The_8472
[04:08:12] *** The_8472 has quit IRC
[04:08:32] *** wadim has joined #bittorrent
[04:08:34] *** wadim is now known as The_8472
[04:30:41] <K`Tetch> yeah, already in process
[04:32:30] <uriel> ah, you guys are fast :)
[04:33:12] <TheSHAD0W> Working on internet time.
[04:37:20] <uriel> hah
[04:49:38] <K`Tetch> i was working on it overnight last night
[04:57:01] <TheSHAD0W> http://www.wired.com/epicenter/2009/11/copyright-time-bomb-set-to-disrupt-music-publishing-industries/
[05:29:49] *** init0_ is now known as init0
[05:37:21] *** _rafi2_ has joined #bittorrent
[05:43:31] *** _rafi_ has quit IRC
[06:48:51] *** _rafi_ has joined #bittorrent
[06:54:40] *** MassaRoddel has quit IRC
[06:55:43] *** _rafi2_ has quit IRC
[07:03:24] *** rrr_ has quit IRC
[07:25:47] *** _rafi2_ has joined #bittorrent
[07:28:18] *** stalled_ has joined #bittorrent
[07:32:10] *** _rafi1_ has joined #bittorrent
[07:36:52] *** stalled has quit IRC
[07:36:52] *** stalled_ is now known as stalled
[07:44:14] *** _rafi_ has quit IRC
[07:46:00] *** _rafi2_ has quit IRC
[07:58:53] *** _rafi1_ has quit IRC
[08:25:51] *** _rafi1_ has joined #bittorrent
[08:35:55] *** MassaRoddel has joined #bittorrent
[08:48:49] *** yehaT has joined #bittorrent
[08:49:44] <yehaT> hey
[08:54:29] *** Andrius has joined #bittorrent
[08:57:14] <yehaT> i want to discuss torrent clients
[08:57:27] <yehaT> anyone awake around here? :)
[08:58:21] <yehaT> have a problem and solution, just it's so obvious that it must be implemented already
[09:01:15] <yehaT> i want to be able to download/seed two+ 99% block identical identical torrent files using same copy of the data
[09:01:58] <yehaT> very minor modification to torrent clients needed to do this
[09:04:25] <yehaT> example, suppose there was two trackers both not allowing duplicate content to be seeded
[09:05:40] <yehaT> they both had the same content except that say a file inside the torrent had slightly different header and the data was offset
[09:07:17] <yehaT> utorrent has feature "force re-check" that could be modified into "force match"
[09:08:33] <yehaT> this "force match" feature will then try match the hashes against all byte offsets in the data but allow user to cancel (as it might take a while)
[09:09:00] <yehaT> if the match succeeds the partial file is then used to store the differential
[09:09:24] <yehaT> between the torrents of the two trackers
[09:09:47] <yehaT> and the non-match of the 2nd tracker is shared from the partial of course
[09:10:33] <yehaT> there are many obvious benefits to this so i won't go into them
[09:11:52] <yehaT> and of course if same tracker had duplicate content you could potentially participate seeding all the duplicated using just one copy of the dasta
[09:11:54] <yehaT> data..
[09:20:37] *** _rafi1_ has quit IRC
[09:20:38] <yehaT> the azureus dev mentions this problem in another context "Another serious realworld issue is swarm fragmentation due to distinct infohashes for identical content, e.g. due to different piece sizes or filenames. Eleminating this and NAT could probably expand the torrent lifecycle much more than xor pieces ever could"
[09:21:30] <yehaT> oh he's here, didn't notice
[09:21:49] <yehaT> anyway there's more benefit than just that fragmentation issue
[09:22:35] <yehaT> and the solution i propose can be implemented without changing protocols, just the popular clients
[09:26:34] <yehaT> another more cleaner solution than the utorrent partial files is to use ntfs streams for the differences
[09:26:53] <yehaT> alternate data streams i mean
[09:27:31] <yehaT> though on second thought i'm not sure i like that even though it's cleaner on the surface ;)
[09:34:47] <Andrius> yehaT, randomly pm'ing people isn't the right way to get attention, and ususally there aren't many active people on this channel during this time of day
[09:35:19] <yehaT> well you were only one i pmd as it seemed quite dead here
[09:35:37] <yehaT> though you might know if this thing i described is already implemented somewhere
[09:35:48] <Andrius> <yehaT> this "force match" feature will then try match the hashes against all byte offsets in the data but allow user to cancel (as it might take a while) < it will take forever if offeset unknown and file is large enough
[09:37:03] <yehaT> well that is obvious of course, it's pretty much the lcs problem
[09:37:28] <yehaT> the idea is it's an added feature for when the user already knows he has a situation where this will find a match
[09:37:30] <Andrius> it would take like a million times more time to hash a file with 1MB piece size
[09:37:35] <Andrius> or I misunderstand something
[09:37:56] <Andrius> because offset is unknown, it's not multiple of piece size
[09:38:20] <yehaT> its not as hard as you make it seem
[09:38:36] <yehaT> i've been doing these matches manually today for some very large torrents
[09:38:55] <yehaT> it's just couple minutes by hand since i know what to do
[09:39:25] <Andrius> you know it, but does that mean bittorrent client knows the same thing?:)
[09:40:16] <yehaT> i said the files would be assumed like 99% identical
[09:40:49] <yehaT> so that reduces the problem
[09:43:04] <Andrius> I don't think it'll be implemented, but you may want to bring this up few hours later. There are usually more active people like 5-10 hours later.
[09:44:07] <Andrius> still, if user doesn't know what he's doing (and average user doesn't), finding the right offset would take forever
[09:44:39] <yehaT> *only* if you always went for "perfection"
[09:45:10] <yehaT> more practical way is to set some threshold of how hard to look
[09:45:32] <yehaT> and the algorithm also look as hard as i did when i manually did that :)
[09:45:35] <yehaT> ie. not very hard ;)
[09:46:30] <yehaT> anyway ive needed this enough time now that i'll probably do it myself just thought it would be nice to have built in
[09:46:41] <yehaT> too lazy to keep doing this stuff by hand :)
[09:48:33] <yehaT> the simple thing i did that worked *well enough*
[09:48:54] <yehaT> this was without taking advantage of the piece hashes btw..
[09:50:55] *** bittwist has quit IRC
[09:50:56] <Andrius> it would probably make more sense to write a tool independant of client to find such data and recreate a file so that it can be rehashed by any bittorrent client
[09:51:08] *** bittwist has joined #bittorrent
[09:51:27] <yehaT> by hand process: was to just pick a string in the file, look for it in another file and it it matched just once then look the offsets and subtract.
[09:51:54] <yehaT> at this point you can just compare the hashes as usual
[09:52:03] <yehaT> if it matches then great, took 1 second..
[09:52:14] <yehaT> if not then it's probably not going to..
[09:52:42] <yehaT> i also ignored all 00 / not downloaded of course during this
[09:53:25] <yehaT> not exact science but works most of the time and takes no time at all
[09:53:31] <Andrius> well that would probably be infinitely faster than hashing torrent the way I imagined
[09:55:19] <yehaT> well yeah .. you could let it keep running if you wanted more of the content matched
[09:55:32] <yehaT> ie - less data downloaded when you start it
[09:57:15] <yehaT> the manual thing i did aligned 80% of the torrent correctly in my scenario
[09:57:32] <yehaT> it was some compressed files
[09:58:02] <yehaT> with the algorithm suggested one could've gone to probably that 99% in maybe 30 seconds
[09:58:28] <yehaT> with file size of around GB
[10:00:05] <yehaT> thing is..
[10:00:16] <yehaT> it doesn't solve the 2 copy issue really
[10:00:40] <yehaT> since now if i want to seed on both trackers i need both copies around
[10:01:04] <yehaT> so it needs to be implemented just like suggested in the clients using the partial files
[10:02:22] <yehaT> or difference files or whatever
[10:02:56] <yehaT> unless of course one implemented a whole new file system replacing ntfs ;)
[10:03:31] <yehaT> with one that didn'd allow duplicate data of block size > x
[10:03:46] <yehaT> seems easier to fix this in the torrent clients ;)
[10:03:46] <Andrius> well another thing is: this is a channel for bittorrent protocol discussion, and your issue/feature request/whatever should probably go to individual client developers, not here.
[10:04:05] <Andrius> <yehaT> with one that didn'd allow duplicate data of block size > x < use something other than windows
[10:04:15] <yehaT> well i didnt' know that, it said "client discussion" in the topic here
[10:17:40] *** burris has quit IRC
[10:57:36] *** goussx_ has joined #bittorrent
[10:59:54] *** goussx has quit IRC
[10:59:54] *** goussx_ is now known as goussx
[11:00:43] *** KyleK_ has joined #bittorrent
[11:07:50] *** goussx has quit IRC
[11:08:17] *** goussx has joined #bittorrent
[11:50:10] *** _rafi2_ has joined #bittorrent
[12:52:09] *** MrRainDrops has joined #bittorrent
[12:52:47] <MrRainDrops> hey everyone, does anyone know why my BITTORRENT6.3 would be downloading so slow when I have such a fast internet
[12:54:01] <The_8472> does your "fast internet" include a high upload speed?
[12:54:12] <The_8472> and does that torrent have a decent number of seeds and peers?
[12:54:28] <The_8472> and does your ISP not limit international traffic, like many australian ISPs do?
[12:56:32] <MrRainDrops> its got seeds 11 (6051) peers 0 (5005)
[12:56:41] <MrRainDrops> ive got really fast internet
[12:58:05] <The_8472> and, if you're only connected to 11 seeds and 0 peers than it's not likely that you'll get high speeds
[12:58:44] <The_8472> <MrRainDrops> ive got really fast internet <- how about providing actual numbers? like... upload speed and download speed as written down in your contract with your ISP?
[13:39:31] *** echelog has joined #bittorrent
[13:39:49] <Andrius> The_8472, but the car is really fast!
[14:05:00] *** MrRainDrops has quit IRC
[16:04:59] <TheSHAD0W> Slashdot is down...
[16:05:03] <TheSHAD0W> I feel disconnected...
[16:09:34] <The_8472> people still use slashdot?
[16:12:42] <TheSHAD0W> Eyyy, it's back up.
[16:14:28] <TheSHAD0W> Well, not entirely.
[17:20:48] *** KyleK_ has joined #bittorrent
[17:38:50] <TheSHAD0W> Okay, I was able to make the post I wanted.
[17:39:06] <TheSHAD0W> http://slashdot.org/comments.pl?sid=1444104&cid=30106028
[17:45:08] *** _rafi2_ has joined #bittorrent
[17:48:19] *** Marc_ has joined #bittorrent
[17:48:41] *** Marc_ has left #bittorrent
[18:19:01] <mpl> heh
[18:24:12] <The_8472> question is if they provided the full source code and machine specs
[18:24:32] <The_8472> black box testing alone doesn't test against attacks that can be performed by insiders
[18:28:03] *** burris has joined #bittorrent
[18:33:57] *** _rafi_ has joined #bittorrent
[18:34:09] *** K`Tetch has quit IRC
[18:34:23] *** K`Tetch has joined #bittorrent
[18:50:42] *** hlindhe has quit IRC
[18:50:42] *** _rafi2_ has quit IRC
[18:50:44] *** andar has quit IRC
[18:50:44] *** spoop has quit IRC
[18:50:46] *** tris has quit IRC
[18:50:46] *** Mazon has quit IRC
[18:50:52] *** yehaT has quit IRC
[18:50:52] *** pajlada has quit IRC
[18:50:52] *** TheSHAD0W has quit IRC
[18:50:54] *** hlindhe_ is now known as hlindhe
[18:55:42] *** burris has quit IRC
[18:59:37] *** hlindhe has quit IRC
[18:59:39] *** hlindhe_ has joined #bittorrent
[19:00:00] *** hlindhe has joined #bittorrent
[19:00:27] *** burris has joined #bittorrent
[19:00:27] *** Mazon has joined #bittorrent
[19:00:27] *** yehaT has joined #bittorrent
[19:00:27] *** pajlada has joined #bittorrent
[19:00:27] *** TheSHAD0W has joined #bittorrent
[19:01:11] *** Switeck has joined #bittorrent
[19:02:31] *** tris has joined #bittorrent
[19:02:36] *** RobertX has joined #bittorrent
[19:03:00] <RobertX> Lately, my Deluge client says "inactive port" to the ports I regularily forward. The port always worked until today.
[19:03:27] *** andar has joined #bittorrent
[19:03:27] *** spoop has joined #bittorrent
[19:04:00] *** stalled has quit IRC
[19:04:37] *** stalled has joined #bittorrent
[19:07:07] <RobertX> Let me make a correction: it's not inactive port, it's the Caution Sign as opposed to a tick.
[19:08:37] <RobertX> It always goes back to a tick sign when I update in the router.
[19:09:45] <RobertX> Until Now
[19:17:02] *** RobertX has quit IRC
[19:20:48] *** burris has quit IRC
[19:20:57] *** burris has joined #bittorrent
[20:30:12] *** burris has quit IRC
[20:30:39] *** burris has joined #bittorrent
[20:57:37] *** burris has quit IRC
[20:58:56] *** burris has joined #bittorrent
[21:27:23] *** _rafi2_ has joined #bittorrent
[21:44:36] *** _rafi_ has quit IRC
[22:18:01] *** _rafi2_ is now known as _rafi_
[22:20:37] *** klapaucjusz has joined #bittorrent
[22:20:39] <klapaucjusz> Oy!
[22:21:10] <klapaucjusz> Excellent news!
[22:21:25] <klapaucjusz> There's an experimental pool of DHT bootstrap nodes.
[22:21:39] <klapaucjusz> You're *not* welcome to hardwire it in your clients.
[22:21:53] <klapaucjusz> Please wait until I've done some testing.
[22:22:09] <klapaucjusz> But you're welcome to test: dht.wifi.pps.jussieu.fr port 6881
[22:22:31] <klapaucjusz> (It's a single DNS name, but two nodes... for now.  I'll be adding more in the future.)
[22:22:35] <klapaucjusz> Thanks to titer.
[22:22:40] <klapaucjusz> He's the one who put the nodes up.
[22:24:17] <The_8472> nice
[22:24:21] <K`Tetch> mainline dht system i assume
[22:24:35] <The_8472> *hardwires it and rolls out a non-beta release to everyone*
[22:24:55] <klapaucjusz> K`Tetch: yep.
[22:24:58] <The_8472> i just rewrote the bootstrap procedure to resolve all IP addresses instead of just one
[22:25:07] <klapaucjusz> Heh.
[22:25:40] <The_8472> which is sortof necesary if a host has A and AAAA records ^^
[22:25:48] * klapaucjusz hasn't had the pleasure of being introduced to K`Tetch, and he's wondering which BT implementation K`Tetch is working on.
[22:26:03] <The_8472> K`Tetch is more of a politician
[22:26:38] <The_8472> or blogger... not sure which role is more important these days
[22:27:06] <klapaucjusz> Blog URL?
[22:27:29] * The_8472 leaves further talking to he person itself
[22:28:33] <The_8472> klapaucjusz, speaking of the DHT. do you ever remove entries from the routing table?
[22:29:45] <klapaucjusz> IPv6 DHT not ready (broken, 3 nodes)
[22:29:47] <klapaucjusz> :-)
[22:30:11] <klapaucjusz> The_8472: yes, after I've pinged them unsuccessfully 3 times.
[22:32:09] <The_8472> well, i'm trying to reduce any removal
[22:32:31] <The_8472> so that the routing table survives transient connectivity problems
[22:32:31] <klapaucjusz> Sorry, need to go.
[22:32:34] *** klapaucjusz has quit IRC
[22:33:23] <yehaT> hey The_8472, did you saw my monologue 13 hours ago here. it was about using same pieces of data to participate in multiple torrents/trackers even though parts of the data had differences (like say header of some binary file)
[22:33:47] <The_8472> yes
[22:33:50] <yehaT> now it doesn't automatically solve the fragmentation issue realyu
[22:33:56] <yehaT> but still helpful imo
[22:34:30] <yehaT> (if i even understand what you meant by that)..
[22:34:33] <The_8472> well, finding 100% identical files is one thing, sortof reasonable to implement
[22:34:53] <The_8472> finding files that are only partially identical, having offsets and similar is not worth the effort imo
[22:35:27] <The_8472> fragmentation == multiple torrents pointing at exactly the same content. thus splitting what could be 1 big swarm into several smaller ones and thus shortening their lifecycle
[22:36:24] <yehaT> do you mean that the whole torrent, all files inside, are identical and identical order etc?
[22:36:35] <yehaT> i've to say i've not seen that that often..
[22:36:45] <yehaT> most of what i see is there's that 1% difference
[22:37:18] <yehaT> and it's annoying to either copy the content twice or pick which tracker you want to seed onj
[22:39:37] <The_8472> i've seen that. like different filenames or different piece sizes
[22:40:39] <yehaT> ok now that i think of it i remember seeing that when i used more trackers these days i use less and see more of the 1% thing
[22:41:08] <yehaT> though
[22:41:14] <yehaT> i still think it was often
[22:41:26] <yehaT> that the torrents had 1 exact identical file
[22:41:36] <yehaT> but maybe there was also some text file or something
[22:41:48] <yehaT> that say had difference line endings or whatever
[22:41:58] *** klapaucjusz has joined #bittorrent
[22:42:01] <klapaucjusz> Sorry for that.
[22:42:03] <yehaT> i think you can manually work around that
[22:42:17] <yehaT> today though..
[22:42:17] <klapaucjusz> What was that, The_8472
[22:42:44] <The_8472> |22:32:09| <The_8472> well, i'm trying to reduce any removal
[22:42:44] <The_8472> |22:32:31| <The_8472> so that the routing table survives transient connectivity problems
[22:42:44] <The_8472> |22:32:32| <klapaucjusz> Sorry, need to go.
[22:43:51] <klapaucjusz> Yeah, I solve that by being a very conservative pinger.
[22:44:05] <klapaucjusz> I usually only ping a node when I have a new node to replace it with.
[22:44:24] <klapaucjusz> No new node -> no ping, so no dropping of nodes during connectivity outages.
[22:44:40] <klapaucjusz> Of course, I don't use old nodes to reply to find_nodes queries.
[22:44:47] <klapaucjusz> s/old/stale/
[22:45:05] <The_8472> yep, same. but there are still some code paths that actively remove nodes... i need to weed those out
[22:45:42] <klapaucjusz> Actually, what I'm describing was the original plan.
[22:46:08] <klapaucjusz> Ehm, no, sorry, getting confused.  It's the current behaviour.
[22:46:16] <The_8472> on the other hand, in the non-full buckets some stale nodes might accumulate, i still want to clean those out
[22:46:43] <klapaucjusz> Why?
[22:46:50] <The_8472> aesthetics ^^
[22:47:05] <klapaucjusz> Are you the guy on port 49001?
[22:47:09] <The_8472> yep
[22:47:31] <klapaucjusz> One of us has Teredo problems.
[22:47:41] <klapaucjusz> You can ping me, but I cannot ping you.
[22:47:46] <The_8472> err, i shouldn't be on a teredo node
[22:48:08] <The_8472> it should be using my 6to4 address
[22:48:18] <klapaucjusz> Er, right.  It's 6to4.
[22:48:29] <klapaucjusz> The point is still valid -- I cannot ping you (from Teredo).
[22:48:35] <klapaucjusz> Hold on, I'll try from native...
[22:48:48] <The_8472> then i would blame congestion. i'm spidering the ipv4 DHT atm
[22:49:07] <The_8472> i need to write a proper spider task though
[22:49:17] <The_8472> i mean one that is efficient
[22:49:39] <klapaucjusz> You're reachable from native IPv6.  Bad latency, but reachable.
[22:49:47] <The_8472> excellent
[22:49:50] <klapaucjusz> So it looks like my Teredo relay is FUBAR.
[22:49:56] <The_8472> but yes, the latency is due to congestion
[22:50:20] <The_8472> what are you using as relay? a HE one?
[22:51:23] <The_8472> btw... the conversion for the version field is that the first 2 bytes are a client shorthand in ascii (like AZ, UT, LT) and the next 2 bytes are some binary encoding of the version number
[22:51:23] <klapaucjusz> Teredo choses the relay dynamically.
[22:52:15] <klapaucjusz> Right.
[22:52:53] <The_8472> well, i'm not seeing "TR" in your version ^^
[22:53:09] <klapaucjusz> In transmission, I use TR followed by the big-endian encoding of the Subversion revision number.
[22:53:13] <klapaucjusz> Yuck, subversion.
[22:53:25] <klapaucjusz> In the standalone implementation, I don't send a v key.
[22:53:31] <klapaucjusz> Note to self: should do.
[22:53:31] <The_8472> ah
[22:55:10] <The_8472> 7 nodes in my routing table so far
[22:55:34] <The_8472> not sure if they're of any use though
[22:56:34] <klapaucjusz> Suggest a version prefix for the standalone implementation.
[22:56:36] *** bbelt16ag has joined #bittorrent
[22:57:08] <klapaucjusz> Using my initials.
[22:57:28] * The_8472 suggests using klapaucjusz's initials
[22:58:38] <klapaucjusz> Better?
[22:59:25] <The_8472> sec, have to dump the routing table
[22:59:44] <The_8472> this thing isn't built for instrumenting it
[23:00:13] <klapaucjusz> That's why it's cool to have a standalone implementation :-)
[23:00:29] <The_8472> yep, i'm seeing JC now
[23:00:57] <klapaucjusz> Nodes found (8+5)
[23:01:17] <klapaucjusz> I see you've fixed your aggressivity.  Well done.
[23:01:37] <The_8472> :3
[23:02:21] <klapaucjusz> (That's a new smiley to me.)
[23:03:34] <yehaT> re: finding offset: take file in the new torrent (secondary) with nothing downloaded, download some amount of the smallest (16 kb?) blocks in vicinity of existing data in the old torrent then seq scan through the file in primary torrent trying to match the dozen or so small blocks. if match found then start using the secondarys piece hashes against the primary file data around the matched with the offset of that match.
[23:03:46] <yehaT> i don't see what's the difficult part there.. :)
[23:04:05] <The_8472> yehaT, then implement it in some open source client
[23:04:13] <klapaucjusz> The_8472: are you changing node-id regularly?
[23:04:42] <The_8472> klapaucjusz, the default is on restart of the instance. but there is an option to never change it, which i'm using
[23:04:59] <The_8472> but i just flushed the datastore, so it changed the ID too
[23:05:17] <klapaucjusz> Ah, ok.
[23:05:28] <klapaucjusz> I save it.  Only way to change it, manually rm dht.dat.
[23:06:29] <The_8472> didn't you mention concerns over long-lived nodes becoming too popular?
[23:06:57] <klapaucjusz> Do you have a PGP-signed message to prove it?
[23:07:06] <The_8472> do i need one?
[23:07:25] <klapaucjusz> Fair enough, I did.
[23:07:33] <klapaucjusz> And I still think that some mechanism is needed.
[23:07:42] <K`Tetch> klapaucjusz - I used to run the US Pirate PArty, and then Pirate APrty International, nowadays I mainly do research, such as for TorrentFreak
[23:07:42] <klapaucjusz> Let's see how the dht.wifi.pps.jussieu.fr nodes evolve...
[23:07:45] <The_8472> i suggested changing node IDs on overload
[23:08:12] <klapaucjusz> K`Tetch: how do you sign your articles for TF?
[23:08:27] <K`Tetch> I do the research, not the writing
[23:08:32] <klapaucjusz> Ah, ok.
[23:08:48] <klapaucjusz> What if I want to write an article for TF?  Do they accept submissions?
[23:09:07] <K`Tetch> yes
[23:09:14] <klapaucjusz> Will you put in a word for me?
[23:09:29] <K`Tetch> well, depends what word you want
[23:09:33] <klapaucjusz> Where do I send a proposal?  How many words do they give me?
[23:09:54] <klapaucjusz> Oh, just mentioning that I'm legit and that I know what I'm speaking about.  Makes the editor's life easier.
[23:10:14] <K`Tetch> a lot of the time like that, i go over the piece, and do some checking
[23:10:16] <K`Tetch> or ernesto does
[23:11:02] <K`Tetch> http://torrentfreak.com/contact/ for contact, and generally 3-400 words, although 800 isn't unknown
[23:11:21] <klapaucjusz> Noted, thanks.  Permission to mention we've met on IRC?
[23:11:27] <K`Tetch> sure
[23:11:56] <klapaucjusz> Ta.
[23:12:27] <klapaucjusz> The_8472: you're abusing want.
[23:12:30] <The_8472> hrrm, thoughts on an exhaustive keyspace crawler? i'm thinking of doing a node lookup and then shifting the target key to the right by the distance spanned between the highest and lowest node ID in the result set
[23:12:36] <The_8472> klapaucjusz, how so?
[23:12:42] <klapaucjusz> You're sending want:[n4, n6] on all requests.
[23:12:53] <klapaucjusz> Pointless if one of your tables is doing fine.
[23:13:09] <The_8472> n6 should be the case, not sure why it's doing n4, i'll look into it
[23:13:28] <The_8472> oh
[23:13:35] <The_8472> you're probably seeing bootstrap lookups
[23:13:43] <The_8472> those always send both
[23:13:50] <The_8472> normal ones shoudln't
[23:13:53] <klapaucjusz> Why are you still bootstrapping IPv4?
[23:14:01] <klapaucjusz> IPv4 should be doing fine.
[23:14:15] <The_8472> it's a bootstrap lookup, they're not table-specific
[23:14:51] <The_8472> i'll see how i can fix it.
[23:15:02] <klapaucjusz> if(ipv4_table.getDelegate().getParent().getPrettyNeighbor().getTable().isFine())
[23:15:12] <klapaucjusz> want = WANT_IPv6;
[23:15:13] <klapaucjusz> else
[23:15:18] <klapaucjusz> want = WANT_BOTH;
[23:15:32] <klapaucjusz> (Am I competent in Java programming style yet?)
[23:15:56] <The_8472> you're getting there ^^
[23:16:11] <The_8472> 					fnr.setWant4(rpc.getDHT().getType() == DHTtype.IPV4_DHT || !fillWithAllBuckets || DHT.getDHT(DHTtype.IPV4_DHT).getNode().getNumEntriesInRoutingTable() < DHTConstants.BOOTSTRAP_IF_LESS_THAN_X_PEERS);
[23:16:11] <The_8472> 					fnr.setWant6(rpc.getDHT().getType() == DHTtype.IPV6_DHT || !fillWithAllBuckets || DHT.getDHT(DHTtype.IPV6_DHT).getNode().getNumEntriesInRoutingTable() < DHTConstants.BOOTSTRAP_IF_LESS_THAN_X_PEERS);
[23:17:31] <The_8472> if a name doesn't describe what is hiding behind it it's not long enough ^^
[23:18:09] <The_8472> unless the javadocs (which happen to pop up the very moment i use autosuggest) describe it in detail
[23:18:19] <The_8472> speaking variable names save you a lot of commenting
[23:18:33] * klapaucjusz 's variables are all called a, b, c.
[23:18:39] <The_8472> lies!
[23:18:43] <klapaucjusz> It was difficult to write, there's no reason why it should be easy to read.
[23:18:50] <The_8472> hahahaha
[23:19:14] <The_8472> "yes, it's released under the GPL. no? why sourcecode? we include a decompiler."
[23:20:28] <DeHackEd> we compiled it with -O0 so the decompiler will minimize the mangling
[23:20:45] *** waldorf_ has joined #bittorrent
[23:22:10] <The_8472> <The_8472> hrrm, thoughts on an exhaustive keyspace crawler? i'm thinking of doing a node lookup and then shifting the target key to the right by the distance spanned between the highest and lowest node ID in the result set <-
[23:22:12] *** _rafi2_ has joined #bittorrent
[23:23:39] <klapaucjusz> Hmm, you're aggressive again.
[23:23:49] <klapaucjusz> Not as aggressive as the other time, but still a little aggressive.
[23:24:00] <klapaucjusz> Your bootstrap should only ping the bootstrap nodes once.
[23:24:23] <klapaucjusz> (I.e. once per advertised address.)
[23:24:34] <klapaucjusz> If bootstrap fails, no point trying again before 15 minutes or so...
[23:24:44] <klapaucjusz> give a chance to PORT messages to fix it.
[23:25:02] <The_8472> it's not pinging
[23:25:13] <mpl> hey. question about the bitfield. the specs says it should be the first message sent after the handshake (if sent at all). is it better to send it after I received the other peer's bitfield, or before?
[23:25:13] <klapaucjusz> It's pinging with find_node messages.
[23:25:16] <The_8472> pings sent: 1 timed out: 1
[23:25:42] <The_8472> those are bootstraps, every 2 minutes
[23:25:55] <klapaucjusz> That's *way* too aggressive.
[23:26:03] <The_8472> it's not
[23:26:12] <The_8472> bootstrap nodes get displaced once the bucket is full
[23:26:26] <The_8472> then they'll not be bothered again until the client is restarted
[23:26:30] <klapaucjusz> If there's a mis-configured firewall in the way, you abuse the bootstrap nodes.
[23:26:42] <The_8472> how so?
[23:26:44] <klapaucjusz> (Think stupid firewall blocks all incoming UDP.)
[23:26:56] <klapaucjusz> (...but allows outgoing UDP.)
[23:26:58] *** GTHK has joined #bittorrent
[23:26:59] <The_8472> doesn't matter, it can keep the tables full even without incoming UDP
[23:27:15] <klapaucjusz> No, not *unsolicited* UDP.  All *incoming* UDP.
[23:27:36] <The_8472> such things exist?
[23:27:45] <DeHackEd> I hope not. DNS breaks completely...
[23:27:50] <The_8472> indeed
[23:29:34] <The_8472> klapaucjusz, i'm parametrizing things for the steady state (i.e. as observed on the IPv4 DHT). some rough edges may appear as long as there are too few v6 nodes, but i hope that'll be over soon
[23:29:54] <klapaucjusz> The_8472: yes, they do.
[23:30:08] <klapaucjusz> DeHackEd: since when are the security drones concerned about breaking things?
[23:30:35] <klapaucjusz> The_8472: that's pretty common experience with NTP servers.
[23:30:35] <DeHackEd> when they break HTTP?
[23:30:44] <klapaucjusz> No, they just tell you to use their proxy.
[23:31:08] <The_8472> well, then you can't use a bittorrent client behind such a firewall anyway
[23:31:10] <klapaucjusz> Or they only allow UDP on port 53.
[23:31:17] <klapaucjusz> Eh?
[23:31:24] <klapaucjusz> BitTorrent works fine with no UDP.
[23:31:47] <DeHackEd> well, if the tracker is by IP and not hostname. but assuming a sane DNS server usually works.
[23:31:51] <The_8472> people who lock down UDP and tell you to use a proxy will block TCP too non-http ports (or maybe even HTTP) too
[23:32:15] <The_8472> *to non-http
[23:32:41] <klapaucjusz> The_8472: I'm really not interested in discussing the creative ways of breaking the Internet that the security drones have devised over the years.
[23:32:57] <klapaucjusz> The point is: if you can conceive of it, it's being done by a security drone somewhere.
[23:33:10] <klapaucjusz> If you cannot conceive of it, it's still being done by a security drone somewhere.
[23:33:18] <The_8472> probably, the question is if it's relevant
[23:33:39] <klapaucjusz> You, The_8472, are not supposed to break the DHT, even if you find yourself behind a firewall with a completely mad configuration.
[23:34:10] <The_8472> not planning on doing so
[23:34:41] <klapaucjusz> Then please make sure that your code is not unduly aggressive even when behind a firewall like the one I've described.
[23:34:59] <The_8472> now there is a leap in your logic
[23:35:20] <klapaucjusz> Okay, I'll put it differently.
[23:35:24] <klapaucjusz> Please humour an old man.
[23:35:30] <klapaucjusz> Make sure that your code is not unduly  aggressive even when behind a firewall like the one I've  described.
[23:35:31] <The_8472> just because such a firewall may exist somewhere and then there is a miniscule chance one of my plugins running there doesn't mean it would break the entire DHT
[23:36:01] <DeHackEd> nor will it go ballistic and anger network admins (above and beyond running BT)
[23:36:07] <klapaucjusz> Find node!
[23:36:08] <klapaucjusz> Sending closest nodes (3).
[23:36:08] <klapaucjusz> Find node!
[23:36:08] <klapaucjusz> Sending closest nodes (3).
[23:36:08] <klapaucjusz> Find node!
[23:36:10] <klapaucjusz> Sending closest nodes (2).
[23:36:13] <klapaucjusz> Find node!
[23:36:15] <klapaucjusz> Sending closest nodes (3).
[23:36:17] <klapaucjusz> Find node!
[23:36:20] <klapaucjusz> Sending closest nodes (3).
[23:36:21] <The_8472> that's not realtime
[23:36:23] <klapaucjusz> Find node!
[23:36:25] <klapaucjusz> Sending closest nodes (3).
[23:36:28] <klapaucjusz> That's you.
[23:36:30] <The_8472> that's not realtime
[23:36:37] <klapaucjusz> No, it's not realtime, but it's the last 12 entries of my log.
[23:37:08] <The_8472> sooo? as i said, every 2 minutes
[23:37:10] <klapaucjusz> About 3/4 of the traffic in the IPv6 DHT are due to your single node.
[23:37:21] <klapaucjusz> Gosh, The_8472.
[23:37:38] <The_8472> which will become insignifiacant once there are other nodes
[23:37:42] <klapaucjusz> Every 2 minutes you send a ping to every bootstrap node.
[23:37:49] <The_8472> NOW
[23:37:50] <The_8472> not THEN
[23:37:54] <The_8472> *sigh*
[23:38:12] <klapaucjusz> If there's a mere 1000 nodes using your implementation behind broken firewalls,
[23:38:21] <klapaucjusz> That's 10 packets a sec.
[23:38:25] <The_8472> which is nothing
[23:38:32] * klapaucjusz gives up
[23:38:42] <The_8472> TPB gets 10k+ packets per second
[23:39:16] *** _rafi_ has quit IRC
[23:39:36] <The_8472> sometimes i'm getting more than that from stray portscans hitting my IP ...
[23:39:39] <The_8472> seriously...
[23:40:54] <The_8472> although i'll probably change it to pick one bootstrap node at random instead of all of them
[23:40:56] * klapaucjusz will stop speaking about efficiency to a Java programmer.
[23:41:33] <The_8472> responsiveness is a form of efficiency too ;)
[23:41:34] * klapaucjusz merely notes that there are 10 million nodes in the IPv4 DHT.
[23:41:56] <The_8472> most of which are not behind a broken firewall
[23:43:05] <The_8472> anyway, if real problems like that occur i'll fix them
[23:43:20] <The_8472> but as long as it serves to get things up and running fast without doing much harm it stays
[23:43:33] <The_8472> things can be made less aggressive once it works
[23:43:37] <The_8472> it doesn't work right now
[23:44:00] <The_8472> goal #1: get it work
[23:44:04] <klapaucjusz> It will work as soon as I get the IPv6 DHT into Transmission.
[23:44:05] <The_8472> goal #2: play ball
[23:44:55] <The_8472> + getting distro maintainers to deploy it
[23:45:00] <The_8472> they're a slowmoving bunch
[23:45:24] <klapaucjusz> Transmission has nightly builds.
[23:45:34] <klapaucjusz> They're pretty popular with the Mac OS crowd.
[23:45:43] <The_8472> ah, ok
[23:47:27] <klapaucjusz> d1:eli202e12:Server Errore1:t4:fn..1:y1:ee
[23:47:33] <klapaucjusz> Sever Error?
[23:47:37] <klapaucjusz> Eek.
[23:48:03] <The_8472> hrm?
[23:48:37] <The_8472> did i send that?
[23:49:38] <klapaucjusz> No.
[23:49:42] <klapaucjusz> Somebody sent me that.
[23:49:48] <klapaucjusz> In reply to a find_nodes.
[23:50:04] <klapaucjusz> (Someone confused by my want flag?)
[23:50:09] <The_8472> 202... server error?
[23:50:19] <The_8472> that's an error msg
[23:50:33] * klapaucjusz is grateful for the explanation ;-)
[23:50:43] <The_8472> i thought they're supposed to have a human-readable message
[23:50:45] <The_8472> as error reason
[23:51:27] <DeHackEd> I think the spec says that you can't include anything else either... just as a technicality...
[23:51:57] <klapaucjusz> ?
[23:52:02] * klapaucjusz is confused
[23:52:34] <klapaucjusz> DeHackEd: what do you mean?
[23:52:34] <DeHackEd> the spec does define a "failure reason" string... which is supposed to be just that...so, yeah.
[23:52:43] *** KyleK_ has quit IRC
[23:52:54] <The_8472> those are tracker announces
[23:52:57] <The_8472> not DHT error messages
[23:53:35] <DeHackEd> they're still bencoded?
[23:53:53] <The_8472> everything is bencoded
[23:54:03] <klapaucjusz> The_8472: are you sure you only send known-good nodes in find_nodes replies?
[23:54:04] <DeHackEd> okay, I'm gonna shut up and go somewhere else.
[23:54:13] <klapaucjusz> What's your definition of known-good?
[23:54:17] <klapaucjusz> DeHackEd: don't go!
[23:55:02] <The_8472> klapaucjusz, seen them within the last 15 minutes.
[23:55:10] <klapaucjusz> Seen?
[23:55:27] <The_8472> if buckets are not full then reachability is not necesarily guaranteed
[23:55:41] <klapaucjusz> Because it really looks like you're sending me stale nodes.
[23:55:49] <The_8472> maybe they're not reachable
[23:56:05] <klapaucjusz> No, they're dead.  Dead as a parrot.
[23:56:16] <klapaucjusz> 2757 seconds old.
[23:57:21] <The_8472> address or node ID?
[23:57:47] <klapaucjusz> 5eeed30efe9f182279e25e25a5cd2a74411bbd5c
[23:58:04] <klapaucjusz> 5da9f3c7eadfaf75e7aa685449ce934d7d3031d9
[23:58:28] <The_8472> 5d is not even in my routing table
[23:58:37] <The_8472> 5e is though
[23:58:51] <klapaucjusz> It's been dead for a long time.





top