[00:02:57] *** chelz has joined #bittorrent
[00:12:30] *** kwinz2 has quit IRC
[00:20:05] *** andar2 has quit IRC
[00:22:45] <chelz> is there any study or work out there on the topic of having two different torrents, with different hashes and different swarms, downloading to the same set of files?
[00:24:20] <The_8472> there are some related studies, but they focus on how to "fix" that problem with some dynamic hashing and set matching techniques
[00:25:02] <chelz> so pieces being grabbed from one of the swarms wouldn't need to be downloaded from the second. it might get tricky if the two torrents have different piece sizes but that wouldn't make it impossible.
[00:25:32] <The_8472> but if you KNOW that the torrents describe the same thing (although you can't really have software verify it due to mismatching hashes) you can just implement it in the clients that the torrents inform each other about the progress of files on byte-offset basis instead of piece basis
[00:26:05] <The_8472> azureus has some very limited support for that kind of thing
[00:26:35] <choykloun> just got home from a wonderful new years party
[00:26:48] <choykloun> time for some dht routing data structure implementation \o
[00:28:56] <chelz> The_8472: is that Az work accessible in the client somehow or just still code in the codebase?
[00:30:36] <The_8472> it's accessible in the client
[00:31:04] <The_8472> http://wiki.vuze.com/index.php/Multiple_torrents_accessing_shared_file%28s%29
[00:31:14] <chelz> awesome! thanks
[00:33:50] <chelz> that's a pretty clever way of going about it
[00:34:06] <The_8472> making it event-driven would be even more clever
[00:34:41] <chelz> hmm
[00:34:41] <chelz> The_8472: do you and other devs usually use the vuze interface thing or just the classic?
[00:35:31] <The_8472> i use the classic interface, idk about the others
[00:36:09] <choykloun> hm am i assuming correct that most lookups for keys in dht will be based on distance ?
[00:36:48] <The_8472> huh?
[00:37:15] <choykloun> for most queries received, the client will look up the proper nodes based on distance
[00:37:21] <choykloun> and not based on the nodeid per se
[00:37:53] <The_8472> distances are always calculated relative to a target
[00:38:07] *** alienvenom has quit IRC
[00:38:25] <The_8472> so other nodes send you the target ID, you deliver the closest nodes in your routing table relative to that target
[00:38:34] <choykloun> yes yes
[00:38:57] <choykloun> i suck at explaining stuff somtimes
[00:39:13] <choykloun> but yeah i guess i'll optimize for the distance case and not the totally random node case then
[00:39:26] <The_8472> i don't see the difference
[00:39:33] *** alienvenom has joined #bittorrent
[00:39:56] <choykloun> total number of XORs and/or comparisions done
[00:40:02] <The_8472> lol
[00:40:15] <choykloun> you know me well enough by now to have expected that
[00:40:23] <choykloun> :)
[00:41:57] <The_8472> should only need a couple of dozen XORs even with a sloppy implementation
[00:42:26] <choykloun> ya but couple of dozens are more than zero
[00:42:37] <choykloun> also it will involve an additional memory access !
[00:42:42] <choykloun> and pipeline penalties !
[00:42:44] <The_8472> and nobody will care
[00:42:57] <choykloun> nope
[00:43:10] <The_8472> my CPU is twiddling thumbs, even when i throw 1080p videos in h.264 at it.
[00:43:54] <choykloun> besides i havent written good large data structures in a while
[00:44:02] <choykloun> i need to do stuff like this to keep sane from all the php at work
[00:44:03] <The_8472> it's fixed size :P
[00:44:35] <choykloun> i'll store every node i encounter
[00:44:37] <choykloun> ewll
[00:44:47] <choykloun> up to 8 per bucket
[00:45:06] <The_8472> you won't need lookups into those lists
[00:45:14] <The_8472> you only need lookups into your actual routing table
[00:45:15] <choykloun> but distance isn't just calculated from the bit
[00:45:23] <choykloun> i want one structure to do it all
[00:45:30] <choykloun> see above about sanity
[00:47:37] <choykloun> and its reusable for other projects ..
[00:50:26] <choykloun> and why not use it to make nice graphs of the dht space :p
[00:51:50] *** MassaRoddel has joined #bittorrent
[00:54:26] *** JudgeSHAD0W has quit IRC
[00:57:05] <choykloun> and basically i just need to care about cpu pipelines instead of gantt charts some days :p
[01:25:16] *** Switeck has joined #bittorrent
[01:41:18] <choykloun> btw regarding the earlier idea about using the 160 bit node id for public key, why not just append the missing 12 bytes somewhere else
[01:41:41] <choykloun> would be a cool idea actually
[01:41:59] <choykloun> would provide a notable but not secure authentication of the key holder also
[01:42:14] <choykloun> as its not that computationally easy to generate a matching pubkey
[01:42:59] <choykloun> or just make it a hash of the key and dont really care about those 12 extra bytes :)
[01:45:53] *** PN has joined #bittorrent
[01:46:15] <choykloun> so much really cool stuff can be done here
[01:47:38] *** mxs has quit IRC
[01:51:09] <choykloun> how many nodes does an average client talk with on a regular basis?
[01:52:15] *** ProperNoun has quit IRC
[01:52:51] *** mxs has joined #bittorrent
[01:56:08] *** trovanon has joined #bittorrent
[01:58:10] *** trovanon has quit IRC
[02:06:16] <The_8472> that's hard to say. they basically talk to everyone in their routing table and then they should receive regular traffic from those nodes that have them in the other's routing table
[02:06:42] <The_8472> regular means that you'll see it more than once within 30 minutes ^^
[02:07:28] <The_8472> anyway, i never counted unique nodes seen.
[02:07:42] <The_8472> during regular DHT operation at least
[02:11:48] <choykloun> was thinking of the feasability of pki stuff
[02:19:11] <chelz> i'm not really sure there's an advantage to integrating pki.  besides something like a decentralized signed trusted feed, possibly related or similar to rss
[02:21:29] <The_8472> PKI is pointless in the DHT itself since there are practically no sources of trust, only sinks
[02:23:45] <The_8472> encryption in general could protect against a few attacks, i.e. force attacks to intercept large amounts of the traffic, but if they're willing to do that we could hardly defend against that
[02:24:17] <The_8472> anyway, there are more important security issues that could be dealt with without using encryption.
[02:24:41] *** bt42 has joined #BitTorrent
[02:25:46] <choykloun> ya
[02:25:51] <choykloun> i'm just philosophizing
[02:26:03] <choykloun> and pki stuff would more be a specific option than something built-in
[02:26:17] <choykloun> but encryption/obfuscation definitely should go mainstream
[02:32:40] *** bittwist has quit IRC
[02:36:02] <chelz> well taking pki+dht to one extreme would be to create some kind of public ratio-enforcement system, similar to what private trackers are used for currently.
[02:37:19] <chelz> a signed reputation system so people who seed a lot get faster speeds or something. although it's a lot of work and a ton of overhead for not really any gain over existing private trackers, besides being decentralized.
[02:37:46] <chelz> the hydra nature of trackers/indexers alone seems to make up for that
[02:38:44] <choykloun> can be used for lots of stuff
[02:38:48] <choykloun> like secure file transfers
[02:42:55] <choykloun> and it could be used for the perfect darknet
[02:43:00] <choykloun> disappear totally in the crowd
[02:43:49] <choykloun> plus, anything involving millions of clients, decentralized networking, and crypto is per definiton worth exploring :)
[02:52:35] <chelz> yeah hmm that makes me wonder if anyone has worked on making a WASTE-like client for the Tor network
[02:53:33] <chelz> something designed to fit well with the hidden services setup of Tor. similar to how Torchat fits with Tor.
[02:54:16] <chelz> OneSwarm and I2P-enabled torrent clients are steps in the direction of anonymous filesharing
[02:55:08] <choykloun> im experimenting with using dht to get entry nodes for tor
[02:55:45] <chelz> oh that's interesting
[02:56:09] <choykloun> also apparently im resonsible for tor's new networking layer that will obfuscate the traffic
[02:56:18] <choykloun> hm maybe that can be combined
[02:56:23] <chelz> i assumed they handled bootstrapping similarly to how Kademlia does it
[02:56:26] <The_8472> that sounds a lot like I2P, from the design aspect i consider I2P far more interesting than tor
[02:56:27] <choykloun> gets keys from the dht lookup
[02:56:46] <swolchok> just don't tell roger about bittorrent over tor
[02:56:52] <swolchok> he doesn't really like that
[02:57:08] <choykloun> heh
[02:57:32] <chelz> choykloun: how are entry nodes acquired if not through the tor network?
[02:57:38] <choykloun> well the head dev of tor has told me personally that it simply wouldnt work
[02:57:44] <choykloun> chelz: external sources
[02:58:00] <choykloun> usually included in the distro
[02:58:05] <Switeck> only part of BitTorrent that should ever be done even partially over tor is tracker updates...
[02:58:10] <choykloun> and it just uses standard ssl
[02:58:13] <chelz> such as.. perhaps web servers?
[02:58:14] <Switeck> and then only as an absolute last resort.
[02:58:20] *** burris has quit IRC
[02:58:26] <swolchok> chelz: P2P reputation doesn't work, see "The Sybil Attack"
[02:58:27] <choykloun> chelz: such as .. http://torir.org ?
[02:58:47] *** burris has joined #bittorrent
[02:59:02] <chelz> oh so an included list
[02:59:46] <chelz> swolchok: thank you very much for that recommendation. i'm reading about it now.
[03:03:53] <swolchok> chelz: it's a disappointing read because distributed reputation is an intuitive and nice idea.
[03:04:54] <chelz> swolchok: well it's a clear exploration of the issue, which up until now i didn't know there was one. the more the problem is understood the more it can be mitigated, even if it can't be prevented entirely.
[03:07:39] <chelz> i'm not really sure that people would try to cheat even if it was possible and fairly easy. an example is how there are several examples BitTorrent clients that can get marginally faster speeds by tweaking/cheating the protocol, but they're not widely used at all
[03:07:54] <chelz> also leecher clients for emule and similar networks
[03:08:08] <chelz> http://en.wikipedia.org/wiki/Sybil_attack#Prevention
[03:10:03] <swolchok> the central authority thing doesn't work for "public" networks
[03:10:26] <swolchok> the social network stuff works as far as it goes, but again doesn't work for public networks
[03:14:43] <chelz> hmm it is an interesting issue.
[03:18:18] <choykloun> anyway dht would provide excellent cover for people actually needing protection
[03:19:02] <choykloun> and lets get the damn thing obfuscated before its put right to bittorrent in chinas filters
[03:22:50] *** kwinz2 has joined #bittorrent
[03:29:09] *** Switeck has quit IRC
[03:37:30] <swolchok> obfuscation is just going to get you in an arms race
[03:38:06] <swolchok> there is always the option of whitelisting traffic if a major government or ISP really wanted to suppress BT
[03:39:14] <choykloun> then we make bt look like [insert favorite protocol here]
[03:47:17] *** kwinz2 has quit IRC
[03:55:15] <The_8472> <swolchok> there is always the option of whitelisting traffic if a major government or ISP really  <- oh, it's not an option. china is already planning to whitelist websites in the coming year instead of blacklisting them.
[03:59:04] *** kwinz2 has joined #bittorrent
[03:59:34] <alus> The_8472: that's going to make it rather hard to get a new website to be popular...
[04:07:25] *** The_8472 has quit IRC
[04:08:14] *** The_8472 has joined #bittorrent
[04:12:25] <choykloun> well, now all new routing is finished.. now i just need to actually make queries use it
[04:12:33] <choykloun> and use it in the way you are supposed too also
[04:12:53] <choykloun> even though ddosing of patrikweb.net was considered a feature by many :)
[04:25:00] *** init0_ has joined #bittorrent
[04:36:08] *** init0 has quit IRC
[04:37:50] <chelz> The_8472: i heard about that but i'm pretty sure it's only regarding sites using chinese TLDs
[04:40:22] *** rrr has quit IRC
[04:41:22] <chelz> swolchok: i was thinking about it and i think a p2p reputation system could work with certain goals. i think as long as the goal isn't to find an objective view of the system and that each person has one other person to really trust, a proper web of trust/reputation can be formed
[04:54:36] *** bt42 has quit IRC
[05:08:36] *** kwinz2 has quit IRC
[05:12:02] <choykloun> theres no natural law saying web of trust cant work
[05:15:54] *** kwinz2 has joined #bittorrent
[05:35:25] <The_8472> actually, there is
[05:36:45] <The_8472> the byzantine generals problem
[05:38:15] <The_8472> well, not exactly. but it forces you to have a source of trust. which i don't really see in a DHT
[05:42:17] *** rrr has joined #bittorrent
[05:56:27] <choykloun> is it customary to ping a node first when contacting it
[05:56:32] <choykloun> or just find_node directly
[06:14:03] <choykloun> the_8472: of course you'd configure trust yourself
[06:49:58] *** MassaRoddel has quit IRC
[09:01:40] *** _rafi_ has joined #bittorrent
[09:13:54] <chelz> "A reputation system's vulnerability to a Sybil attack depends on how cheaply Sybils can be generated, the degree to which the reputation system accepts input from entities that do not have a chain of trust linking them to a trusted entity, and whether the reputation system treats all entities identically."
[09:14:41] <chelz> all of those points just need to be taken into account and a system should do pretty well. might be limited or slow in some ways, but it would work.
[09:19:59] *** goussx has joined #bittorrent
[09:41:26] *** norc_ has joined #bittorrent
[09:41:39] <norc_> Can somebody layout the difference between announce and scrape requests to me?
[09:42:53] <norc_> I just don't really get the necessity of scrape ;)
[09:46:57] <norc_> The way I see it, it just serves for frequent statistical updates on the screen of the client (possibly to make it aware of changes in the swarm)
[09:49:54] <chelz> sometimes there's a clear need for requesting info about a swarm and being clear that the requesting party doesn't want to join
[09:50:08] <chelz> an indexing site querying stats on a torrent it has for example
[09:51:04] <norc_> So it might be for clients to determine the statistics of several torrents, to check whether it may be needed (and thus makes an announce in that case) in case of low seeds?
[09:51:21] <norc_> (The indexing part is not my concern here)
[09:52:27] <norc_> chelz, I think I get the basic idea
[09:53:07] <chelz> yeah, a client knowing if it should join back into a queued torrent if seeds are needed is another good example
[09:56:07] *** goussx has quit IRC
[09:56:11] *** goussx_ has joined #bittorrent
[09:56:21] *** goussx_ is now known as goussx
[10:02:34] *** goussx has quit IRC
[10:07:35] *** burris has quit IRC
[10:13:37] *** burris has joined #bittorrent
[10:43:12] *** MassaRoddel has joined #bittorrent
[10:56:06] <chelz> are there any other ways to make a server/hosting system resilient besides making it decentralized?
[11:16:16] <norc_> rent security?
[11:24:28] <chelz> hah
[11:24:53] <chelz> yeah it seems the only answer is some form of decentralization, ie implied redundancy
[11:25:27] <alus> nah. cannons
[11:41:36] *** ania has joined #bittorrent
[11:47:05] *** ania has quit IRC
[12:23:52] *** norc__ has joined #bittorrent
[12:44:08] *** norc_ has quit IRC
[13:01:09] *** bittwist has joined #BitTorrent
[13:24:50] *** twanny796 has joined #bittorrent
[13:25:29] <twanny796> I want a simple bittorrent client to run on linux - no gui?
[13:26:17] <norc__> How simple shall it be?
[13:26:40] <chelz> rtorrent, transmission cli, bittornado
[13:26:44] <norc__> ncurses? no ncurses? cheap? expensive? red?
[13:27:05] <chelz> i use rtorrent
[13:27:14] <norc__> chelz, know what I just figured by the way?
[13:27:49] <norc__> I'll write my own tracker protocol, and let users set up local proxies that decode the transmissions and serialize them into bencoded format
[13:27:56] <norc__> This way I wont have any pain tracker side-wise.
[13:29:07] <norc__> Just a couple bytes of a tcp connection
[13:29:17] <norc__> no expensive bencoding needed and reduced bandwidth.
[13:32:41] <chelz> hmm i don't really get the rationale of a lot of what's done, but i'm pretty sure there was a reason things were bencoded
[13:32:56] <chelz> and the 'local proxy' sounds like it should be part of a torrent client
[13:33:05] <chelz> if it's a good idea n all
[13:34:09] <norc__> nope, it'll be just a local tcp server
[13:34:16] <norc__> acting as a tracker
[13:34:29] <norc__> listening on localhost
[13:34:44] <norc__> so the torrent files will have to be modified accordingly
[13:34:44] <norc__> the problem with bencoding is:
[13:34:49] <norc__> 1. It's over HTTP (WTF?)
[13:35:01] <norc__> 2. Bencoding is like the worst serialization format in existence
[13:35:10] <norc__> (It's neither human readable, nor nicely parsable)
[13:35:30] <norc__> 3. Bdecoding is expensive in terms of cpu cycles
[13:35:40] <alus> it's over http because it's easy. see: xmlrpc, jsonrpc, etc
[13:35:54] <alus> bencoding is space efficient, and relatively human readable
[13:36:01] <alus> and so much better than json or xml for this task
[13:36:10] <alus> imaging having to escape characters
[13:36:19] <norc__> Yeah, because bram seems to be incapable of doing anything properly
[13:36:46] <alus> which serialization format would you prefer?
[13:36:52] <norc__> none at all.
[13:37:02] <norc__> For the torrent files yaml would suffice
[13:37:04] <twanny796> source for rtorrent
[13:37:06] <alus> yaml!
[13:37:07] <twanny796> ?
[13:37:12] * alus laughs at norc__
[13:37:34] <norc__> It's human readable, easy to parse, and libraries for yaml exist in most languages.
[13:37:45] <norc__> But that's besides the point
[13:37:56] <norc__> I'm just talking about the announce/scrape bencoding part.
[13:38:11] <alus> also slow as hell, has multiple ways to encode the same thing, and has some ambiguous structures
[13:38:25] <norc__> Like speed really matters for the torrent files.
[13:38:54] <alus> could you look at your #3 up there?
[13:39:01] <norc__> Besides the bencoding format is pretty much retarded and inconsistent
[13:39:09] <alus> inconsistent how?
[13:39:45] <norc__> Like an integer must be prefixed with i, strings with the length (wtf?) and a colon (even more wtf?)
[13:40:30] <alus> length prefixing is the way to go for strings on the wire, in my book
[13:40:39] <alus> escaping is a pain in the ass
[13:41:02] <norc__> In any case, my points (including #3) were for the scrape/announce protocol.
[13:41:34] <norc__> I mean it's nice for all those retarded php developers who don't understand anything else
[13:41:45] <norc__> But in terms of anything else it's just pointless
[13:41:53] <alus> ok, how would you serialize a bunch of hashes, numbers, and compact IP/port data?
[13:41:59] <norc__> Not at all.
[13:42:13] <alus> that doesn't mean anything
[13:42:21] <alus> you need to write the bytes on the wire. how would you do that?
[13:42:29] <norc__> You don't need hashes
[13:42:37] <alus> you don't need torrent identifiers?
[13:42:59] <norc__> one second
[13:44:27] <norc__> Just make it like first 20 bytes for the info hash, 2 bytes for the port the client is listening on, one byte as a bitfield for the state
[13:44:44] <norc__> doesnt matter
[13:44:48] <norc__> just some form of convention
[13:44:55] <alus> the announce protocol is in url query args, not bencode
[13:45:14] <alus> the response is bencoded
[13:45:14] <norc__> Point is
[13:45:21] <norc__> no need for that either
[13:45:33] <chelz> norc__: in terms of getting users to adopt something, they're much more likely to update their client than to install a local tcp server
[13:45:37] <alus> and can contain optional keys, like 'warning message' and other stuff
[13:45:46] <norc__> just set up a tcp server, send the request in binary form over, give a response packed in a binary format.
[13:45:46] <norc__> And be done with it.
[13:45:55] <alus> ok..
[13:46:01] <alus> bencode is packed binary format
[13:46:04] <norc__> chelz, yeah and getting them to use such a protocol is impossible
[13:46:09] <norc__> alus, it is not.
[13:46:16] <norc__> bencode is serialization which is the opposite.
[13:46:21] <alus> uh
[13:46:35] <alus> you maybe should look up
[13:46:37] <alus> "serialization"
[13:47:07] <alus> that's my project for you, for 2010
[13:48:17] <norc__> chelz, such a change must come from a large bittorrent tracker or a popular client to become widely accepted
[13:48:22] <norc__> (or from Bram)
[13:49:03] <alus> no, changes from Bram do not make them popular
[13:49:04] <chelz> or a bep
[13:49:18] <norc__> alus, semantics. What I was referring to was human readable serialization then
[13:49:20] <chelz> i'm pretty sure promoting good ideas is the purpose of beps
[13:49:23] <alus> but BT Inc does work on one of the most popular BT clients (if not the most)
[13:49:36] <alus> chelz: yes!
[13:49:46] <norc__> in any case, bencoding for tracker communication is pretty much pointless
[13:49:54] <norc__> ESPECIALLY the part when it goes over HTTP.
[13:49:59] <alus> ok, you can keep saying it, but you're not correct about that
[13:50:16] <alus> take a look at the train-wreck that is the UDP tracker spec
[13:50:26] <chelz> http://torrentfreak.com/thunder-blasts-utorrents-market-share-away-091204/ - latest stats on client markestshare that i know of
[13:50:26] <norc__> well, give me one argument why bencoding is *good* in its task
[13:50:44] <norc__> considering its slow, bloated, and oversized big in terms of bandwidth consumption
[13:50:59] <alus> you need a serialization format, or you end up with broken crap like the UDP tracker spec.
[13:51:14] <alus> becoding and decoding is not slow enough to matter on the server or client side
[13:51:30] <alus> fast tracker implementations bump up against TCP-handling speed in the kernel first
[13:51:32] <norc__> the server side? sure./
[13:51:33] <chelz> wait, this might be painfully obvious but, why doesn't plaintext work?
[13:51:51] <norc__> If you get hammered with over 5,000 requests per second you'd wish you better not bencode and bdecode them.
[13:52:07] <alus> chelz: the data structure is key/value, which some binary data. you need a way to serialize that which is parsable
[13:52:15] <norc__> chelz, bencoding is like plaintext, just worse.
[13:52:39] <chelz> oh
[13:52:39] <norc__> oh right, nevermind the bdecoding part ;)
[13:53:12] <alus> norc__: first, request are not in bencode, so maybe learn that. second, 5k rps is slow and if bencoding matters you have probably failed to write a bencoder
[13:53:12] *** twanny796 has quit IRC
[13:53:49] <norc__> Oh great. So we just pointlessly generate strings, because it doesn't matter?
[13:54:09] <alus> norc__: the great thing about bencode, is that creating a bencoded message does not take time proportional to the number of bytes you are encoding. this is not true of xml, json, or yaml
[13:54:14] <norc__> put a http header over it
[13:54:17] <norc__> and send it over the network
[13:54:31] <norc__> just so the http header is COMPLETELY ignored, the string is then torn apart again
[13:54:57] <norc__> Im not suggesting xml, json, yaml or any other human readable serialization format.
[13:55:08] <norc__> Imagine the peer wire protocol was bencoded
[13:55:12] <norc__> Would that be cool too?
[13:55:16] <alus> http is a human readable serialization format
[13:55:22] <norc__> After all, it would be human readable!
[13:55:32] <alus> sure, it could use pure bencode.
[13:55:40] <alus> however, http load balancers are nice
[13:55:56] <alus> also https
[13:56:10] <alus> also the http response codes are useful
[13:56:30] <norc__> heh.. you're not listening to yourself are you?
[13:56:51] <norc__> I really cannot see how anyone could possibly consider bencoding over http as a sane way of dealing with these things.
[13:57:10] <chelz> i do like the idea of building on existing standards and getting the benefit of existing work/software/tools
[13:57:15] <alus> norc__: I sure am. http is a reasonable transport, and the body is used for application data. hooray.
[13:57:33] <chelz> norc__: what would you propose?
[13:57:43] <alus> norc__: unless you feel like adding bittorrent fields to TCP, which is not entirely unlike the UDP tracker spec
[13:57:52] <norc__> chelz, some random binary format over pure tcp. No other layers in between.
[13:58:03] <alus> norc__: in which case, why use TCP?
[13:58:16] <alus> norc__: http://bittorrent.org/beps/bep_0015.html
[13:58:25] <norc__> Because it's a fairly common standard.
[13:58:47] <alus> norc__: but what is the benefit over UDP?
[13:58:52] <norc__> I read about UDP as a tracker protocol, which is pretty retarded in my opinion
[13:58:59] <alus> norc__: why?
[13:59:20] <norc__> connectionlessity is a useless feature there. I rather incur a little overhead to protect against package loss.
[13:59:39] <alus> norc__: you clearly have not operated a high-volume tracker.
[14:00:07] <norc__> And yet you claim that HTTP with bencoding ontop of it is a good way to go and by no means a bottleneck.
[14:00:10] <alus> norc__: giving up bencoding and keeping TCP would get you no-where in terms of performance
[14:00:23] <alus> norc__: the bottleneck is not http or bencode
[14:02:12] <alus> norc__: so you're trying to optimize away flexibility for no reason, which is exactly what the UDP tracker spec did, but you're not even solving the real issue in the meantime
[14:04:32] <norc__> I really don't see how I'm giving away any flexibility at all ;)
[14:05:05] <alus> norc__: then read about the UDP tracker spec, and about how it is inflexible
[14:05:47] <alus> norc__: when you've done that, try to come up with a serialization format which is not inflexible
[14:05:56] <alus> norc__: when you've done that, compare it against bencode
[14:06:05] <alus> norc__: when you've done that, propose it
[15:10:23] <choykloun> hey is it proper to send a ping query to a node just addeed (i.e. when bootstrapping) ?
[15:10:29] <choykloun> or find_node directly
[15:40:25] <MassaRoddel> node ids change, you cannot be sure that ids which you got from find_nodes are still valid
[15:44:40] <MassaRoddel> therefor, you might get unexpected (useless) results if you send find_node immediately
[16:06:26] <choykloun> thanks
[16:06:41] <choykloun> wow, my new proper routing code seems to actually work :)
[16:06:58] <choykloun> old was ehm not to standards
[16:07:43] *** rrr has quit IRC
[17:52:25] *** GTHK has quit IRC
[17:52:30] *** GTHK has joined #bittorrent
[17:54:26] *** Gottaname has joined #bittorrent
[18:11:49] <The_8472> <choykloun> or just find_node directly <- that. pinging is mostly for routing table maintenance. or when you want to learn its node ID because you only have ip+port of that node and no ID
[18:13:11] <The_8472> <norc__> I'll write my own tracker protocol, and let users set up local proxies that decode the transmissions and serialize them into bencoded format <- lol ^^
[18:13:36] <The_8472> |13:35:06| <norc__> 2. Bencoding is like the worst serialization format in existence
[18:13:36] <The_8472> |13:35:15| <norc__> (It's neither human readable, nor nicely parsable)
[18:13:52] <The_8472> it's human-readable if you get used to it and it's parseable even with a single-pass parser
[18:13:59] <The_8472> i.e. no backtracking required
[18:16:11] <norc__> The_8472, sure, binary formats are human readable too.. if you get used to it. ;)
[18:16:31] <The_8472> binary formats require you to display them in hex or whatever
[18:16:45] <The_8472> bencoding is human-readable even if you display it in ascii or utf8
[18:16:54] <The_8472> it's not human-friendly, but human-readable
[18:17:51] <norc__> Hah!
[18:18:47] <The_8472> but humans are only supposed to read it for debugging purposes, i.e. devs or powerusers. therefore it's not a problem
[18:19:15] <DWKnight> for the most part, the people that need to read it are able to
[18:21:12] <MassaRoddel> there are ppl who can actually read utf8 encoded chinese?
[18:21:43] <The_8472> <MassaRoddel> node ids change, you cannot be sure that ids which you got from find_nodes are still valid <- sure, but you'll learn about the ID change from the find_node response too, so the ping is sortof wasted in 99.9% of the cases.
[18:22:19] <The_8472> just set your text editor to utf8
[18:22:50] <MassaRoddel> nah, that conflicts with the binary strings like ids
[18:23:10] <The_8472> so? you usually don't care about the binary parts anyway
[18:27:46] <norc__> The_8472, in that case we shouldn't make it easy for them. Otherwise we could just pick a popular format, one that is easily readable and understandable without even thinking. ;)
[18:29:08] <The_8472> those usually have other problem. like being overly verbose or requireing you to un-escape characters or whatever
[18:30:33] <norc__> Not really sure what's wrong with JSON/YAML :)
[18:30:45] <norc__> But anyway
[18:30:45] <The_8472> we're not web developers.
[18:31:18] <The_8472> we throw lots of binary data around too
[18:32:00] <MassaRoddel> :)
[18:32:02] <norc__> I'm sorry, did you actually like my idea of a workaround for clients that do not support UDP yet?
[18:32:57] <The_8472> no
[18:33:12] <The_8472> they can either use HTTP or be updated.
[18:34:17] <norc__> Well.. just figured after what you just said. ;)
[18:35:32] <The_8472> the udp tracker protocol certainly doesn't fit all that well into the BT landscape, but writing a udp-http proxy seems even worse
[18:37:46] <norc__> I really don't see the point. Let's just strip away my performance arguments. Merely from a design perspective, the bencoding serialization format, and the tracker announce protocol is just horrible.
[18:38:19] <The_8472> i don't see the issue with them
[18:41:13] <The_8472> what exactly is wrong with it as a network protocol? supports binary, it's not overly verbose, mostly human-readable, easy to parse, ...
[18:42:45] *** Gottaname has quit IRC
[18:43:52] *** Gottaname has joined #bittorrent
[18:44:49] <norc__> Or hey. We could use XML. It's also human readable, not overly verbose. Even easier to parse. And even better, it's so popular that you can open up with some nice xml viewer!
[18:44:58] <norc__> I'm sorry, but I fail to see the point in all of this.
[18:45:01] <The_8472> ...
[18:45:05] <norc__> Should not have raised this idea loudly here.
[18:45:19] <The_8472> xml IS overly verbose and hard to parse...
[18:45:40] <The_8472> and it doesn't support binary
[18:45:42] * Gottaname throws in a misformed tag to seal the deal...
[18:45:55] <Gottaname> :D
[19:48:14] <choykloun> norc: wow, someone who agrees with me :)
[20:11:15] <The_8472> everyone is complaining, but nobody is really able to spell out what's actually wrong with it
[20:37:11] <DWKnight> if there's anything actually wrong with it at all
[20:37:36] *** rrr_ has joined #bittorrent
[21:12:39] *** PN has quit IRC
[21:13:49] *** ProperNoun has joined #bittorrent
[21:19:30] *** MassaRoddel has quit IRC
[21:20:12] *** PN has joined #bittorrent
[21:26:11] *** ProperNoun has quit IRC
[21:34:17] *** MassaRoddel has joined #bittorrent
[21:35:10] *** klapaucjusz has joined #bittorrent
[21:35:18] <klapaucjusz> New BEP in progress.
[21:35:37] <klapaucjusz> Not ready for publication yet, but early draft ready for public review.
[21:35:41] <klapaucjusz> http://www.pps.jussieu.fr/~jch/software/bittorrent/bep-dht-minor-extensions.html
[21:36:05] <klapaucjusz> The_8472: ^
[21:36:23] * The_8472 descends upon the prey like a vulture
[21:38:20] <The_8472> i like the taste of its bone marrow
[21:39:23] <choykloun> and here i thought it was some call to arms that would turn the channel into a warzone
[21:40:10] <The_8472> in the context of scrapes i would even go further and put a cap on how many values any node should store before stopping to provide a token. assuming we can agree on how they should work at some point.
[21:41:06] <The_8472> oh, another thing. nodes should prefer changing node IDs every once in a while instead of sending drop requests
[21:41:39] <The_8472> that should level out their load too without having any unpredictable impact on the routing table
[21:42:30] <The_8472> every once in a while meaning once or twice per day, at most
[21:43:11] <The_8472> i like the distinction between drop=1 and drop=2 though :)
[21:44:20] <The_8472> you should mention in which dictionary the drop key has to be included ;)
[21:44:27] <MassaRoddel> i would prefer words there
[21:45:27] <choykloun> argh, need to clean up my fucking code now
[21:45:44] <The_8472> DHT = UDP = we try to restrict everything in size
[21:45:45] <choykloun> a single file cant do everything from routing to transaction tracking ffs
[21:46:21] * The_8472 prods klapaucjusz
[21:46:38] <klapaucjusz> Back in five minutes, sorry.
[21:49:07] <klapaucjusz> Ok.
[21:49:23] <klapaucjusz> MassaRoddel: good point.  Any suggestions for the values of the drop key?
[21:49:23] <The_8472> 'A node receiving a reply to get_peers with no "token" parameter should not send an announce_peers request, and may optionally perform widening.' <- a) widening has to be explained and it should be a "should" instead of a "may optionally" since no new values could be stored otherwise.
[21:49:52] <klapaucjusz> I'm also hesitating between "drop" and "overload".
[21:50:31] <klapaucjusz> I'm not sure about changing node ids, since I don't fully understand how quickly the DHT recovers from an ID change.
[21:50:43] <klapaucjusz> We might include it as a last-ditch effort to reduce load, though.
[21:51:06] <The_8472> that depends on the implementation i guess. i have some explicit node ID change detection code in my logic
[21:51:09] <klapaucjusz> Setting a hard limit on number of values, I'm against.
[21:51:33] <The_8472> it rould range in the thousands
[21:51:52] <klapaucjusz> The_8472: about the  dictionary in which the key is: yes, the details of the syntax have to be fleshed out.
[21:51:59] <The_8472> kk
[21:52:24] <klapaucjusz> Anyway, are we in agreement on the principle?  (I need three yeses, one for each of the proposed extensions.)
[21:53:01] <The_8472> omitting token: yes, needs refinement though
[21:53:15] <The_8472> always sending nodes in addition to values: yes
[21:53:35] <klapaucjusz> choykloun: "i thought it was some call to arms" -- I'm not keen for conflict, except with so-called "security experts".
[21:54:18] <The_8472> drop/overload: hrrrm. i'm not quite sold on that one. esp. the necessity aspect.
[21:54:45] <klapaucjusz> "omitting token: yes, needs refinement though" -- I'll be grateful if you can suggest something.
[21:54:55] <klapaucjusz> Posting on forum.bittorrent.org if no objection.
[21:55:03] <The_8472> sure
[21:56:41] <klapaucjusz> Done.
[21:57:01] <The_8472> as for the token, well... as i said. perimeter widening is a should, not a may imo. and it has to be explained.
[21:57:11] <klapaucjusz> Suggest moving the discussion to http://forum.bittorrent.org/viewtopic.php?id=148
[21:57:48] <The_8472> *sigh* ok
[22:08:05] <The_8472> http://forum.bittorrent.org/viewtopic.php?pid=898#p898
[22:09:01] <The_8472> *edits*
[22:10:52] * The_8472 edits some more
[22:12:09] <The_8472> done
[22:17:14] *** BentMyWookie has quit IRC
[22:17:32] <The_8472> ahhh, now i remember :)
[22:17:36] * The_8472 edits again
[22:17:39] <klapaucjusz> Too late.
[22:17:52] <The_8472> see, that's why things should be done in IRC :P
[22:17:54] <klapaucjusz> Please make a new posting, so chronology is preserved.
[22:18:00] *** BentMyWookie has joined #bittorrent
[22:18:01] <The_8472> blergh
[22:18:09] * klapaucjusz dislikes forums, because there's no record of history.
[22:18:17] <klapaucjusz> Mistakes happen, and they should belong to the history.
[22:18:26] * The_8472 loves the edit button
[22:20:27] <klapaucjusz> There's a "preview" button.  Please respect your reader, don't force us to read half-finished postings.
[22:25:46] <The_8472> well... new posts!
[22:28:31] <The_8472> hah! another idea :)
[22:33:45] <The_8472> this stuff has more implications than i thought
[22:51:19] <klapaucjusz> I'll keep monitoring the forum.
[22:51:21] <klapaucjusz> Bye for now.
[22:51:22] *** klapaucjusz has quit IRC
[23:54:18] *** goussx has joined #bittorrent





top