Switch to DuckDuckGo Search
   August 2, 2013  
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | >

Toggle Join/Part | bottom
[01:15:33] *** mikeal has joined #manta
[01:16:13] <mikeal> using the node.js client, how do i pass the input objects to job creation?
[01:20:37] <mcavage> mikeal: https://github.com/joyent/node-manta/blob/master/test/client.test.js#L223-L233
[01:21:03] <mcavage> you call that API as many times as you like and then ultimately call: https://github.com/joyent/node-manta/blob/master/test/client.test.js#L260
[01:22:23] <mcavage> we should probably make a JobStream or something like that for this. i.e., foo.pipe(client.createJob());
[01:22:26] <mikeal> ahhhhh
[01:22:52] <mikeal> would be cool if createJob returned a writable stream you could just pipe to
[01:22:54] <mcavage> but for now - that's the api - each invocation of that is an API round trip, so if you have lots, chunk up your inputs into blocks of 1k or something (that's what mjob does)
[01:23:02] <mcavage> yeah - i'll file a GH issue for it.
[01:23:19] <mikeal> well, in that case, it's probably better to just pass this an array then
[01:24:05] <mcavage> https://github.com/joyent/node-manta/issues/102
[01:24:08] <mcavage> yeah- you give it an array.
[01:24:43] <mcavage> just saying - if you give it one at a time it's going to hurt if you have a lot of inputs. much faster to give it a managable list at a time.
[01:25:10] <mikeal> is createJob('ls', {id:'myname'}) how i name the job explicitely?
[01:26:21] <mcavage> no, you'd want it to look like what's in the comments: https://github.com/joyent/node-manta/blob/master/lib/client.js#L1285-L1317
[01:26:32] <mcavage> that createJob('ls') is just sugary short-hand
[01:26:50] <mcavage> side point: apologies the SDK docs aren't online. I'll get on that again.
[01:27:52] <mcavage> so you'd want client.createJob({name: 'foo', phases: [ { exec: 'ls' }] }, function (err, job) { ... });
[01:37:31] <mcavage> mikeal: ok, the node docs are now formatted: https://apidocs.joyent.com/manta/nodesdk.html
[01:42:01] <mikeal> nice
[01:42:19] *** dap has quit IRC
[01:45:50] *** dap has joined #manta
[01:57:21] *** AvianFlu has joined #manta
[01:58:08] <mikeal> i tried {name and it still gives me back a random uuid as the id
[02:06:46] *** bixu_ has joined #manta
[02:07:18] <bixu_> Are there plans to have ACLs in Manta (so that I can have a user access a dir or something but not my whole Joyent infrastructure)
[02:08:03] <wesolows> it's at the top of the list.
[02:08:05] <nahamu> bixu_: you missed discussion earler about just that
[02:08:30] <mikeal> is there a way to get the headers for $MANTA_INPUT_OBJECT
[02:08:57] <mikeal> i need to copy them over to my mput
[02:17:12] <mikeal> actually, better question, how do I copy a header from one manta object to another
[02:18:16] <tjfontaine> I feel like the answer is mln
[02:19:57] <mikeal> only the headers?
[02:20:05] <mikeal> i thought mln was basically the manta version of a sym link
[02:20:35] <tjfontaine> I guess I didn't read enough scrollback for this window
[02:21:58] <mikeal> i need to get the header from inside a job for $MANTA_INPUT_OBJECT so i can pass it to my mput
[02:22:27] <mikeal> or, alternatively, i could set it after the mput but i'd still need to get it inside the job
[02:23:05] <mikeal> wish there was a $MANTA_INPUT_HEADER_*
[02:23:19] <mikeal> so i could just do $MANTA_INPUT_HEADER_CONTENT_TYPE
[02:23:24] <nfitch> mikeal: try HEADER_VAL=$(mls -j $MANTA_INPUT_OBJECT | json <header>)?
[02:23:51] <nfitch> For example: [nate@dev node-manta]$ mls -j /nfitch/stor/a.txt | json type
[02:23:51] <nfitch> text/plain
[02:24:58] <mikeal> the json property is just called "type", strange
[02:25:13] <mikeal> cool, i'll try this
[02:26:48] <bixu_> nahamu: Is there a log somewhere or can you recap?
[02:27:25] <nahamu> bixu_: as wesolows said, it's at the top of their list of features to implement.
[02:27:55] <bixu_> Thanks.
[02:28:03] <wesolows> the specifics are interesting, so mark's take is useful. But I don't know of any loggers in here.
[02:28:08] <wesolows> Maybe echelog has one.
[02:28:17] <nahamu> I have my logs, hang on
[02:28:27] <wesolows> yep
[02:28:34] <wesolows> http://echelog.com/logs/browse/manta/1375394400
[02:29:43] <nahamu> oh, cool
[02:29:50] <nfitch> mikeal: We actually delete the headers explicitly before printing the object: https://github.com/joyent/node-manta/blob/master/bin/mls#L187
[02:30:01] <nfitch> That's weird. We should have an option that leaves them in...
[02:30:28] <mikeal> that worked
[02:30:29] <mikeal> sweet!
[02:30:38] <mikeal> the rest should be easy :)
[02:31:19] <nfitch> mikeal: Good deal. I've opened this to track the ability to leave the headers in: https://github.com/joyent/node-manta/issues/103
[02:32:10] *** nfitch has quit IRC
[02:32:20] *** mikeal has quit IRC
[02:34:48] *** yunong has quit IRC
[02:34:50] <mcavage> nahamu/wesolows/bixu: also, the logs are on libuv as well (I find the interface slightly nicer. YMMV): http://logs.libuv.org/manta/latest
[02:35:37] *** fredk has quit IRC
[03:14:27] *** mikeal has joined #manta
[03:28:28] *** abraxas has joined #manta
[03:45:04] *** dap has quit IRC
[04:03:21] <mikeal> strange error in two images https://gist.github.com/mikeal/a15ef4a56a78148bead3
[04:03:28] <mikeal> doing convert from imagemagick
[04:03:57] <mikeal> looks like imagemagick might not be entirely configured properly in manta
[04:09:51] <mikeal> is there a way to get rid of old jobs? even after i cancel them i can see them
[04:21:14] *** mikeal has quit IRC
[04:56:24] *** mikeal has joined #manta
[05:21:50] *** nfitch has joined #manta
[05:48:22] *** mikeal has quit IRC
[06:16:12] *** mikeal has joined #manta
[06:30:25] *** bixu_ has quit IRC
[06:51:06] *** mikeal has quit IRC
[06:52:38] *** mikeal has joined #manta
[07:13:10] *** bixu___ has quit IRC
[07:16:26] *** mikeal has quit IRC
[07:43:23] *** mikeal has joined #manta
[07:56:39] *** AvianFlu has quit IRC
[08:13:31] *** mamash has joined #manta
[08:23:03] *** ghostbar has quit IRC
[08:39:27] *** nfitch has quit IRC
[09:09:59] *** ghostbar has joined #manta
[12:04:43] *** abraxas has quit IRC
[12:20:31] *** ghostbar has quit IRC
[13:37:46] *** AvianFlu has joined #manta
[15:00:22] *** chorrell has joined #manta
[15:42:52] *** AvianFlu has quit IRC
[16:39:17] *** mikeal has quit IRC
[17:12:15] *** mikeal has joined #manta
[17:32:40] *** nfitch has joined #manta
[17:34:22] *** nfitch has quit IRC
[17:39:50] *** mikeal has quit IRC
[17:46:59] <mcavage> mikeal: we'll look at the imagemagick thing. To your question of how to remove old jobs:
[17:47:08] <mcavage> jobs are "archived" to objects once they're done. Running this:
[17:47:17] <mcavage> mls -rt /$MANTA_USER/jobs
[17:47:33] <mcavage> will list jobs in ascending order. So to get rid of old ones, I typically do something like this:
[17:48:03] <mcavage> mls -rt /$MANTA_USER/jobs | head -50 | sed 's|^|/mark.cavage/jobs/|' | parallel -j 20 mrm -r {}
[17:52:21] <nahamu> s/\/mark.cavage/~~\// ;)
[17:53:33] <mcavage> yeah given the new CLI.
[17:53:46] <mcavage> i just wanted to batch up a few more fixes before I publish to npm
[17:57:55] <nahamu> fair
[18:03:47] *** mikeal has joined #manta
[18:07:07] *** trentm has joined #manta
[18:09:13] *** trentm has quit IRC
[18:11:38] *** nfitch has joined #manta
[18:13:40] *** fredk has joined #manta
[18:13:57] *** fredk has joined #manta
[18:21:41] *** mikeal has quit IRC
[18:25:27] *** dap has joined #manta
[18:26:35] *** yunong has joined #manta
[18:46:25] *** nfitch has quit IRC
[18:51:43] *** mamash has left #manta
[18:53:07] *** yunong has quit IRC
[18:59:37] *** mikeal has joined #manta
[19:10:34] *** bixu has joined #manta
[19:14:20] *** bixu has quit IRC
[19:14:47] *** bixu has joined #manta
[19:25:09] *** bixu_ has joined #manta
[19:28:24] *** bixu_ has quit IRC
[19:29:14] *** ghostbar has joined #manta
[19:29:53] *** bixu has quit IRC
[19:34:00] <mcavage> mikeal: we looked into your transcode problem -- your images aren't images. they're "rate limit" 403 error pages.
[19:34:50] *** nfitch has joined #manta
[19:38:37] <mikeal> yeah
[19:38:49] <mikeal> i'm working it out, it's a bug in the generation/scrape code
[19:39:05] <mikeal> there's definitely a check in their for statusCode but there must be a bug
[19:39:11] <mikeal> there
[19:39:20] <mcavage> you might want to sanity check content as well (or at least content-type)
[19:39:22] <mikeal> been talking to jperkin on twitter
[19:39:32] <mcavage> since you're saving HTML as .png right now.
[19:39:33] <mikeal> so many image content types :(
[19:39:33] <mcavage> k
[19:39:43] <mikeal> well, i don't actually stick a .png on there :)
[19:40:01] <mikeal> and i use the content-type given from the http response in the scraper
[19:44:51] *** yunong has joined #manta
[19:52:38] *** papertigers_ has joined #manta
[19:53:32] *** papertigers has quit IRC
[19:53:33] *** papertigers_ is now known as papertigers
[20:09:23] *** bixu has joined #manta
[20:09:34] *** bixu_ has joined #manta
[20:17:12] *** sbraswe1 has joined #manta
[20:41:36] *** papertigers has quit IRC
[21:02:15] *** papertigers has joined #manta
[21:09:29] <mcavage> I know a few people have asked here as well -- heads up that I just landed "client.createReadStream" and "client.createWriteStream" in master of node-manta.
[21:10:27] <mcavage> https://github.com/joyent/node-manta/commit/4e73a6c0e7b5a1de9ad4471e5a3a7d0405f8b495
[21:11:47] *** yunong1 has joined #manta
[21:20:23] *** yunong has quit IRC
[21:46:38] *** bixu_ has quit IRC
[21:47:06] *** bixu_ has joined #manta
[22:24:53] <nahamu> mcavage: might it be possible for one of the tools (mfind strikes me as the obvious choice) to spit out the full http (or https) URL for the results?
[22:25:09] <nahamu> clearly one can ad-hoc sed the output, or make an alias or function to do it
[22:25:25] <mcavage> you mean instead of /foo/stor/bar https://us-east.manta.joyent.com/foo/stor/bar?
[22:25:31] <nahamu> yes
[22:25:50] <mcavage> i suppose it could be optional (mfind is the only one that spits out full paths, iirc) -- I guess is it worth it over | sed ?
[22:26:20] <nahamu> I feel like I do it often enough that I'm tempted to make a funcion in my .bashrc
[22:26:30] <mcavage> hmm
[22:26:33] <nahamu> if I feel like doing that, others might as well and then it might as well be a part of the tool.
[22:26:40] <mcavage> ooc: why?
[22:26:51] <mcavage> unless it's /public you can't pipe that to a browser or curl or anytihng?
[22:27:14] <nahamu> I use /public as a dumping ground for uploading stuff to share.
[22:27:44] <mcavage> oh ok so it is /public.
[22:27:50] <nahamu> other option would be for mput or muntar to spit out a url at the end.
[22:28:32] <mcavage> ok - well, file a GH issue, and i can make it optional.
[22:28:44] <mcavage> i know at least *I* don't want it doing that all the time ;)
[22:29:01] <nahamu> certainly not as default behavior, agreed.
[22:29:54] <nahamu> (and probably not worth implmenting unless other people are asking for it)
[22:37:36] <nahamu> https://github.com/joyent/node-manta/issues/104
[22:37:59] <nahamu> can I not label it myself? sad...
[22:40:07] <mcavage> oh, i don't know - i never use GH labels anyway :)
[22:40:39] <nahamu> fair enough.
[22:40:44] <tjfontaine> you can't label yourself, no.
[22:41:47] <nahamu> now I need to file and RFE on github itself. I should be able to mark an issue I submit as some sort of low priority... but too lazy. enough bug filing for one day
[22:54:44] *** mamash has joined #manta
[22:58:38] <jperkin> fwiw I'd definitely +1 that, echo'ing $MANTA_URL and then copy/pasting that + the filename is somewhat annoying
[22:59:43] *** sbraswe1 has left #manta
[23:00:05] <nahamu> jperkin: +1 it on github. :)
[23:05:35] *** yunong1 has quit IRC
[23:05:48] *** yunong has joined #manta
[23:09:42] *** mamash has left #manta
[23:24:19] *** mikeal1 has joined #manta
[23:24:59] *** mikeal has quit IRC
[23:34:43] *** bixu has quit IRC
[23:35:11] *** bixu has joined #manta
top

   August 2, 2013  
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | >