October 1, 2013 
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | >
October 1, 2013 [00:03:56] <irker408> spec-files-extra [5476] tom68 SFEhadoop.spec: change (Build)Requires to %{pnm_buildrequires_SUNWant}, %{pnm_requires_java_runtime_default}, %{pnm_requires_java_runtime_default}, add Requires: SFEprotobuf[00:09:06] *** randw has quit IRC[00:10:56] <irker408> spec-files-extra [5477] tom68 SFEperl-extutils-cbuilder.spec: re-create by script, keep in mind changed version-number whan upgrading[00:11:51] *** randw has joined #openindiana[00:14:18] <irker408> spec-files-extra [5478] tom68 experimental/SFEmplayer-fatbuildprep.spec: change SUNWsmba to %{pnm_buildrequires_SUNWsmba}, remove osdistro.inc, add %include packagenamemacros.inc[00:15:47] *** nikolam has quit IRC[00:16:40] *** andy_js has quit IRC[00:16:46] <irker408> spec-files-extra [5479] tom68 SFEvlc-2.0.7.spec: change (Build)Requires: SFEfaac-gpp(-devel), enable_x11_xcb on S12, remove extra -L -R/lib from LDFLAGS, remove tdestroy from func tests in configure.ac (OS provided doesn't link properly)[00:19:05] <irker408> spec-files-extra [5480] tom68 SFEmp3c.spec: change (Build)Requires to %{pnm_buildrequires_SUNWncurses_devel}, %include packagenamemacros.inc, add IPS_package_name, add Group[00:20:27] <irker408> spec-files-extra [5481] tom68 SFEbox2d.spec: change (Build)Requires to %{pnm_buildrequires_x11_library_freeglut}, %include packagenamemacros.inc[00:21:09] *** gwr has quit IRC[00:22:25] *** szaydel has quit IRC[00:24:46] *** ngchk1 has quit IRC[00:25:10] *** ngchk1 has joined #openindiana[00:26:41] *** randw has quit IRC[00:27:16] <irker408> spec-files-extra [5482] tom68 base-specs/packagenamemacros.spec: add Solaris 11.0 11.1 12 (new variables), add featureflags: etc_security_directorylayout, add example for freeglut package names[00:28:26] *** randw has joined #openindiana[00:29:37] <irker408> spec-files-extra [5483] tom68 SFEdoxygen.spec: fix %files for man pages[00:31:32] *** nikolam has joined #openindiana[00:35:26] <irker408> spec-files-extra [5484] tom68 SFElame.spec: export CC=gcc[00:38:46] <irker408> spec-files-extra [5485] tom68 SFEplib.spec: change (Build)Requires to %{pnm_buildrequires_SUNWaudh}, %include packagenamemacros.inc[00:39:19] <tomww> tsoome: yes :-)[00:39:22] <tomww> normally not.[00:39:30] <tomww> we'll survive it.[00:39:37] <irker408> spec-files-extra [5486] tom68 SFEquesoglc.spec: change (Build)Requires to %{pnm_buildrequires_x11_library_freeglut}, %include packagenamemacros.inc[00:39:39] <cpn> holy damn tomww hah[00:44:47] <irker408> spec-files-extra [5487] tom68 SFElibsamplerate.spec, SFEbc.spec: change (Build)Requires to %{pnm_*}[00:45:46] <irker408> spec-files-extra [5488] tom68 SFEperl-test-simple.spec: use find for removal of perllocal.pod[00:46:35] <alanc> someone is clearly overdosing on the coffee[00:54:08] <irker408> spec-files-extra [5489] tom68 SFElibvisual.spec: fix for /usr/share/locale[00:55:21] <tomww> hehe :)[00:57:41] <tomww> too many changes left, I'll suspend the irker filter for this channel. you might already know that contains a lot of stuff.[00:58:08] <tomww> and please excuse this many log lines.[00:59:45] <alanc> some of us just bundle up a ton of change into a single mega-push instead: https://java.net/projects/solaris-x11/lists/commits/archive/2013-09/message/12[01:00:05] <alanc> but that's mostly because it's less paperwork to get in the gate that way[01:02:38] *** Anton_ has quit IRC[01:07:56] *** Anton_ has joined #openindiana[01:15:54] *** InTheWings has quit IRC[01:17:48] <tomww> yeah, some changes might be bundled, but thats more work then editing .procmailrc and comment out the irker command for the channels other then pkgbuild[01:18:00] <tomww> so thats what I'm going with for the moment.[01:23:02] *** szaydel has joined #openindiana[01:25:00] *** hunter has quit IRC[01:31:51] <alanc> heh, that works[01:32:16] <alanc> not that #opensolaris has had any traffic other than irker yet this week[01:35:04] <tomww> yeah :)[01:49:32] *** movement has joined #openindiana[01:52:09] *** held has joined #openindiana[01:55:45] *** heldus has quit IRC[02:19:00] *** jihyun has quit IRC[02:22:05] *** ball has joined #openindiana[02:30:35] *** flight16 has joined #openindiana[02:32:16] *** Vutral has quit IRC[02:32:36] *** flight16 has quit IRC[02:48:26] *** ball has quit IRC[02:48:37] *** _Tenchi_ has joined #openindiana[02:54:57] *** Vutral has joined #openindiana[02:58:32] *** davenz has joined #openindiana[03:36:44] *** ngchk1_ has joined #openindiana[03:40:08] *** ngchk1 has quit IRC[03:44:18] *** smrt has quit IRC[03:51:12] *** randw has quit IRC[03:57:26] *** WormDrink has quit IRC[03:58:50] *** rlaager has joined #openindiana[04:12:46] *** nikolam has quit IRC[04:13:51] *** nikolam has joined #openindiana[04:16:32] *** jihyun has joined #openindiana[04:43:26] *** irker408 has quit IRC[04:51:58] *** kdavyd has joined #openindiana[05:12:15] *** Lee- has quit IRC[05:12:33] *** Lee- has joined #openindiana[05:17:04] *** SonikkuAmerica has joined #openindiana[05:18:24] *** szaydel has quit IRC[05:22:27] *** szaydel has joined #openindiana[05:34:59] *** SonikkuAmerica has quit IRC[05:35:54] *** SonikkuAmerica has joined #openindiana[05:45:48] *** SonikkuAmerica has quit IRC[06:33:41] *** alex1 has joined #openindiana[06:53:00] *** nsuperbus has joined #openindiana[06:54:36] *** davenz has quit IRC[07:00:19] *** szaydel has quit IRC[07:17:48] *** basic6_ has quit IRC[07:18:01] *** basic6_ has joined #openindiana[07:26:08] *** kshannon has quit IRC[07:26:58] *** kshannon has joined #openindiana[07:33:28] *** ianj has quit IRC[07:33:35] *** Lee- has quit IRC[07:44:06] *** rfm has quit IRC[08:13:07] *** syoyo has joined #openindiana[08:13:19] *** syoyo has quit IRC[08:13:56] *** syoyo has joined #openindiana[08:25:58] *** syoyo_ has joined #openindiana[08:29:04] *** syoyo has quit IRC[08:29:27] *** spanglywires has joined #openindiana[08:45:40] *** lblume has joined #openindiana[08:52:56] *** Seony has joined #openindiana[08:57:16] *** spanglywires has quit IRC[08:58:07] *** Worsoe has joined #openindiana[09:08:02] *** Triskelios has quit IRC[09:16:14] *** Seemone has joined #openindiana[09:20:02] *** texarcana has quit IRC[09:22:07] *** texarcana has joined #openindiana[09:22:50] *** alanc has quit IRC[09:23:09] *** alanc has joined #openindiana[09:23:10] *** ChanServ sets mode: +o alanc[09:34:02] *** ianj has joined #openindiana[09:48:49] *** tsoome has quit IRC[10:02:52] *** alcir has joined #openindiana[10:04:07] *** andy_js has joined #openindiana[10:11:42] *** spanglywires has joined #openindiana[10:13:41] *** ngchk1 has joined #openindiana[10:15:51] *** tsoome has joined #openindiana[10:16:04] *** ngchk1_ has quit IRC[10:33:50] *** sjorge has quit IRC[10:35:02] *** sjorge has joined #openindiana[10:35:19] *** spanglywires has quit IRC[10:36:35] *** ivann has joined #openindiana[10:36:41] <ivann> is there some way to increase the performance when using find -type f -mtime and similar commands on zfs with lots of files?[10:39:58] *** tommy_e has joined #openindiana[10:41:18] *** atrius has quit IRC[10:41:48] *** atrius has joined #openindiana[10:42:32] <tomww> not really. only you could try atime=off, as I believe it would do updates to directory entries when find goes through them. would be worth a try.[10:43:19] <tomww> else have enough RAM to hold the directory structures in fast memory. alternatively have a SSD as read cache could help (have no numbers).[10:43:56] *** djfirefox has joined #openindiana[10:44:15] <djfirefox> greetings all[10:44:32] <djfirefox> Anyone have experience with fixing mbuffer problems?[10:45:05] <tomww> oh, what kind of mbuffer problems (I'm using it to buffer zfs send | zfs recv)[10:45:53] <djfirefox> I have a zfs send recieve that pipes throught pigz and mbuffer, this worked fine until I upgraded to 151 a8, also to throw in the mix, i changed my zfs FS's to iSCSI volumes. Just tried last night to send recieve for the first time since the update[10:46:01] *** Zenos has joined #openindiana[10:46:18] <djfirefox> and since I changed to using iSCSI volumes[10:46:36] <djfirefox> correction using zfs volumes as Iscsi luns[10:47:00] <tomww> you are sending the zvol block data or the filesystem content?[10:47:53] <tomww> and what kind of problem do you see? if it doesn't work with "mbuffer --options" replaced by "cat" then the problem isn't mbuffer (just to mention it, not that I think you did it wrong).[10:47:57] <djfirefox> after the help provided from the guys here yesterday, I got the zvols turned into sparse so I was able to snapshot them, and it is those I'm sending, not the underlying root zfs FS pool[10:48:47] <djfirefox> i'll paste the command i'm using for send recieve and see what you think, its part of a script so please excuse the variable names[10:49:06] <ivann> tomww: ok. thanks for the info[10:49:58] <djfirefox> so, I start mbuffer on the remote host:-[10:50:45] <djfirefox> ssh -f $SSHRHOST "$RMBUFFER -q -I 12999 -m 128M -s 128k | $PIGZ -d | sudo /sbin/zfs receive -d -F $sshdestfsroot1"[10:50:55] <djfirefox> and then send from the local host:-[10:50:58] <tomww> ivann: I hope it helps. you could check disk activity while running find zwice, zpool iostat -v 5 should should almost no activity with the second run if RAM / SSD cache is large enough/present[10:51:13] <djfirefox> $LZFS send -v $localnewsnap2 | $PIGZ -9 | $LMBUFFER -s 128k -m 128M -O $RHOST:12999[10:51:39] <ivann> tomww: yeah... i noticed it only took 1 second on the second try... compared to 40 on the first try[10:51:52] <ivann> i was just hoping it could be fast on the first try too :)[10:53:07] <tomww> ivann: great :), so you only have to wait until the cache got warmed up[10:53:16] <tomww> djfirefox: and whats the kind of error you are seeing?[10:55:15] <djfirefox> just running now to get the exact error[10:56:19] <djfirefox> I was getting a broken pipe error from zfs recieve but cant remember the exact error I was getting from mbuffer[10:56:21] *** tn8or has joined #openindiana[10:57:21] <tomww> you could verify the mbuffer connection? e.g. with recveive with "$RMBUFFER -q -I 12999 -m 128M -s 128k"[10:57:35] <tomww> and send with "date | $LMBUFFER -s 128k -m 128M -O $RHOST:12999"[10:58:04] <tomww> I'll be back in a short while[10:58:41] <djfirefox> ok here we go - pigz abort: corrupted input -- invalid deflate data: <stdin>[10:59:26] *** tommy_e has quit IRC[10:59:36] <djfirefox> mbuffer: error: outputThread: error writing to remotehostxxx:12998 at offset xxxxxx: broken pipe[11:00:33] <tsoome> remove mbuffer temporarily and see if you still will get an errir[11:00:37] <tsoome> error*[11:01:02] <djfirefox> I'm thining it might be pigz thats at fault here, but i'll try both[11:01:56] <tsoome> yep, just use plain zfs send/receive, if thats ok, then add other components one by one[11:07:45] *** tn8or has quit IRC[11:09:22] <tomww> or send the "date" output through the chain and see the surpriding output on the other side :)[11:11:03] *** tommy_e has joined #openindiana[11:11:10] <djfirefox> aii good idea to do that first, on it now[11:15:47] <djfirefox> date seems to be recieved, how do I send a stream of dates to test this over a longer period of time[11:15:59] <djfirefox> or does that confirm mbuffer is fine[11:16:42] *** WormDrink has joined #openindiana[11:19:57] <tomww> it does confirm that mbuffer is right at least for 10 bytes of data :-)[11:20:08] <tomww> I would not expect mbuffer to do wrong.[11:20:34] <tomww> you could send over a larger binary file, make the checksum on both sides and compare[11:20:47] <tomww> e.g. digest -a md5 /usr/bin/largestbinaryicanfind[11:21:01] <tomww> on the receiving side you could use[11:21:36] <tomww> ssh -f $SSHRHOST "$RMBUFFER -q -I 12999 -m 128M -s 128k | $PIGZ -d > /tmp/temporarybinary and checksum this file afterwards[11:22:19] <tomww> there has been an error with *older* mbuffer versions, so please check if you run the version which has the fix for checksums (on the website[11:22:44] <djfirefox> im running the mbuffer from package manager?[11:22:53] <tomww> http://www.maier-komor.de/mbuffer.html "Versions 20120505 to 20130209 have a bug that can lead to data corruption if you use multiple outputs or one output with hashing. "[11:23:03] <tomww> pkg info mbuffer[11:23:27] <djfirefox> 20130220[11:23:36] <lblume> It was only with hashing though.[11:23:45] <tomww> yes[11:23:57] <tomww> djfirefox: that version looks good.[11:29:43] <djfirefox> ok testing without pigz, the MD5 chek would be good, but i dont have a binary at 8Gb in size and it only breaks at about 7-8 Gb[11:29:54] <djfirefox> I'm at 5gb and counting :)[11:31:34] <djfirefox> ok something incredibly weird happened,[11:32:02] <djfirefox> bassically the output from the zfs send started spewing to the terminal!![11:32:09] <djfirefox> had to kill the terminal[11:35:23] *** ball has joined #openindiana[11:35:54] *** InTheWings has joined #openindiana[11:37:09] *** newbie has joined #openindiana[11:37:32] *** Zenos has quit IRC[11:37:34] *** newbie is now known as Guest90589[11:37:35] *** Guest90589 is now known as Zenos[11:39:47] <djfirefox> ok here we go guys, error appears to be coming from zfs receive - cannot receive new filesystem stream: invalid backup stream[11:40:06] <djfirefox> this occurred 15.4G into sending[11:40:18] <djfirefox> i'm really baffled[11:40:48] <djfirefox> going to remove everything other than zfs send receive now[11:44:09] *** alex1 has quit IRC[11:44:18] <tomww> so what about using a different compression tool then pigz?[11:44:24] *** alex1 has joined #openindiana[11:44:52] <djfirefox> oh i removed pigz from the equation looks like its zfs itself[11:46:51] <djfirefox> straight zfs send receive over ssh running now....[11:46:59] <tomww> enough space / quota on the receiving side?[11:47:07] <djfirefox> exact same storage[11:47:18] <djfirefox> both sides[11:47:26] <tomww> and same admin on both sides :-)[11:47:34] <tomww> snapshots might haave eaten up all free space..[11:47:35] <djfirefox> same zil and cache config everything, total replication of sending server[11:47:53] <djfirefox> storage on remote side unused, one big large 3.53TB pool[11:49:02] <djfirefox> no snapshots on storage on local side except the one i'm sending, and we coverted these volumes to sparse volumes yesterday to theres about 1.44TB free on the local storage[11:51:39] *** held has quit IRC[11:55:26] <djfirefox> ok so straight zfs send receive working fine i'm up to 16.5Gb transferred so far and that is the biggest yet[11:55:49] <djfirefox> so it looks like it is mbuffer that is breaking things, I just don't know why???[11:57:18] <djfirefox> @tomww: Any ideas ?[12:01:53] <tsoome> thats about debugging mbuffer itself. you can try first by feeding some amount of data in it without zfs send/receive and see if the error is still there, if it is - grab the source and debugger...[12:02:42] <djfirefox> tsoome: thats a bit over my head, wouldn't know where to start.[12:02:45] <tomww> djfirefox: so whats in the chain at the moment?[12:03:02] <tsoome> wll, the first step is to check how repeatable that issue is[12:03:09] <lblume> I do send ~40GB using mbuffer those days, works nicely[12:03:10] <tomww> zfs send | mbuffer | (network) | mbuffer | zfs recv ... nothing else?[12:03:24] <djfirefox> tomww: simply - zfs send .... | ssh xxxx@xxxxxx zfs receive ...[12:03:53] <djfirefox> still going strong at 32.3Gb transferred[12:03:53] <lblume> why isn't mbuffer before ssh?[12:04:21] <tomww> so you already had a run with mbuffer but without pigz ?[12:04:26] <djfirefox> lblume: Sorry i dont understand your question?[12:04:43] <djfirefox> tomww: yes tried mbuffer without pigz and it failed about 8Gb in[12:04:55] <tomww> aha.[12:05:25] <djfirefox> ok here we go guys, error appears to be coming from zfs receive - cannot receive new filesystem stream: invalid backup stream[12:05:35] <djfirefox> sorry thats what i posted earlier[12:05:52] <djfirefox> but that was the error when running mbuffer[12:06:42] <lblume> You use mbuffer to send to a port? Not through ssh?[12:07:19] <djfirefox> lblume: correct, but i use ssh to start mbuffer on the remote side via a script[12:08:27] <lblume> Can you try the more classic zfs send | mbuffer | ssh zfs receive? To establish if it could be the connection option of mbuffer breaking (I've never used that)[12:09:04] <djfirefox> lblume: i'll give it a try[12:10:58] <djfirefox> lblume: so checking syntax - zfs send | mbuffer -s 128k -m 128M | ssh ...... zfs receive ....[12:11:12] <tomww> if CPUs are fast enough, it should not harm throughput too much when piping through ssh.[12:11:32] <tomww> zfs send | mbuffer | ssh root@remote "mbuffer | zfs recv"[12:12:42] <djfirefox> tomww: Is that not the same as what I've been doing originally?[12:15:29] <djfirefox> although actualy, my script does the ssh bit first with the -O, whereas tom, are you suggesting skipping -O but still pipe through mbuffer on both sides?[12:16:56] <djfirefox> and the ssh bit just starts the command whereas here i'm pipe over ssh?[12:18:43] <djfirefox> sshd using a fair amount of cpu time[12:20:53] *** Seony has quit IRC[12:24:11] <djfirefox> ok so --- $LZFS send -v $localnewsnap1 | $LMBUFFER -s 128k -m 128M | ssh $SSHRHOST "sudo /sbin/zfs receive -d -F $sshdestfsroot1"[12:24:19] <djfirefox> This is working without a problem[12:24:54] <djfirefox> going to try the tom version with mbuffer on both sides of the ssh[12:29:05] *** ball has quit IRC[12:29:59] <djfirefox> tomww: i'm currently running like this 'please excuse script variables :-[12:30:00] <djfirefox> $LZFS send -v $localnewsnap1 | $LMBUFFER -s 128k -m 128M | ssh $SSHRHOST "sudo /sbin/zfs receive -d -F $sshdestfsroot1"[12:30:30] <djfirefox> correction thats the wrong one[12:30:32] *** held has joined #openindiana[12:30:56] <djfirefox> this is the right one:-[12:30:57] <djfirefox> $LZFS send -v $localnewsnap1 | $LMBUFFER -s 128k -m 128M | ssh $SSHRHOST "$RMBUFFER -m 128M -s 128k | sudo /sbin/zfs receive -d -F $sshdestfsroot1"[12:32:23] <djfirefox> ssh is taking up a fair bit of cpu % time on both hosts[12:32:23] *** newbie1 has joined #openindiana[12:33:54] <tomww> yeah, encryption by ssh[12:34:39] *** Zenos has quit IRC[12:35:15] *** ivann has left #openindiana[12:35:16] <tomww> I would not necessarily set -s 128k. I think that doesn't help too much speeding up, only may add extra latency. But maybe it doesn't harm too much.[12:35:53] <tomww> I would more give mbuffer a defined ammount of memory, e.g. -m 1G if you have that many memory left.[12:36:03] *** spanglywires has joined #openindiana[12:36:58] <tomww> 100MBs is used up in unter one second, so 10 times more buffer isn't wrong. ZFS might need some time to get the send stream assembled while going though the zvol or zfs filesystem.[12:37:02] *** newbie1 has quit IRC[12:37:13] *** held has quit IRC[12:38:46] <djfirefox> tom: i used to use 512M, i tried 1G previously and it kept dying so used 512M, which was working up until now[12:38:55] *** held has joined #openindiana[12:39:30] <djfirefox> that being said, pipeing through ssh is working at the moment, no drop out or broken pipe error, so it looks like its an issue with the mbuffer -O connection[12:41:02] <djfirefox> going to go back to original script and just see if it still breaks[12:46:57] *** spanglywires has quit IRC[12:48:45] <lblume> I'd also advise 512 as buffer, but tbh, if you don't have a bandwidth limit, it's not making a huge difference.[12:49:16] <lblume> If you can prove that it's indeed the use of -I that breaks it, be sure to drop a line to the author :-)[12:52:47] <djfirefox> yup defo when I use -I -O just tried it again[12:53:06] <djfirefox> must be the 20130220 version :([12:53:21] <djfirefox> how do I contact the author? or is it an OI bug report?[12:53:42] <djfirefox> might try the optCSW version :)[12:57:12] <lblume> I package that one. If it has the same issue, then yep, contact the author. His contact is on his website :-)][12:58:38] <djfirefox> is there a way to downgrade the version and try the older version[12:58:40] *** syoyo has joined #openindiana[12:59:02] <tsoome> you can always build one on your own for test at least[12:59:06] *** syoyo has quit IRC[13:00:13] <djfirefox> ah tsoome: i love your hope in my unix abilities, lol unfortunately I dont have a clue[13:00:53] <djfirefox> well thats a lie, i've a bit of and idea, something regarding configure and make[13:00:55] <tsoome> all you need is to install compiler and headers, then it should be more or less just ./configure; make[13:00:58] <lblume> With OpenCSW, you can go to the /allpkg directory and manually download/install an older version[13:01:08] <lblume> /allpkgs, even[13:01:18] <tsoome> or that[13:01:38] <djfirefox> lblume: how do you goto the allpkg dir i normall just use pkgutil -i[13:02:01] <lblume> On mirror.opencsw.org/opencsw/[13:02:12] *** syoyo_ has quit IRC[13:02:15] <djfirefox> ooh hang on, do you mean goto the web site and do a wget of the pkg from the allpkg dir and then pkgutil install from that?[13:02:18] <lblume> Download/gunzip/pkgadd -d[13:03:10] <djfirefox> will pkgadd not overwrite the install mbuffer from OI?[13:03:36] <lblume> hmmmm, there shouldn't be any conflict, different directories[13:03:51] <lblume> Just be sure to call the right one, don't rely on the PATH[13:07:39] <djfirefox> oki doke i'll try this[13:13:39] *** Zenos has joined #openindiana[13:17:39] *** heldus has joined #openindiana[13:20:37] *** spanglywires has joined #openindiana[13:20:45] *** held has quit IRC[13:22:49] <djfirefox> ok old version installed and trying now[13:27:24] <tomww> OpenCSW mostly installs into --prefix=/opt/csw[13:27:43] <djfirefox> yup /opt/csw/bin/mbuffer :)[13:28:01] <djfirefox> we're at 9Gb and going so far so good[13:28:20] <djfirefox> if it gets past 15.4gb I'll know its mbuffer version[13:28:40] <djfirefox> there was an update to libmhash as well as mbuffer[13:28:52] <djfirefox> the isaexec is the same version[13:29:11] <djfirefox> sorry that mainly for lblume's benefit as he packages it[13:29:44] <tomww> would it make sense to test mbuffer with, e.g. libumem loaded? just an idea.[13:30:01] <djfirefox> indeed a good idea,[13:30:42] <djfirefox> i'll wait and see if this gets to 20G, then I'll stop it and try and update libmhash[13:31:25] <djfirefox> just need to figure out how to update a single package only[13:31:56] *** irker219 has joined #openindiana[13:31:56] <irker219> spec-files-extra [5496] tom68 SFEperl-extutils-cbuilder.spec: add SFEperl-extutils-cbuilder.copyright[13:32:24] *** lblume has quit IRC[13:32:30] *** szaydel has joined #openindiana[13:33:04] <djfirefox> ok 15.5Gb and going so it looks like the older version is working :)[13:35:15] *** lblume has joined #openindiana[13:36:00] <lblume> Still going?[13:36:11] <djfirefox> oh yeh baby! 20.3 gb[13:36:26] <djfirefox> gonna stop it now and update the libmhash package and try it again[13:36:58] <djfirefox> lblume: do you know how to update a specific pkg with pkgutil?[13:37:00] <lblume> it's more recent than OI's? Weird, that version is lile 5 years old[13:37:09] <lblume> pkgutil -u <package?[13:37:14] <lblume> > even[13:37:22] <lblume> it might pull in needed deps, though[13:37:39] <djfirefox> there arent any needed deps for libmhash[13:37:42] <djfirefox> i think[13:37:46] <lblume> Do you also get the error with the latest OpenCSW package?[13:38:58] <djfirefox> havent tried actually[13:40:47] <djfirefox> ok, libmhash updated to latest release mbuffer still on 20120505[13:40:51] <djfirefox> running again[13:42:42] *** patdk-lap has quit IRC[13:43:06] *** syoyo has joined #openindiana[13:43:27] <djfirefox> 1Gb so far, 8 Gb will be where it most likely dies but has happened at 15.1G so if it reaches 20Gb we know its not the libmhash library and defo the mbuffer package 20130220[13:44:44] <djfirefox> ...4Gb[13:45:26] <tsoome> broken pipe, or similar, means one side did stop, probably by application crash. if so, you may have core around.[13:49:12] *** Zenos has quit IRC[13:50:16] <djfirefox> 11Gb :) - Looks like the actually mbuffer package, i'll let it run to 20, then i'll update the CSW mbuffer package and re-run[13:50:50] <djfirefox> if we get the same issue with the OpenCSW version then its application for definate if not then its OI package of mbuffer in 151a8[13:53:51] <tsoome> could be compiler version issue[13:54:03] <tsoome> or optimization level[13:56:40] <tomww> oh, about the mbuffer version 20120505.... "Versions 20120505 to 20130209 have a bug that can lead to data corruption if you use multiple outputs or one output with hashing."[13:57:00] <lblume> Don't use hashing.[13:57:04] *** spanglywires has quit IRC[13:57:07] <tomww> so, if hashing is not involved, then data is consitent?[13:57:13] <tomww> aha.[13:57:27] <lblume> yes[13:57:42] <lblume> I spent a while pinpointing that bug.[13:58:19] <tomww> could opencsw push the new version to the stable package collection?[14:02:46] <djfirefox> ok so 28Gb no problem, the issue must be with mbuffer packe, going to update openCSW package to latest ver and try again[14:02:47] <lblume> There's been talks about that. Right now, I believe that unstable is the best choice. There should be a more automated package migration from unstable to stable coming up.[14:03:57] <tomww> ah, okay. at least if people can get that newer version.[14:05:25] <djfirefox> ok here we go ver - 20130220,REV=2013.02.21[14:05:41] <djfirefox> 1Gb so far, 8 Gb is where it normally bombs out[14:06:02] <djfirefox> its died folks[14:06:20] <djfirefox> brokemn pipe[14:06:28] <tsoome> any core?[14:06:33] <tomww> is there a non-ZFS test case to check the mbuffer setup against this data stream corruption?[14:06:36] <djfirefox> mbuffer 20130220,REV=2013.02.21 cannot do -O | -I[14:07:11] <tomww> I'm asking because I would really like the SFEmbuffer.spec tested against this.[14:07:28] <JT-EC> So are you lot finding an issue with the a8 mbuffer?[14:07:53] <tsoome> what you can do is to run it, and then attach truss to each mbuffer process, so you can see what they are doing on last moments[14:07:55] <djfirefox> pastebin.com/cqxGd6sa[14:08:53] <tsoome> ah, so it finds corruption in input and will stop there.[14:09:29] *** nikolam has quit IRC[14:09:31] <djfirefox> i've no idea how to help you guys any further[14:09:39] <tsoome> invalid deflate - its compressed stream between mbuffer?[14:10:01] <djfirefox> doesnt matter whether I compress stream or not, it corrupts eventually[14:10:19] <djfirefox> I've tried with pigz gzip, straight send[14:10:26] *** jihyun has quit IRC[14:10:30] <djfirefox> uncompressed send i mean[14:10:53] <djfirefox> if I dont use -O and -I to send receive, it works fine[14:11:36] <djfirefox> network:port send receive that is[14:11:51] <djfirefox> if I use mbuffer over ssh, it works fine[14:12:22] <djfirefox> is there anything else I can do to help before I revert to the old version and start the process of trnasferring 3TB :D[14:12:33] <tsoome> ah right, its pigz complaint about corruption[14:12:43] <djfirefox> tsoome: correct[14:14:22] <tsoome> ou, well, so the direct network link is broken but sending stdout is working?[14:15:09] <djfirefox> I would think its the send that must be breaking then it dies and the remote side freaks out[14:15:44] <tomww> crazy idea to use one version on the sender side, another version on the receiver? that way there is a small[14:16:05] <tsoome> well, the chain is simple - once pigz finds corruption, it will stop and obviously the whole pilepine is broken[14:16:11] <tsoome> pipeline*[14:16:15] <djfirefox> ah you want me to try old version to send and new to receive then vice versa?[14:16:16] <tomww> well, would that help to see if the send or rceive is broken?[14:17:10] <tomww> I don't know if mbuffer uses a protocol, if not, then one side could be netcat as well. to have a completely different piece of software which is assumed to be clean.[14:17:14] <djfirefox> place your bets folks, send or receive is broken or both ;)[14:17:29] <tsoome> i as thinking about netcat as well:D[14:17:52] <tsoome> hm, i keep messing up with my keyboard:D[14:18:33] <djfirefox> i'll downgrade send now[14:31:27] <djfirefox> running again[14:31:49] <djfirefox> sending with 20120505 receiving with 20130220[14:32:11] <tomww> cool, well be back later and read the backlog.[14:37:40] <djfirefox> hmmm interesting 8Gb still not broke yet[14:39:24] *** heldus has quit IRC[14:48:38] <djfirefox> ok 23Gb and fine, going to downgrade receiving side and upgrade sending side[14:52:07] <djfirefox> sending with 20130220 receiving with 20120505[14:54:07] *** Zenos has joined #openindiana[14:55:03] <djfirefox> And there she blows[14:55:39] <djfirefox> sending with new version receiving with old version, therefore as we thought the sending via host:port on version 20130220 is buggy[14:56:33] <djfirefox> if you guys need anymore help you can find me on the illuminos bug site as djfirefox, send me a mail or something :)[14:56:45] <djfirefox> now back to work :D[14:58:08] *** djfirefox has quit IRC[15:30:12] *** jihyun has joined #openindiana[15:34:27] *** djfirefox has joined #openindiana[15:36:05] *** syoyo has quit IRC[15:36:39] <djfirefox> ah well, it doesn't work now with the old version when transferring without compresseion, if I zfs send receive via mbuffer without going through pigz, it now crashes out[15:39:50] <lblume> Report the issue to the author and for now, use it through ssh or through netcat if you want to avoid crypting[15:41:12] <djfirefox> how do i use it through netcat?[15:41:24] <djfirefox> never used netcat before[15:43:52] <lblume> me either :-)[15:44:00] <lblume> I only know it's doable :-)[15:44:22] <djfirefox> i'll ssh and see how it looks[15:44:45] <djfirefox> man should get me author details for mbuffer?[15:49:21] <djfirefox> i get almost half the transfer rate with ssh :([15:50:10] <djfirefox> lol no man entry for netcat[15:51:42] <lblume> Wasn't it posted before? maier-komor.de[15:52:21] <djfirefox> altogether now 'I Love Google' - http://blog.smartcore.net.au/fast-zfs-send-with-netcat/[15:52:33] <djfirefox> quite simple really[15:54:45] *** jim80net has joined #openindiana[15:59:12] *** syoyo has joined #openindiana[16:10:49] *** timclassic has quit IRC[16:11:09] *** timclassic has joined #openindiana[16:15:24] <tomww> and mbuffer could still make the stream flow nicely while netcat pipes the data to the other side[16:16:05] *** cpn has quit IRC[16:16:26] *** joltman has joined #openindiana[16:17:36] *** gregburd has quit IRC[16:18:06] *** jihyun has quit IRC[16:31:16] <djfirefox> This is very strange, when trying to use netcat to send i get 'Cannot receive: failed to read from stream[16:31:53] *** irker219 has quit IRC[16:32:09] *** Worsoe has quit IRC[16:38:06] *** hunter has joined #openindiana[16:41:20] *** jihyun has joined #openindiana[16:42:16] <djfirefox> anyone successfully zfs send receive via netcat?[16:42:34] *** gregburd has joined #openindiana[16:43:42] <tomww> does it transfer the "date" test form above?[16:49:02] <djfirefox> well it sends the date ok, dont know how to check if it receives it[16:49:15] <djfirefox> all i know is date displays in source terminal[16:49:39] *** kdavyd has quit IRC[16:52:37] <tomww> on the receiving side the date should be displayed[16:54:12] <tomww> on the receiver: nc -w 120 -l -p 8023[16:54:23] <djfirefox> where on the receiving side?[16:54:58] <tomww> on the sender: date | nc -w 20 192.168.1.14 8023[16:55:07] <tomww> both just types into shell windows[16:55:27] <tomww> oh replace the IP Address with your receiver's IP Address[16:55:50] <tomww> the receiver will just print the date output[16:57:08] *** rfm has joined #openindiana[16:58:53] <djfirefox> yes that works fine[17:00:36] *** zerobsd has quit IRC[17:00:49] *** sebasp_ is now known as sebasp[17:01:42] <djfirefox> ok it works fine when i type the commands in manually at either end, if I try and start nc with ssh -f "start nc command etc" it fails[17:02:40] <djfirefox> surely it cant be too complicated.[17:02:47] <djfirefox> from the source i do:-[17:02:49] <djfirefox> ssh -f admin@muunixnfs02 "nc -w 30 -l -p 12990 | /opt/csw/bin/mbuffer -s 128k -m 1G | sudo /sbin/zfs receive -d -F vmbkpdr"[17:02:59] <djfirefox> then i do:-[17:03:10] <djfirefox> sudo /sbin/zfs send -v vmbkp/iSCSI-00@20131001-15:09:04 | /opt/csw/bin/mbuffer -s 128k -m 1G | /bin/nc muunixnfs02 12990[17:03:28] *** master_of_master has joined #openindiana[17:03:45] *** master_o1_master has quit IRC[17:04:11] <tomww> doesn't look too wrong.[17:04:23] *** DanaG has joined #openindiana[17:05:27] <DanaG> Argh, just accidentally added two drives to a mirror pool in a non-mirrored manner. Is there any way to remove them? Same sort of thing as in this post, but under OpenIndiana, not Linux: http://ubuntuforums.org/showthread.php?t=1665088[17:05:36] <djfirefox> doesnt work, 'Cannot receive: failed to read from stream'[17:06:41] *** tsoome has quit IRC[17:07:20] <djfirefox> I am totally baffled[17:07:32] <djfirefox> none of this makes sense[17:08:47] <djfirefox> it has to be something to do with executing nc from ssh -f[17:09:10] <djfirefox> as if i log in to the remote machine and setup the nc receive manually it works fine[17:09:28] <djfirefox> any ideas[17:09:51] <djfirefox> my ssh auto logs in using the rsa key trick and sudo requires no password[17:09:55] *** DanaG has quit IRC[17:10:01] *** DanaG has joined #openindiana[17:10:43] <tomww> you could wrap "nc" on the receiving side and cut off env variables.[17:10:57] <djfirefox> explain?[17:11:06] <tomww> 1sec :)[17:11:13] <djfirefox> no problem :)[17:11:35] <tomww> ssh -f admin@muunixnfs02 "env -i nc -w 30 -l -p 12990 | /opt/csw/bin/mbuffer -s 128k -m 1G | sudo /sbin/zfs receive -d -F vmbkpdr"[17:11:58] <tomww> but I would try the date test instead the zfs send[17:12:05] *** hunter has quit IRC[17:12:34] <tomww> for the receiver: ssh -f admin@muunixnfs02 "env -i nc -w 30 -l -p 12990 | /opt/csw/bin/mbuffer -s 128k -m 1G "[17:12:37] <tomww> and the sender[17:12:52] <tomww> date | nc muunixnfs02 12990[17:13:35] <tomww> (I believe, 1GB receive buffer should be enough to hold the date string)[17:14:39] *** hunter has joined #openindiana[17:15:04] <djfirefox> works for date , ie i see the message coming back on the source side via ssh -f[17:15:10] <djfirefox> fails for zfs[17:15:14] <djfirefox> same error as before[17:17:07] *** alcir has quit IRC[17:18:38] <djfirefox> correction fails for date[17:18:56] <djfirefox> Address family not supported by protocol family[17:19:25] <tomww> aha. could you try adding an IP address for the receiving "nc" command?[17:19:55] <djfirefox> thats -i yes?[17:19:59] <tomww> or add "-4" to only use ipv4 addresses[17:20:07] <djfirefox> ahh sounds good[17:20:13] <djfirefox> or both just to make sure[17:21:32] <tomww> yes. that would be "nc -4 -l recv_ip_addr recv_port_number"[17:22:09] <tomww> I'm away now for a while, I hope the other people here can jump in[17:22:58] <djfirefox> is this on the nc command executed on the receving side or on the nc command executed on source side?[17:23:34] <tomww> the example I gave is for the receiving part, in your case inside the ssh command[17:24:20] <djfirefox> same error[17:25:04] <tomww> I'm sorry for leaving now, but if you pastebin the both commands, maybe someone else can have a look.[17:25:18] <djfirefox> sure no probs thx anyway[17:26:50] <djfirefox> works without mbuffer by the way :D[17:28:31] *** Patrickdk has quit IRC[17:30:11] *** hunter has quit IRC[17:30:15] *** patdk-wk_ has quit IRC[17:33:10] *** Patrickdk has joined #openindiana[17:35:11] *** patdk-wk_ has joined #openindiana[18:02:21] *** DanaG has quit IRC[18:05:36] *** tsoome has joined #openindiana[18:12:42] *** cpn has joined #openindiana[18:23:09] *** hunter has joined #openindiana[18:37:38] *** hunter has quit IRC[18:51:17] *** Seemone has quit IRC[18:54:43] *** movement has quit IRC[18:55:37] *** movement has joined #openindiana[18:57:50] *** InTheWings has quit IRC[18:58:25] *** InTheWings has joined #openindiana[19:19:31] *** zerobsd has joined #openindiana[19:28:11] *** Lee- has joined #openindiana[19:30:53] *** zerobsd has quit IRC[19:34:39] *** zerobsd has joined #openindiana[19:36:47] *** SonikkuAmerica has joined #openindiana[19:37:06] *** sjorge has quit IRC[19:39:22] *** zerobsd has left #openindiana[19:46:03] *** deet has quit IRC[19:47:05] *** keremet has joined #openindiana[19:47:17] *** SonikkuAmerica has quit IRC[19:47:17] *** syoyo has quit IRC[19:51:02] *** deet has joined #openindiana[19:55:09] *** spanglywires has joined #openindiana[20:05:33] *** WormDrink has quit IRC[20:10:40] *** lennard has quit IRC[20:10:50] *** macbar has quit IRC[20:12:04] *** lennard has joined #openindiana[20:17:04] *** Triskelios has joined #openindiana[20:19:32] *** hunter has joined #openindiana[20:19:43] *** Vutral has quit IRC[20:20:55] *** sjorge has joined #openindiana[20:21:15] *** Triskelios has quit IRC[20:24:31] *** Vutral has joined #openindiana[20:27:09] *** alex1 has quit IRC[20:31:48] *** alex1 has joined #openindiana[20:32:25] *** spanglywires has quit IRC[20:52:20] *** alex1 has quit IRC[20:54:00] *** gwr has joined #openindiana[20:56:26] *** SonikkuAmerica has joined #openindiana[21:15:56] *** held has joined #openindiana[21:18:14] *** theason_ has joined #openindiana[21:19:24] *** ctrlbreak_x has joined #openindiana[21:20:28] *** Gugge_ has joined #openindiana[21:20:30] *** SonikkuAmerica has quit IRC[21:21:23] *** alp_ has joined #openindiana[21:21:50] *** ivan`_ has joined #openindiana[21:22:30] *** keremet has quit IRC[21:22:35] *** APTX_ has joined #openindiana[21:24:46] *** EisNerd_ has joined #openindiana[21:25:01] *** system5_ has joined #openindiana[21:27:05] *** Weedstock has joined #openindiana[21:27:10] *** lpsmith_ has joined #openindiana[21:27:16] *** sjorge has quit IRC[21:27:16] *** lennard has quit IRC[21:27:17] *** InTheWings has quit IRC[21:27:17] *** doug_ndndn has quit IRC[21:27:18] *** xenol has quit IRC[21:27:19] *** CVLTCMK0 has quit IRC[21:27:21] *** APTX has quit IRC[21:27:22] *** alp has quit IRC[21:27:23] *** system5 has quit IRC[21:27:23] *** EisNerd has quit IRC[21:27:24] *** Gugge has quit IRC[21:27:26] *** ivan` has quit IRC[21:27:28] *** ctrlbreak has quit IRC[21:27:29] *** patdk-wk_ has quit IRC[21:27:30] *** Patrickdk has quit IRC[21:27:35] *** Warod has quit IRC[21:27:35] *** theason has quit IRC[21:27:39] *** Woodstock has quit IRC[21:27:40] *** lpsmith has quit IRC[21:27:42] *** threadlock has quit IRC[21:27:43] *** lpsmith_ is now known as lpsmith[21:27:45] *** Weedstock is now known as Woodstock[21:27:45] *** lennard_ has joined #openindiana[21:28:56] *** InTheWings has joined #openindiana[21:31:45] *** sjorge has joined #openindiana[21:38:00] *** CVLTCMK0 has joined #openindiana[21:38:42] *** Warod has joined #openindiana[22:05:07] *** threadlock has joined #openindiana[22:18:09] *** Triskelios has joined #openindiana[22:20:55] *** worsoe has joined #openindiana[22:28:31] *** Gugge_ is now known as Gugge[22:30:25] *** WormDrink has joined #openindiana[22:31:22] *** patdk-wk_ has joined #openindiana[22:31:53] *** sjorge has quit IRC[22:40:24] *** worsoe has quit IRC[22:44:16] *** hunter has quit IRC[22:47:24] *** hunter has joined #openindiana[23:10:48] *** welby is now known as welby_away[23:27:24] *** bruges_ has joined #openindiana[23:29:28] *** bruges has quit IRC[23:35:13] *** andy_js has quit IRC[23:35:31] *** nsuperbus has quit IRC[23:37:35] *** xenol has joined #openindiana[23:47:55] *** movement has quit IRC[23:56:24] *** hunter has quit IRC