August 15, 2016 
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | >
August 15, 2016 [00:49:45] *** Kurlon <Kurlon!~Kurlon@cpe-104-228-216-170.maine.res.rr.com> has joined #omnios[00:50:23] *** andy_js <andy_js!~andy@90.219.225.136> has quit IRC (Quit: andy_js)[01:25:10] *** Botanic <Botanic!~Botanic@tita/admin/Botanic> has joined #omnios[01:37:16] *** Botanic <Botanic!~Botanic@tita/admin/Botanic> has quit IRC (Read error: Connection reset by peer)[01:53:27] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has joined #omnios[02:00:53] *** Guest1538741 <Guest1538741!~Guest1538@proxy-109-190-254-32.ovh.net> has quit IRC (Ping timeout: 258 seconds)[02:13:11] *** Botanic <Botanic!~Botanic@tita/admin/Botanic> has joined #omnios[02:19:56] *** Botanic <Botanic!~Botanic@tita/admin/Botanic> has quit IRC (Read error: Connection reset by peer)[02:21:11] *** Botanic <Botanic!~Botanic@199.180.152.220> has joined #omnios[02:21:11] *** Botanic <Botanic!~Botanic@199.180.152.220> has quit IRC (Changing host)[02:21:11] *** Botanic <Botanic!~Botanic@tita/admin/Botanic> has joined #omnios[02:24:09] *** xzi11a <xzi11a!~robert@pool-71-244-148-68.bltmmd.fios.verizon.net> has joined #omnios[02:28:52] *** xzi11a <xzi11a!~robert@pool-71-244-148-68.bltmmd.fios.verizon.net> has quit IRC (Ping timeout: 240 seconds)[02:28:58] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[02:47:51] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[02:59:15] *** v_a_b1 <v_a_b1!~volker@p5B3701E3.dip0.t-ipconnect.de> has quit IRC (Ping timeout: 264 seconds)[02:59:15] *** v_a_b <v_a_b!~volker@p5B3701E3.dip0.t-ipconnect.de> has quit IRC (Ping timeout: 264 seconds)[03:01:41] *** v_a_b <v_a_b!~volker@p5B37114C.dip0.t-ipconnect.de> has joined #omnios[03:02:57] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[03:03:01] *** v_a_b1 <v_a_b1!~volker@p5B37114C.dip0.t-ipconnect.de> has joined #omnios[03:09:53] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has quit IRC (Ping timeout: 258 seconds)[03:10:22] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[03:25:39] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[03:36:42] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[03:46:51] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[04:07:50] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[04:09:54] *** xzi11a <xzi11a!~robert@pool-71-244-148-68.bltmmd.fios.verizon.net> has joined #omnios[04:15:01] *** xzi11a <xzi11a!~robert@pool-71-244-148-68.bltmmd.fios.verizon.net> has quit IRC (Ping timeout: 252 seconds)[04:16:05] *** synegy34 <synegy34!~gatekeep@2001:470:8b45:400:7578:bb49:4a1f:bf90> has joined #omnios[04:18:44] *** gatekeep <gatekeep!~gatekeep@wls/staff/gatekeep> has quit IRC (Ping timeout: 250 seconds)[04:20:41] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[04:43:49] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[04:56:53] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[05:00:15] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has joined #omnios[05:04:11] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has quit IRC (Remote host closed the connection)[05:06:14] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has joined #omnios[05:09:21] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[05:14:29] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has quit IRC (Quit: tozhu)[05:17:37] *** xzi11a <xzi11a!~robert@pool-71-244-148-68.bltmmd.fios.verizon.net> has joined #omnios[05:53:04] *** bahamat <bahamat!~bahamat@ip72-199-207-63.sd.sd.cox.net> has joined #omnios[06:10:53] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has joined #omnios[06:33:56] *** openzfsslackbot <openzfsslackbot!~nobody@ec2-52-23-198-104.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)[06:34:47] *** openzfsslackbot <openzfsslackbot!~nobody@ec2-52-23-198-104.compute-1.amazonaws.com> has joined #omnios[06:39:57] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[06:40:08] *** bahamat <bahamat!~bahamat@ip72-199-207-63.sd.sd.cox.net> has quit IRC (Quit: Leaving.)[06:44:38] *** slx86 <slx86!~slx86@46-129-44-239.dynamic.upc.nl> has joined #omnios[07:11:29] *** chrustler <chrustler!~chrustler@94-137-110-36.customers.ownit.se> has quit IRC (Remote host closed the connection)[07:13:14] *** chrustler <chrustler!~chrustler@94-137-110-36.customers.ownit.se> has joined #omnios[07:21:53] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has quit IRC (Remote host closed the connection)[07:22:13] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has joined #omnios[07:30:27] *** bahamat <bahamat!~bahamat@ip72-199-207-63.sd.sd.cox.net> has joined #omnios[07:40:18] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[07:43:46] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has quit IRC (Remote host closed the connection)[07:44:16] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has joined #omnios[07:45:28] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Ping timeout: 244 seconds)[08:00:17] *** jfqd1 <jfqd1!~Adium@pD95DD89A.dip0.t-ipconnect.de> has joined #omnios[08:12:47] *** bahamat <bahamat!~bahamat@ip72-199-207-63.sd.sd.cox.net> has quit IRC (Quit: Leaving.)[08:41:26] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has quit IRC (Ping timeout: 265 seconds)[09:06:27] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has joined #omnios[09:13:39] *** tozhu <tozhu!~Thunderbi@218.88.84.11> has quit IRC (Ping timeout: 264 seconds)[09:43:21] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[09:48:56] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Ping timeout: 244 seconds)[09:55:43] *** slx86 <slx86!~slx86@46-129-44-239.dynamic.upc.nl> has quit IRC (Quit: leaving)[09:55:48] *** dmip <dmip!~Thunderbi@95.143.115.50> has joined #omnios[09:56:47] *** ptribble <ptribble!~ptribble@cpc92320-cmbg19-2-0-cust3282.5-4.cable.virginm.net> has joined #omnios[09:59:28] *** andy_js <andy_js!~andy@90.219.225.136> has joined #omnios[10:33:28] *** bens1 <bens1!~bens1@161.122.2.81.in-addr.arpa> has joined #omnios[11:46:15] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[11:50:21] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Ping timeout: 244 seconds)[12:09:17] *** tozhu <tozhu!~Thunderbi@223.85.203.28> has joined #omnios[12:26:10] *** tozhu <tozhu!~Thunderbi@223.85.203.28> has quit IRC (Ping timeout: 244 seconds)[12:27:17] *** tozhu <tozhu!~Thunderbi@223.85.203.28> has joined #omnios[12:45:39] *** tozhu <tozhu!~Thunderbi@223.85.203.28> has quit IRC (Ping timeout: 258 seconds)[12:48:51] *** Toasterson <Toasterson!~toast@68.30.150.83.dyn.dsl.as8758.net> has joined #omnios[13:00:05] *** tozhu <tozhu!~Thunderbi@223.85.203.28> has joined #omnios[13:21:40] *** Toasterson <Toasterson!~toast@68.30.150.83.dyn.dsl.as8758.net> has quit IRC (Ping timeout: 250 seconds)[13:26:01] *** tozhu <tozhu!~Thunderbi@223.85.203.28> has quit IRC (Quit: tozhu)[13:34:24] *** tozhu <tozhu!~Thunderbi@223.85.203.28> has joined #omnios[13:42:07] *** tozhu <tozhu!~Thunderbi@223.85.203.28> has quit IRC (Ping timeout: 244 seconds)[13:48:07] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[13:53:20] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Ping timeout: 244 seconds)[14:23:28] *** Toasterson <Toasterson!~toast@68.30.150.83.dyn.dsl.as8758.net> has joined #omnios[14:49:15] *** Kurlon <Kurlon!~Kurlon@cpe-104-228-216-170.maine.res.rr.com> has quit IRC (Ping timeout: 265 seconds)[14:51:30] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[14:55:50] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Ping timeout: 244 seconds)[15:34:38] *** Kurlon <Kurlon!~Kurlon@bidd-pub-03.gwi.net> has joined #omnios[15:40:45] *** tdowg1 <tdowg1!~quassel@pool-96-244-201-225.bltmmd.fios.verizon.net> has joined #omnios[15:57:57] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[16:02:12] *** xzi11a <xzi11a!~robert@pool-71-244-148-68.bltmmd.fios.verizon.net> has quit IRC (Ping timeout: 240 seconds)[16:08:14] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[16:20:14] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[16:20:57] *** jfqd1 <jfqd1!~Adium@pD95DD89A.dip0.t-ipconnect.de> has quit IRC (Quit: Leaving.)[16:27:33] *** Toasterson <Toasterson!~toast@68.30.150.83.dyn.dsl.as8758.net> has quit IRC (Ping timeout: 240 seconds)[16:31:25] *** xzi11a <xzi11a!~robert@static-108-48-124-82.washdc.fios.verizon.net> has joined #omnios[16:49:04] *** danmcd <danmcd!~danmcd@static-71-174-113-16.bstnma.fios.verizon.net> has joined #omnios[16:49:35] *** synegy34 <synegy34!~gatekeep@2001:470:8b45:400:7578:bb49:4a1f:bf90> has quit IRC (Read error: Network is unreachable)[16:49:58] *** synegy34 <synegy34!~gatekeep@2001:470:8b45:400:7578:bb49:4a1f:bf90> has joined #omnios[17:20:01] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[17:23:26] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[18:00:44] *** Toasterson <Toasterson!~toast@2a02:aa10:e100:4b00:3798:d028:fa3e:2cf8> has joined #omnios[18:23:29] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[18:33:02] *** ptribble <ptribble!~ptribble@cpc92320-cmbg19-2-0-cust3282.5-4.cable.virginm.net> has quit IRC (Quit: Leaving)[18:49:15] *** bens1 <bens1!~bens1@161.122.2.81.in-addr.arpa> has quit IRC (Quit: bens1)[18:53:35] *** bahamat <bahamat!~bahamat@ip72-199-207-63.sd.sd.cox.net> has joined #omnios[19:28:37] *** ptribble <ptribble!~ptribble@cpc92320-cmbg19-2-0-cust3282.5-4.cable.virginm.net> has joined #omnios[19:36:05] *** dmip <dmip!~Thunderbi@95.143.115.50> has quit IRC (Ping timeout: 250 seconds)[19:53:36] <justif> any one here who can help with a disk performance issue?[20:03:07] <danmcd> Maybe. I'm not a storage guy per se, but try me.[20:06:14] <justif> c1t0d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0[20:06:14] <justif> Vendor: SanDisk Product: LT0400WM Revision: D40Z Serial No:[20:06:14] <justif> Size: 400.09GB <400088457216 bytes>[20:06:14] <justif> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0[20:06:14] <justif> Illegal Request: 366 Predictive Failure Analysis: 0[20:06:15] <justif> c1t1d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0[20:06:15] <justif> Vendor: SanDisk Product: LT0400WM Revision: D40Z Serial No:[20:06:16] <justif> Size: 400.09GB <400088457216 bytes>[20:06:16] <justif> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0[20:06:17] <justif> Illegal Request: 364 Predictive Failure Analysis: 0[20:06:31] <justif> seeing illegal requests on my zil drives[20:06:38] <justif> and poor performance[20:06:56] <justif> high IO wait and await times in vms hosted on this storage[20:08:54] <justif> have similar systems with no issues that still have errors but it seems the rate is much lower[20:13:00] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[20:16:41] <danmcd> "zpool status" have anything to say?[20:17:21] <danmcd> And this is your slog device, right?[20:17:35] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Ping timeout: 258 seconds)[20:18:58] <justif> https://www.irccloud.com/pastebin/NN484m9t/[20:19:02] <justif> yup[20:22:14] <justif> https://www.irccloud.com/pastebin/XMThnpQM/[20:23:01] <justif> this also looks bad, 182 writes a sec at 15Mbps a sec and 80% utilized?[20:23:05] <justif> for ssds?[20:36:07] <danmcd> Seems weird, but there may be a reason. (How big are the writes, for one thing?)[20:41:50] <justif> iscsi mounts to a kvm box running the vms - got an easy way to find out?[20:46:01] <danmcd> Could it be iSCSI is the problem?[20:46:23] <justif> same setup in another DC and seems to be fine[20:46:32] <justif> same hardware, same software[20:46:35] <justif> same config[20:47:13] <danmcd> Well, if it's all the same elsewhere and working, then your drive is likely the culprit, OR what your drive is attached to.[20:47:48] <justif> next step I was going to do was to try and remove the zil[20:47:56] <justif> and see if it is the SSDs[20:49:02] <patdk-wk> remove zil?[20:49:10] <patdk-wk> just set sync=none[20:49:22] <justif> guess that works also[20:50:01] <patdk-wk> what are the slog drives?[20:50:23] <justif> SAS write intensive SSDs[20:51:28] <patdk-wk> hmm, asvc_t is in the 20ms range[20:51:33] <patdk-wk> wonder why that isn't <1[20:53:25] <patdk-wk> mine is 0.3ms at 14mb/sec[20:57:37] <justif> yea perf seems much better with sync=disabled[20:58:00] <patdk-wk> than your disks don't flush very nicely[20:58:12] <patdk-wk> or slog drives are having issues[20:58:21] <justif> so ssds=the problem?[20:58:29] <justif> now that I look at the other boxes[20:58:37] <justif> they have Toshiba ssds[20:58:45] <justif> and the problem ones are SanDisk[20:59:06] <justif> now going to check for firmware updtes[20:59:10] <justif> *updates[21:11:25] *** _Tenchi_ <_Tenchi_!~phil@207-255-68-080-dhcp.aoo.pa.atlanticbb.net> has quit IRC (Ping timeout: 252 seconds)[21:21:08] *** matejz <matejz!~matejz@element.planetq.org> has joined #omnios[21:23:30] <danmcd> justif: is that helping? (And thanks patdk-wk for the extra clue.)[21:23:53] *** synegi <synegi!~gatekeep@2001:470:8b45:400:7578:bb49:4a1f:bf90> has joined #omnios[21:24:19] <justif> danmcd: it is, just need to resolve the ssd performance issue[21:24:27] <danmcd> Cool.[21:24:49] <patdk-wk> 20ms is GREEN HD speeds :)[21:27:27] *** synegy34 <synegy34!~gatekeep@2001:470:8b45:400:7578:bb49:4a1f:bf90> has quit IRC (Ping timeout: 250 seconds)[21:43:47] *** matejz <matejz!~matejz@element.planetq.org> has quit IRC (Quit: matejz)[21:44:51] *** Toasterson <Toasterson!~toast@2a02:aa10:e100:4b00:3798:d028:fa3e:2cf8> has quit IRC (Ping timeout: 264 seconds)[21:50:38] *** dmip <dmip!~Thunderbi@84.51.125.126> has joined #omnios[22:17:19] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[22:31:57] *** ptribble <ptribble!~ptribble@cpc92320-cmbg19-2-0-cust3282.5-4.cable.virginm.net> has quit IRC (Quit: Leaving)[22:37:28] *** synegy34 <synegy34!~gatekeep@2001:470:8b45:400:7578:bb49:4a1f:bf90> has joined #omnios[22:40:40] *** synegi <synegi!~gatekeep@2001:470:8b45:400:7578:bb49:4a1f:bf90> has quit IRC (Ping timeout: 250 seconds)[22:47:39] *** mfournier <mfournier!~marc@2001:4b98:dc2:41:216:3eff:fe6d:dc0b> has quit IRC (Ping timeout: 260 seconds)[22:53:58] *** fwp <fwp!~fwp@www.deepthought.com> has joined #omnios[22:59:13] *** Kurlon_ <Kurlon_!~Kurlon@bidd-pub-04.gwi.net> has joined #omnios[23:02:02] *** mfournier <mfournier!~marc@xvm-109-226.dc2.ghst.net> has joined #omnios[23:03:38] *** Kurlon_ <Kurlon_!~Kurlon@bidd-pub-04.gwi.net> has quit IRC (Ping timeout: 250 seconds)[23:03:45] *** Kurlon <Kurlon!~Kurlon@bidd-pub-03.gwi.net> has quit IRC (Ping timeout: 276 seconds)[23:04:32] <richlowe> danmcd: any luck?[23:04:40] <danmcd> illumos-omnios built.[23:04:51] <danmcd> OOod is a 4.5 hour run, and it's finishing up in ~45 mins.[23:04:59] <richlowe> k[23:05:08] <richlowe> at least if that one fails it's your fault not mine ;)[23:05:10] <danmcd> I'll do testing tonight or tomorrow. I think I'm going to have less hard of a time than OI, because I never got the mediators for sunssh right in the first place.[23:08:40] *** Botanic_ <Botanic_!~Botanic@107.150.184.246> has joined #omnios[23:08:54] *** Botanic_ <Botanic_!~Botanic@107.150.184.246> has quit IRC (Changing host)[23:08:54] *** Botanic_ <Botanic_!~Botanic@tita/admin/Botanic> has joined #omnios[23:10:03] *** Botanic <Botanic!~Botanic@tita/admin/Botanic> has quit IRC (Ping timeout: 240 seconds)[23:31:23] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Remote host closed the connection)[23:32:25] *** fwp_ <fwp_!~fwp@75-149-197-38-Illinois.hfc.comcastbusiness.net> has joined #omnios[23:33:57] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has joined #omnios[23:34:58] *** fwp <fwp!~fwp@www.deepthought.com> has quit IRC (Ping timeout: 252 seconds)[23:38:30] *** natkin <natkin!~natkin@cpe-70-117-102-168.austin.res.rr.com> has quit IRC (Ping timeout: 265 seconds)[23:41:11] *** Kurlon <Kurlon!~Kurlon@cpe-104-228-216-170.maine.res.rr.com> has joined #omnios[23:43:30] *** patdk-wk <patdk-wk!~dswett@2001:470:e0ba:5:f55f:621d:64c1:d9c8> has quit IRC (Ping timeout: 258 seconds)[23:46:21] *** danmcd <danmcd!~danmcd@static-71-174-113-16.bstnma.fios.verizon.net> has quit IRC (Quit: danmcd)[23:53:07] <richlowe> I think that shouldn't matter, hard-time wise[23:53:20] <richlowe> the only user-facing downside I could find is that IPS is unfriendly if you set a mediator to something that goes away[23:53:24] <richlowe> it, for some reason, still honours that mediator[23:53:28] <richlowe> (and leaves the links dangling)[23:55:47] <richlowe> but anyone who _explicitly_ chose to use sunssh deserves what they get :)