Switch to DuckDuckGo Search
   January 31, 2012  
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | >

Toggle Join/Part | bottom
[00:00:59] *** ira has joined #openindiana
[00:03:15] *** ira has quit IRC
[00:03:57] *** ivan` has joined #openindiana
[00:18:16] *** joes_ has joined #openindiana
[00:18:31] *** joes_ has left #openindiana
[00:18:40] *** vados has quit IRC
[00:18:40] *** vados_ has joined #openindiana
[00:18:45] *** nvez has joined #openindiana
[00:19:37] *** joes_ has joined #openindiana
[00:19:51] *** joes_ has left #openindiana
[00:23:13] *** cis has quit IRC
[00:40:08] *** Hexi has quit IRC
[00:45:55] *** Hexi has joined #openindiana
[00:54:09] <CentralStates> Does anyone have experience with any kind of MPIO with NFS / ESXi through different switches (10gb / 1gb) ?
[00:58:20] *** Zenos has quit IRC
[01:05:15] <Patrickdk> you don't do mpio with nfs
[01:05:20] <Patrickdk> it's not possible
[01:06:03] <Patrickdk> if you want mpio on esxi you must use fc or iscsi
[01:06:48] <Triskelios> maybe CentralStates just means link aggregation, not MPIO?
[01:06:56] <CentralStates> That is what I have figured so far
[01:07:32] <CentralStates> No - I dont want to use LAG - I have a 10gb switch, and a 1gb swich... I was some kinda protection if the 10gb switch fails
[01:09:15] *** Aengus has joined #openindiana
[01:10:13] <CentralStates> I guess, if the 10gb switch fails, I can recreate NFS datastores fairly quickly and get the VM back up, but there has been mention of failover setups without LAG, through seperate switches (researching through google)
[01:10:36] *** vados_ has quit IRC
[01:13:57] *** joffe has quit IRC
[01:15:23] <Triskelios> you could use IPMP
[01:16:42] <CentralStates> Ok - lemme read up on that
[01:16:46] <CentralStates> Thanks much
[01:36:08] *** gea has quit IRC
[01:43:09] *** InTheWings has quit IRC
[01:45:57] *** enmand has quit IRC
[01:45:57] *** CVLTCMK0 has quit IRC
[01:48:54] *** nvez has quit IRC
[02:12:58] *** nvez has joined #openindiana
[02:12:58] *** nvez has joined #openindiana
[02:15:47] *** CVLTCMK0 has joined #openindiana
[02:21:12] *** vados has joined #openindiana
[02:21:49] *** CVLTCMK0 has quit IRC
[02:26:51] *** brougham has quit IRC
[02:27:06] *** brougham has joined #openindiana
[02:29:08] *** CVLTCMK0 has joined #openindiana
[02:31:55] *** brougham has quit IRC
[02:36:57] *** brougham has joined #openindiana
[03:06:38] *** vados has quit IRC
[03:10:12] *** vados has joined #openindiana
[03:25:25] *** nvez has quit IRC
[03:25:26] *** Shep_Book has quit IRC
[03:25:26] *** heldus has quit IRC
[03:25:26] *** axisys has quit IRC
[03:25:26] *** OMV-User has quit IRC
[03:25:26] *** CentralStates has quit IRC
[03:25:26] *** geoff_ has quit IRC
[03:25:26] *** Alasdairrr has quit IRC
[03:25:26] *** russiane39 has quit IRC
[03:25:26] *** SH0x has quit IRC
[03:25:26] *** Tempt has quit IRC
[03:25:26] *** [noc] has quit IRC
[03:25:26] *** descipher has quit IRC
[03:25:26] *** Tenzer has quit IRC
[03:25:26] *** justinsb has quit IRC
[03:25:26] *** disharmony has quit IRC
[03:25:26] *** Gugge has quit IRC
[03:25:26] *** fbettag has quit IRC
[03:25:27] *** mixis_ has quit IRC
[03:25:27] *** FireflyST has quit IRC
[03:25:27] *** IDoNotCare has quit IRC
[03:25:27] *** deraps has quit IRC
[03:25:27] *** docsteel has quit IRC
[03:25:27] *** ltz has quit IRC
[03:25:27] *** Hoaxter has quit IRC
[03:25:31] *** IDoNotCare has joined #openindiana
[03:25:32] *** axisys has joined #openindiana
[03:25:33] *** mixis has joined #openindiana
[03:25:33] *** FireflyST has joined #openindiana
[03:25:33] *** Tempt has joined #openindiana
[03:25:35] *** Alasdairrr has joined #openindiana
[03:25:37] *** ChanServ sets mode: +o Alasdairrr
[03:25:39] *** SH0x has joined #openindiana
[03:25:40] *** Shep_Book has joined #openindiana
[03:25:40] *** russiane39 has joined #openindiana
[03:25:43] *** heldus has joined #openindiana
[03:25:44] *** [noc] has joined #openindiana
[03:25:44] *** deraps has joined #openindiana
[03:25:50] *** CentralStates has joined #openindiana
[03:25:51] *** Hoaxter has joined #openindiana
[03:25:51] *** justinsb has joined #openindiana
[03:25:55] *** descipher has joined #openindiana
[03:25:59] *** nvez has joined #openindiana
[03:26:00] *** nvez has joined #openindiana
[03:28:44] *** voidcoder has quit IRC
[03:28:45] *** OMV-User has joined #openindiana
[03:30:40] *** Tenzer has joined #openindiana
[03:31:23] *** vados has quit IRC
[03:39:45] *** geoff_ has joined #openindiana
[03:40:01] *** Gugge has joined #openindiana
[03:45:15] *** voidcoder has joined #openindiana
[03:45:43] *** cis has joined #openindiana
[03:51:38] *** POloser has joined #openindiana
[03:53:14] *** Licenser_ has joined #openindiana
[03:54:42] *** Licenser has quit IRC
[03:54:42] *** Licenser_ is now known as Licenser
[04:02:31] *** Botanic has quit IRC
[04:02:45] *** Botanic has joined #openindiana
[04:02:45] *** Botanic has quit IRC
[04:02:45] *** Botanic has joined #openindiana
[04:16:36] *** kart_ has joined #openindiana
[04:21:01] *** ira has joined #openindiana
[04:37:53] *** DanaG has joined #openindiana
[04:40:21] *** ira has quit IRC
[04:50:54] *** Seony has quit IRC
[05:05:32] *** Seony has joined #openindiana
[05:18:48] *** Shep_Book has quit IRC
[05:30:46] <DanaG> Say, what group do I need to add for my "vbox" user (that is, username is "vbox") to have rawdisk access?
[05:33:21] *** voidcoder has quit IRC
[05:34:03] *** voidcoder has joined #openindiana
[05:45:40] <POloser> it is possible to disable zfs prefetch on pool basis?
[05:47:22] <DanaG> hokay, so what's the OI equivalent of udev?
[05:47:38] *** xdnny_ has quit IRC
[05:59:37] *** Botanic has quit IRC
[06:01:38] <DanaG> say, where the heck are lsusb and lspci?
[06:02:00] <alanc> sitting on a system running the Linux kernel they were written for
[06:02:59] *** lennard has quit IRC
[06:03:08] <alanc> OI has HAL for handling device adding/discovery type stuff instead of udev, and libraries like libdevinfo for walking the device tree
[06:03:11] *** lennard has joined #openindiana
[06:03:29] <alanc> lshal should find usb devices, scanpci should find PCI devices
[06:04:25] <Agnar> DanaG: generally devfsadmd will care
[06:04:39] <DanaG> ah yeah, they still use hal.
[06:04:53] <DanaG> Now I just need those damn disk permissions.
[06:04:54] <Agnar> "still"
[06:05:16] <DanaG> Oh, and is there a pulseaudio for OI, or am I better off handing my USB headset adapter to the Linux guest?
[06:05:21] <DanaG> When I try vbox's audio, it's mono!
[06:05:27] *** LizardKing has quit IRC
[06:05:29] <alanc> yes, as all the HAL replacements were written with the attitude "Only Linux matters, fuck all other kernels"
[06:05:35] <DanaG> speaker-test -c2.... comes out mono on the host.
[06:06:19] <DanaG> ah, I think I will just pass the guest the headset.
[06:06:24] <Agnar> DanaG: you should be able to just chmod/chown the disk devices - it stays persistant
[06:06:30] <Agnar> devfs will care
[06:06:34] <DanaG> Spiffy. No udev.
[06:06:40] <Agnar> no stupid udev rules needed
[06:06:48] <DanaG> hmm, but if I give one guest the headset, the other guest won't get audio.
[06:07:24] <Agnar> DanaG: and just to give you a good advice - if you want OI being Linux, just install Linux, hm? ;)
[06:08:32] <DanaG> nah, I have the Linux guest for things like PA and Deluge.
[06:09:23] <DanaG> I also need to figure out a safe way to integrate my Ubuntu user with my OI user.
[06:09:38] <DanaG> Right now I'm using nfs mapping, but that means anyone named "dana" who tries to connect will become me.
[06:10:00] <DanaG> I'd make it do that on the host-only network adapter... but the host sees vboxnet0 at 0.0.0.0.
[06:10:38] <DanaG> Oh yeah, and what's the equivalent of alsamixer?
[06:10:49] <Agnar> DanaG: why not simply using the same uid?
[06:11:18] <alanc> OI audio is based on OSS if that helps you google for equivalents
[06:11:29] <RobbieCrash> How can I get OI to use SMB2 instead of SMB1? This post: https://www.illumos.org/boards/1/topics/743 seems to point to a patch for it, but I'm not sure that this isn't just a fix for OSX which I don't care about SMB with. I would, however, rather move to smb2 for my Windows clients that support it.
[06:11:34] <DanaG> or wait, maybe that's what I did.
[06:11:34] <DanaG> Seems like a hack, though.
[06:11:42] <Agnar> DanaG: solaris uses OSS. actually I don't know what what alsamixer is, but gnome has volume-control-foobar
[06:12:06] <DanaG> My microserver is headless.
[06:12:23] <DanaG> Has a remote access card... but the remote access card is a piece of crap that sometimes can't be remotely accessed.
[06:13:20] <RobbieCrash> Additionally, I have a pool with 2 RAIDZ arrays in it. The last disk in the second array (c7d0) died and was just replaced. Now, the same disk (c7d0) is giving me checksum errors. This time, when I do iostat -En, there are no errors reported on it, or any other device. The array has already resilvered, and has been functioning properly since the drive was replaced. However, it's only been
[06:13:20] <RobbieCrash> resilvered for about 30 hours.
[06:14:01] <DanaG> I made the mistake of handing my VM the zfs disks when the host was still remounting them even when I unmounted them deliberately.
[06:14:23] <DanaG> Totally hosed the pools... paniced ZFSonLinux, OI, and even Solaris proper.
[06:16:31] <DanaG> oh yeah, and where is the program "gdate"? The vbox SMF service definition needs it.
[06:20:47] <alanc> in the gnu-coreutils package, as "pkg search gdate" should find for you
[06:22:54] <DanaG> root@microserver:~# pkg search gdate
[06:22:54] <DanaG> root@microserver:~#
[06:23:01] <DanaG> i.e. no results.
[06:23:44] <DanaG> and if I pkg install gnu-coreutils: No updates necessary for this image.
[06:24:03] *** Botanic has joined #openindiana
[06:24:03] *** Botanic has joined #openindiana
[06:24:45] *** LizardKing has joined #openindiana
[06:25:40] <Agnar> DanaG: I bet gdate should be a link to /usr/gnu/bin/date
[06:26:07] *** lennard has quit IRC
[06:26:36] <alanc> oh, maybe the link is only in newer pkg versions
[06:26:44] *** lennard has joined #openindiana
[06:30:07] *** nvez has quit IRC
[06:30:48] *** brougham has quit IRC
[06:31:09] *** brougham has joined #openindiana
[06:32:01] <DanaG> Thanks.
[06:48:50] *** DrLou_ has quit IRC
[06:51:33] *** brougham has quit IRC
[06:52:08] *** brougham has joined #openindiana
[06:52:13] *** keremet has joined #openindiana
[06:52:53] *** jgrafton has joined #openindiana
[07:01:07] *** jgrafton_ has joined #openindiana
[07:03:15] <DanaG> argh, how do I find a package I've installed, but that doesn't show up in pkg search?
[07:03:20] <DanaG> pkg search oss... gives nothing.
[07:03:20] *** gregburd has quit IRC
[07:03:33] *** jgrafton has quit IRC
[07:04:32] *** gregburd has joined #openindiana
[07:04:48] <Agnar> DanaG: oss is not a package
[07:05:24] <DanaG> well, it's a .pkg file -- oss4, that is.
[07:05:31] <Agnar> ah
[07:05:33] <DanaG> But won't load the usb sound driver.
[07:05:48] <Agnar> pkginfo - it's a legacy srv4 package
[07:05:49] <DanaG> oss-solaris-v4.2-2005-i386.pkg
[07:06:08] <DanaG> argh, why is there no tab-complete on package names?
[07:06:13] <DanaG> Ubuntu has really spoiled me about that.
[07:06:22] <DanaG> pkginfo oss<tab><tab><tab><tab, damnit>
[07:06:35] <DanaG> oh, pkginfo | grep
[07:07:30] <DanaG> so, where's the command-line equivalent of alsamixer?
[07:07:46] <Agnar> DanaG: you are welcome to write a lot of bash tab completion-code
[07:09:18] <Agnar> no idea if there is a command line tool for changing sound volume.
[07:09:23] <DanaG> oh, I see.. I need to use wildcards.
[07:09:27] <DanaG> or something like that.
[07:09:36] <DanaG> I get nothing for pkg search mix
[07:09:47] <DanaG> I get tons of actual package contents for "pkg search *mix*"
[07:10:35] <Agnar> there used to be some scripts together with CDE in /usr/dt/bin - but they are gone, since CDE went away
[07:11:13] <Agnar> maybe there's something in SoX
[07:12:49] <DanaG> argh, fine, i'll ssh -X and run gnome-volume-control...
[07:13:40] <Agnar> do I want to know why you need to change the sound-volume on a box you are not sitting in front of?
[07:13:48] <DanaG> well, it's about 3 feet away.
[07:13:58] <DanaG> But has a remote-access card because I don't have a monitor for it.
[07:14:12] <DanaG> I'm thinking of just _getting_ a monitor, and setting the Radeon back as primary GPU.
[07:14:51] <DanaG> great, so the oss4 uninstaller left me with no sound modules at all.
[07:14:53] <Agnar> DanaG: I have no idea how long a feet is - I'm from the metric world. anyway - have you tried sox?
[07:15:18] <DanaG> ah, say, 1.5 meters? That's not equivalent, but I'm rethinking the distance.
[07:15:25] <DanaG> Nah, more like 1.0.
[07:15:31] <Agnar> ok, so very close
[07:16:30] <DanaG> And I want to be able to stgream to it via pulseaudio
[07:16:32] <Agnar> but headless, ok. well, sound-support was designed for workstations...and usually the user sits in front of it - so I guess nobody needed a cmd-line tool
[07:16:46] <DanaG> If that's via a Linux guest, that's fine too.
[07:17:29] <Agnar> DanaG: maybe just google for a cmd-line OSS mixer tool and compile it
[07:17:48] <DanaG> well, now the oss4 package script has taken out the existing usb audio driver.
[07:18:23] <DanaG> so now I need to figure out how to get it back.
[07:18:24] <RobbieCrash> How can I figure out why I'm getting absolutely horrible delete performance? Like '15 minutes to delete 4 files totaling 2GB' horrible
[07:18:30] <Agnar> DanaG: do you have the filename of the driver? then you can pkg search the package, remove and install it
[07:19:29] *** hajma has quit IRC
[07:19:29] <Agnar> RobbieCrash: dtrace
[07:19:49] <Agnar> or truss
[07:19:51] <RobbieCrash> I'm not showing a load on anything, I've got 5GB RAM free, CPU is 98% idle, 0.0 iowait. zpool iostat shows minimal activity.
[07:19:51] <Agnar> or mdb
[07:19:56] <RobbieCrash> haha, thanks
[07:20:02] <DanaG> ah, it's "usb_ac"
[07:20:51] <Agnar> RobbieCrash: I'm sure the unlink() systemcall is waiting for a lock or something
[07:21:18] <Agnar> or it is simply waiting to return from the disk
[07:22:21] <DanaG> package is driver/usb, but I need to reinstall it.
[07:22:48] <Agnar> DanaG: pkg uninstall, pkg install should work
[07:23:47] <DanaG> Too many dependencies.
[07:23:48] <RobbieCrash> Agnar thanks. It's just delete that is horrid right now. Reads and writes to the same FS, even same directory are zippy, even with files that are too big to be cached. So I dunno if it's disk wait, but I'ma go try to figure out dtrace now. THanks for the pointer.
[07:24:16] <DanaG> pkg://openindiana.org/driver/serial/usbser_edge at 0 dot 5.11,5.11-0.151.1:20110912T031042Z pkg://openindiana.org/driver/serial/usbsacm at 0 dot 5.11,5.11-0.151.1:20110912T031042Z pkg://openindiana.org/driver/network/urtw at 0 dot 5.11,5.11-0.151.1:20110912T031036Z pkg://openindiana.org/driver/network/zyd at 0 dot 5.11,5.11-0.151.1:20110912T031039Z pkg://openindiana.org/driver/graphics/usbvc at 0 dot 5.11,5.11-0.151.1:20110912T030945Z pkg://openindiana.org/
[07:24:24] <DanaG> hey, that's all one line. =þ
[07:25:38] <Agnar> RobbieCrash: maybe truss will be helpful enough. tell truss to write time deltas and see where it spends most time
[07:29:47] <Agnar> DanaG: just the usb audio package should be enough
[07:32:10] <madwizard> Coffee
[07:32:33] <DanaG> oh hey, the file is still there.
[07:32:36] <DanaG> /kernel/drv/usb_ac
[07:32:44] <DanaG> Now I just need to rebind it to the device.
[07:33:22] <Agnar> madwizard: good idea
[07:33:42] *** Seony has quit IRC
[07:33:52] *** Seony has joined #openindiana
[07:33:53] <DanaG> but I have no idea how to do so.
[07:34:02] <DanaG> root@microserver:~# modload /kernel/drv/usb_ac
[07:34:02] <DanaG> can't load module: No such device or address
[07:34:22] <Agnar> man modload
[07:34:24] <Agnar> man add_drv
[07:35:19] <Agnar> (easiest way could be trying a reboot...)
[07:37:15] <Agnar> I'd like to have a "chat" with the developers of live-upgrade...
[07:37:18] <Agnar> grrrr.
[07:38:14] <Agnar> and while we are there...could please somebody send me the vxvm/vxfs guys too?
[07:38:48] <DanaG> devfsadm: driver failed to attach: usb_ac
[07:38:48] <DanaG> Warning: Driver (usb_ac) successfully added to system but failed to attach
[07:40:20] <Agnar> that means the driver did not found a device
[07:40:32] <Agnar> maybe attached to the wrong pci-id
[07:41:06] <Agnar> usually add_drv/modload is not a good idea if you have no idea what you are doing...
[07:41:08] *** Nitial has joined #openindiana
[07:42:16] <DanaG> hmm, should I just reboot?
[07:42:34] <DanaG> I'm trying it, with "touch /reconfigure" first.
[07:45:40] <DanaG> nope, still no /dev/dsp
[07:47:54] <Agnar> with add_drv/modload you have modified your /etc/path_to_inst - that way OI knows on reboot where the driver used to be attached to.
[07:48:34] <Agnar> you could try to remove just the line for usb_ac from /etc/path_to_inst and reboot -- -r
[07:50:08] <DanaG> I see, I got usb_ac, but not usb_as
[07:50:37] <DanaG> nope, that's a red herring
[07:51:19] *** trn has quit IRC
[07:54:16] <DanaG> nope, still no dsp or audio.
[07:54:38] *** trn has joined #openindiana
[07:55:30] *** kart_ has quit IRC
[07:56:06] *** jw_urodoc has joined #openindiana
[07:56:15] *** kart_ has joined #openindiana
[07:56:18] *** DanaG has quit IRC
[07:59:34] *** Okona has joined #openindiana
[08:02:19] *** DanaG has joined #openindiana
[08:08:18] *** kdavyd has joined #openindiana
[08:08:45] <DanaG> So, still no-go.
[08:08:49] *** _404_Error_ has quit IRC
[08:17:35] <DanaG> Oh, duh, i can zfs ro;lbback.
[08:19:13] *** girafe has quit IRC
[08:23:05] *** gea has joined #openindiana
[08:26:34] *** brougham has quit IRC
[08:27:06] *** brougham has joined #openindiana
[08:45:22] *** gea has quit IRC
[08:50:23] *** Onno has quit IRC
[08:57:44] *** bens1 has joined #openindiana
[09:01:58] *** dijenerate has quit IRC
[09:02:09] *** Worsoe has joined #openindiana
[09:10:03] *** dijenerate has joined #openindiana
[09:11:09] *** Zenos has joined #openindiana
[09:14:52] *** TomJ has quit IRC
[09:14:56] *** |AbsyntH| has joined #openindiana
[09:16:49] *** Botanic has quit IRC
[09:17:22] *** jMCg has quit IRC
[09:18:14] *** jMCg has joined #openindiana
[09:26:06] *** bens1 has quit IRC
[09:32:44] *** Licenser has quit IRC
[09:41:35] *** Onno has joined #openindiana
[09:53:11] *** DanaG has quit IRC
[09:55:06] *** Onno has quit IRC
[10:05:22] <RobbieCrash> Does OI support 3TB drives?
[10:06:19] *** trentster has joined #openindiana
[10:07:36] *** LizardKing has quit IRC
[10:08:39] <trentster> Hi all, I am trying to install openindiana 151a text from a usb cd-rom drive, it gets as far as " All rights reserved" and then just stops there indefinitely. I have tried other ISO cd's like Freebsd 9 on same carom and boots just fine. Any ideas what I can do to get this to work.?
[10:10:57] <RobbieCrash> what chipset? z68?
[10:11:51] <Worsoe> trentster: try add -v to the kernel line in grub
[10:12:32] <trentster> RobbieCrash: yes its a Z68
[10:13:16] <RobbieCrash> good luck. I tried 5 different z68mobos before giving up and getting a p67. I couldn't find anyone that was using a z68 successfully.
[10:15:13] <trentster> RobbieCrash: aaargh thats bad news indeed, I just bought this mono this morning especially for this setup....
[10:15:27] *** CentralStates has quit IRC
[10:22:59] *** Whoopsie has joined #openindiana
[10:23:00] *** ChanServ sets mode: +v Whoopsie
[10:25:33] *** brougham has quit IRC
[10:25:40] <RobbieCrash> Yeah, I did the same thing. It failed at the same point with each of the mobos. When I enabled verbose kernel debug messages I found it failed at the same point with each; as soon as it started loading ISA modules.
[10:25:44] *** jgrafton_ has quit IRC
[10:26:00] *** brougham has joined #openindiana
[10:26:09] *** tsoome has quit IRC
[10:35:23] *** LizardKing has joined #openindiana
[10:37:33] *** bens1 has joined #openindiana
[10:48:19] *** movement has joined #openindiana
[10:52:14] *** tsoome has joined #openindiana
[10:57:05] *** Bdragon28 has quit IRC
[10:59:08] *** raichoo has joined #openindiana
[11:01:32] *** raichoo has quit IRC
[11:01:39] *** raichoo has joined #openindiana
[11:01:39] *** Bdragon28 has joined #openindiana
[11:04:16] *** lucalenardi has joined #openindiana
[11:05:22] *** vados has joined #openindiana
[11:08:01] *** Licenser has joined #openindiana
[11:17:25] *** TomJ has joined #openindiana
[11:17:36] *** TomJ has quit IRC
[11:17:37] *** TomJ has joined #openindiana
[11:18:08] *** TomJ has quit IRC
[11:23:54] *** master_of_master has quit IRC
[11:24:12] <floogy> Is there an alternative to "veto oplock files" for the CIFS server, like in samba?
[11:25:01] *** laogrymt has joined #openindiana
[11:26:03] *** master_of_master has joined #openindiana
[11:28:33] *** TBCOOL has quit IRC
[11:32:05] *** heldus has quit IRC
[11:33:52] *** movement has quit IRC
[11:34:33] *** hajma_ has joined #openindiana
[11:35:29] *** noahmehl has quit IRC
[11:39:48] *** mikw has joined #openindiana
[11:41:19] *** TBCOOL has joined #openindiana
[11:47:42] *** TBCOOL has quit IRC
[11:52:05] *** Bdragon28 has quit IRC
[11:52:19] *** heldus has joined #openindiana
[11:56:39] *** TBCOOL has joined #openindiana
[12:00:41] *** Bdragon28 has joined #openindiana
[12:08:57] *** InTheWings has joined #openindiana
[12:15:33] *** bongo has quit IRC
[12:28:03] *** _404_Error_ has joined #openindiana
[12:35:34] *** movement has joined #openindiana
[12:42:36] *** POloser has left #openindiana
[12:45:53] *** movement has quit IRC
[12:54:46] *** movement has joined #openindiana
[12:56:33] *** RobbieCrash has quit IRC
[12:58:26] <heldus> how can i zfs send/receive with keeping the properties in OI? the solaris way (http://docs.oracle.com/cd/E19963-01/html/821-1448/gbchx.html#gjonv) does not seem to apply
[13:04:05] *** movement has quit IRC
[13:08:29] *** kart_ has quit IRC
[13:09:34] *** kart_ has joined #openindiana
[13:12:37] *** enmand has joined #openindiana
[13:25:34] *** CVLTCMK0 has quit IRC
[13:26:37] *** myrkraverk has joined #openindiana
[13:26:37] *** myrkraverk has joined #openindiana
[13:32:43] *** CVLTCMK0 has joined #openindiana
[13:44:31] *** cis has quit IRC
[13:45:09] *** brougham has quit IRC
[13:45:41] *** brougham has joined #openindiana
[13:51:09] *** enmand has quit IRC
[14:13:43] *** enmand has joined #openindiana
[14:14:16] *** brougham has quit IRC
[14:14:35] *** brougham has joined #openindiana
[14:24:59] *** cis has joined #openindiana
[14:27:51] *** voidcoder has quit IRC
[14:28:09] *** voidcoder has joined #openindiana
[14:34:54] *** DrLou_ has joined #openindiana
[14:34:55] *** ChanServ sets mode: +o DrLou_
[14:36:35] *** vados has quit IRC
[14:38:14] *** Seony has quit IRC
[14:44:27] *** nvez has joined #openindiana
[14:44:27] *** nvez has joined #openindiana
[14:54:22] *** CoilDomain has joined #openindiana
[14:55:17] <ianj> has anyone accidentally enabled cursor themes and had to change back to the default?
[15:03:31] *** brougham has quit IRC
[15:09:25] *** brougham has joined #openindiana
[15:13:24] *** joffe has joined #openindiana
[15:27:55] *** flyz has joined #openindiana
[15:32:44] *** Shep_Book has joined #openindiana
[15:36:19] *** Infin1ty has joined #openindiana
[15:37:44] <Infin1ty> i'm using comstar as an iscsi target, and as a test i exposed a lun to a centos 6.2 intiator, it all goes over a 1GbE Switch, The target (oi) has an aggregated link (2 x 1 GbE), all works well, i get a huge performance penalty while doing reads from the LUN, writes are fast but reads are very very slow, about x10 from the writes.
[15:38:04] <Infin1ty> If i use NFS on that server (the one that is the target), read and write are fast, equally almost
[15:51:08] *** noahmehl has joined #openindiana
[15:52:41] <tsoome> first thing is that aggregated link does not help you with single client;)
[15:53:41] <tsoome> second thing to test is pure network layer - if your nic is the actual issue.
[15:54:45] <tsoome> altho your nfs test sort of excludes that
[15:55:23] <tsoome> but there are some but(s) still
[15:57:27] <tsoome> the thing is that nfs itself defaults to 32k blocks (i believe - you should verify it) and once request hits actual filesystem layer, it will deal with filesystem blocks (128k default). iscsi if set up on top of zvol, is using zvol block size, which is 8k, which is indeed 16 times smaller.
[15:58:21] <tsoome> so, if you did read 128k block from nfs, the zfs had to do 1 read, and from zvol it had to do 16 reads....
[15:58:32] <tsoome> (assuming the data is not cached)
[16:02:53] <tsoome> what you can do is first, test with cached data - read same chunk of data over and over and see if second read is faster (iostat in server should confirm you have no reads from disks), just make sure the read amount is large enough to give meaningful results and will fit into cache.
[16:03:25] <tsoome> and second thing - you can create zvol with larger recordsize to se if that will improve the results.
[16:03:35] <Krazypoloc> Also I would drop the aggregate link and do multipathing
[16:03:59] <Krazypoloc> AFAIK its not a best practive to do link aggregation with iSCSI
[16:04:42] <mui> depends
[16:05:19] <tsoome> well, im not sure if the aggregation will really matter there or not.
[16:05:52] <tsoome> i mean, if the traffic will use only one physical link then it really shouldn't matter.
[16:06:28] <mui> Infin1ty: you didnt mention what kind of disk configuration you have?
[16:07:12] <mui> and do you have more than one reader when testing?
[16:07:20] *** elgar has joined #openindiana
[16:08:26] <patdk-wk> I haven't had any issues with iscsi over aggregate links
[16:10:30] *** tsoome has quit IRC
[16:10:50] <patdk-wk> but I'm doing aggregate links, cause I have 6 nic, and running both nfs and iscsi
[16:11:03] <Krazypoloc> Yeah I have 4
[16:11:20] <Krazypoloc> Use 2 for LAN (aggregate) and 2 for iSCSI
[16:11:32] <Krazypoloc> Multipathing works well
[16:11:32] <patdk-wk> hmm?
[16:11:46] <patdk-wk> I just have mine in 3 sets of aggregates
[16:11:52] *** Worsoe has quit IRC
[16:12:01] <patdk-wk> just roundrobin the nfs mount dns, and iscsi multipath over all 3 aggregates
[16:13:13] <patdk-wk> my new setup, I'm just about done testing, is much more annoying
[16:13:21] <patdk-wk> adding vlans ontop of the aggregates
[16:13:42] <patdk-wk> then ipmp ontop of the vlans
[16:14:43] <Krazypoloc> Why would you do that?
[16:14:59] <Krazypoloc> Why not do the vlan'ing on the switch?
[16:15:11] <patdk-wk> heh?
[16:15:43] <patdk-wk> how am I going access my unit on 5 vlans?
[16:15:55] <patdk-wk> one nic per vlan? that is insane, and kills any change of failover
[16:16:00] <patdk-wk> chance
[16:16:13] <Krazypoloc> routing
[16:16:41] <patdk-wk> I really want to put multi-gigabit of nfs traffic over a router? not really
[16:16:55] <Krazypoloc> ....
[16:16:58] <Krazypoloc> Not router
[16:17:06] <Krazypoloc> IP routing via the switches
[16:17:15] <patdk-wk> that would turn the switch into a router, so yes, a router
[16:17:26] <Krazypoloc> Correct
[16:17:34] <patdk-wk> that would also require a l3 switch
[16:17:45] <patdk-wk> and also require having a l3 switch that is able to handle routing that much traffic
[16:17:58] <Krazypoloc> Correct
[16:18:16] * patdk-wk wonders if the l3 switch, will adjust mtu correctly
[16:18:16] <Krazypoloc> I just assumed these were in place
[16:18:17] <Krazypoloc> :)
[16:18:31] <patdk-wk> actually, this setup is all l2 switchs
[16:18:47] <patdk-wk> plus, keeping traffic on the correct vlan, is nice
[16:18:57] <Krazypoloc> I guess I'm so used to my HP 5406's...I thought everyone had L3 switches at this point
[16:18:58] <patdk-wk> nfs traffic is not jumboframes, yet, atleast
[16:19:01] <patdk-wk> iscsi is jumbo frames
[16:19:33] <patdk-wk> I have l3 switchs in some places, but most of them can only handle a 100mbit or so across it
[16:19:48] <patdk-wk> I don't think I use the l3 features of any of my switchs
[16:33:36] *** keremet has left #openindiana
[16:35:32] *** vados has joined #openindiana
[16:52:09] *** DrFart has quit IRC
[16:55:41] *** cwebber has joined #openindiana
[16:56:56] <Infin1ty> Can i use multipathing without configuring the target as multipath?
[16:57:03] <cwebber> any suggestions on finding a more current pkg for rubgems?
[16:57:09] <Infin1ty> sorry fo the delay :)
[16:58:30] <Infin1ty> mui, i have 5 vdevs, each vdev is a RAIDZ made out of 4 x 1TB WDC 7.2k DISKS (Enterprise ones), i also have a log cache (mirror) 32GB SSD (SLC), and a L2ARC cache which is 4 x 256GB SSD.
[16:58:39] *** raichoo has quit IRC
[16:59:41] <Infin1ty> tsoome, i did read a 16GB file at once (copied a 16G from the exposed disk into the local file system, which can handle about 100MB/s without a problem)
[17:00:07] <Infin1ty> one more thing, i noticed people are doing LVM on top of the exposed iscsi disks, is there any performance reason?
[17:00:56] <Tenzer> Infin1ty: It's probably in order to support a "disk" changing size
[17:01:12] <Tenzer> LVM is quite flexible in that regard
[17:01:51] *** _404_Error_ has quit IRC
[17:01:56] <Infin1ty> Tenzer, i see, because i know that by default, if i use a software raid and run lvm on top of it, the lvm knows to identify the stripe and width size, i assume i don't need special mount options (like sunit,swidth)
[17:02:20] <Tenzer> Oh, I don't know about that :>
[17:02:38] <Infin1ty> Oh and i don't have jumbo frames
[17:02:51] <Infin1ty> I am getting a 10GbE network setup next week to test if it will improve
[17:03:04] <Tenzer> I try not to use iSCSI, and generally don't like the extra layer of complexity LVM can provide on systems with physical disks
[17:03:26] <Infin1ty> it's based on the Intel 82599ES so it should be good i assume
[17:05:41] <Infin1ty> any other things i can and need to tweak if someone can give me a hint
[17:08:58] *** nvez_ has joined #openindiana
[17:09:39] *** nvez_ has joined #openindiana
[17:09:41] *** nvez has quit IRC
[17:09:41] *** nvez_ is now known as nvez
[17:11:06] *** Bdragon28 has quit IRC
[17:11:09] *** Bdragon|work has joined #openindiana
[17:18:35] *** nvez_ has joined #openindiana
[17:20:10] *** nvez has quit IRC
[17:21:10] *** Shep_Book has quit IRC
[17:21:40] *** vados has left #openindiana
[17:27:51] *** tsoome has joined #openindiana
[17:36:29] *** mikaeld has quit IRC
[17:37:06] *** APTX has quit IRC
[17:37:16] *** APTX has joined #openindiana
[17:38:11] *** mikaeld has joined #openindiana
[17:43:01] *** vados has joined #openindiana
[17:43:30] *** Darkproger has quit IRC
[17:44:08] <Infin1ty> tsoome, creating the zvol with a 128k blocksize did the trick, if i plan to use 8k (which i think is more effecient in iSCSI, if recall well), i have to create the zpool with a 8k recordsize as well?
[17:44:36] <tsoome> no, recordsize is dataset property
[17:44:45] *** noahmehl has quit IRC
[17:45:37] <Infin1ty> tsoome, I can create a zpool with a lower recordsize?
[17:46:16] <tsoome> not pool itself, the recordsize is filesystem or zvol property
[17:46:32] <tsoome> but, what you need to think of is an balance
[17:47:09] <tsoome> you can make recordsize smaller, but it means the zfs needs to make more IOPS to serve same amount of data
[17:47:15] <Infin1ty> tsoome, oh, it's the default property when i create a zpool, so i have to modify it
[17:47:37] <Infin1ty> i see, so for getting performance with iSCSI, creating it with 128k is prefered over 8k
[17:48:08] <tsoome> however, if all your host IO is very small and the blocks are large, then you have inflated reads/writes.
[17:48:15] *** Licenser has quit IRC
[17:49:51] <tsoome> also what you need to keep in mind is that systems will do various optimisations; for example, if your app will do 8 times 1k sequential write, the os can consolidate them into one 8k write.
[17:51:18] <tsoome> and such optimisations are the reason why in general you wanna have your back end blocks somewhat larger, thats why 128k records can be preferred even if your host filesystem block is , say, 8k
[17:52:18] <Infin1ty> tsoome, the filesystem on the host (default ext4/xfs) is 4k, the zvol is 8k though
[17:52:23] *** Infin1ty has quit IRC
[17:52:32] *** Infin1ty has joined #openindiana
[17:52:36] <Infin1ty> tsoome, i see i can set recordsize for a zvol (tank/test) as example, correct?
[17:52:39] *** Infin1ty has quit IRC
[17:52:39] *** Infin1ty has joined #openindiana
[17:52:42] <tsoome> yes
[17:53:01] <tsoome> but in case of zvols, you can set it only on creation.
[17:53:14] *** |AbsyntH| has quit IRC
[17:53:23] <Infin1ty> is recordsize different than blocksize? or is it the same?
[17:53:26] <Infin1ty> on the zvol :)
[17:53:34] <tsoome> its the same thing
[17:53:43] <Infin1ty> oh ok :) thanks
[17:54:28] <tsoome> there is one special case when you wanna match the zvol recordsize with host filesystem block size
[17:54:56] <tsoome> that case is when you are going to use deduplication
[17:55:12] <Infin1ty> as host filesystem, you mean the initiators?
[17:55:17] <tsoome> yes
[17:55:34] <Infin1ty> oh, so if i use NTFS as example, it uses 4k blocks, so it prefered to match if i use dedup (currently not in mind)
[17:56:05] *** hajma_ has quit IRC
[17:56:30] <Infin1ty> Are there also any consideration on partition alignment (fdisk on the initiator?) or is it not needed that much?
[17:56:34] <tsoome> and thats because deduplication is working on zfs records, so if the initiator host will write just 1 block, you dont wanna change more than just that block in storage
[17:57:01] <tsoome> but thats special case anyhow, and not very common.
[17:57:14] <Infin1ty> i see :)
[18:05:22] *** gea has joined #openindiana
[18:13:04] *** raichoo has joined #openindiana
[18:14:57] *** Whoopsie has quit IRC
[18:16:10] *** floogy_ has joined #openindiana
[18:16:37] *** newbie1 has joined #openindiana
[18:16:47] *** Coil-Domain has joined #openindiana
[18:17:31] *** Infin1ty|work has joined #openindiana
[18:17:32] *** pothos_ has joined #openindiana
[18:17:33] *** wcj has joined #openindiana
[18:18:07] *** Qkona has joined #openindiana
[18:20:51] *** SH0x_ has joined #openindiana
[18:21:09] *** brougham has quit IRC
[18:21:34] *** robins has joined #openindiana
[18:21:35] *** init3_ has joined #openindiana
[18:22:25] *** voidcoder has quit IRC
[18:22:46] *** voidcoder has joined #openindiana
[18:22:49] *** brougham has joined #openindiana
[18:23:50] *** pjama1 has joined #openindiana
[18:24:27] *** santoroj has joined #openindiana
[18:24:28] *** Infin1ty|work has quit IRC
[18:24:28] *** Infin1ty|work has joined #openindiana
[18:24:46] *** CoilDomain has quit IRC
[18:24:49] *** Infin1ty has quit IRC
[18:24:49] *** pjama has quit IRC
[18:24:50] *** SH0x has quit IRC
[18:24:50] *** jedi4ever has quit IRC
[18:24:50] *** [gorgo] has quit IRC
[18:24:50] *** init3 has quit IRC
[18:24:51] *** laogrymt has quit IRC
[18:24:51] *** Zenos has quit IRC
[18:24:51] *** floogy has quit IRC
[18:24:51] *** Okona has quit IRC
[18:24:51] *** calmh has quit IRC
[18:24:51] *** gea has quit IRC
[18:24:51] *** cis has quit IRC
[18:24:51] *** mikw has quit IRC
[18:24:51] *** gregburd has quit IRC
[18:24:51] *** sponix has quit IRC
[18:24:51] *** pothos has quit IRC
[18:24:52] *** robinsmidsrod has quit IRC
[18:25:14] *** pothos_ is now known as pothos
[18:25:29] *** Coil-Domain is now known as CoilDomain
[18:25:50] *** Infin1ty|work is now known as Infin1ty
[18:26:05] *** [gorgo] has joined #openindiana
[18:28:43] *** CoilDomain has quit IRC
[18:28:45] *** calmh has joined #openindiana
[18:30:54] *** mikw has joined #openindiana
[18:31:34] *** laogrymt has joined #openindiana
[18:33:04] *** CoilDomain has joined #openindiana
[18:33:32] *** Botanic has joined #openindiana
[18:33:32] *** Botanic has joined #openindiana
[18:37:56] *** sponix has joined #openindiana
[18:38:10] *** eryc has quit IRC
[18:39:51] *** vados has left #openindiana
[18:40:25] *** gregburd has joined #openindiana
[18:43:04] *** robins is now known as robinsmidsrod
[18:44:30] *** lblume has quit IRC
[18:45:01] *** sensille has left #openindiana
[18:45:37] <patdk-wk> is there some way I can see what blocksize a file is using?
[18:45:44] <patdk-wk> like if it's all 128k blocks, or 16k?
[18:46:14] *** lblume has joined #openindiana
[18:49:35] *** eryc has joined #openindiana
[18:51:57] *** lblume has quit IRC
[18:55:07] *** lblume has joined #openindiana
[18:55:53] *** Darkproger has joined #openindiana
[19:02:44] *** santoroj has quit IRC
[19:03:29] *** lblume has quit IRC
[19:03:42] *** heldus has quit IRC
[19:05:27] *** girafe has joined #openindiana
[19:05:44] *** lblume has joined #openindiana
[19:08:12] *** sommerfeld has quit IRC
[19:09:26] *** jw_urodoc has quit IRC
[19:11:12] *** bens1 has quit IRC
[19:12:31] <tsoome> patdk-wk: zdb i would assume….
[19:12:40] *** cwebber has left #openindiana
[19:13:07] <patdk-wk> zdb only does pools
[19:13:40] <tsoome> ?
[19:16:51] <tsoome> well, there is sample data mining demo available, but true, its not just zdb, but also mdb
[19:17:22] *** lblume has quit IRC
[19:17:55] *** lblume has joined #openindiana
[19:18:04] <kdavyd> You could parse the output of zdb -divvv pool/dataset to get information on each file
[19:18:26] <kdavyd> I'm sure there's a more efficient command line option, but -divvv gives you all the info you need
[19:19:18] <kdavyd> Example: Object lvl iblk dblk dsize lsize %full type
[19:19:18] <kdavyd> 9 1 16K 512 0 512 0.00 ZFS plain file
[19:19:18] <kdavyd> 168 bonus System attributes
[19:19:19] <kdavyd> dnode flags: USERUSED_ACCOUNTED
[19:19:19] <kdavyd> dnode maxblkid: 0
[19:19:19] <kdavyd> path /test.blah
[19:20:39] <tsoome> thats neat. good to know
[19:24:06] <tsoome> dblk is the data block size?
[19:24:22] *** kimc has joined #openindiana
[19:24:22] *** kimc has quit IRC
[19:24:22] *** kimc has joined #openindiana
[19:27:19] <kdavyd> tsoome, yes
[19:27:35] <kdavyd> 1M file for example:
[19:27:35] <kdavyd> Object lvl iblk dblk dsize lsize %full type
[19:27:35] <kdavyd> 10 2 16K 128K 1.00M 1M 100.00 ZFS plain file
[19:28:04] <tsoome> yep, just looking on my files:D
[19:28:22] <tsoome> iblk is increment? (wild guessing:)
[19:29:08] *** mikw has quit IRC
[19:30:13] <kdavyd> inode block size, maybe?
[19:30:48] *** santoroj has joined #openindiana
[19:32:22] <tsoome> zdb_nicenum(doi.doi_metadata_block_size, iblk);
[19:34:02] <tsoome> matadata it is. inode shouldnt be the case because zfs has dnodes:)
[19:35:35] *** lblume has quit IRC
[19:35:56] *** nvez_ has quit IRC
[19:35:57] *** nvez_ has joined #openindiana
[19:36:01] *** nvez_ is now known as nvez
[19:36:48] *** Shep_Book has joined #openindiana
[19:37:20] *** lblume has joined #openindiana
[19:41:00] *** heldus has joined #openindiana
[19:50:42] *** CoilDomain has quit IRC
[19:53:11] *** elgar has quit IRC
[19:58:35] *** merzo has joined #openindiana
[19:59:01] *** Andrew1 has quit IRC
[20:00:45] *** merzo has quit IRC
[20:02:48] *** merzo has joined #openindiana
[20:10:07] *** Seony has joined #openindiana
[20:15:43] *** gwr has joined #openindiana
[20:18:23] *** elgar has joined #openindiana
[20:25:49] *** lucalenardi has quit IRC
[20:26:05] *** lucalenardi has joined #openindiana
[20:26:13] *** easy-iPad has joined #openindiana
[20:26:31] <Wraithhh> How can I tell if a crash dump was saved in /rpool/dump?
[20:27:53] *** Botanic has quit IRC
[20:30:29] *** lucalenardi has quit IRC
[20:30:49] <richlowe> run savecore
[20:32:13] <patdk-wk> yay, my zdb finished
[20:36:13] *** Andrew1 has joined #openindiana
[20:38:59] <Wraithhh> savecore: bad magic number 0
[20:39:49] <Triskelios> Wraithhh: no crash dump then
[20:39:52] <richlowe> if it said nothing else, no there's no dump.
[20:40:00] <richlowe> there's at least one bug in savecore where it just decides the magic is bad
[20:40:04] <richlowe> regardless of the real error, and in addition to it.
[20:40:45] <richlowe> my notes say: "run savecore -vf vmdump.0 while the crash directory doesn't exist, for eg"
[20:54:11] *** laogrymt has quit IRC
[20:54:52] *** Shep_Book has quit IRC
[20:56:01] <tsoome> mkay. anyone knows an spell to wipe LSI raids out of this universe?:P
[20:56:53] *** Botanic has joined #openindiana
[20:56:54] *** Botanic has joined #openindiana
[20:58:45] <patdk-wk> raid cards? raid arrays? or just all hba's?
[20:59:17] <tsoome> raid card....
[20:59:45] <tsoome> i had to fsck bloody / and it had massive amount of issues:(
[21:01:32] *** PiotrSikora has quit IRC
[21:03:18] *** PiotrSikora has joined #openindiana
[21:05:28] *** Shep_Book has joined #openindiana
[21:12:24] *** kart_ has quit IRC
[21:12:43] *** CoilDomain has joined #openindiana
[21:20:06] *** Shep_Book has quit IRC
[21:31:05] *** gea has joined #openindiana
[21:33:46] *** Darkproger has quit IRC
[21:39:12] *** _Tenchi_ has joined #openindiana
[21:44:57] *** easy-iPad has quit IRC
[21:53:17] *** Darkproger has joined #openindiana
[21:57:30] *** Licenser has joined #openindiana
[21:59:37] *** Botanic has quit IRC
[21:59:51] *** Botanic has joined #openindiana
[21:59:51] *** Botanic has joined #openindiana
[22:05:16] *** disharmony has joined #openindiana
[22:06:24] *** dijenerate has quit IRC
[22:07:24] *** MrSambo has joined #openindiana
[22:10:01] *** xenol has quit IRC
[22:10:15] *** xenol has joined #openindiana
[22:14:49] *** dijenerate has joined #openindiana
[22:20:48] *** raichoo has quit IRC
[22:21:30] *** papertigers has quit IRC
[22:21:43] *** tsoome has quit IRC
[22:22:40] *** tsoome has joined #openindiana
[22:31:54] *** auronandace has quit IRC
[22:33:27] *** enmand has quit IRC
[22:39:23] *** Darkproger has quit IRC
[22:44:05] *** auronandace has joined #openindiana
[22:53:06] *** Botanic has quit IRC
[22:53:20] *** Botanic has joined #openindiana
[22:53:21] *** Botanic has joined #openindiana
[22:54:32] *** papertigers has joined #openindiana
[22:56:36] *** Seony has quit IRC
[22:57:28] *** ira has joined #openindiana
[23:00:52] *** enmand has joined #openindiana
[23:02:11] *** Shep_Book has joined #openindiana
[23:02:45] *** Shep_Book has quit IRC
[23:03:01] *** Shep_Book has joined #openindiana
[23:10:14] *** santoroj has left #openindiana
[23:13:18] *** enmand has quit IRC
[23:16:44] *** Darkproger has joined #openindiana
[23:34:40] *** [1]kdavyd has joined #openindiana
[23:35:13] *** kdavyd has quit IRC
[23:35:13] *** [1]kdavyd is now known as kdavyd
[23:42:33] *** wcj has quit IRC
[23:42:47] *** cis has joined #openindiana
[23:48:08] *** InTheWings has quit IRC
[23:58:17] *** CoilDomain has quit IRC
[23:58:57] *** newbie1 has quit IRC
top

   January 31, 2012  
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | >