[00:01:02] <FoxBuru> That moment on where you do an upgrade, and then your dashboard stops working =(
[00:08:24] <killfill> trentster: oh not really.. :P but im getting sick not to push it to somewhere.. :P
[00:08:36] <killfill> FoxBuru: fifo dashboard?
[00:09:12] <FoxBuru> killfill: Yep, it seems that the api/0.1.0/cloud url causes a 500 error
[00:09:30] <killfill> oh
[00:09:39] <killfill> other things work?
[00:10:52] <Licenser> killfill yap do it in a seperate branch and then merge(overwrite) master)
[00:11:29] <Licenser> trentster thanks mate I'm planning to I'm sure it'll be awesome
[00:12:58] <killfill> FoxBuru: and it worked ones?
[00:15:09] <FoxBuru> killfill: Yep, other things work on UI.
[00:16:19] <FoxBuru> killfill: It worked before the upgrade? yep, i was on the sniffle-20131015 and wiggle on 20131006
[00:17:17] <FoxBuru> killfill: Also, i don't know since when, but doing "fifoadm datasets list" now just give me an empty response. But in the UI the datasets show correctly.
[00:18:10] <killfill> oh.. hm.. will update my fifo setup to check. (in a coulpd of hours..)
[00:18:13] <Licenser> FoxBuru the newest sniffle comes with a new database thingy
[00:18:57] <FoxBuru> Licenser: Oh ok o.o So I need to do something to work it out? ^^
[00:18:57] <killfill> new database?
[00:19:25] <Licenser> killfill yes and no, it will default to no changes but you can select other databases now
[00:20:49] <killfill> Licenser: you mean one can select wich database (datastore?) to use in sniffle?
[00:21:01] <Licenser> killfill yup and snarl
[00:21:44] <killfill> that sounds wired.. looks you have a hidder reason somewhere why to do something like that.. :P
[00:23:10] <Licenser> killfill the maintainer of hanoidb kind of dropped the project so its dead
[00:23:24] <Licenser> I like the database but it has issues and I don't have the manpower to take it over
[00:23:30] <Licenser> so I plan on gradually migrating away from it
[00:25:11] <killfill> ooh that really sucks.
[00:25:28] <Licenser> yea it does
[00:25:46] <Licenser> it was active when we started using it but since then everyone involved dropped dead
[00:26:06] <killfill> :(
[00:26:06] <Licenser> there are two open pull requests with partially critical fixes that never got merged :(
[00:26:58] <killfill> hope next db of fifo will garantuee it doesnt happend again... (riak?.. :P)
[00:28:46] <Licenser> killfill hanoidb was a candidate for riak backends that's why I picked it in the first place
[00:28:58] <Licenser> both eleveldb and bitcask are actively maintained by basho since they use it in riak already
[00:29:34] <killfill> ah.
[00:30:00] <killfill> so its probably a good idea not to update to latest dev.. :P
[00:30:12] <killfill> maybe FoxBuru you can downgrade so your previous versions.. :P
[00:30:26] *** FoxBuru has quit IRC
[00:30:37] <trentster> killfill: I think once new refactored version is tied to dev, I would suggest that we slowly phase out old version updates etc. Its too much for one person (aka killfill ) to maintain alone.
[00:31:06] <Licenser> current dev should still work I ran the whole test suite against every database supported and they all work
[00:31:23] <trentster> Licenser: can we have some kind of basic instructions on how to switch DB's, I plan to to do this now sooner rather than later and get it over and done with.
[00:31:37] <trentster> best to scare out the bugs now ;-)
[01:53:19] <pantech> sorry to hear the bad news about hanoidb... :-(
[02:26:17] <Licenser> pantech +1 to that
[02:26:42] <Licenser> it will still stay in so you can keep using
[02:26:43] <Licenser> it
[02:27:04] <Licenser> The problem is that all databases have their problems, leveldb for example is a memory hog,
[02:27:35] <Licenser> bitcask is slooow, hanoidb is not maintained :(
[02:27:58] <pantech> It's a pitty!
[02:28:12] <Licenser> it is
[02:28:21] <Licenser> from all the db's hanoidb behaves best
[02:28:22] <pantech> Because hanoidb seemed very promissing.
[02:28:35] <Licenser> it has a rather small memory footprint, is fast, and pure erlang
[02:28:50] <pantech> Then I can't understand why basho isn't picking it up.
[02:29:02] <Licenser> I guess because leveldb is faster ;)
[02:29:16] <pantech> Yes, that's true.
[02:29:55] <Licenser> and the huge memory footprint usually don't makes that much of a issue. I don't think anyone minds setting up riak and having to give it a few G of memory
[02:30:02] <Licenser> but people already complain when the fifo zone gets up to 1G :P
[02:30:19] <pantech> haha
[02:30:50] <Licenser> that's the big issue really leveldb would be fine if you give the zone 8G memory, but very few people want to send that much :P
[02:30:57] <Licenser> especially since the ammount of data in fifo is much smaller
[02:31:14] <Licenser> so I see the reluctance for that
[02:31:50] <pantech> what is your average amount in fifo? I haven't played around with it yet.
[02:32:40] <pantech> But I played around with hanoidb... If I make a range query on rather large datasets, it takes some time...
[02:32:51] <Licenser> On data. Well images taken out (since they're in a seperate datastore which is bitcask based) I would guess a few hundred megabytes at tops and mostly because of historical data for conflict resoplution
[02:33:14] <Licenser> yup but at least it an do range queries ;) neither bitcask nor eleveldb allow for thsat
[02:33:28] <pantech> that's true!
[02:34:16] <pantech> btw, you have a real nice project with fifo. I like it!
[02:34:17] <Licenser> I made a wrapper for the three databases and folding over a bucket in eleveldb/eleveldb is implemented by folding over all data and just discarding all the other buckets
[02:34:25] <Licenser> thanks mate :)
[02:35:53] <pantech> The first time I heard about SmartOS :) I used to maintain Oracle DBs on SunOS and SPARC some ages ago... I really liked the unmatched reliability!
[02:36:41] <Licenser> I totally love solaris, and SmartOS is very very good in making it hypervisor like
[02:37:46] <pantech> Yea I had machines, which where never rebooted during my five years I worked there. But they were all doing some rather heavy work...
[02:39:58] <pantech> Very cool, I have to have look at SmarOS when I have more time.
[02:40:54] <Licenser> ^^
[02:43:52] <killfill> :)
[04:09:00] *** FoxBuru has joined #project-FiFo
[04:09:34] *** ipalreadytaken has joined #project-FiFo
[04:30:35] *** ipalreadytaken has quit IRC
[04:31:09] *** ipalreadytaken has joined #project-FiFo
[04:35:23] *** ipalreadytaken has quit IRC
[04:37:15] *** pantech has quit IRC
[04:50:18] *** pantech has joined #project-FiFo
[04:50:52] * konobi waves
[07:31:02] <MerlinDMC> morning
[08:12:48] *** FoxBuru has quit IRC
[08:14:29] <trentster> morning
[09:52:11] *** mgls has joined #project-FiFo
[12:35:28] *** ira has joined #project-FiFo
[12:37:38] *** aortmann has joined #project-FiFo
[13:11:18] *** g-flemming has joined #project-FiFo
[13:16:54] <trentster> g-flemming: so you still having issues?
[13:17:05] <trentster> you been through the problem checklist?
[13:18:03] <trentster> g-flemming: you tried to restart the services and teh fifo zone as well as the chunter services in the GZ?
[13:20:12] <trentster> svcadm restart snarl sniffle howl wiggle
[13:20:31] <g-flemming> I tell you, we are 3 guys working on it, we even tried restart the entire server. only the fiffle error log is gving some information
[13:20:35] <trentster> and then in the GZ of your nodes " svcadm restart chunter"
[13:21:17] <trentster> the odd thing is that it was working and stopped by itself, you sure you did not try anything, like perhaps you booted to a newer version of smartos on the nodes?
[13:22:02] <g-flemming> nope, same fifo and smartos version, nothing changed (to my knowledge)
[13:23:41] <trentster> hmm, ok before we hit the logs, are you guys using chrome? and if so, have you tried any other browsers?
[13:24:04] <trentster> and is your chrome autoupdated itself to v30?
[13:24:22] <trentster> as the chrome update broke websocket support
[13:26:24] <g-flemming> no, we don't use chrome - FiFo is not working with crome.
[13:26:37] <g-flemming> FF or IE
[13:29:48] <trentster> you got plenty of free space in your fifo zone?
[13:29:51] <trentster> df -h
[13:30:20] <trentster> and you have memory available " zonememstat "
[13:32:46] <g-flemming> # df -h
[13:32:46] <g-flemming> Filesystem Size Used Avail Use% Mounted on
[13:32:46] <g-flemming> zones/11cc7dbb-a900-490c-aa6c-4937a78cdf5b 91G 72G 20G 79% /
[13:32:49] <g-flemming> swap 2.0G 603M 1.5G 30% /etc/svc/volatile
[13:32:50] <g-flemming> swap 1.0G 8.0K 1.0G 1% /tmp
[13:32:51] <g-flemming> swap 2.0G 603M 1.5G 30% /var/run
[13:33:27] <g-flemming> zonememstat
[13:33:27] <g-flemming> ZONE RSS(MB) CAP(MB) NOVER POUT(MB)
[13:33:27] <g-flemming> 11cc7dbb-a900-490c-aa6c-4937a78cdf5b 506 2048 0 0
[13:35:07] <g-flemming> so there you have it
[13:35:41] <g-flemming> df -h
[13:35:41] <g-flemming> Filesystem Size Used Avail Use% Mounted on
[13:35:41] <g-flemming> zones/11cc7dbb-a900-490c-aa6c-4937a78cdf5b 91G 72G 20G 79% /
[13:35:41] <g-flemming> lib 249M 227M 23M 92% /lib
[13:35:41] <g-flemming> lib/svc/manifest 125G 701K 125G 1% /lib/svc/manifest
[13:35:41] <g-flemming> lib/svc/manifest/site 91G 72G 20G 79% /lib/svc/manifest/site
[13:35:41] <g-flemming> sbin 249M 227M 23M 92% /sbin
[13:35:42] <g-flemming> usr 377M 358M 19M 95% /usr
[13:35:42] <g-flemming> usr/ccs 91G 72G 20G 79% /usr/ccs
[13:35:43] <g-flemming> usr/local 91G 72G 20G 79% /usr/local
[13:35:43] <g-flemming> swap 2.0G 603M 1.5G 30% /etc/svc/volatile
[13:35:44] <g-flemming> usr/lib/libc/libc_hwcap1.so.1 377M 358M 19M 95% /lib/libc.so.1
[13:35:44] <g-flemming> swap 1.0G 8.0K 1.0G 1% /tmp
[13:35:45] <g-flemming> swap 2.0G 603M 1.5G 30% /var/run
[13:50:39] *** g-flemming has quit IRC
[13:50:55] *** g-flemming has joined #project-FiFo
[13:51:13] <trentster> g-flemming: ok time to look at logs, you want to gist them.
[13:51:22] <trentster> please dont paste in the channel
[13:52:11] <g-flemming> oh, sorry - this is my first IRC chat since 1998
[13:52:32] <g-flemming> which logs do you want?
[13:54:25] <trentster> cd /var/log; tar cvzf logs.tar snarl sniffle howl wiggle
[13:55:30] <trentster> then attach those logs to a trouble ticket, so Licenser can have a look and determine what your issue is caused by.
[13:56:59] <trentster> g-flemming: also if your setup is not production, then consider moving your install over to dev.
[13:57:15] <trentster> Most of us are on dev now anyway, and a lot of bugs have been fixed there.
[13:57:24] <MerlinDMC> dev is the new prod!
[13:57:51] <trentster> if you are going to do that you will also have to migrate your fifo zoen to a newer 13.2.x dataset
[13:58:37] <trentster> MerlinDMC: yeah, for fifo it kinda is I guess :-P
[13:58:44] <MerlinDMC> (if it never really ran - just destroy it and recreate it with a newer base image ... might be easier)
[13:58:54] <g-flemming> dev is the new prod :-) it always is!
[13:59:28] * MerlinDMC new todo ... asap: chilli dog
[13:59:34] <trentster> MerlinDMC: I think g-flemming has data there, if not can just blow away fifo and install new dev version.
[13:59:54] <trentster> g-flemming: also remember to change your chunters to dev as well if you decide to go this route.
[14:00:30] <trentster> *new todo … go to bed its midnight here*
[14:00:48] <MerlinDMC> trentster, how is the fifo release cycle actually? ... is that planned somehow?
[14:01:15] <trentster> MerlinDMC: yes kinda…
[14:01:35] <g-flemming> we are testing under production like circumstances i.e. with real users, so we would like to retain the fifo db/repository
[14:02:50] <trentster> MerlinDMC: it follows a staggered release with every even number being a stability release and odd number being a new features release.
[14:02:58] <g-flemming> ok goodnight
[14:03:43] <trentster> g-flemming: then I suggest you log a ticket and hopefully licenser will be around soon to ahve a look at it, he is normally pretty fast with these kind of things :^)
[14:05:06] <trentster> omg! it takes forever to compile stuff on this joyent public cloud freebsd vps. its like the train that goes "i think I can, I think I can"
[14:07:07] <trentster> and g-flemming " irc " is now back in favour, its the new "black" ;-)
[14:08:28] <MerlinDMC> trentster, fbsd on smartos had problems with the virtio drivers
[14:08:40] <MerlinDMC> io on the disks was awefully slow
[14:09:03] <MerlinDMC> at least the time I did build my image
[14:33:39] *** ira has quit IRC
[14:48:43] *** jim80net has joined #project-FiFo
[15:24:26] *** noahmehl has joined #project-FiFo
[17:50:01] *** echelog has joined #project-FiFo
[18:31:08] *** Licenser has quit IRC
[18:36:05] *** g-flemming has joined #project-FiFo
[19:08:32] <MerlinDMC> killfill, currently just zabbix ... mostly because the servers are at 4% load in 95% of the week ... and in the 5% they are dead because the load explodes ;)
[19:17:06] <killfill> heh
[19:17:34] <killfill> oh and there is a dataset for it.
[19:18:03] <MerlinDMC> yeah ... little old but working
[19:19:26] <MerlinDMC> i'll start building mibe templates for all the datasets trenster and me made when I got my stuff done here ... or at least to a point where I'm happy with it ;)
[19:33:52] *** FoxBuru has joined #project-FiFo
[19:34:18] <killfill> zabbix looks scrary at first.. :S
[19:40:12] <MerlinDMC> works fine if you have it running ... and the agent is very light
[19:40:44] <MerlinDMC> trentster can possibly help with setting that up he seems to use that really regularly :P
[19:41:35] <MerlinDMC> I'm just fine with having a little overview + backlog (and I like the possibility to extend the agent with own commands)
[19:41:59] *** FoxBuru has quit IRC
[19:44:49] *** FoxBuru has joined #project-FiFo
[20:01:07] *** FoxBuru has quit IRC
[20:04:40] <killfill> :)
[20:31:55] *** pantech has quit IRC
[20:41:02] *** pantech has joined #project-FiFo
[20:45:33] *** g-flemming has quit IRC
[21:09:38] *** conan_the_destro has quit IRC
[21:10:53] *** conan_the_destro has joined #project-FiFo
[21:11:44] *** noahmehl has joined #project-FiFo
[21:21:33] *** FoxBuru has joined #project-FiFo
[22:53:26] *** conan_the_destro has quit IRC
[22:53:55] *** conan_the_destro has joined #project-FiFo
[23:09:36] *** ira has quit IRC
[23:10:00] *** mgls has quit IRC
[23:14:56] *** mgls has joined #project-FiFo
[23:15:21] *** mgls has quit IRC
[23:18:31] <killfill> dammit, grunt-connect-proxy doesnt support proxin websocket.. cannot connect to howl.. :S
[23:29:26] <trentster> :-(
[23:42:58] <trentster> This is ridiculous, my install of powerdns is still compiling on freebsd in the joyent public cloud, its been running for about 10 hours now!
[23:44:21] <trentster> feels like 1992 again :-P
[23:52:45] <benbangert> the fifo UI says its deleting a VM
[23:52:49] <benbangert> vmadm says it doesn't exist
[23:52:59] <benbangert> the PF UI has said its being deleted for like.... a few days now
[23:56:14] <trentster> benbangert: yeah its a quirky bug, killfill knows about it.
[23:56:30] <trentster> benbangert: just delete it via fifoadm vms delete uuid
[23:58:50] <benbangert> cool, that got rid of it