November 6, 2014 
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | >
November 6, 2014 [00:14:50] *** dasjoe has left #project-FiFo[00:51:36] *** glshrike has quit IRC[01:28:17] *** conan_the_destro has quit IRC[01:59:05] *** jim80net1 has joined #project-FiFo[01:59:25] *** jim80net has quit IRC[02:33:29] <trentster> aysah: how did it go?[02:59:46] *** aysah has quit IRC[03:22:18] *** aysah has joined #project-FiFo[03:22:46] <aysah> hey tren! umm things are good (thanks for suggesting those zpool changes) im still stuck on one thing tho - the issue that originally started all of this.. im on node 2 and a server on node 1 has a snapshot that I'd like to image and use as a dataset[03:23:11] <aysah> everytime i run it - it just keeps saying pending 0%[03:23:27] <aysah> it's my windows sysprep image[04:00:36] *** trentster has quit IRC[04:44:11] <aysah> nevermind, the import just took a long time[04:44:16] <aysah> thanks again for all your help tren[05:19:28] *** jim80net1 has quit IRC[05:41:07] *** aysah has quit IRC[05:45:29] *** ipalreadytaken has joined #project-FiFo[05:45:29] *** ChanServ sets mode: +o ipalreadytaken[05:48:00] *** aysah has joined #project-FiFo[05:49:14] *** jim80net has joined #project-FiFo[05:51:09] *** ipalreadytaken has quit IRC[06:04:50] *** ipalreadytaken has joined #project-FiFo[06:04:52] *** ChanServ sets mode: +o ipalreadytaken[06:07:56] *** ipalreadytaken has quit IRC[06:40:32] *** trentster has joined #project-FiFo[06:40:36] *** ChanServ sets mode: +o trentster[06:40:52] <trentster> aysah: howdy you still there?[06:41:07] <aysah> hey[06:43:03] <trentster> I was in the garage putting in a backplane into a server.[06:43:07] <trentster> how goes it?[06:43:59] <aysah> =) good times man, everything is going well i think here - it finally stabled off[06:44:19] <trentster> how did the install go yesterday after I left you?[06:44:48] <aysah> went well - its noticeably faster; you were right[06:45:10] <aysah> can i use that guide to make anoher fifo instance and cluster the two together?[06:46:05] <aysah> not a high priority, just wondering[06:47:10] <trentster> aysah: actually - its a question quite a few of us wonder about - the single install has been configured in such a way - I am not sure its possible to add to the cluster.[06:47:53] <trentster> I tried to do it yesterday - but did not have much luck. I tried to ping the main dev to see if its possible. - but afaik according to licenser he said he is not sure the all-in-one can be transitioned[06:48:39] <aysah> haha all good - i'll leave it to the leets lol.... i have my jr sys guy rotate drives every month and do hardware check so im not overly concerned about it to be honest[06:48:42] <aysah> was just curious[06:48:57] <aysah> i did hit one quirky issue tho[06:49:48] <aysah> obviously i have some machines with ip addresses already from the previous installation of FiFo.. after reinstalling everything and giving FiFo the same network range, it's creating overlapping ip addresses[06:50:08] <aysah> i thought it would check through the pre-existing machines to see the next available ip[06:52:33] <trentster> nah its not a dhcp server[06:52:59] <trentster> it basically just hands out from a pool and when it destroys a machine that has an ip in the pool range it returns it[06:54:35] <aysah> yeah thats what I did as well - is there any plans on giving us an option on the create machine screen to enter our own static ip/subnet/gateway ?[06:54:41] <aysah> i think that would be very handy[06:56:30] <trentster> aysah: not the ip no, I dont think so.[06:57:06] <trentster> Its designed to be a big multitenant cloud orchestration suite, so would not want people handing out ips manually.[06:58:27] <aysah> yeah it just seems like its in a gray area at the moment.. you can't do static but it doesnt track ip addresses - like other orchestrators[06:58:51] <aysah> not trying to nitpick, just a few things i observed[06:59:12] <aysah> still love the performance gain hands down compared to my last endeavors[07:00:22] <trentster> yeah - there is also a way to claim a single ip so it does not get used.[07:00:48] <aysah> hmm now that sounds interesting[07:01:15] <trentster> in the situation you have an existing machine x.5 and the next ip that fifo is gonna hand out will be x.5 you can use fifoadm to claim it and it will skip it and hand out 6[07:01:59] *** ipalreadytaken has joined #project-FiFo[07:01:59] *** ChanServ sets mode: +o ipalreadytaken[07:02:10] <trentster> aysah: so what was the issue with your windows dataset provisioning, did you have the wrong stuff in the manifest when you imported it?[07:02:40] <aysah> no it was like 40 GBs[07:02:49] <aysah> i totally forgot about the size[07:02:56] <aysah> so it was just tkaing forever to copy[07:03:20] <aysah> i thought it was locked up until i looked at my network traffic[07:03:48] <aysah> i went for sushi i came back and it was done[07:05:09] <trentster> aysah: yeah the first time it deploys a dataset its not a super fast thing it has to zfs send the chunks to the node.[07:05:48] <trentster> once its there when you provision a vm off the same dataset it will be very fast[07:05:58] <aysah> yeah i should've thought of that - but im happy to say its working now... i do have a bunch of unknowns in my list of machines[07:06:11] <aysah> is there anyway i can change it to say windows ?[07:06:46] <aysah> the dataset has a different UUID than before since i took a new snapshot and i never got a chance to import my old dataset so it could be recognized[07:07:11] <trentster> aysah: did they get created from a windows dataset?[07:07:36] <ipalreadytaken> fwiw. There are a couple of reasons the ip address assignment happens that way: 1) fifo is meant for clouds (read: lots of vms being created and destroyed rapidly) so in this case its not realistic to manually assign vms. 2) each hypervisor gets a subset of the pool of ip addresses, so that it does not have to "check out" one from a central server, and the hypervisor expects to have exclusive[07:07:36] <ipalreadytaken> rights to these ips (so it doesnt expect other people to use them. Having to ping an address would add time to the creation of zones, besides a simple ping is a bad way to know for sure if the ip is in use.[07:07:37] <aysah> yeah from my broken fifo install[07:08:05] <trentster> have you imported the datasets into the fifo dataset store?[07:08:47] <aysah> @ipalreadytaken thank you for the run down on the expected architecture.. i'll dedicate subsets of pools to each hypervisors accordingly going forward[07:09:13] <ipalreadytaken> you can have 4k vlans (use them!)[07:09:15] <aysah> @trentster no - I wiped the broken installatino[07:09:15] <ipalreadytaken> :P[07:09:25] <aysah> haha valid point[07:10:36] <ipalreadytaken> aysah: what type of network are you on? 40gb shod be pretty quick, might have a setting wrong...[07:11:28] <aysah> 1GB Agrregated on Two Nics (LACP )[07:11:41] <trentster> aysah: if you are going to be using SmartOS / Fifo going forward. I would highly suggest keeping all your datasets in your own dsapi server, that way if you ever have to blow fifo away or whatever, your datasets are still available via dsapi and can be imported.[07:12:01] <trentster> its the way I use it, and have set it up like that for a lot of people.[07:12:27] <ipalreadytaken> aysah: do some nc test for speed on your switch and server over that lag, i bet soemthing is wrong with it[07:12:29] <trentster> also the dsapi server is super efficient only uses like 40MB of ram so you can run it in a 256MB zone[07:13:08] <aysah> @ipalreadytaken - i was hoping you wouldnt say that but i have a hunch as well.. i'll save that for sunday lol[07:13:31] <aysah> @trentster - that would be awesome? is there a guide similar to the one you showed me earlier on hwo to set it up?[07:13:53] <ipalreadytaken> quick and dirty test is to remove the lag, and see if just one link is faster. usually a single link is faster than two in a bad config[07:15:27] <trentster> aysah: there is a blog post about it coming out shortly - its pretty easy - have you been to datasets.at ?[07:15:29] <aysah> noted - do you guys find aggregation helps alot or is it overkill for a mid-range box?[07:16:02] <trentster> ipalreadytaken: thats funny I told him to do the same thing yesterday, plug a single nic into a non lacp port[07:16:09] <trentster> but the DC is far from him[07:16:12] <aysah> lol shh[07:17:03] <ipalreadytaken> not worth the pain, if you need the speed get 10g, otherwise dont bother. Besides if you are running a small setup you are likely just doing simple websites in zones, and your wan uplink is WAY slower than gig anways, so you have another bottle neck that is way bigger[07:17:06] <aysah> @trendster: i've seen datasets.at - i'd like to have my own server that could still import from the public server and allow me to upload my private datasets[07:17:18] <trentster> aysah: it helps when correctly configured - but it will not really give you 2X speed, its more for redundency if one nic goes down. I dont use them personally[07:17:57] <aysah> understood guys[07:18:18] <aysah> and it really is mostly website with a handful of custom web apps/databases[07:18:27] <aysah> nothing spectacular[07:18:32] <trentster> aysah: yup so dsapid dataset is the exact same system that datasets.at use. You can then configure your own dsapi server to sync every dataset that datasets.at has or selectively sync the stuff you want. You can also of course put your own datasets in there as well.[07:18:44] <trentster> Bonus is you have everything locally and its super fast[07:19:12] <trentster> If you no too perturbed about space the innitial sync will use about 70GB of space[07:19:33] <trentster> http://datasets.at/status[07:19:56] <trentster> That will show you how many datasets there are and in total 70GB of space[07:20:03] <aysah> 70gb i can live with, i was just confused on where to start last time i googled this lol: https://github.com/MerlinDMC/dsapid[07:20:12] <aysah> but if there is a guide coming soon i'll wait[07:20:27] <trentster> I can help you if you like - its pretty simple[07:20:44] <trentster> just import the latest dsapi dataset from datasets.at[07:20:53] <aysah> if you dont mind that'd be great[07:21:06] <trentster> then deploy a zone - give it 256MB ram and maybe 100GB disk[07:21:07] <aysah> k[07:21:18] <trentster> or more depending on how big your own images will be[07:21:28] <aysah> should i make it bigger because of my windows images?[07:21:33] <aysah> those are like 40gb[07:21:37] <aysah> k[07:22:01] *** jim80net has quit IRC[07:22:15] *** jim80net has joined #project-FiFo[07:22:24] <trentster> yeah remember zones space can be changed on the fly - its not like KVM, so if you need more space you just change the package for the zone to a package that says 300GB and immediately its available.[07:22:43] <aysah> ahh - thats nice... i forgot about that[07:30:29] <aysah> @trentster: k its done[07:32:40] <trentster> aysah: you are too fast for me mate! ;-)[07:32:49] <aysah> haha just eager[07:34:45] <trentster> ok you want me to show you or tell you what to do?[07:35:04] <ipalreadytaken> trentster: tell him so we can have a record of it ;)[07:35:10] <aysah> haha[07:35:16] <aysah> majority has spoken[07:36:22] <trentster> ok give me 5 pls just gonna grab a cup of Joe[07:36:46] <aysah> alright man, take your time - watching walking dead[07:40:32] <trentster> kk[07:41:35] <aysah> k[07:52:26] <trentster> ok so all datasets will get stored in /data/files/uuid[07:53:19] <trentster> the main config file is /data/config.json[07:54:06] <trentster> If you want to be able to push new datasets to other dsapi server then you can configure users and auth tokens in /data/users.json.json[07:54:17] <trentster> but for 99% of people they will never need that[07:54:18] <trentster> —[07:54:24] <trentster> aysah: you with me?[07:54:37] <trentster> or you watching zombies try eat people?[07:54:41] <trentster> :-P[07:54:57] <aysah> yup following - zombies have been paused haha... yeah i dont need to push to other dsapi servers[07:55:02] <aysah> just need my own personal library[07:58:40] <trentster> aysah: open web browser to http://fusion.onyxit.net:13377[07:58:59] <trentster> let me know once you are there, ipalreadytaken can jump on too if you like[07:59:11] <aysah> thats pretty cool[07:59:12] <aysah> im there[08:00:48] <trentster> Under every UUID directory you will find the the file and the manifest[08:01:35] <aysah> k[08:02:00] <trentster> now that config is all that is needed for a full auto sync of every dataset on datasets.at[08:02:10] <trentster> when a new one is added it will sync[08:02:31] <aysah> the address 0.0.0.0 acts like a wildcard?[08:02:34] <aysah> no need to set it?[08:02:37] <trentster> to selectively choose what to sync[08:02:54] <trentster> yeah 0.0.0.0 means it will listen on everything[08:03:02] <aysah> k[08:03:28] <trentster> but you will connect to it with a hostname preferably I will show you where to change that in jingles config file[08:03:39] <aysah> k[08:03:41] <trentster> you want to see what a selective sync looks like?[08:03:49] <aysah> pelase[08:03:54] <trentster> or you not gonna do selective sync?[08:04:07] <aysah> more than likely just going to do datasets.at[08:04:56] <trentster> https://gist.github.com/trentster/b9644aea3f0edc4d667a[08:05:04] <trentster> ok there is the config file I am using[08:05:10] <trentster> just adjust it accordingly[08:05:20] <aysah> k[08:05:22] <aysah> ty[08:06:11] <trentster> once your config has been setup enable the daemon if it has not been enabled already[08:06:41] <aysah> how do i force the first refresh?[08:06:46] <aysah> sync*[08:06:52] <trentster> then ls /data/files and you should see each uuid directory being created[08:07:26] <trentster> svcadm restart dsapid[08:07:36] <aysah> nice[08:07:41] <trentster> but first make sure its enabled[08:07:41] <aysah> i see it coming in now[08:07:45] <aysah> yup[08:07:50] <trentster> svcadm enable dsapid[08:08:08] <trentster> see how ram its using[08:08:18] <aysah> amazing[08:08:30] <trentster> yup its written in GO[08:08:44] <aysah> never heard - but im impressed[08:10:07] <trentster> "/var/svc/log/application-dsapid\:default.log"[08:10:16] <trentster> that log normally shows you what is going on.[08:10:57] <aysah> k looks like its trucking along[08:10:58] <trentster> btw its also running a webserver so if you connect to the ip of your dsapi zone - you will see the exact same webUI as datasets.at[08:11:08] <trentster> easy peasy wasnt it?[08:11:34] <aysah> yeah way easier than i thought ...hopefully uploading my custom datasets are just as easy lol[08:12:30] <trentster> ok yeah - its simply a matter of adding the gz file and making sure the manifest is correct[08:12:58] <trentster> you can look at sample manifests when your stuff comes down[08:13:51] <aysah> so this is different than the kvm json file in the instructions of smartos (setupwindows.json) ?[08:17:33] <trentster> you want me to paste the json?[08:17:51] <trentster> the KVM uses disk_0 and Disk_1[08:17:53] <aysah> its fine im looking at the manifest now[08:18:07] <trentster> and the manifests have some differences[08:18:21] <aysah> is the requirements section always the same?[08:18:35] <trentster> if you look at one of the kvm images when it comes down you will see the difference[08:18:58] <aysah> ahh okay so kvm and zones are different manifests[08:19:19] <trentster> yup[08:20:11] <aysah> so i get the overall concept here...tar + manifest + uuid folder name[08:20:45] <aysah> again im a noob at this behind the scenes stuff soooo im not sure where I find the dataset that I need to tar[08:21:05] <aysah> where would i find the img that got imported into fifo?[08:22:08] <trentster> http://monosnap.com/image/8zzW7RzgpmLX8nGrxOgqlDCq4u6tyt[08:22:23] <trentster> 1) is the uuid for your dataset[08:23:02] <trentster> 2) is your own uuid - make sure you use the same one for all of your images. if you dont have one just generate one with the "uuid" command[08:23:06] <aysah> nice thank you for the illustration[08:23:21] <trentster> 3) the name of the image/file as it exists on disk[08:23:40] <trentster> 4) size of the file "ls filename"[08:24:11] <trentster> 5) and 6) sha1 hash and the md5 hash of the file[08:24:43] <trentster> Version is for you to keep track of the version - just increment it when you release an image of the same type[08:25:17] <trentster> urn change the local part to your own identifier[08:26:13] <trentster> e.g. smartos:wowit:wsr:0.1.0[08:26:13] <aysah> the urn is what im a little confused on still..[08:26:35] <aysah> so its not tied to anything? whatever I want to call it?[08:26:51] <trentster> just keep the wowit part consistent across your datasets[08:27:02] <trentster> yup - call it whatever[08:27:09] <aysah> ok understood[08:27:45] <trentster> and obviously give it a good description of what the dataset is - this will also show in fifo under the dataset description[08:28:07] <trentster> ok make sense?[08:28:22] <aysah> yeah this is pretty straight forward so far[08:28:25] <trentster> gonna close my web console now - unless you have any other questions?[08:28:53] <aysah> just the one still regarding the custom dataset[08:29:04] *** ipalreadytaken has quit IRC[08:29:15] <aysah> where do i find the custom dataset directory that i need to compress?[08:29:33] <aysah> e.g, windows server[08:30:29] <trentster> is it a dataset already?[08:30:39] <trentster> how did you create it? within fifo?[08:30:55] <aysah> yeah i created a snapshot with fifo and then created an image from there[08:30:57] <aysah> yup[08:31:06] <trentster> so just export it out[08:32:00] <aysah> oh i see the button - didnt realize there was an export button haha[08:32:07] <aysah> thanks man[08:32:28] <trentster> let me know if it works, I havent tested it myself[08:32:51] <aysah> will do - thanks for the tutorial man[08:32:55] <aysah> this was very helpful[08:32:58] <trentster> np[09:00:04] *** alcir has joined #project-FiFo[09:57:49] <mattronix> rdenis: hi :P[10:00:53] *** marianoguerra has quit IRC[10:08:09] *** marianoguerra has joined #project-FiFo[10:35:08] *** alcir has quit IRC[11:15:41] *** alcir has joined #project-FiFo[14:55:31] <trentster> DigiDaz: ping prv[14:55:36] <trentster> mattronix: howdy[15:01:01] <mattronix> trentster: yo[15:38:37] *** TechByTom has joined #project-FiFo[16:09:58] *** noahmehl has joined #project-FiFo[16:38:58] *** Conti_ has joined #project-FiFo[16:39:59] *** conan_the_destro has joined #project-FiFo[17:14:39] *** Conti_ has quit IRC[17:20:09] *** Conti_ has joined #project-FiFo[17:35:58] <konobi> https://www.joyent.com/blog/sdc-and-manta-are-now-open-source[17:38:57] <trentster> konobi: awesome news![17:41:00] *** marianoguerra has quit IRC[17:42:17] *** marianoguerra has joined #project-FiFo[17:42:17] *** marianoguerra has joined #project-FiFo[17:45:13] *** Conti has quit IRC[17:45:14] *** Conti_ is now known as Conti[18:04:49] *** alcir has quit IRC[18:10:03] *** ipalreadytaken has joined #project-FiFo[18:10:04] *** ChanServ sets mode: +o ipalreadytaken[18:22:07] *** olauzon has joined #project-FiFo[18:26:05] <aysah> @trentster: hey tren - the dataset server is working - everything seems to be synced. I repointed FiFo to use the local dataset server and it shows the list fine but when I try to import sniffle gives me the following error:[18:26:06] <aysah> 2014-11-06 17:21:18.113 [error] emulator Error in process <0.4073.0> on node 'sniffle at 10 dot 120.10.4' with exit value: {{case_clause,{error,nxdomain}},[{sniffle_dataset,read_image,8,[{file,"src/sniffle_dataset.erl"},{line,216}]}]}[18:26:37] <ipalreadytaken> is there a route between them?[18:27:20] <aysah> yup, I can ping the dataset server fine from the FiFo box[18:29:44] <aysah> weird thing is if I restart fifo, then i get an empty dataset shell in my local dataset tab in the gui of what i was trying to import but it never downloads[18:30:10] <ipalreadytaken> what version of siffle are you running[18:30:19] <ipalreadytaken> oh what is your leo gateway hostname?[18:31:42] <aysah> EDI-DAT-FIF01.dc.local, and based on the guide I was following I had to create additional endpoints like s3.dc.local[18:32:30] <ipalreadytaken> can the sniffle server ping the leo gateway by hostname?[18:32:49] <ipalreadytaken> thats would be waht "nxdomain" seems to indicate[18:35:32] <aysah> i just went on my dsapid server and was able to ping EDI-DAT-FIF01.dc.local & s3.dc.local[18:35:52] <aysah> so i think it can[18:37:48] <ipalreadytaken> aysah: im sorry I was talking to somebody else and mixed you guys up in my head. Lets start over. This has nothing to do with leo[18:37:59] <ipalreadytaken> ok can you ping dsapid from your sniffle server[18:39:14] <aysah> on sec - getting different behavior at the moment.. i restarted the dsapid server and now in fifo i can click import without an error but the progress sits a 0%[18:39:38] <aysah> yup I can ping dsapid from fifo[18:39:44] <ipalreadytaken> by hostname?[18:39:59] <ipalreadytaken> the error you are getting is a hostname look up problem[18:40:32] *** olauzon has quit IRC[18:41:58] <aysah> not by hostname - I'll update that in my dns and hopefully that resolves the issue.. I originally made an A-Record called datasets.dc.local that point to the same IP address and thought that would work[18:42:14] <ipalreadytaken> just add it to your hosts file[18:43:11] <aysah> k added it[18:43:18] <aysah> i'll try again[18:43:32] <ipalreadytaken> the way it works is your browser is looking at what datasets the dsapi server has advailable and then passes the url to sniffle when you click "Create". so just becasue you can see it in the browser doesnt mean fifo can access it[18:51:44] <aysah> hmm still getting the same thing - i wish i can figure what address it cant reach[18:51:55] <DigiDaz> How different to SDc is this project?[18:51:57] <aysah> the hostname is resolvable now[18:51:58] <DigiDaz> C[18:52:56] <DigiDaz> Or maybe another way of putting it is, will this project die out now joyent has open sourced SDC?[18:54:03] <ipalreadytaken> aysah: anything in the logs?[18:54:04] *** noahmehl has joined #project-FiFo[18:54:18] <aysah> 2014-11-06 17:51:05.601 [error] emulator Error in process <0.18464.0> on node 'sniffle at 10 dot 120.10.4' with exit value: {{case_clause,{error,nxdomain}},[{sniffle_dataset,read_image,8,[{file,"src/sniffle_dataset.erl"},{line,216}]}]}[18:54:21] <aysah> same[18:54:29] <ipalreadytaken> DigiDaz: it will probably grow as a result.[18:54:44] <ipalreadytaken> aysah: anything in debug log?[18:54:58] <aysah> that was my debug log[18:55:02] <aysah> same event over and over again[18:55:06] <DigiDaz> OK, I'm just curious, I haven't even used smartos until this week so haven't a clue what the differences are etc[18:55:08] <aysah> does the GZ need to resolve as well?[18:56:13] <ipalreadytaken> DigiDaz: the architecture is very different. Fifo is built for 0% downtimes.[18:56:31] <ipalreadytaken> aysah: are you using leofs for ds storage?[18:57:00] <aysah> yes 1.1.5[18:57:20] <DigiDaz> trenster has helped me and today I've achieved something that for me is amazing so I'm gonna make a feature request already[18:57:31] <ipalreadytaken> then gz shouldnt matter[18:57:40] <aysah> hmm thought so[18:57:43] <ipalreadytaken> DigiDaz: shoot. we love feature request[18:57:57] <ipalreadytaken> aysah: can you svc reload fifo[18:58:05] <ipalreadytaken> actaully restart[18:58:18] <aysah> sure[18:59:21] <ipalreadytaken> what version of fifo are you running?[19:03:43] <aysah> as of yesterday the latest now: https://docs.project-fifo.net/en/latest/general/singlezoneleofs.html[19:04:08] <aysah> still getting that nxdomain error just kinda weird everything seems resolvable[19:04:54] <ipalreadytaken> aysah: so release not dev correct?[19:05:44] <aysah> release yes[19:07:18] *** conan_the_destro has quit IRC[19:07:45] <ipalreadytaken> *digging give me a few minutes*[19:08:12] *** conan_the_destro has joined #project-FiFo[19:09:27] <aysah> coffee time, brb[19:11:34] <ipalreadytaken> aysah: when you get back try to use curl to download one of the datasets to your sniffle zone from your local datasets server. make sure to use the hsot name[19:14:54] <DigiDaz> I've made my request https://jira.project-fifo.net/browse/FIFO-557?filter=-4[19:17:18] <ipalreadytaken> DigiDaz: that is very similar to https://jira.project-fifo.net/browse/FIFO-548?filter=-4[19:19:04] <ipalreadytaken> I stand by my comment that its trying to gain HA at the wrong level, but I think your request brings up an interesting point, slow restore times.[19:20:47] <DigiDaz> ipalreadytaken: I fully understand the comment but already many of us are using systems where that is just not an option in the short term.[19:21:26] <ipalreadytaken> I understand the pain of legacy systems.[19:21:44] <ipalreadytaken> I think the idea of cold staging a vm is interesting.[19:22:08] <ipalreadytaken> basically doing it at set intervals (or at the vms request)[19:22:17] <DigiDaz> I'm coming from a vmware were we have high availability using shared storage, the SAN at the moment being the single point of failure[19:22:45] <ipalreadytaken> one of the big issues is that a lot of systems dont expect to be booted from an unknown state.[19:23:24] <ipalreadytaken> so lets use mysql db as a simple example.[19:23:41] <DigiDaz> Yep, I completely understand it can cause problems but I think for many people though far from perfect, it will get the job done[19:24:00] <ipalreadytaken> you are running along jsut fine, making and shipping snapshots every 30 minutes (or less whatever) and then the vm goes down[19:24:55] <ipalreadytaken> operator notices and clicks "bring up" on the cold staged one, but now not only could the writes be only half done, but you are not really sure what you have lost (was it 5 minutes or 2 minutes)[19:25:12] <aysah> @ipalreadytaken: sure not a problem - when you can, can you provide me with a sample download path? I'm still becoming familiar with the directory structures here[19:25:47] <ipalreadytaken> aysah: are you logged into your fifo zone (i assume you are not clustered)?[19:26:04] <aysah> not clustered and i am logged in[19:27:06] <ipalreadytaken> DigiDaz: I guess my problem is that if you give people the illusion of HA they make bad application design decisions, and eventually it will break and then it comes back to bite you.[19:27:59] <ipalreadytaken> aysah: what is the domain of your datasets api server?[19:29:19] <aysah> the dsapid hostname is EDI-DAT-DSA01 and in the config file I'm using the url http://datasets.dc.local/" which is an alias that points to EDI-DAT-DSA01[19:30:13] <ipalreadytaken> so try downloading http://datasets.dc.local//api/export/9a382924-5c2f-11e4-81c8-0b38c6aca636[19:32:03] <ipalreadytaken> err http://datasets.dc.local/api/export/9a382924-5c2f-11e4-81c8-0b38c6aca636[19:33:32] <aysah> its downloading the znc tar file[19:34:31] <ipalreadytaken> darn.[19:34:35] <ipalreadytaken> you can cancel that download[19:35:47] <aysah> when i do it from FiFo: curl http://datasets.dc.local/api/export/9a382924-5c2f-11e4-81c8-0b38c6aca636/ I get a lot of random shit in my putty session and then it locks up and i get disconnected[19:37:18] *** Conti has quit IRC[19:37:20] *** noahmehl has quit IRC[19:38:37] <ipalreadytaken> oh thats becasue you are not saving it to a file (use -o file.tar.gz)[19:38:56] <aysah> thanks[19:38:57] <aysah> lol[19:40:39] *** noahmehl has joined #project-FiFo[19:40:51] <aysah> well it works in fifo as well[19:40:58] <ipalreadytaken> there are some interesting reads about that out there. it shouldnt lock up your shell, but none of them interpret characters correctly.[19:41:16] <ipalreadytaken> what do you mean "it works in fifo", as in you are not having the issue any more?[19:42:04] <aysah> sorry that was misleading - I can download the tar with curl in fifo so its resolving the hostname fine in fifo[19:42:17] <aysah> but in the fifo webui i still have the same issue[19:43:48] <ipalreadytaken> your web console isnt giving you any errors right?[19:47:07] <aysah> no it gives me the quick notification that is downloading dataset and then the dataset sits at 0%[19:47:10] <aysah> when i switch it back to datasets.at it works fine[19:47:14] <aysah> kinda weird[19:49:37] <ipalreadytaken> i think its night time where trent is but we might need to compare some of your configs to his.[19:49:51] <ipalreadytaken> where are you located?[19:50:18] <aysah> US \ new jersey[19:50:37] <aysah> i can wait - its not high priority..[19:50:51] <aysah> thanks for your help tho man[19:51:49] <ipalreadytaken> you have bandwidth problems from datasets.at?[19:53:12] <aysah> none - i just have quite a few custom datasets that i would like stored in a local repository. i learned the hard way that storing it in FiFo exclusively isnt the best option because if there is a configuration issue and i have to blow it away, i lose my custom datasets as well[19:54:14] <ipalreadytaken> aysah: configure it right the first time and then enjoy 0 downtime. :P[19:54:24] <DigiDaz> Can fifo be nuked at any time and reinstalled without affecting any vms?[19:54:35] <ipalreadytaken> DigiDaz: yes[19:54:42] <aysah> haha now your taking jabs lol[19:54:46] <aysah> im still a newbie[19:54:48] <DigiDaz> Thats good[19:54:58] <aysah> yeah digi, my vms were all fine[19:55:13] <ipalreadytaken> aysah: make snapshots before you make changes and then if you have to jsut roll back the snapshot[19:55:31] <ipalreadytaken> But really the config shouldnt need much tweeking[19:55:49] <aysah> yeah I do that I just didnt realize how messed up my fifo was until trent looked at it haha[19:56:14] <DigiDaz> I have put my first box in the datacenter, second is going in tomorrow and I'll start migrating. As soon as I have enough machines I want to get the 5 server cluster going[19:56:22] <aysah> either way I like the idea having an inhouse dataset server - eventually I want to share with my other location over the vpn[19:56:55] <ipalreadytaken> you can use leo for image storage as well, but you have to have fifo working first...[19:57:22] <DigiDaz> Here's an uptime to beat http://arstechnica.com/information-technology/2013/03/epic-uptime-achievement-can-you-beat-16-years/[19:57:42] <ipalreadytaken> leo is nice because its replicated. dsapi is not[20:00:38] <aysah> true but from what i understand clustering isn't working yet with the guide i used: https://docs.project-fifo.net/en/latest/general/singlezoneleofs.html. I know trent and licenser are working/testing that ( i think )[20:01:12] <aysah> "Note: Do NOT use this dataset for a clustered FiFo installation."[20:02:00] <ipalreadytaken> if you use the dataset ymmv. but its very easy to install into a couple of datasets[20:02:34] <ipalreadytaken> leo clusters just fine, i think its just that the dataset is for getting people up and running quickly, but it sacrafices a couple of things to do that[20:03:42] <aysah> ahh okay understoond[20:06:01] *** Conti has joined #project-FiFo[20:07:43] <DigiDaz> I think trentster did that so us noobs had half a chance :)[20:08:42] <ipalreadytaken> there is a tough balance between being fully scaled out and not having a million setup steps...[20:10:13] *** noahmehl has quit IRC[20:10:35] *** noahmehl has joined #project-FiFo[20:11:13] *** noahmehl has quit IRC[20:12:28] *** olauzon has joined #project-FiFo[20:15:30] *** liyu has joined #project-FiFo[20:21:45] *** olauzon has quit IRC[20:34:28] <MerlinDMC> ipalreadytaken, is there a decent case where erlang would raise nxdomain if you give it an IP to load from?[20:35:27] <ipalreadytaken> MerlinDMC: not sure, but he is using a hostname.[20:35:36] <ipalreadytaken> oh i see where you are going with this[20:36:13] <MerlinDMC> hackney does a request to http://1.2.3.4/foo and raises nxdomain[20:36:13] <ipalreadytaken> aysah: you want to change your config to use the ip address of you dsapi server and then svc reload?[20:36:59] <aysah> sure - i didnt do a reload yet[20:37:03] <aysah> one sec[20:37:18] <MerlinDMC> which is totally weird for me but yeah i'm not deep into erlang and telco ;)[20:38:25] <ipalreadytaken> MerlinDMC: are you saying that is behavior you have seen, or you are asking if tahts what happens?[20:39:00] <MerlinDMC> ipalreadytaken, I told aysah to change to the dsapid ip as erlang does raise nxdomain and curl is working fine[20:39:12] <MerlinDMC> still after using the ip we see nxdomain[20:39:45] <ipalreadytaken> he said he didnt reload fifo. it only looks at configs at startup[20:39:46] <MerlinDMC> don't know if still after the reload tho ... but afaik sniffle gets the complete url fed from jingles so a reload should not change that much[20:40:06] <MerlinDMC> is the dataset source not in the config.js anymore?[20:40:23] <ipalreadytaken> caching?[20:40:38] <MerlinDMC> fresh unused IE for testing - no cache[20:40:55] <aysah> sec did a reboot on fifo and "shutting down" is taking forever now[20:41:03] <aysah> oh fifo why do you hate me[20:41:07] <aysah> lol[20:41:45] <MerlinDMC> aysah, it took me 4h yesterday to get an IPMI connection ... solution was to reset the BMC[20:42:02] <aysah> lol nice[20:43:26] <ipalreadytaken> MerlinDMC: does dsapi look for any host headers or anything like that? I just dont understand why it works when he switches to *.at[20:43:54] <ipalreadytaken> to me that means that its a problem either with network or his dsapi install[20:44:24] <MerlinDMC> ipalreadytaken, it even works on his local instance via curl[20:44:37] <MerlinDMC> and no there is no host header check or anything - it's a dumb service[20:44:52] <ipalreadytaken> which is why i was thinking that mabey we are not sending a correct header or something[20:45:22] <MerlinDMC> you can even get the stuff via telnet and bare HTTP/1.0[20:46:25] <ipalreadytaken> yea im lost.[20:47:27] <ipalreadytaken> there is no https config switch, where it redirects to https right?[20:47:56] <MerlinDMC> nop ... you can setup https ... but both run in parallel - no forced https[20:48:20] <MerlinDMC> it's just mounting the complete router 2 times ... one via https socket and one via http socket[20:53:57] <ipalreadytaken> aysah: can you post your jingles conf?[20:54:32] <aysah> the only other behavior i can replicate is when I restart the FiFo zone and then import a dataset it fails but then if I click it one more time going forward it always says Importing Dataset and then I get the 0% progress dataset in my local datasets[20:54:45] <aysah> sure i'll get it now[20:57:31] <aysah> sec brb[21:00:01] *** liyu has quit IRC[21:08:56] <ipalreadytaken> looks fine[21:09:36] <ipalreadytaken> you mind trying to enable https on dsapi and in your jingles config[21:09:46] <ipalreadytaken> also change the jingles config back to using hostname[21:10:32] <aysah> k, on it[21:11:29] <ipalreadytaken> tea time[21:11:35] *** ipalreadytaken has quit IRC[21:12:01] *** ipalreadytaken has joined #project-FiFo[21:12:01] *** ChanServ sets mode: +o ipalreadytaken[21:14:52] *** ipalread_ has joined #project-FiFo[21:16:15] *** ipalreadytaken has quit IRC[21:19:37] <aysah> so guys its working now and really fucking fast lol[21:19:50] <aysah> i changed it to ssl and it then restarted the service and it didnt work[21:20:00] <aysah> and then i changed it back to http and now it works[21:22:55] <aysah> either way thanks again for everyone help on this.. i know where to look in the logs and configs now going forward[21:27:46] <ipalread_> ok so my working therory is that its setup only for https[21:27:58] <ipalread_> i will check with lice to see, but we should make a note of that in the docs[21:28:54] <ipalread_> come to think of it, there was some discussion a while ago about being worried about images being tampered with across the wire, and how there was no checksum. so i think https was the next best thing...[21:30:10] *** ipalread_ has quit IRC[21:30:36] *** ipalreadytaken has joined #project-FiFo[21:30:37] *** ChanServ sets mode: +o ipalreadytaken[21:30:48] <aysah> well its currently working on http for me[21:30:58] <aysah> its when i switched to https and then back to http that it started working[21:31:03] <aysah> which is kinda weird[21:34:32] *** ipalread_ has joined #project-FiFo[21:35:25] *** ipalreadytaken has quit IRC[21:35:28] *** ipalread_ has joined #project-FiFo[21:40:25] *** ipalread_ has quit IRC[21:50:08] *** marianoguerra has quit IRC[22:06:22] <mattronix> https://www.joyent.com/blog/sdc-and-manta-are-now-open-source[22:06:29] <mattronix> OMG OMG OMG OMG OMG OMG :P[22:16:45] *** marianoguerra has joined #project-FiFo[22:26:33] *** ipalreadytaken has joined #project-FiFo[22:26:33] *** ChanServ sets mode: +o ipalreadytaken[22:27:44] <ipalreadytaken> mattronix: wrong channel. you are looking for #joyent/#smartos[22:28:43] <mattronix> the manta stuff is sweet[22:28:44] <mattronix> :P i inteded to post it here[22:28:59] <mattronix> I dont think its going to replace fifo now its open[22:29:04] <mattronix> whole different set of requirments[22:29:34] <ipalreadytaken> :)[22:30:31] <mattronix> its quite bulkey[22:30:31] <mattronix> but manta being OS is sweet[22:32:27] <ipalreadytaken> for people running their own private systems its very limited use case[22:55:02] *** aysah has quit IRC[22:55:05] *** aysah_ has joined #project-FiFo[22:55:13] *** aysah_ has left #project-FiFo[22:55:22] *** aysah_ has joined #project-FiFo[22:57:27] *** aysah_ has quit IRC[22:57:39] *** aysah has joined #project-FiFo[23:31:05] *** marianoguerra has quit IRC