November 11, 2014 
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | >
November 11, 2014 [00:08:15] *** qchris has quit IRC[00:08:48] *** signed8b_ has joined #gerrit[00:11:51] *** signed8bit has quit IRC[00:17:24] *** signed8b_ has quit IRC[00:31:51] *** russt has joined #gerrit[01:56:27] *** firemanxbr has joined #gerrit[02:37:51] *** HeOS_ has quit IRC[02:39:35] *** HeOS_ has joined #gerrit[02:51:59] *** xnox_ has joined #gerrit[02:54:07] *** xnox has quit IRC[02:54:08] *** xnox_ is now known as xnox[02:56:52] *** jelmer has quit IRC[02:57:18] *** Camel has quit IRC[02:57:28] *** Camel has joined #gerrit[02:57:34] *** leepa_ has quit IRC[02:59:28] *** leepa_ has joined #gerrit[03:02:25] *** jelmer has joined #gerrit[03:03:59] *** jelmer has quit IRC[03:06:37] *** jelmer has joined #gerrit[03:16:12] *** topagae_ has joined #gerrit[03:20:05] *** topagae has quit IRC[03:24:23] *** miqui has joined #gerrit[03:33:50] *** Vampire0_ has joined #gerrit[03:37:42] *** Vampire0 has quit IRC[04:18:07] *** jrnieder has quit IRC[04:44:03] *** signed8bit has joined #gerrit[06:03:54] *** firemanxbr has quit IRC[06:11:44] *** miqui has quit IRC[06:49:01] *** signed8bit has quit IRC[08:07:41] *** JMZ_DMZ has joined #gerrit[08:07:46] <JMZ_DMZ> anyone here use gerrithub?[08:07:49] <JMZ_DMZ> does it work well?[08:27:46] *** jelmer has quit IRC[08:29:03] *** srenatus has joined #gerrit[08:29:11] *** ansiwen has quit IRC[08:32:00] *** HeOS has quit IRC[08:48:37] *** jelmer has joined #gerrit[09:08:56] *** ansiwen has joined #gerrit[09:15:34] *** mountains__ has joined #gerrit[09:24:36] *** franred has joined #gerrit[09:31:44] *** HeOS has joined #gerrit[09:32:53] *** HeOS has quit IRC[09:57:16] *** qchris has joined #gerrit[10:24:18] *** HeOS has joined #gerrit[10:25:29] *** teran has joined #gerrit[10:32:44] *** timothy has joined #gerrit[10:58:52] *** teran_ has joined #gerrit[11:00:12] *** teran has quit IRC[11:29:46] *** JMZ_DMZ has quit IRC[11:35:54] *** JMZ_DMZ has joined #gerrit[11:39:14] *** franred has quit IRC[11:48:18] *** mindjiver has joined #gerrit[11:52:03] *** JMZ_DMZ has quit IRC[11:58:55] *** teran_ has quit IRC[12:02:42] *** franred has joined #gerrit[12:38:08] *** plinio has joined #gerrit[12:40:10] *** Vampire0_ is now known as Vampire0[13:42:55] *** denizt___ has joined #gerrit[13:42:55] *** denizt__ has joined #gerrit[13:42:55] *** denizt_ has quit IRC[13:42:55] *** denizt has quit IRC[13:45:27] *** tty1 has joined #gerrit[13:46:29] <tty1> hey guys.. so im having hte same problem for weeks.. after about 24 hours i get an internal server error when i try to commit and need to restart tomcat daily.. nothing at all shows up in the logs though (this is a problem in genral.. i havent had any errors since i installed this thing ctually get logged to a log file)[13:46:38] *** sqlnoob has joined #gerrit[13:46:40] <tty1> so two things 1) how can i get errors to be logged...[13:46:55] <tty1> 2) how can i fix my actual problem (which i suspect willb e easier to diagnose after #1)[14:04:54] *** qchris has quit IRC[14:13:28] *** zenithdk has joined #gerrit[14:34:51] *** j`ey_w has joined #gerrit[14:34:54] <j`ey_w> hi[14:35:01] <j`ey_w> is there a way to look at "submodule subscriptions"?[14:42:49] <tty1> j`ey_w: what do you mean by subscription?[14:44:18] <j`ey_w> https://review.openstack.org/Documentation/user-submodules.html[14:44:52] <j`ey_w> I have 2 branches. One of them gets commits when a submodule is updated, the other doesnt[14:44:59] <j`ey_w> Im trying to understand why[14:46:15] <tty1> j`ey_w: i had similar problems when i first setup submodules.. though it works for me now (my main repo always updates whena submodule updates)[14:46:26] <tty1> j`ey_w: did you check your .gitmodule file? make sure its in there in both cases?[14:46:31] <j`ey_w> it is[14:46:42] <tty1> j`ey_w: and i assume the submodule and main repo are both on gerrit?[14:46:48] <j`ey_w> yep[14:47:09] <tty1> j`ey_w: not too sure then.. ive always found when i have my git modules configured correctly gerrit seemed to always handle it for me[14:47:27] <tty1> j`ey_w: i know i had a problem similar to yours when i first set up gerrit but i dont recall now what caused it[14:47:34] <j`ey_w> thing is, I dont want it to update automatically[14:47:47] <j`ey_w> and I dont see why it works for one branch and not the other[14:47:50] <tty1> j`ey_w: oh in that case i dunno.. i always found tht feature one of the best parts of gerrit[14:48:08] <tty1> j`ey_w: submodules are a hassle and almost unmanagable without that feature IMO[14:48:33] <j`ey_w> yes, but it makes them hard to sync[14:48:41] <j`ey_w> if I have a change to the submodule and the main repo[14:48:45] <j`ey_w> how do I do that at the same time?[14:49:20] <tty1> j`ey_w: just submit them at the same time[14:49:28] <tty1> they will show up as seperate changesets but show up at the same time[14:49:49] <j`ey_w> anyway[14:49:52] <j`ey_w> I dont want it[14:49:58] <j`ey_w> I want to do it manually![14:50:01] <tty1> j`ey_w: i understand, I dont know how[14:50:08] <j`ey_w> but I cant find much info about submodules and gerrit :/[14:50:18] <tty1> j`ey_w: ive always had it enabled and never sought a way to disable it.. so i cant help you[14:50:33] <tty1> j`ey_w: im sure there must be a way though, just gotta find it int he config docs :)[14:51:33] <tty1> j`ey_w: usually though id say submodules shouldnt have dependencies with their parent repo.. each repository should be able to be compiled on its own IMO.. but thats a matter of opinion[14:51:34] *** hugares has joined #gerrit[14:51:38] <j`ey_w> https://git.wiki.kernel.org/index.php/Gerrit_Submodules_Support[14:51:45] <j`ey_w> where is this "configure gerrit" file?[14:52:04] <j`ey_w> they dont, but a parent can have a depdency on a submodule[14:52:13] <tty1> j`ey_w: depends on your installation.. for me it would be in /opt/gerrit/etc[14:52:38] <j`ey_w> ok thanks, Ill take a look[14:52:50] <tty1> j`ey_w: true a parent can depend on its submodule.. but if it does i always make sure the submodule gets approved prior to the parent repo in that case[14:54:24] <j`ey_w> and if the submodule is committed first, the parent cant build with that version![15:03:07] *** echelog has joined #gerrit[15:13:59] <j`ey_w> tty1: do you have any commit restrictions?[15:14:09] <j`ey_w> we have some "submit" restrictions on reviews[15:14:16] <j`ey_w> I wonder if that's what's affecting the push[15:16:14] <tty1> j`ey_w: not anymore, i used to[15:16:41] <tty1> j`ey_w: once i moved over to gerrit i didnt find i needed hook-level restrictions.. i just review that by hand in gerrit now[15:36:10] *** signed8bit has joined #gerrit[15:43:11] <HeOS> Hi to all! Could anyone show example of gerrit hooks for patchset-created for example?[15:52:38] <pedroalvarez> Hi! I'm trying to automate the creation of a gerrit instance, and the creation of some users. I wonder if there is any way I can create the first user, which is going to be the admin user, in a script. Do I have to do it manually in the database?[15:56:00] <tty1> pedroalvarez: probably manually int he DB I would think..[15:56:16] <tty1> HeOS: i dont know what your asking[15:58:07] <pedroalvarez> tty1: thanks! I'll try to take a look at gerrit's code to see if I can do something else, though[15:58:35] <tty1> pedroalvarez: gerrit is pretty bare bones.. doesnt even provide a built-in way to delete a repository[15:58:46] <tty1> pedroalvarez: not even with using the web interface[15:59:07] <pedroalvarez> I didn't know that[15:59:09] <tty1> pedroalvarez: you pretty much needto manually do it ont he server (I think that entails editing the database and manually deleting the repo)[15:59:39] <tty1> pedroalvarez: there is a plugin you can add thatcan allow you to delete repos fromt he web interface.. but i have never been able to get it to compile[16:06:04] <dougk> tty1: logs should be (by default) under the logs/ folder... you should usually have an error_log and sshd_log. but I'm not sure how running in tomcat will play with that[16:06:36] <tty1> dougk: yea all i have under there is sshd_log no error_log[16:06:45] <dougk> if all else fails, yesterday I posted a gist which is the "default" set of logging: https://gist.github.com/vangdfang/686b10d514ce7a4bd6a5[16:06:53] <tty1> dougk: thats the main problem i cant find an error_log anywhere[16:07:17] <dougk> try setting the log4j.properties file to that; see if that starts to give you an error log (after you fix up the file paths in there)[16:09:05] <tty1> dougk: hmm not sure where i would have to put that log4j file.. obviously in the classpath somewhere.. just not sure of the proper spot for that[16:09:18] <dougk> actually, you can put it about anywhere[16:09:54] <dougk> https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#container[16:10:31] <dougk> you just have to make sure log4j.configuration is defined in the java options, I think.[16:10:35] <tty1> dougk: that says it only applies if i start gerrit as a standalone server, not tomcat[16:10:39] <HeOS> tty1, by default gerrit has a patchset-created hook, isn't it? But in the directory $gerrit_path/hooks no scripts which used this hook.[16:10:39] <dougk> correct.[16:10:58] <dougk> but can you pass log4j.configuration when starting Gerrit in tomcat?[16:11:19] <tty1> HeOS: oh i dunno.. i only ever used the commit hook i have to use to get hte change-id in there.. i dont know much about the gerrit server-side hooks though[16:11:26] <HeOS> In gerrit documentation I have not found examples for this hook.[16:11:45] <HeOS> tty1, ok, thanks. )[16:12:25] <dougk> I think you'd use the -Dlog4j.configuration=.... in CATALINA_OPTS?[16:12:37] <tty1> dougk: that seems likely[16:13:00] <dougk> (sorry, I've got practically zero experience running Tomcat, other than what the Atlassian apps use)[16:13:44] <tty1> dougk: ironically atlassian i couldnt get to work on tomcat so ran it standalone[16:14:31] <dougk> haha, I thought their "standalone" installs use tomcat in some pre-packaged form[16:15:25] <tty1> dougk: it might i didnt have to dig too deep into the internals[16:15:38] <dougk> hehe[16:15:49] <tty1> dougk: JIRA is a huge PITA to keep up and running[16:16:04] <tty1> dougk: so is gerrit but thats just cause i havent fixed logging.. once i do i expect it wont be so bad[16:16:38] <dougk> hehe, I find Gerrit easier than JIRA. the database schema is at least logical :)[16:18:03] <tty1> dougk: if i have to dig into the database then the app has already not lived up to expectations :)[16:18:13] <tty1> dougk: dont get me wrong, i liek gerrit way better than JIRA[16:18:27] <tty1> dougk: but that is mostly cause i like the direction its headed in :)[16:18:46] <tty1> dougk: and like i said 80% of my problem is just not having a log file anyway[16:21:56] <dougk> someone here found a bug in JIRA: if you attach a corrupt image, the thumbnailer tries to run on it and dies, making the attachment unavailable.[16:22:24] <dougk> so, I had to crawl around, find the attachment ID, and mark it as not thumbnailable[16:23:07] <tty1> dougk: sounds nasty![16:23:25] <dougk> just a bit annoying :)[16:25:01] <dougk> I'm curious how trying that log4j.properties goes, though... just a warning, it does turn off a few more warnings that I had bugging me--but they're documented at the bottom[16:26:32] <tty1> dougk: im doing it now will be done any second[16:27:03] *** mountains__ has quit IRC[16:27:06] *** j`ey_w has quit IRC[16:29:00] <tty1> dougk: just need to restart tomcat and pray it works :)[16:31:53] <tty1> dougk: no change, no error_log is showing up.. but im not sure if that just means there is no error[16:32:46] <dougk> hmmm, you *should* get Gerrit start messages at the least.[16:32:47] <tty1> dougk: CATALINA_OPTS="-Dlog4j.configuration=/opt/gerrit/etc/log4j.properties"[16:33:12] <tty1> dougk: and i set CATALINA_OPTS in the proper place as it was already there (just an empty string)[16:34:00] <tty1> dougk: i can try putting it into the java_opts variable instead...[16:35:38] <tty1> dougk: it doesnt appear to work in JAVA_OPTS either[16:36:54] <dougk> hmm[16:37:14] <dougk> you modified the hard-coded paths in that configuration, right?[16:38:59] <tty1> dougk: i did yes[16:39:20] <tty1> dougk: i searched for /data/ and replaced it with /opt/ which matched my path[16:40:30] <dougk> ok[16:40:48] <dougk> and, /opt/gerrit/logs/ is writable by the gerrit user?[16:41:03] <tty1> dougk: its writable by tomcat user[16:41:14] <tty1> dougk: and the sshd log is still created fine[16:41:21] <tty1> but the error_log is nowhere to be found[16:41:34] <dougk> this is going out on a limb... but... are you running selinux?[16:42:01] <tty1> dougk: no gentoo[16:43:22] <dougk> no, I mean the "security enhanced linux" options (mandatory access control)... I know redhat and fedora enable it by default, and you can build it for gentoo[16:44:06] <dougk> the only reason I say that is because selinux has a way of mucking with file creation and stuff :)[16:44:44] <tty1> dougk: oh no, nothing like that[16:44:58] <tty1> it isnt a hardened install or anything like that[16:45:05] <tty1> doesnt even uyse ACL[16:45:40] <dougk> hmm... the only other thing I can think of... does Tomcat have its own logs? maybe it's ending up there?[16:46:30] <tty1> dougk: it does have its own logs, and i reviewed those logs in the past and never found anything from gerrit in them[16:47:17] <dougk> ok[16:48:05] <tty1> dougk: im surpsied there isnt anything about this on the internet[16:48:11] <dougk> (sounds like I should try running tomcat just to see what happens) :)[16:48:16] <tty1> dougk: dont a good number of people use gerrit? Id think this is a common thing[16:49:10] <tty1> dougk: i know ive setup gerrit on 3 seperate servers over the past year, always running off of tomcat, and always had this problem[16:49:25] <tty1> dougk: so either hte installation docs i used were missing something, or there is a bug[16:49:26] <dougk> yeah, but usually the recommended setup is using the internal jetty container.[16:49:32] <dougk> both are possible :)[16:54:58] <tty1> dougk: lame :([16:55:07] <tty1> dougk: this has been a thorn in my side forever :([16:55:24] <tty1> dougk: id run it on jetty but my resources ont he box are strained and the box hosts all tomcat webapps[16:55:31] <tty1> so id have to setup another box for that :([16:55:32] <dougk> aaah[16:57:08] <dougk> let me see what happens if I try an install :)[16:57:19] <tty1> dougk: kk, incase thgis helps http://wiki.apache.org/tomcat/FAQ/Logging#Q1[16:59:57] <tty1> dougk: take note of any tomcat configuration files you change.. if somehow it works for you id like to compare our config files[17:00:04] <dougk> sure[17:00:37] <dougk> note, I'm kinda starting from a blank slate, having never done anything with tomcat :) so, I'll let you know how it goes.[17:00:53] <dougk> (I'm just doing this on a dev server)[17:01:28] <tty1> dougk: let me know if i canhelp with that[17:02:17] <dougk> cool, thanks :)[17:05:13] <dougk> actually, just a question: what version of tomcat are you running?[17:09:26] <tty1> dougk: 7[17:18:15] <dougk> thanks[17:18:57] <tty1> dougk: np[17:27:36] <tty1> dougk: i found some, potentially relevent log lines.. just warnings though which explains why i overlooked it before[17:29:18] <tty1> dougk: judge for yourself: https://gist.github.com/freemo/c95f13ecfbd17c8061ce[17:40:06] *** HeOS has quit IRC[17:41:21] <dougk> OK, so here's the deal. I can get Gerrit partially running under tomcat, and it looks like the default is spewing messages to catalina.out[17:43:03] <dougk> tty1: I think I saw that same message... but it doesn't seem to do anything[17:43:53] <tty1> dougk: i have a catalina.out file but thats where i got the warning messages from[17:44:33] <tty1> although on gentoo its called catalina.<date>.log but same thing[17:54:29] <tty1> if its working for you perhaps we can compare our server.xml files[17:54:45] <tty1> dougk: thats where catalina logging is configured in a global way[17:57:35] <dougk> I literally kept a default server.xml, then added the context there.[17:57:49] <dougk> so nothing special[17:58:12] <tty1> dougk: yea but im using gentoo, im worried the gentoo package uses a non standard server.xml[17:58:20] <dougk> ah[17:58:35] <tty1> dougk: at this point i want to bethurough because im out of ideas otherwise anyway[17:58:51] <dougk> gotcha[17:59:09] <dougk> let me try again, see if I can actually get the web UI to come up :P last time I got stuck in a redirect loop[17:59:27] *** dborowitz has quit IRC[17:59:39] <tty1> dougk: if yout trying to redirect the port i use apache to proxy for that[18:00:02] <dougk> yeah, we use the same.[18:00:10] <dougk> for some reason, I got a double-slash[18:00:53] <tty1> dougk: i dont think i ran into that problem[18:01:01] <tty1> if my config files could be of any help to you let me know[18:01:02] *** ashp has joined #gerrit[18:01:14] <tty1> dougk: but once youg et it working if i could see yoru server.xml it might shed some light on this[18:01:24] <dougk> sure :)[18:01:26] <tty1> dougk: im trying to dig deeper into tomcat to see if there might be something i missed there[18:04:13] <dougk> tty1: [2014-11-11 11:03:57,741] WARN com.google.gerrit.server.documentation.QueryDocumentationExecutor : No index available[18:04:20] <dougk> that's what I get :([18:04:48] <dougk> it's trying to redirect with a double-slash[18:04:56] <tty1> I dont think ive seen that one :([18:05:10] <tty1> would it help if i shared my apache proxy setup for gerrit?[18:05:16] <ashp> reading changelogs since 2.4 is painful. Something has changed so my CI group can no longer push tags into gerrit[18:05:40] <ashp> Anyone here know off the top of their head if anything was added to specifically handle tags from an acl viewpoint[18:15:30] <dougk> tty1: okay, I got it: catalina.properties has gerrit.site_path defined, and I'll provide you my server.xml.[18:15:51] <dougk> ashp: hmm, I think annotated tags and regular tags were separated at some point? but I could be mistaken.[18:16:13] <ashp> dougk: yeah, I find all sorts of answers online, from requiring forging onwards :/[18:16:18] <ashp> I'm trying several things, no luck so far[18:16:28] <ashp> https://code.google.com/p/gerrit/issues/detail?id=2012 for example[18:16:56] <ashp> frustrating, no matter what I get it's always:[18:16:57] <ashp> ! [remote rejected] releases/2013.1.7.3 -> releases/2013.1.7.3 (prohibited by Gerrit)[18:17:24] <dougk> tty1: https://gist.github.com/vangdfang/d167786493bf2b0fefe5[18:17:39] <dougk> ashp: can you tell if the tag is annotated or not?[18:18:21] <ashp> I don't think they are annotated, I don't ... actually know how to tell, but I'm fairly sure we don't annotate tags at all, hang on[18:18:48] <dougk> do you know how the tag is created?[18:19:08] <tty1> dougk: hmm i do see a slight different between our server files.. lets hope it makes a difference[18:19:59] <ashp> oh they -are- annotated[18:20:20] <ashp> i have push annotated tags on for refs/* however[18:21:42] <ashp> I added create reference as mentioned in https://code.google.com/p/gerrit/issues/detail?id=1048 as well, then tried both forging options[18:23:20] <tty1> dougk: ok NOW i am getting stuff in the logs.. but its mostly jsut severe errors.. not sure if what i did fixed the logging or jsut added a new problem though[18:23:36] <dougk> tty1: lol, what'd you change? :)[18:23:48] <tty1> <Context docBase="/data/gerrit/bin/gerrit.war" path="/code-review" reloadable="false" useHttpOnly="true" />[18:23:53] <tty1> dougk: just had to add that line[18:23:57] <tty1> of course i changed /data to /opt[18:24:22] <dougk> tty1: oh, I'm not sure that's the right way to do it. context.xml seems to be "preferred"[18:24:28] <dougk> I was just lazy[18:24:29] <ashp> https://gerrit-review.googlesource.com/Documentation/error-prohibited-by-gerrit.html based on this it should be working but... I clearly am missing something :)[18:24:53] <tty1> dougk: ahh let me see if i have something similar in context.xml[18:25:20] <tty1> dougk: it certainly is dumping exceptions to the log now though :)[18:25:20] <dougk> ashp: we grant only push annotated tag to refs/tags/* for our bots[18:25:41] <tty1> dougk: and they are gerrit specific exceptions[18:25:48] <ashp> Yeah, I've inherited an existing setup, this had most privs on refs/*, let me try and build a refs/tags/* that works. :/[18:25:49] <dougk> wooo! \o/[18:26:03] <ashp> dougk: I don't suppose you could do me a massive favor and screenshot just the refs/tags/* bit if that's sharable?[18:26:07] <ashp> just so I can be sure[18:26:12] <dougk> also, maybe check there aren't any exclusive rights[18:26:23] <ashp> we have nothing set exclusive[18:26:27] <tty1> dougk: i do seem to have stuff in context.xml but only a reference to the datasource[18:26:31] <dougk> our ACLs are long and complicated; it's unlikely to help :)[18:27:05] <dougk> what may help is https://gerrit.googlesource.com/plugins/admin-console[18:27:24] <dougk> it has a means to collect ACLs for a particular user on a particular project; maybe that'll help.[18:29:17] <tty1> dougk: i may need your help with the new exceptions :([18:29:56] <dougk> I'll try? it's lunchtime now, though.[18:30:07] <dougk> should be back in about an hour.[18:30:54] <tty1> dougk: ok.. tomcat is coming bakc up (wipped the log) so ill post it in about 5 minutes[18:30:57] <tty1> enjoy your lunch ill be here[18:31:02] <dougk> cool! good luck :)[18:33:15] <tty1> dougk: https://bpaste.net/show/8a7de9fa2e78[18:34:01] <tty1> dougk: bear in mind despite all thos eexceptions gerrit does run, at the moment, ok (on the surface).. it is just that about after 24 hours i start getting internal server errors if i try to make a commit. Hopefully these exceptions are related to that[18:42:06] *** dborowitz has joined #gerrit[18:42:16] *** pedahzur has joined #gerrit[18:43:17] *** timothy has quit IRC[18:47:43] <pedahzur> We have a gerrit install with 54 projects/repositories. Total size of all projects is 3.4G. I have max heap set to 6G and packedGitLimit set to 4G. In theory, all repos should be able to be in memory. The server load is never over 1 or 2 (on a 4 CPU system) with next to no I/O load a indicated by top. Clones/pushes/pulls (over gerrit ssh) can take on order of several minutes, sometimes 10 or 20 (it will just hang for a LONG time,[18:47:45] <pedahzur> then suddenly resume and finish). Any ideas for debugging this? This is really frustrating us.[18:50:58] <ashp> argghhhhhhh, I'm going to eat the permissions system, it's driving me insane[18:52:39] <tty1> pedahzur: have you tried running it locally directly cloning the repos through SSH without going through gerrit (just to see if the slowdown is because of anything gerrit does)[18:53:03] <tty1> pedahzur: my understanding is that when you clone a git repo through gerrit that gerrit really doesnt do much, shouldnt be different than cloning anything else via SSH[18:53:14] <tty1> pedahzur: but that would be one way to check[18:55:13] *** plinio has quit IRC[19:02:05] <pedahzur> tty1: I'll try that.[19:06:37] <tty1> pedahzur: by the way when you said the size of yoru git repo.. you did mean your ENTIRE repo not just the latest commit right?[19:07:09] <pedahzur> tty1: That 3.4G is the total on-disk size of *all* repos in our gerrit system.[19:07:25] <tty1> pedahzur: ok then[19:07:45] <tty1> pedahzur: generally i dont like to let me repos get too big.. mostly because its annoying for the devs[19:08:19] <pedahzur> tty1: yeah. We do have some repos with binaries in them, but even then, we're not talking multi-gigs.[19:09:25] <tty1> pedahzur: good to hear[19:10:06] <tty1> pedahzur: ive went to great lengths to keep binaries out of main repos.. usually my rule is.. if i absolutely must include binaries in a repo i make sure to put them in a submodule so i can cut it out of the repo without it being int he history of the main[19:10:12] <tty1> pedahzur: but that doesnt seemt o be a huge issue here[19:10:57] <pedahzur> tty1: Yeah, not my decision, and I really didn't have the the backing to say "no," sadly.[19:11:27] <tty1> pedahzur: yea it happens..[19:14:30] <pedahzur> OK, here's a fun one. I'm trying to do a git clone, over ssh, to the gerrit server and getting this error: "Disconnected; key exchange or algorithm negotiation failed (Key exchange failed.)." This is some "enterprise" ssh that I'm being forced to use: wrq-rsit-server-6.1.2.1-0.3005.sles10 Anyone seen this before?[19:16:45] <tty1> pedahzur: not me.. sounds like it isnt listening to your public key though[19:18:46] <tty1> pedahzur: your use of proprietary SSH could be the source of all your woes.. maybe it is just slow at how it handles a lot of data over SSH[19:31:10] <pedahzur> The proprietary ssh is on the server running Gerrit, thus the issue there...[19:33:11] *** srenatus has quit IRC[19:49:37] <dougk> pedahzur: just a question, have you repacked recently? and what gerrit version?[19:49:58] <dougk> ALSO, which client are you using? :)[19:51:00] *** jrnieder has joined #gerrit[19:51:29] <dougk> tty1: yeah, interesting; I didn't get any of those[19:51:33] <pedahzur> dougk: We repack once a week. We use the git pack; should look at using 'gerrit pack'. Gerrit 2.8.5. Clients are various versions of git. >= 1.7, I think.[19:51:46] <tty1> dougk: i think they are what was my originally problem[19:51:48] <dougk> Windows?[19:52:01] <pedahzur> dougk: All Linux or Mac.[19:52:18] <dougk> tty1: interesting[19:52:20] <tty1> dougk: since it takes 24 horus to break a memory leak makes sense as a problem[19:52:28] <dougk> pedahzur: :( damn. that's one easy problem I can't blame[19:52:36] <tty1> dougk: im thinking this may be a bug in gerritat this point.. do you agree?[19:53:03] <dougk> OH, the possible memory leaks? it may be a leak, but we've not encountered any memory leaks that cause issues.[19:53:23] <dougk> it may also just be how injection works, but I'm not certain. I *did* see those messages.[19:54:25] <pedahzur> dougk, tty1: Seems I'm not the only one hitting this: https://groups.google.com/forum/#!topic/repo-discuss/Xa7UxKIgu0w[19:54:33] <pedahzur> (Old messgae, though)[19:56:15] <dougk> pedahzur: the ForwardAgent no?[19:56:29] <pedahzur> I'll try that.[19:58:55] <pedahzur> dougk: There is also something about replacing Gerrit ssh keys, but I can't do that right now. Not sure why ForwardAgent no would change things, but I'll give it a go.[19:59:13] <pedahzur> Nope.[19:59:28] <pedahzur> So, is this a bug in Gerrit (or the SSH part of Gerrit) or in Reflections Secure IT?[19:59:33] <dougk> hmm, so, clone and push are extremely slow? what connection method do you use? ssh only, or have you tried HTTP?[20:00:47] <pedahzur> ssh only. No, we've not tried http.[20:01:20] <pedahzur> Hmm...in my SITE_DIR/etc/ I have a ssh_host_dsa_key file...I wonder why it's not using that.[20:03:18] <dougk> (I'd say http would be a way to remove variables, but it's a bit tricky if you use a proxy in front of Gerrit)[20:04:47] <dougk> if it's possibly an issue with binaries causing headaches (usually when this happens, jstack indicates that you have threads stuck in the inflate delta step, I think)... you can do a bit of magic to fix things up.[20:05:32] <pedahzur> dougk: Well, the repos having issues aren't necessarily the ones with binaries in them. It's the site as a whole.[20:05:57] <dougk> ah, but once you have one or two requests that are "stuck" -- it can quickly bog down the server. jstack output is useful here.[20:06:47] <dougk> ALSO, 2.9 doesn't have the "slow path" streaming delta support any more, so it may remove the issue.[20:08:00] <pedahzur> dougk: Ah, I see. Yeah, even http cloning is slow.[20:08:15] <pedahzur> dougk: Is there good page that shows how to do the jstack debugging?[20:08:36] <dougk> well, the basics is find your pid, and then sudo -u (gerrit_user) jstack (pid)[20:09:00] <dougk> that'll print a wall of text. then, you can peek and see which threads are running.[20:13:41] <pedahzur> dougk: Ah, OK. Thanks.[20:13:54] <pedahzur> dougk: jstack: command not found. Where do I find it? :)[20:14:29] <dougk> look under $JAVA_HOME/bin/ ?[20:14:42] <dougk> it may only be with the JDK[20:15:06] <pedahzur> dougk: Ah...right, I have an oracle RPM installed...let me look.[20:15:20] <tty1> dougk: so those errors in my log, the ones that say a memory leak is possible.. you dont think those actually cause a memory leak?[20:15:44] <pedahzur> dougk: Shoot...no jstack. Time to go find it... :)[20:15:46] <dougk> not necessarily[20:17:51] <dougk> at least, if there's a leak, it's not seriously hit us :)[20:30:40] *** hugares1 has joined #gerrit[20:33:56] *** hugares has quit IRC[20:38:47] <tty1> dougk: well i guess ill see if any new exceptions come up when the problem hits again in a day[20:38:59] <dougk> tty1: good luck![20:39:33] <tty1> dougk: thanks, youll probably hear from me in a day :)[20:39:56] <dougk> haha, well thanks for challenging me to set up tomcat :)[20:41:23] <pedahzur> dougk: Are you a gerrit dev?[20:41:44] <dougk> pedahzur: other than I've hacked on it? no :)[20:42:04] <pedahzur> dougk: OK[20:42:18] <dougk> (I do have a CLA, have submitted patches, etc. but not officially affiliated) :)[20:43:07] <tty1> dougk: the weird thing is, my gerrit server is pretty low traffic.. im surprised it fails at all :) anyway ill keep you informed[20:43:38] <dougk> yeah, I mean, we handle ~50k pulls, maybe 2k pushes daily[20:50:40] <pedahzur> dougk: you said "you can do a bit of magic to fix things up." What might that magic be? :)[20:51:21] <dougk> the steps I found... set core.bigFileThreshold low, like 10mb, then find the "big" objects in the repo[20:51:33] <dougk> I used http://naleid.com/blog/2012/01/17/finding-and-purging-big-files-from-git-history/ to look for them...[20:52:12] <dougk> then, I add those filetypes to gitattributes with the -delta option (i.e. echo "*.dll -delta" > info/gitattributes)[20:52:25] <dougk> finally, git repack -a -d -f --window-memory 100m --max-pack-size 20m[20:52:34] <pedahzur> Gotcha.[20:52:55] <dougk> that just keeps you out of the streaming delta case.[20:53:54] <pedahzur> dougk: Yeah, I don't know if we'll be able to actually remove things from the repo, since old versions might rely on the binaries being in the repo, but going forward for new pulls and fetches, hopefully we can do away with the large files.[20:54:20] <dougk> yeah, it's not necessarily to remove them from the repo[20:54:29] <dougk> but, you repack them so they're not delta-compressed[20:54:46] <pedahzur> Gotcha![20:55:40] <pedahzur> dougk: So, if I add that max-pack command to my weekly repack, will it speed everything up in the end? Or is there more work required? :)[20:56:15] <dougk> that "finding and purging big files" article is useful if you wanted to delete them... but at the same time, you can use it to find those "giant" objects and not remove them.[20:56:48] <dougk> ... I don't think I've used max-pack-size for everything... but let me check.[20:57:12] <dougk> (for our nightly repacks, I actually just use "gerrit gc"[20:57:16] <pedahzur> dougk: Oh! We do a weekly git gc...we don't do a weekly repack....[20:58:03] <dougk> nope, our old scripts only did "git gc --auto --prune" -- new one runs gerrit gc after that became available in 2.7[20:58:18] <dougk> I only did the repack on an as-needed basis when things broke.[20:58:55] <pedahzur> dougk: So, if I can't convince the particular dev team to remove binaries, will git repack .... --max-pack-size mitigate the problem since there won't be all the uncompression work when it comes time to clone/pull/fetch?[21:00:15] <dougk> it can, yes[21:00:41] <dougk> and the problem is, you probably can't remove things that are already inside the repo, since you'd change every sha1 to rewrite it.[21:05:40] <dougk> (I get to fight our developers on this, too... part of why I was so hot to remove the slow streaming delta code.) :)[21:16:46] <pedahzur> dougk: Hmm...maybe I'll repack with a max pack size of only 10 or 5MB. We aren't short on disk space...[21:17:05] <dougk> exactly :) it's a bit of a size trade-off... but yeah.[21:23:35] *** hugares1 has quit IRC[21:43:03] <pedahzur> dougk: Yay...got jstack working. So, I'll start doing periodic dumps. What am I looking for to see if there are threads holding up the process?[21:43:51] <dougk> you'll see something like InflateZip I think... I usually would notice problems with CPU load; htop would show it extremely busy as well.[21:44:27] <dougk> then, I'd run jstack, capture the output, and look to see what threads were listed as RUNNABLE[21:48:03] *** firemanxbr has joined #gerrit[21:52:05] <pedahzur> dougk: That's the thing...the CPU usage is never particulartly high... :|[21:52:26] <dougk> hmmm, interesting. low load average, too?[21:52:40] <dougk> (must have missed that early on)[21:52:47] <pedahzur> dougk: Usually..1 to 2 (on a four CPU system).[21:53:46] <dougk> OK. I mean, we also throw gobs of RAM and CPU at Gerrit... and I know you said you've got a lot of memory, but ...[21:54:05] <dougk> there's some tuning aspect from what I remember.[21:54:35] *** HeOS has joined #gerrit[21:55:04] <pedahzur> dougk: Yeah, I'm sure there is some tuning we could do. We also started having these slow-down issues when the binaries were added to the repo, so I'm suspecting them as well.[21:55:20] <dougk> interesting.[21:56:30] <dougk> the tuning bits I found helped me were database connections, http threads, ssh threads, etc.[21:56:52] <volker-> we set the keepalive of sshd to 1sec (don't ask me why, some issues before I was here). now there was a problem with the git clone that just lost the connection cause of it... also based on big data blobs in the repository.[21:57:39] <dougk> yeah, that sounds familiar.[21:58:45] *** docwhat is now known as zz_docwhat[21:58:49] <dougk> I think I keep that value extremely high, but still low enough to actually be useful.[22:05:07] *** firemanxbr has quit IRC[22:07:43] *** firemanxbr has joined #gerrit[22:36:11] <pedahzur> dougk: Another question about that git repack. If I do that with a very low MB setting, that is "sticky," right? It won't be reverted or otherwise changed by our weekly git gc runs, will it?[22:38:07] <pedahzur> dougk: Also, I can't find a core.bigFileThreshold in the docs.[22:41:24] <pedahzur> dougk: Hmm... --max-pack-size=<n> seems to have to do with the maximum size of a pack file, not so much the maximum size of a file that it will compress. Maybe another option?[22:49:00] *** zz_docwhat is now known as docwhat[23:04:30] *** dusted has quit IRC[23:30:45] *** qchris has joined #gerrit[23:39:09] <pedahzur> Hmm...so 2.9 swaps the position of the commit message and the owner/reviewers/etc. with no way to switch it back. Why the change?[23:50:55] *** miqui has joined #gerrit[23:57:15] *** signed8bit has quit IRC