[00:06:39] <grundler> [ 6673.900166] Unhandled prefetch abort: breakpoint debug exception (0x002) at 0
[00:06:39] <grundler> x40807a64
[00:06:40] <grundler> [ 6674.220065] init: update-engine main process (10742) killed by TRAP signal
[00:06:40] <grundler> [ 6674.228542] init: update-engine main process ended, respawning
[00:06:48] <grundler> known bug?
[00:08:40] <saintlou> it seems that stumpy-binary has been in 'offline' status since last night.
[00:10:49] *** vmil86 has quit IRC
[00:20:38] *** D|sT has quit IRC
[00:26:15] *** D|sT has joined #chromium-os
[00:29:17] *** FusionX has quit IRC
[00:33:54] *** FusionX has joined #chromium-os
[00:36:51] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "tegra2 seaboard full" from 35694152dd616f5da76d6b383ad54efe7a541a62: Darin Petkov <petkov at chromium dot org>)'
[00:37:20] <cmasone> hm...escalate to troopers?
[00:38:47] *** chocobo__ has quit IRC
[00:39:10] <saintlou> cmasone: which issue issue my question from 3:02 or the tree is closed
[00:39:59] <cmasone> saintlou: the latter
[00:40:08] <saintlou> cmasone: who are the CrOS troopers I only know kliegs and he is on Eastern time. My 2 emails to the troopers at chromium dot org alias have been unanswered
[00:40:22] <cmasone> crosbot: troopers?
[00:40:23] <crosbot> cmasone: troopers: djmm, scottz, bradnelson, maruel (EST), kliegs (EST)
[00:40:34] <saintlou> cools, thanks cmasone
[00:40:42] <cmasone> saintlou: though, it's a problem that emails to troopers at chromium dot org are going unanswered
[00:40:53] <saintlou> cmasone: I would agree with that.
[00:40:57] <cmasone> oh...you know what?  They're all at an offsite :-P
[00:41:03] <nsanders> i think the stumpy builder is down, but also the stumpy build is broken
[00:41:08] <cmasone> except maruel, who's EST
[00:41:14] <saintlou> cmasone: seriously?
[00:41:48] <cmasone> saintlou: well, the chrome-infrastructure team is, which is mostly who's on troopers@chromium, and scottz is attending to a family situation
[00:42:38] <cmasone> davidjames: sosa could one of you glance at the stumpy builder?  I don't know how to check on the slave and such
[00:43:05] <sosa> cmasone: i'll do it <acquires lock>
[00:44:16] <saintlou> thanks sosa
[00:44:41] <cmasone> saintlou: as for the upload prebuilts failure...I got nothin'.  Maybe file a bug and assign to davidjames/anush?
[00:44:52] <cmasone> http://build.chromium.org/p/chromiumos/builders/arm%20generic%20full/builds/175/steps/UploadPrebuilts/logs/stdio
[00:45:02] <cmasone> that log has some info at the end
[00:45:28] <saintlou> cmasone: I have pinged djmm
[00:46:13] <cmasone> saintlou: sounds good
[00:46:31] <sosa> (the builder is back up...it's starting with the canary then moving to the binary)
[00:46:37] <saintlou> thanks sosa
[00:46:45] <saintlou> thanks cmasone I see error: RPC failed; result=52, HTTP code = 0 :-(
[00:48:03] <cmasone> saintlou: yah, no idea what that means, but that's what I saw too
[00:48:19] <crosbot> tree became 'Possible transient failure uploading prebuild -- tree is opem'
[00:48:22] *** micahc has joined #chromium-os
[00:48:22] *** ChanServ sets mode: +v micahc
[00:48:35] <crosbot> tree became 'Possible transient failure uploading prebuilts -- tree is opem'
[00:50:37] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "arm generic PFQ" from b6767ff3b6eb621c8a49c0ad8539094a506be7ff: Chris Masone <cmasone at chromium dot org>)'
[00:51:19] <cmasone> uh
[00:51:40] <cmasone> Traceback (most recent call last):
[00:51:40] <cmasone>   File "/b/build/slave/arm-generic-PFQ/build/chromite/buildbot/cbuildbot_background.py", line 95, in run
[00:51:43] <cmasone>     step()
[00:51:45] <cmasone>   File "/b/build/slave/arm-generic-PFQ/build/chromite/buildbot/cbuildbot_stages.py", line 658, in _RunVMTests
[00:51:48] <cmasone>     os.nice(-10)
[00:51:51] <cmasone> OSError: [Errno 1] Operation not permitted
[00:51:53] <cmasone> ???
[00:51:56] <sosa> <- davidjames
[00:52:08] <sosa> err rather ^^^ davidjames
[00:52:26] <sosa> he just committed a CL related to that
[00:52:42] <cmasone> davidjames: shall we revert?
[00:53:04] <cmasone> http://gerrit.chromium.org/gerrit/6208
[00:53:51] <gauravsh> what user is this run as?
[00:55:13] <gauravsh> you can't renice to negative unless you are r00t
[00:56:13] <cmasone> I vote we revert, unless someone can raise david on chat
[00:56:38] <saintlou> agreed cmasone
[00:56:50] <gauravsh> http://gerrit.chromium.org/gerrit/6259
[00:57:00] <gauravsh> i had always wanted to try the gerrit revert
[00:57:10] <cmasone> gauravsh: don't forget to push
[00:57:18] <gauravsh> shall I?
[00:57:21] <gauravsh> sheriffs?
[00:57:23] <davidjames> gauravsh: Yeah sure, we can revert, I guess they don't like os.nice(-10)
[00:57:37] <davidjames> gauravsh: But it doesn't push, haha
[00:57:38] <gauravsh> woohoo
[00:57:38] <gauravsh> done
[00:57:50] <davidjames> gauravsh: I'll juist remove os.nice(-10), easy change
[00:58:06] <saintlou> I reverted
[00:58:08] <saintlou> http://gerrit.chromium.org/gerrit/6261
[00:58:28] <davidjames> saintlou: Double fail, does not merge :)
[00:58:43] <saintlou> well I thought I was the sherriff
[00:58:59] <gauravsh> davidjames: yeah, the test wouldn't have caught that because of the mocks.
[00:59:05] <saintlou> I appreciate gauravsh help though....
[00:59:08] <quiche> davidjames: was the intent to give the process higher priority, or lower?
[00:59:14] <cmasone> saintlou: gaurav is usurping your authority!
[00:59:29] <gauravsh> saintlou: sorry about that. i had always wanted to use gerrit revert banhammer.
[00:59:34] <saintlou> cmasone: OK gauravsh you are now deputized!
[00:59:43] <jhorwich_> quiche: comment leads me to believe it's to make it higer priority
[00:59:45] <saintlou> gauravsh: understood
[00:59:54] <davidjames> saintlou: I'll just remove os.nice(-10), should just take a second
[01:00:05] <gauravsh> saintlou: next time I will respect your authoritah! :)
[01:00:14] <quiche> jhorwich_: ah, ok. (thought it might have been a simple sign error.)
[01:00:24] <saintlou> gauravsh: since I am a lone west coast sheriff you are more than welcome to have you help :-) I am still very new at all this stuff (as cmasone can attest)
[01:00:58] <davidjames> saintlou: Ok I just killed the os.nice
[01:01:05] <saintlou> thanks davidjames
[01:01:07] <davidjames> That seems easier than reverting it
[01:01:56] <davidjames> os.nice(-10) works on my machine, maybe I was running as root
[01:02:16] <gauravsh> davidjames: yes, unless you messed up somewhere else in your CL. :P
[01:02:29] <jhorwich_> davidjames: any value in wrapping the os.nice call in something that catches the os.error exception (disclaimer: I'm not a python programmer)
[01:02:33] <quiche> davidjames: increasing prio does require root
[01:02:35] <saintlou> davidjames: are you going to kill the builds that are doomed as per FAQ or should I unstruct gauravsh to do so :-)
[01:02:37] <jhorwich_> ?
[01:03:09] <davidjames> saintlou: Doing that now
[01:03:16] <saintlou> thanks davidjames
[01:03:28] <saintlou> just checking :-)
[01:03:29] <gauravsh> saintlou: it's been a pleasure feeling like a sherriff for 5 minutes, and failing at the revert catastrophically. i am signing out in shame. :)
[01:04:01] <saintlou> gauravsh: I will make sure you are on the next rotation ;-)
[01:05:35] <davidjames> saintlou: Ok just finished killing all builders in sight :)
[01:05:55] <saintlou> thanks davidjames I see the purple status :-)
[01:05:56] <crosbot> tree became 'Tree is open (os,nice issue fixed, watching builders -> davidjames)'
[01:06:11] <crosbot> tree became 'Tree is open (os.nice issue fixed, watching builders -> davidjames)'
[01:07:12] <crosbot> tree became 'Tree is closed (one more builder to kill)'
[01:07:28] <crosbot> tree became 'Tree is open (os.nice issue fixed, watching builders -> davidjames)'
[01:07:46] <davidjames> saintlou: Ok hope I got them all!
[01:07:52] <saintlou> thanks davidjames
[01:07:57] <davidjames> saintlou: Let me know if you spot any doomed builders still hanging around
[01:08:04] <saintlou> sure will
[01:08:06] <saintlou> thanks
[01:08:28] <davidjames> saintlou: BTW the reason why it works for me is because my shell was already running with 'nice 10'
[01:08:45] <davidjames> saintlou: So nice -10 works for me, but if you are running at nice 0 then nice -10 takes you to too high a priority
[01:09:39] <saintlou> good to know davidjames
[01:11:17] <davidjames> saintlou: Just kicked off PFQ to get things rolling again
[01:11:33] <saintlou> that is 'nice' davidjames ;-) thanks!
[01:15:32] <grundler> uh...known outage? "Oops! Google Chrome could not connect to feedback.corp.google.com"
[01:17:25] *** benchan has quit IRC
[01:18:37] <gauravsh> is it safe to push, or should I wait to make sure david's fix works?
[01:19:02] *** sadrul has quit IRC
[01:19:27] <davidjames> gauravsh: should be fine to push, kaen binary already passed test phase
[01:19:41] <jhorwich_> grundler: I had no problem visiting feedback using chrome (via http / https, on my desktop chrome). Transient error perhaps?
[01:19:56] <davidjames> (it doesn't run tests but it does hit the code path that would have failed before)
[01:20:26] *** jennb has joined #chromium-os
[01:20:27] *** ChanServ sets mode: +v jennb
[01:22:02] <grundler> jhorwich_, oh...I'm not signed into corp account. I can't access from the windows that have chromium.org auth. :(
[01:22:40] <grundler> that means I need one auth to update the bug which is different from the auth necessary to view the original data
[01:22:46] * grundler tests his theory
[01:23:34] <grundler> jhorwich_, yeah. It wants an OTP and all that. :(
[01:23:43] <grundler> It == feedback.corp
[01:24:32] *** aaronp_ is now known as aaronp
[01:25:55] <grundler> grrh. Now I can't close the tabs that have "Chrome could not connect" messages in them.
[01:27:03] <grundler> hrm "Close Tab" action worked but clicking on the 'x' in tab would not.
[01:27:55] <cmasone> grundler: yeah, chrome linux dev channel has that bug
[01:27:58] <cmasone> it's aggravating
[01:28:04] <quiche> grundler: known issue
[01:28:26] <grundler> 'k
[01:29:22] <grundler> ta :)
[01:30:19] <grundler> btw, I've not paid attention to build tools before...but are we using gold for linking everything on x86?
[01:31:21] <grundler> I'm looking at 10+ bugs opened in Oct 2010 for "Toolchain" team about that and I suspect those could be closed as "fixed" (or converted to "ARCH=ARM" bugs)
[01:31:57] <grundler> http://code.google.com/p/chromium-os/issues/list?q=label:gold
[01:32:37] <grundler> anyway...done with current round of bug scrubbing.
[01:37:10] *** JasonO has quit IRC
[01:52:37] <crosbot> tree became 'Tree is open'
[01:57:53] *** Styx has quit IRC
[02:02:03] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot_master" on "x86 generic PFQ" from de707ffe700218687c830f83a0df49ad0b3ff716: Ben Chan <benchan at chromium dot org>, Chris Masone <cmasone at chromium dot org>, Gaurav Shah <gauravsh at chromium dot org>, J. Richard Barnette <jrbarnette at chromium dot org>, Katie Roberts-Hoffman <katierh at chromium dot org>, Zdenek Behan <zbehan at chromium dot org>, mcgrathr at chromium dot org <mcgrathr at chromium dot org@fcba33aa-ac0c-11dd-b9e7-8d5594d729c2
[02:02:49] <crosbot> tree became 'sheriff is looking into: Tree is closed (Automatic: "cbuildbot_master" on "x86 generic PFQ" from de707ffe700218687c830f83a0df49ad0b3ff716: Ben Chan <benchan at chromium dot org>, Chris Masone <cmasone at chromium dot org>, Gaurav Shah <gauravsh at chromium dot org>, J. Richard Barnette <jrbarnette at chromium dot org>, Katie Roberts-Hoffman <katierh at chromium dot org>, Zdenek Behan <zbehan at chromium dot org>, mcgrathr at chromium dot org <mcgrathr at chromium dot org@fcba33aa-ac
[02:04:08] <cmasone> that's me
[02:04:47] <saintlou> cmasone: we were just getting to that conclusion
[02:04:54] <saintlou> you wanna fix or revert?
[02:04:59] <cmasone> saintlou: already reverted
[02:05:07] <saintlou> awesome :-)
[02:05:09] <saintlou> thanks cmasone
[02:06:23] <crosbot> tree became 'Tree is open and expected to cycle green ([cmasone:6260] reverted)'
[02:08:46] <cmasone> i'm glad that interns are checking in with a closed tree :-/
[02:09:47] <saintlou> yes that is nice of them, they are just trying to help
[02:09:51] <saintlou> :-(
[02:10:45] <davidjames> doesn't gerrit warn on that now?
[02:11:06] *** deshantm has quit IRC
[02:11:36] <cmasone> davidjames: I'd heard that, but clearly there's some bug
[02:23:57] <nirnimesh_> On ToT chrome, anyone know why I see a blank login iframe when using "--webui-login --webui-gaia-login"?
[02:24:51] *** lipsinV2 has joined #chromium-os
[02:26:02] <nirnimesh_> And I see this in the chrome logs: "#### Authenticator.onMessage: unknown message + origin!?", source: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html?hl=en-US (135)
[02:32:57] *** wfrichar has quit IRC
[02:35:57] *** Sarten-X has quit IRC
[02:44:33] *** petermayo has left #chromium-os
[02:45:05] *** zmedico has joined #chromium-os
[02:48:05] *** sadrul has joined #chromium-os
[02:48:05] *** ChanServ sets mode: +v sadrul
[02:49:53] <davidjames> saintlou: Google Storage is acting up right now, tegra2 pfq is about to fail I think
[02:50:33] <saintlou> davidjames: ok thanks. I also code a few 500 on coderview.chromium.org...
[02:50:34] <davidjames> Wow, amazing, despite all these failures it might actually go through
[02:50:47] <davidjames> success!
[02:51:05] <saintlou> cool!
[02:51:05] <zbehan> passing despite failures, sounds like a bug :)
[02:51:41] <davidjames> zbehan: Well, it retried enough times that it succeeded, but there's a service problem if it's failing this often
[02:52:25] <zbehan> davidjames: yeah, not opposed, just thought the statement was funny, something like "winning despite losing"
[02:56:14] *** thieule has quit IRC
[03:01:36] *** Sarten-X has joined #chromium-os
[03:09:23] *** ihf has joined #chromium-os
[03:10:46] <crosbot> tree became 'Tree is open (sheriff is off-duty)'
[03:10:53] * saintlou is off-duty
[03:15:56] *** D|sT has quit IRC
[03:28:53] *** Daxvex has joined #chromium-os
[03:30:06] *** GTAXL has joined #chromium-os
[03:31:33] *** Keybuk has quit IRC
[03:35:01] *** saintlou has quit IRC
[03:35:04] *** jhorwich_ has quit IRC
[03:41:23] <crosbot> tree became '?'
[03:41:39] <crosbot> tree became 'Tree is open (sheriff is off-duty)'
[03:46:10] <crosbot> tree became '?'
[03:46:25] <crosbot> tree became 'Tree is open (sheriff is off-duty)'
[03:47:04] *** sakoman has quit IRC
[03:47:36] *** nirnimesh_ has quit IRC
[03:53:00] *** deshantm has joined #chromium-os
[03:57:51] <crosbot> tree became '?'
[03:58:06] <crosbot> tree became 'Tree is open (sheriff is off-duty)'
[03:58:22] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "x86 pineview full" from b6767ff3b6eb621c8a49c0ad8539094a506be7ff: Ben Chan <benchan at chromium dot org>, Chris Masone <cmasone at chromium dot org>, scherkus at chromium dot org <scherkus at chromium dot org@4ff67af0-8c30-449e-8e8b-ad334ec8d88c>)'
[04:02:53] <crosbot> tree became '?'
[04:03:08] <crosbot> tree became 'Tree is open (Flaky infrastructure, repo sync on x86-pineview-full took 2h30)'
[04:07:24] <crosbot> tree became '?'
[04:11:55] <crosbot> tree became 'Tree is open (Flaky infrastructure, repo sync on x86-pineview-full took 2h30)'
[04:16:10] *** D|sT has joined #chromium-os
[04:16:41] <crosbot> tree became '?'
[04:17:16] *** jrbarnette has quit IRC
[04:29:43] <crosbot> tree became 'Tree is open (Flaky infrastructure, repo sync on x86-pineview-full took 2h30)'
[04:30:23] *** sakoman has joined #chromium-os
[04:33:55] *** sadrul has quit IRC
[04:53:44] <cmasone> zbehan: ping
[04:54:12] <crosbot> tree became '?'
[04:56:08] *** D|sT has quit IRC
[04:57:06] *** ihf has quit IRC
[04:58:44] <crosbot> tree became 'Tree is open (Flaky infrastructure, repo sync on x86-pineview-full took 2h30)'
[05:00:40] *** cmasone has quit IRC
[05:36:17] *** D|sT has joined #chromium-os
[05:37:11] *** oc80z has quit IRC
[05:37:11] *** oc80z has joined #chromium-os
[05:38:28] *** sadrul has joined #chromium-os
[05:38:29] *** ChanServ sets mode: +v sadrul
[05:52:06] <crosbot> tree became 'Tree is open (Flaky git -> http://crbug.com/90675)'
[06:09:55] *** sadrul has quit IRC
[06:11:15] *** Daxvex has quit IRC
[06:14:15] *** sadrul has joined #chromium-os
[06:14:15] *** ChanServ sets mode: +v sadrul
[06:15:45] *** Daxvex has joined #chromium-os
[06:26:18] *** Daxvex has quit IRC
[06:28:54] *** deshantm has quit IRC
[07:01:19] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "x86-alex-binary" from c402b1d5a908e6cd29fd209a13beb4c1495ee7c7: akalin at chromium dot org <akalin at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>, sreeram at chromium dot org <sreeram at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>)'
[07:14:00] *** vmil86 has joined #chromium-os
[07:15:06] *** arun has quit IRC
[07:45:04] *** phh has quit IRC
[07:46:36] *** yoshiki has quit IRC
[07:51:55] *** snickersnack has joined #chromium-os
[07:56:04] *** sergiu has quit IRC
[08:13:05] *** snickersnack has quit IRC
[08:22:02] *** jochen__ has quit IRC
[08:22:07] *** jochen__ has joined #chromium-os
[08:22:07] *** ChanServ sets mode: +v jochen__
[08:23:57] <zbehan> cmasone: late pong
[08:30:14] <crosbot> tree became 'Tree is open (Flaky test on alex-binary, next run was green)'
[08:46:47] *** sadrul has quit IRC
[08:53:23] *** Styx has joined #chromium-os
[09:11:38] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "tegra2_arthur-binary" from ce2b8c6253bfa63311e117670764a0787f78288a: _third_party_ at chromium dot org)'
[09:15:59] *** nirnimesh_ has joined #chromium-os
[09:17:31] *** jujugre has joined #chromium-os
[09:33:00] *** sadrul has joined #chromium-os
[09:35:11] *** ScriptRipper has quit IRC
[09:35:34] *** ScriptRipper has joined #chromium-os
[09:42:29] *** ScriptRipper has quit IRC
[09:42:29] *** ScriptRipper has joined #chromium-os
[09:57:46] *** oc80z has quit IRC
[09:58:27] *** oc80z has joined #chromium-os
[10:45:20] *** patcito has quit IRC
[11:40:59] *** superm1 has quit IRC
[11:41:04] *** superm1 has joined #chromium-os
[11:41:04] *** superm1 has joined #chromium-os
[12:40:49] *** Flameeyes has quit IRC
[13:13:35] *** Flameeyes has joined #chromium-os
[13:14:59] *** Flameeyes has joined #chromium-os
[13:15:28] *** Flameeyes has quit IRC
[13:16:25] *** Flameeyes has joined #chromium-os
[14:15:42] *** oc80z has quit IRC
[14:15:42] *** oc80z has joined #chromium-os
[14:16:22] *** GodoPPL|Away has quit IRC
[14:16:27] *** GodoPPL|Away has joined #chromium-os
[14:16:31] *** dots has quit IRC
[14:16:37] *** dots has joined #chromium-os
[14:16:43] *** reveman has quit IRC
[14:18:49] *** reveman has joined #chromium-os
[14:38:06] *** SoulRaven has joined #chromium-os
[14:55:44] *** rharrison_chrome has joined #chromium-os
[14:55:44] *** ChanServ sets mode: +v rharrison_chrome
[15:07:24] <crosbot> tree became 'Tree is open (sjg: tegra2_arthur-binary failed but has since succeeded)'
[15:27:46] <reveman> east coast sheriff reporting for duty
[16:08:20] *** SoulRaven has quit IRC
[16:15:27] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "tegra2_arthur-binary" from 63f0fd4c1921e1fb669f40b0abbfe7620b92f351: _third_party_ at chromium dot org)'
[16:23:54] <crosbot> tree became 'Tree is closed (reveman investigating "tegra2_arthur-binary" from 63f0fd4c1921e1fb669f40b0abbfe7620b92f351: _third_party_ at chromium dot org)'
[16:26:13] *** sadrul has quit IRC
[16:33:38] <reveman> tegra2_arthur-binary failing to build chromeos-u-boot
[16:33:40] <reveman> chromeos-u-boot-0.0.1-r172: cmd_sf.c:97: error: unused variable 'end'
[16:38:32] <reveman> I suspect one of sjg's commits caused this
[16:41:50] *** sadrul has joined #chromium-os
[16:41:50] *** ChanServ sets mode: +v sadrul
[16:44:12] *** jaharkes has quit IRC
[16:46:09] <reveman> 0dd46030619520657ebdf2da7a1896b7e97943cb: Add 'sf update' command to do smart SPI flash update
[16:46:16] <reveman> is the problem
[16:47:45] *** ScriptRipper has quit IRC
[16:50:39] *** GodoPPL|Away is now known as GodoPPL
[16:51:32] *** jaharkes has joined #chromium-os
[16:52:14] *** wfrichar has joined #chromium-os
[16:52:15] *** ChanServ sets mode: +v wfrichar
[16:53:09] <ellyjones> revert! :D
[16:56:26] *** SoulRaven has joined #chromium-os
[16:56:37] *** SoulRaven has joined #chromium-os
[16:56:45] *** SoulRaven has joined #chromium-os
[16:56:55] *** SoulRaven has joined #chromium-os
[16:56:56] *** SoulRaven has quit IRC
[16:57:04] *** SoulRaven has joined #chromium-os
[16:57:12] *** SoulRaven has joined #chromium-os
[16:57:27] *** SoulRaven has joined #chromium-os
[16:57:32] *** SoulRaven has quit IRC
[16:57:41] *** SoulRaven has joined #chromium-os
[16:58:03] <reveman> I'm about to submit a fix..
[16:58:12] <ellyjones> woo
[17:13:34] <crosbot> tree became 'Tree is open ("tegra2_arthur-binary" fix submitted -> crosbug.com/19385)'
[17:16:08] <crosbot> tree became 'Tree is open (u-boot fix submitted -> crosbug.com/19385)'
[17:19:12] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "tegra2 seaboard full" from 11f9ac2b85ebece59eba16ae4f7681b53c03755c: bsalomon at google dot com <bsalomon at google dot com@2bbb7eff-a529-9590-31e7-b0007b416f81>)'
[17:24:20] <crosbot> tree became 'Tree is open (u-boot fix submitted -> crosbug.com/19385)'
[17:28:03] *** jrbarnette has joined #chromium-os
[17:28:26] <crosbot> tree became 'Tree is open (tegra2_arthur-binary, tegra2_kaen-binary, tegra2 seaboard full: u-boot fix submitted -> crosbug.com/19385 should cycle green)'
[17:32:23] *** jrbarnette_ has joined #chromium-os
[17:36:26] *** jrbarnette has quit IRC
[17:36:26] *** jrbarnette_ is now known as jrbarnette
[17:37:52] *** jaharkes has quit IRC
[17:47:53] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "tegra2_seaboard-tangent-binary" from cbcdc40ba3c71c2a01ae29fae593bff8f097b4c7: _third_party_ at chromium dot org, jochen at chromium dot org <jochen at chromium dot org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>, pfeldman at chromium dot org <pfeldman at chromium dot org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>, rkc at chromium dot org <rkc at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>, tfarina at chromium dot org <tfarina at chromium dot org@
[17:53:11] *** ScriptRipper has joined #chromium-os
[17:53:28] *** ScriptRipper has quit IRC
[17:53:29] *** ScriptRipper has joined #chromium-os
[17:54:43] *** D|sT has quit IRC
[17:57:40] <reveman> hm, u-boot fix doesn't seem to be included in builds. how is the chromeos-u-boot ebuild updated to include this fix?
[17:58:11] <ellyjones> are those builds that started before your fix?
[17:58:22] <crosbot> tree became 'Tree is closed ("tegra2_seaboard-tangent-binary" -> crosbug.com/19385)'
[17:59:06] <reveman> ellyjones: it looks like they started after
[17:59:13] <ellyjones> hm, sadnessa
[17:59:37] <marcheu> reveman: doesn't the chromeos bot uprev the ebuild?
[18:01:01] *** petermayo has joined #chromium-os
[18:01:01] *** ChanServ sets mode: +v petermayo
[18:02:09] <reveman> marcheu: that's what I thought was supposed to happen but it looks like the latest builds are using the same ebuild version as before my fix
[18:02:21] *** jujugre has left #chromium-os
[18:03:32] <marcheu> if you make a dumb change to 9999 it'll uprev? ;)
[18:05:12] *** saintlou has joined #chromium-os
[18:05:12] *** ChanServ sets mode: +v saintlou
[18:07:46] * saintlou is back on duty as sheriff
[18:07:54] *** D|sT has joined #chromium-os
[18:08:14] <saintlou> varunjain, reveman: let me know when you leave for lunch
[18:08:29] <saintlou> or signoff at the end of day.
[18:09:26] <saintlou> varunjain, reveman: I have a meeting from 11.30 to 1.30 (Pacific), might be shorter, but can you please cover? Thanks :-)
[18:09:57] *** yoshiki has joined #chromium-os
[18:10:38] <reveman> saintlou: sure, no prob. tree is currently closed as I'm trying to figure out why the chromeos-u-boot ebuild didn't get uprev from my u-boot fix
[18:11:44] <saintlou> ok reveman thanks. If the issue is specific to seaboard-tangent then petermayo or kliegs-the-trooper might be able to help too
[18:13:31] <reveman> saintlou: affects all tegra2 builds
[18:13:58] <saintlou> reveman: ok thanks
[18:15:10] *** jaharkes has joined #chromium-os
[18:16:46] *** jrbarnette_ has joined #chromium-os
[18:17:10] *** jrbarnette has quit IRC
[18:17:16] *** jrbarnette has joined #chromium-os
[18:21:37] *** jrbarnette_ has quit IRC
[18:21:49] *** jaharkes has quit IRC
[18:23:00] *** jaharkes has joined #chromium-os
[18:25:35] <reveman> saintlou: looks like the u-boot ebuild has correct rev now
[18:25:58] <saintlou> cool reveman should we wait for a green cycle?
[18:27:54] <reveman> saintlou:  I think we can open tree. just need keep an eye on the tegra2 builds
[18:27:56] <saintlou> reveman: have you determined that that from looking at the logs of new builds?
[18:28:14] <saintlou> reveman: ok go ahead, please do the honors :-)
[18:29:19] <crosbot> tree became 'Tree is open (tegra2_arthur-binary, tegra2_kaen-binary, tegra2 seaboard full, tegra2_seaboard-tangent-binary: u-boot fix submitted -> crosbug.com/19385 should cycle green)'
[18:34:57] <reveman> saintlou: fyi, grabbing lunch
[18:35:14] <saintlou> reveman: have a good one :-) thanks for the update and working on the tree
[18:36:00] <davidjames> saintlou: x86 alex canary is about to fail, see http://crosbug.com/19386 and http://crosbug.com/19387 which I just filed
[18:37:12] <saintlou> davidjames: It seems that the second one could be transient, not sure what to do about the first one.
[18:37:36] <davidjames> saintlou: They're both transient
[18:37:57] <saintlou> davidjames: even better :-) :-(
[18:38:06] <saintlou> thanks for the heads up
[18:41:35] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "x86-alex-binary" from cbcdc40ba3c71c2a01ae29fae593bff8f097b4c7: _third_party_ at chromium dot org, rkc at chromium dot org <rkc at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>)'
[18:43:15] *** jhorwich_ has joined #chromium-os
[18:43:15] *** ChanServ sets mode: +v jhorwich_
[18:44:24] <crosbot> tree became 'Tree is open (transient failures crosbug.com/19386 and crosbug.com/19387 for x86-alex-binary, tegra2_* u-boot fix submitted -> crosbug.com/19385 should cycle green) )'
[18:44:39] <crosbot> tree became 'Tree is open (transient failures crosbug.com/19386 and crosbug.com/19387 for x86-alex-binary, tegra2_* u-boot fix submitted -> crosbug.com/19385 should cycle green)'
[18:50:00] <davidjames> saintlou: That's alex binary, it's got a different failure than alex canary
[18:50:31] <saintlou> davidjames: ok sorry got confused. What is the failur eif you have it handy?
[18:50:34] <davidjames> saintlou: In that case looks like the failure is unit tests
[18:50:53] <davidjames> saintlou: update_engine fails unit tests, maybe a recent change
[18:50:57] <saintlou> seems that we should close the tree then
[18:51:02] <saintlou> until we know for sure
[18:51:18] <davidjames> saintlou: +1
[18:51:49] <crosbot> tree became 'Tree is closed => Investigating: "cbuildbot" on "x86-alex-binary" from cbcdc40ba3c71c2a01ae29fae593bff8f097b4c7: _third_party_ at chromium dot org, rkc at chromium dot org <rkc at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>'
[18:52:06] <saintlou> thanks for catching this davidjames
[18:52:29] <davidjames> saintlou: np, glad to help :)
[18:55:46] <saintlou> adlr: can you give us a hand here
[18:56:08] * adlr looks
[18:56:25] <saintlou> adlr: davidjames is saying that from experience: that often changes to other packages break the unit tests
[18:56:31] <saintlou> thanks adlr
[18:56:33] <adlr> is there a log i can see?
[18:56:37] <adlr> if you have a link handy
[18:56:55] *** dkrahn has left #chromium-os
[18:57:15] <davidjames> adlr: http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex-binary/builds/961/steps/Test/logs/stdio
[19:05:40] <adlr> looking
[19:06:27] <adlr> hm, for some reason, we were unable to start our mini http server
[19:06:42] <adlr> maybe something was using the port? let me check which port it uses
[19:06:45] <davidjames> adlr: It's happening reliably on multiple builders, same issue on stumpy builder
[19:07:05] <adlr> okay
[19:07:20] <adlr> I don't think there was any updater change recently, tho
[19:07:29] <adlr> so there is likely nothing to revert :(
[19:07:36] <davidjames> adlr: Yeah I think it's an unrelated change that broke the tests
[19:07:52] <adlr> i suspect it's a bot change that is somehow preventing our http server from running, but let me get more info
[19:08:38] <adlr> it uses port 8088
[19:08:40] <saintlou> thanks adlr
[19:09:28] <adlr> can anyone ssh into the bots and run the tests manually?
[19:09:29] <davidjames> adlr: It was working at 7:53am this morning and broken at 8:09am this morning
[19:09:33] <adlr> hm
[19:10:31] <kliegs> adlr: still need someone to login to bots?
[19:10:36] <adlr> yes
[19:10:42] <kliegs> adlr: which bot?
[19:10:50] <adlr> x86-alex-binary?
[19:11:49] <davidjames> adlr: It's running tests right now so you can see it run :)
[19:12:19] <adlr> well, if someone can ssh in, and has the failure reproducing, we can see why it's failing
[19:12:22] <adlr> i can already see the log
[19:13:03] *** arun_ has quit IRC
[19:13:12] <kliegs> adlr: sorry - you'd think I'd remember the passwords by now instead of looking up each time.  on alex-binary now
[19:13:31] <kliegs> what do you need me to check?
[19:13:38] <davidjames> kliegs: Might not be good to run the tests while it's actively running
[19:13:45] <davidjames> kliegs: As you'll steal the port :)
[19:13:51] *** arun has joined #chromium-os
[19:14:14] <kliegs> davidjames: true
[19:14:20] <adlr> the unittest is showing "Connecting to 127.0.0.1:8088... failed: Connection refused."
[19:15:30] <adlr> hm, it spins up the test http server
[19:15:38] <adlr> then waits up to ~100ms for it to be ready
[19:15:45] <adlr> maybe the bots are very loaded?
[19:15:56] <davidjames> adlr: They are very loaded
[19:15:58] <kliegs> tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN
[19:16:04] <kliegs> there is a process listening on that port at least
[19:16:09] <adlr> can you find the process?
[19:16:10] <kliegs>  10:09:13 up 34 min,  1 user,  load average: 17.71, 23.07, 15.56
[19:16:49] <davidjames> adlr: Unit tests run with idle priority so you can't depend on any specific timing
[19:16:50] <kliegs> adlr: I can't - the process exited already
[19:17:55] <davidjames> adlr: I'll suggest reopening the tree if it's just flakiness, but can we get rid of anything that's timing dependent?
[19:18:10] <adlr> well, we can increase the timeout to infinity, but that seems bad
[19:18:23] <adlr> maybe to 20 seconds?
[19:19:07] <kliegs> is this just all the tests running in parallel causing this load?
[19:19:41] <davidjames> kliegs: Yeah, plus the unit tests are nice'd and ionice'd so they let VM tests take priority, it runs a lot faster that way
[19:20:10] <kliegs> ok.  showing that alex failed in update_engine test just now
[19:20:20] <davidjames> adlr: 20 secs might work, but yeah there's no guarantee it'll run in 20 seconds either
[19:20:32] <adlr> davidjames: how's 20 minutes then?
[19:20:45] <davidjames> adlr: Yeah that should definitely work :)
[19:21:06] <adlr> the issue is we want to spin up a test http server, and validate the actual fetching code (w/ libcurl) works
[19:21:25] <adlr> i'm open to suggestion on the timeout
[19:21:52] <adlr> but, it would be nice to not hang forever if there's a bug that prevents the test_http_server from starting
[19:22:05] <davidjames> adlr: When the VM tests run they hog the disk for a while so you may be queued for even as long as 10-20 minutes
[19:22:17] <adlr> wow
[19:22:21] *** rharrison_chrom1 has joined #chromium-os
[19:22:35] <davidjames> adlr: I timed it and it takes ~20 minutes for unit tests to start running before it runs a single test
[19:23:00] <adlr> okay, i'll add exponential backoff, up to about 30 minutes then
[19:23:01] <davidjames> adlr: (That's doing some dependency analysis, but yeah that stuff usually happens in ~1 minute)
[19:23:36] <davidjames> adlr: Yeah that would be great
[19:23:44] *** rharrison_chrome has quit IRC
[19:23:44] <adlr> for now, i say open the tree
[19:23:49] *** lipsinV2 has quit IRC
[19:23:54] <davidjames> saintlou: ^
[19:23:59] <adlr> since the timeout is currently 100ms, it seems likely that's the issue
[19:24:03] <saintlou> OK will do
[19:24:48] <crosbot> tree became 'Tree is open (adlr still investigating x86-alex-binary failure)'
[19:25:47] <davidjames> saintlou: FYI looks like it's a flaky failure, we try twice and I guess it usually works one of those two times, that's what happened on stumpy
[19:26:05] <saintlou> ok davidjames
[19:26:13] *** jrbarnette_ has joined #chromium-os
[19:26:19] <davidjames> saintlou: BTW any idea what's happening on zgb binary? That's been red for a long time
[19:26:31] *** rharrison_chrom1 has quit IRC
[19:26:49] <saintlou> davidjames: it was like that yesterday already, AFAIK
[19:26:51] <davidjames> saintlou: Since Wednesday (build 900) actually
[19:27:06] <crosbot> tree became 'Tree is open (x86-alex-binary was flaky)'
[19:27:18] <saintlou> yes let me check with previous sheriffs
[19:27:37] <davidjames> saintlou: thanks!
[19:29:51] *** jrbarnette has quit IRC
[19:30:00] <reveman> davidjames, saintlou: afaik zgb binary -> crosbug.com/19298
[19:31:11] *** jrbarnette_ has quit IRC
[19:31:18] <davidjames> reveman, saintlou: Can we get that assigned to someone / increased in priority? Since it's reliably failing
[19:34:24] *** thieule has joined #chromium-os
[19:34:24] *** ChanServ sets mode: +v thieule
[19:34:26] <reveman> davidjames: yes, something needs to be done about that problem. previous sheriffs reported it as coming and going but it seems to be reliably failing now
[19:37:24] *** D|sT has quit IRC
[19:43:15] *** rharrison_chrome has joined #chromium-os
[19:43:16] *** ChanServ sets mode: +v rharrison_chrome
[19:44:58] *** deshantm has joined #chromium-os
[19:47:14] *** jrbarnette has joined #chromium-os
[19:47:15] *** ChanServ sets mode: +v jrbarnette
[19:47:43] *** oc80z has quit IRC
[19:48:58] *** oc80z has joined #chromium-os
[19:51:20] *** GTAXL has quit IRC
[19:52:56] <saintlou> davidjames: that bug is already at P0 it can get much higher than that.
[19:53:29] *** Keybuk has joined #chromium-os
[19:53:30] *** ChanServ sets mode: +v Keybuk
[19:53:58] <davidjames> saintlou: Yes looks like reveman raised it :)
[19:55:14] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "arm generic full" from 11f9ac2b85ebece59eba16ae4f7681b53c03755c: bsalomon at google dot com <bsalomon at google dot com@2bbb7eff-a529-9590-31e7-b0007b416f81>)'
[19:55:28] <adlr> (testing my timeout change, btw)
[19:56:23] <davidjames> saintlou: That arm-generic fail might be crosbug.com/19387
[19:57:53] <saintlou> davidjames: seems like we need to raise the priority of that one too
[19:58:06] <davidjames> saintlou: Yeah, that one is a flaky fail
[19:58:27] <saintlou> davidjames: so we should reopen, right?
[19:58:31] <davidjames> saintlou: Yup
[19:59:04] <crosbot> tree became 'Tree is open (crosbug.com/19387 need to be addressed)'
[20:04:58] <adlr> update_engine timeout change is in
[20:08:43] <saintlou> thanks adlr !!
[20:14:46] *** D|sT has joined #chromium-os
[20:21:33] *** chocobo__ has joined #chromium-os
[20:21:42] *** ChanServ sets mode: +v chocobo__
[20:27:10] <jhorwich_> adlr: Just noticed an oddity the change to http_fetcher_unittest. The code (as it was written before) will sleep even if the call succeeds. Probably not a big deal in the old case (sleep was 10ms), but if we get a successful one after the timeout has backed off to "minutes" (32 min?), we'll slow things down
[20:27:45] *** Styx has quit IRC
[20:30:09] <jhorwich_> aldr: the old code (and the new) also seem to have a small flaw in that if it succeeds in the last iteration (rc=0) it'll still be seen as a failure of sorts (started_ = false and a logged error)
[20:36:14] <nirnimesh> I'm looking at crosbug.com/19298. who knows about cros_au_test_harness?
[20:39:04] *** sakoman_ has joined #chromium-os
[20:39:10] *** xiyuan_ has joined #chromium-os
[20:39:10] *** ChanServ sets mode: +v xiyuan_
[20:39:25] *** awong_ has joined #chromium-os
[20:39:29] *** sadrul1 has joined #chromium-os
[20:39:29] *** ChanServ sets mode: +v sadrul1
[20:39:30] *** patcito has joined #chromium-os
[20:40:51] * saintlou is stepping away, leaving sheriffing in hands of capable east coast collagues
[20:41:25] *** Calchan_ has joined #chromium-os
[20:45:23] *** arun has quit IRC
[20:45:24] *** sadrul has quit IRC
[20:45:25] *** sakoman has quit IRC
[20:45:26] *** McMAGIC--Copy has quit IRC
[20:45:26] *** elly has quit IRC
[20:45:26] *** cooled has quit IRC
[20:45:26] *** radhermit has quit IRC
[20:45:28] *** xiyuan has quit IRC
[20:45:29] *** awong has quit IRC
[20:45:29] *** Calchan has quit IRC
[20:46:44] *** radhermit has joined #chromium-os
[20:47:31] <sosa> nirnimesh:  Don and I are taking a look at the bot
[20:47:47] <sosa> it looks like cros_sdk i.e. enter_chroot is just hanging after build_packages on just that machine
[20:48:01] <sosa> we're going to go ahead and attempt to clobber the build and hopefully fix the issue
[20:48:30] <sosa> since it's only happening (and consistently) on one builder it seems like there's something up with either the configuration of that builder or its build environment
[20:50:04] <nirnimesh> sg. let me know if you need me to look more into the bug.
[20:51:54] <sosa> thanks :)
[20:53:33] *** arun has joined #chromium-os
[21:02:38] *** oc80z has quit IRC
[21:02:39] *** oc80z has joined #chromium-os
[21:08:52] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "x86-zgb-binary" from None: )'
[21:12:12] <crosbot> tree became 'Tree is open (ZGB builder hating, needed clobber but ran into sync bug and died)'
[21:13:08] <sosa> haters gonna hate
[21:23:13] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "x86-alex-binary" from fcede9452da8e2a6112770ee7b80fad53283c37f: Gaurav Shah <gauravsh at chromium dot org>, avi at chromium dot org <avi at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>, marja at chromium dot org <marja at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>)'
[21:29:06] <crosbot> tree became 'Tree is closed (investigating "x86-alex-binary" from fcede9452da8e2a6112770ee7b80fad53283c37f: Gaurav Shah <gauravsh at chromium dot org>, avi at chromium dot org <avi at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>, marja at chromium dot org <marja at chromium dot org@0039d316-1c4b-4281-b951-d872f2087c98>)'
[21:32:59] <reveman> davidjames: is the current x86-alex-binary failure the same flaky problem we've seen earlier today?
[21:33:55] <davidjames> reveman: Yes
[21:34:07] <davidjames> reveman: I think adlr fixed it but maybe adlr's fix hasn't been picked up yet by that bot
[21:35:02] <davidjames> reveman: Yeah, adlr's fix got pushed by PFQ at 11:24 and alex binary started at 10:50am
[21:35:15] <crosbot> tree became 'Tree is open (x86-alex-binary was flaky)'
[21:37:04] <reveman> davidjames: ok, thanks
[21:40:27] *** vmil86 has quit IRC
[21:47:37] *** rochberg has joined #chromium-os
[21:47:37] *** ChanServ sets mode: +v rochberg
[21:48:24] <sosa> has syncing gotten significantly more error prone?
[21:48:41] <sosa> i've clobbered zgb binary  twice and it has failed while syncing from scratch twice
[21:57:37] <saintlou> sosa: from what I hear from other sheriffs and tropper yes syncing issues are pretty common
[21:57:46] *** rbyers_ has joined #chromium-os
[22:00:49] <reveman> looks like x86-zgb-binary failed on sync again
[22:00:52] *** rbyers___ has quit IRC
[22:02:50] *** saintlou has quit IRC
[22:15:23] <crosbot> tree became 'Tree is closed (Automatic: "cbuildbot" on "x86-mario canary" from None: )'
[22:22:57] <sosa> yeah...i'm on the bot now and doing sync manually until i get it synced
[22:23:04] <sosa> clobbering this thing is never gonna work
[22:24:13] <zbehan> isn't it related to the issues that googlecode had?
[22:26:23] <crosbot> tree became 'Tree is open ("x86-mario canary" was flaky. fixed pushed by PFQ at 11:24)'
[22:36:39] *** saintlou has joined #chromium-os
[22:36:45] *** ChanServ sets mode: +v saintlou
[22:48:12] *** GTAXL has joined #chromium-os
[22:55:54] *** katier has left #chromium-os
[23:04:31] *** katier has joined #chromium-os
[23:04:31] *** ChanServ sets mode: +v katier
[23:12:54] <nirnimesh> when building chrome from source, do the breakpad symbols get saved somewhere?
[23:18:08] *** JasonO has joined #chromium-os
[23:32:23] *** cros_ has joined #chromium-os
[23:37:27] *** Ruetobas has quit IRC
[23:44:25] *** Calchan_ is now known as Calchan
[23:44:46] *** petermayo has quit IRC
[23:44:51] *** sergiu has joined #chromium-os
[23:44:51] *** sergiu has joined #chromium-os
[23:45:17] *** Ruetobas has joined #chromium-os
[23:47:05] <sergiu> hey, I have the following problem: Did a repo start, made 3 commits, uploaded them for review. Got LGTM for review 1 and 3, have to make some changes to CL 2. Is there a way in which I can reupload only the second CL? repo upload would reupload all of them, which will make me lose the LGTM's...
[23:47:37] <sergiu> for CL 1 and 3 (not review 1 and 3)
[23:48:12] <sergiu> One option was doing a cherry pick and specifing the change-id, wondering if there's anything more simple
[23:49:40] *** JasonO_ has joined #chromium-os
[23:51:02] *** JasonO has quit IRC
[23:51:18] *** JasonO_ is now known as JasonO
[23:56:31] *** cros_ has quit IRC





top