Switch to DuckDuckGo Search
   April 8, 2020
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30

Toggle Join/Part | bottom
[00:08:38] *** nalind <nalind!~nalin@209.6.216.123> has quit IRC (Quit: bye)
[00:18:41] *** misuto <misuto!~misuto@45.15.16.198> has quit IRC (Ping timeout: 265 seconds)
[00:23:33] *** misuto <misuto!~misuto@45.15.16.198> has joined #fedora-coreos
[00:34:48] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has joined #fedora-coreos
[01:06:11] *** bgilbert <bgilbert!~bgilbert@cpe-98-14-226-113.nyc.res.rr.com> has joined #fedora-coreos
[01:09:25] *** yuvalt <yuvalt!~yturgema@93-173-229-182.bb.netvision.net.il> has quit IRC (Quit: Leaving.)
[01:10:39] *** bgilbert <bgilbert!~bgilbert@cpe-98-14-226-113.nyc.res.rr.com> has quit IRC (Ping timeout: 258 seconds)
[01:14:31] *** bgilbert <bgilbert!~bgilbert@cpe-98-14-226-113.nyc.res.rr.com> has joined #fedora-coreos
[01:20:43] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has quit IRC (Ping timeout: 265 seconds)
[01:21:28] *** bexelbie <bexelbie!~bexelbie@fedora/bexelbie> has quit IRC (Quit: Leaving)
[01:32:19] *** bgilbert <bgilbert!~bgilbert@cpe-98-14-226-113.nyc.res.rr.com> has quit IRC (Ping timeout: 265 seconds)
[01:36:31] *** nordic_devops <nordic_devops!~balthyr@p200300D1670C70FC4968A51AB9E3F3FC.dip0.t-ipconnect.de> has joined #fedora-coreos
[02:05:27] *** paragan <paragan!~paragan@unaffiliated/paragan> has joined #fedora-coreos
[02:07:53] *** DanyC <DanyC!~DanyC@cpc72331-sotn15-2-0-cust18.15-1.cable.virginm.net> has quit IRC (Remote host closed the connection)
[02:11:03] *** gustavold <gustavold!~gustavold@189.110.165.51> has quit IRC (Quit: Leaving.)
[02:17:39] *** nordic_devops <nordic_devops!~balthyr@p200300D1670C70FC4968A51AB9E3F3FC.dip0.t-ipconnect.de> has quit IRC (Remote host closed the connection)
[02:41:19] *** mheon <mheon!~mheon@pool-71-117-146-79.prvdri.fios.verizon.net> has quit IRC (Ping timeout: 260 seconds)
[03:14:28] *** bgilbert <bgilbert!~bgilbert@cpe-98-14-226-113.nyc.res.rr.com> has joined #fedora-coreos
[03:25:18] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has joined #fedora-coreos
[04:00:29] *** bgilbert <bgilbert!~bgilbert@cpe-98-14-226-113.nyc.res.rr.com> has quit IRC (Ping timeout: 250 seconds)
[04:28:36] *** johnsonshi <johnsonshi!~johnsonsh@52.156.152.164> has joined #fedora-coreos
[04:32:20] *** johnsonshi <johnsonshi!~johnsonsh@52.156.152.164> has quit IRC (Client Quit)
[04:32:36] *** johnsonshi <johnsonshi!~johnsonsh@52.156.152.164> has joined #fedora-coreos
[04:38:13] *** johnsonshi <johnsonshi!~johnsonsh@52.156.152.164> has quit IRC (Quit: Leaving)
[04:48:10] *** rvd <rvd!~rdave@45.124.48.105> has joined #fedora-coreos
[05:43:03] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has quit IRC (Remote host closed the connection)
[05:43:40] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has joined #fedora-coreos
[05:47:54] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has quit IRC (Ping timeout: 240 seconds)
[06:02:38] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has joined #fedora-coreos
[06:20:14] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has quit IRC (Remote host closed the connection)
[06:20:52] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has joined #fedora-coreos
[06:23:25] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has quit IRC (Remote host closed the connection)
[06:23:39] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has joined #fedora-coreos
[07:34:37] *** suanand <suanand!~suanand@182.68.101.185> has joined #fedora-coreos
[08:12:11] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has quit IRC (Remote host closed the connection)
[08:17:50] *** tmartin <tmartin!~TMA68@aorleans-657-1-19-154.w90-19.abo.wanadoo.fr> has joined #fedora-coreos
[08:30:00] *** gtema <gtema!~gtema@p200300F97F07F918F14B2C99A9CEF004.dip0.t-ipconnect.de> has joined #fedora-coreos
[08:33:59] *** RaphGro <RaphGro!~raphgro@fedora/raphgro> has joined #fedora-coreos
[08:36:49] *** fos <fos!~Thunderbi@ipbcc31214.dynamic.kabel-deutschland.de> has joined #fedora-coreos
[08:38:13] *** gtema <gtema!~gtema@p200300F97F07F918F14B2C99A9CEF004.dip0.t-ipconnect.de> has quit IRC (Remote host closed the connection)
[08:39:28] *** tmartin <tmartin!~TMA68@aorleans-657-1-19-154.w90-19.abo.wanadoo.fr> has quit IRC (Ping timeout: 256 seconds)
[08:40:05] *** gtema <gtema!~gtema@p200300F97F07F918F14B2C99A9CEF004.dip0.t-ipconnect.de> has joined #fedora-coreos
[08:45:03] *** gtema <gtema!~gtema@p200300F97F07F918F14B2C99A9CEF004.dip0.t-ipconnect.de> has quit IRC (Remote host closed the connection)
[08:45:23] *** gtema <gtema!~gtema@p200300F97F07F918F14B2C99A9CEF004.dip0.t-ipconnect.de> has joined #fedora-coreos
[09:00:10] *** fldu1 <fldu1!~fldu1@165.225.73.26> has joined #fedora-coreos
[09:00:36] *** fldu1 <fldu1!~fldu1@165.225.73.26> has left #fedora-coreos
[09:44:53] *** gtema <gtema!~gtema@p200300F97F07F918F14B2C99A9CEF004.dip0.t-ipconnect.de> has quit IRC (Remote host closed the connection)
[09:58:28] *** gtema <gtema!~gtema@p200300F97F07F918F14B2C99A9CEF004.dip0.t-ipconnect.de> has joined #fedora-coreos
[10:04:45] *** gtema <gtema!~gtema@p200300F97F07F918F14B2C99A9CEF004.dip0.t-ipconnect.de> has quit IRC ()
[10:05:59] *** pej <pej!~pej@eth0.kube-worker003.gv2.exoscale.net> has joined #fedora-coreos
[10:07:14] *** gtema <gtema!~gtema@p200300F97F07F918ACC15E3B41680819.dip0.t-ipconnect.de> has joined #fedora-coreos
[10:12:49] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has joined #fedora-coreos
[10:15:28] *** DanyC <DanyC!~DanyC@cpc72331-sotn15-2-0-cust18.15-1.cable.virginm.net> has joined #fedora-coreos
[10:17:32] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has quit IRC (Ping timeout: 265 seconds)
[10:22:32] *** jdoss <jdoss!sid287091@gateway/web/irccloud.com/x-uuwrnwxzwucsugop> has quit IRC (Read error: Connection reset by peer)
[10:22:43] *** jdoss <jdoss!sid287091@gateway/web/irccloud.com/x-rxeauzvbdvzxhekg> has joined #fedora-coreos
[10:44:32] *** rvd <rvd!~rdave@45.124.48.105> has quit IRC (Remote host closed the connection)
[11:02:43] *** harrymichal <harrymichal!~harrymich@88.83.234.226> has joined #fedora-coreos
[11:24:53] *** harrymichal <harrymichal!~harrymich@88.83.234.226> has quit IRC (Remote host closed the connection)
[11:25:34] *** chiefgoat <chiefgoat!~chiefgoat@80-162-53-70-cable.dk.customer.tdc.net> has quit IRC (Quit: ZNC 1.7.5 - https://znc.in)
[11:26:08] *** chiefgoat <chiefgoat!~chiefgoat@80-162-53-70-cable.dk.customer.tdc.net> has joined #fedora-coreos
[11:33:41] *** narispo <narispo!~narispo@2a01:e0a:2a2:1590:9665:2dff:fea4:9486> has quit IRC (Ping timeout: 272 seconds)
[12:15:40] *** harrymichal <harrymichal!~harrymich@88.83.234.226> has joined #fedora-coreos
[12:28:04] *** harrymichal <harrymichal!~harrymich@88.83.234.226> has quit IRC (Remote host closed the connection)
[13:55:00] *** DanyC <DanyC!~DanyC@cpc72331-sotn15-2-0-cust18.15-1.cable.virginm.net> has quit IRC (Remote host closed the connection)
[14:02:07] *** DanyC <DanyC!~DanyC@cpc72331-sotn15-2-0-cust18.15-1.cable.virginm.net> has joined #fedora-coreos
[14:21:33] *** mosen <mosen!~mosen@119-18-3-119.771203.syd.nbn.aussiebb.net> has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
[14:23:46] *** preethi_ <preethi_!~preethit@2600:1700:7e0:6e20:4580:fab9:8879:cf3> has joined #fedora-coreos
[14:39:45] *** mosen <mosen!~mosen@119-18-3-119.771203.syd.nbn.aussiebb.net> has joined #fedora-coreos
[14:40:15] *** mosen <mosen!~mosen@119-18-3-119.771203.syd.nbn.aussiebb.net> has quit IRC (Client Quit)
[14:47:24] *** gtema <gtema!~gtema@p200300F97F07F918ACC15E3B41680819.dip0.t-ipconnect.de> has quit IRC (Ping timeout: 265 seconds)
[14:49:03] *** gtema <gtema!~gtema@p5B072113.dip0.t-ipconnect.de> has joined #fedora-coreos
[14:57:19] *** tmartin <tmartin!~TMA68@aorleans-657-1-19-154.w90-19.abo.wanadoo.fr> has joined #fedora-coreos
[15:01:16] *** fldu1 <fldu1!6881c26d@104.129.194.109> has joined #fedora-coreos
[15:05:29] <fldu1> Hello, I'm looking for fcct templates. I quickly searched on https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/ ut I was not able to find them. Has someone some hints for me ?
[15:06:57] <jlebon> fldu1: check out https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/ and following sections
[15:07:00] *** mheon <mheon!~mheon@pool-71-117-146-79.prvdri.fios.verizon.net> has joined #fedora-coreos
[15:07:03] *** mheon <mheon!~mheon@pool-71-117-146-79.prvdri.fios.verizon.net> has quit IRC (Client Quit)
[15:07:18] <jlebon> dustymabe: ouch re. lockfile troubles
[15:08:05] *** mosen <mosen!~mosen@119-18-3-119.771203.syd.nbn.aussiebb.net> has joined #fedora-coreos
[15:08:05] <jlebon> ok, so i think i'm going to prioritize https://github.com/coreos/rpm-ostree/pull/1858 over getting the next-devel fully working
[15:09:57] *** mheon <mheon!~mheon@pool-71-117-146-79.prvdri.fios.verizon.net> has joined #fedora-coreos
[15:10:37] *** mosen <mosen!~mosen@119-18-3-119.771203.syd.nbn.aussiebb.net> has quit IRC (Client Quit)
[15:10:56] <kaeso[m]> dustymabe, jlebon: moin! I think there is one more package (brotli) to untag from the pool, how does that work?
[15:11:20] <kaeso[m]> (that affects both testing and stable)
[15:12:24] <fldu1> jlebon: thanks a lot for your answer
[15:12:40] <jlebon> ahh yup i see it. let me try untagging it
[15:13:17] *** mosen <mosen!~mosen@119-18-3-119.771203.syd.nbn.aussiebb.net> has joined #fedora-coreos
[15:13:51] *** mosen <mosen!~mosen@119-18-3-119.771203.syd.nbn.aussiebb.net> has quit IRC (Client Quit)
[15:14:27] <jlebon> done. dustymabe: let's do another build?
[15:15:37] <jlebon> kaeso[m]: or were you going to take over?
[15:18:05] <kaeso[m]> jlebon: yes, I am, I was just waiting to check/discuss the brotli bit
[15:18:47] <dustymabe> hey jlebon kaeso[m]
[15:18:55] <jlebon> actually, let's hold on a bit more for a sec
[15:19:07] <jlebon> dustymabe: hi!
[15:19:20] <dustymabe> i really tried to get the builds out last night :)
[15:19:22] <dustymabe> oh well
[15:19:34] <jlebon> so, i don't quite understand why libbrotli stayed in there when libcurl was back to fc31
[15:20:16] <dustymabe> jlebon: neither do I, but I think we should update the yum repo for stable and testing to point at a version of the distrepo from before the f32 packages got tagged
[15:20:28] <dustymabe> that will let us get past this mess
[15:20:39] <dustymabe> WDYT?
[15:21:05] <jlebon> dustymabe: if that works, yeah. i think that's the most foolproof approach
[15:21:15] *** gtema <gtema!~gtema@p5B072113.dip0.t-ipconnect.de> has left #fedora-coreos
[15:21:19] <dustymabe> one sefc
[15:21:28] <jlebon> i did untag libbrotli, but we're essentially racing against the tagger tagging it back in
[15:23:12] *** abai <abai!~abai@CPE00fc8d78a413-CM00fc8d78a410.cpe.net.fido.ca> has joined #fedora-coreos
[15:29:51] *** nalind <nalind!~nalin@209.6.216.123> has joined #fedora-coreos
[15:31:42] <dustymabe> jlebon: https://github.com/coreos/fedora-coreos-config/pull/336
[15:31:45] <dustymabe> kaeso[m]: ^^
[15:37:25] <jlebon> ahh ok, brotli mystery solved
[15:37:58] <jlebon> libcurl f31 requires libbrotlidec.so.1()(64bit), which is provided by both brotli f31 and libbrotli f32, which were both in the pool
[15:38:06] <jlebon> so libsolv just picked the highest
[15:38:55] <kaeso[m]> jlebon: but it didn't for libcurl f32
[15:38:57] <jlebon> i knew we would eventually get to needing strict lockfiles, but totally did not foresee triggering that yesterday after merging the f32 patch
[15:39:15] <jlebon> kaeso[m]: this is after dustymabe tagged libcurl f32 out
[15:39:23] <jlebon> before that, it did pull in libcurl too
[15:40:13] <jlebon> libcurl-minimal to be more precise
[15:41:08] *** mosen <mosen!~mosen@119-18-3-119.771203.syd.nbn.aussiebb.net> has joined #fedora-coreos
[15:41:18] <kaeso[m]> jlebon: ah, I see the f31 was -minimal while f32 was -full, so brotli got dropped and re-added at the wrong version
[15:41:50] <kaeso[m]> ehm, viceversa
[15:41:51] *** gustavold <gustavold!~gustavold@189.110.165.51> has joined #fedora-coreos
[15:42:42] <jlebon> right yeah :)
[15:44:28] <kaeso[m]> does the repo change also propagate to client-side rpm-ostree installs/composes?
[15:45:13] *** crobinso <crobinso!~crobinso@redhat/crobinso> has joined #fedora-coreos
[15:45:37] *** suanand_ <suanand_!~suanand@2401:4900:4455:c377:896a:2938:61fd:b32d> has joined #fedora-coreos
[15:45:51] <jlebon> kaeso[m]: no it only affects the server side. client repos are just regular files like any other installed by RPMs
[15:45:56] <jlebon> in this case fedora-repos
[15:46:01] <dustymabe> jlebon: kaeso[m]:
[15:46:09] <dustymabe> https://github.com/coreos/fedora-coreos-config/pull/337
[15:47:49] *** suanand <suanand!~suanand@182.68.101.185> has quit IRC (Ping timeout: 265 seconds)
[15:49:06] *** paragan <paragan!~paragan@unaffiliated/paragan> has quit IRC (Ping timeout: 265 seconds)
[15:50:51] <jlebon> ok i filed https://github.com/coreos/fedora-coreos-tracker/issues/454 to have something to link to from now on
[15:52:06] <dustymabe> +1
[15:53:30] *** gtema <gtema!~gtema@p5b072030.dip0.t-ipconnect.de> has joined #fedora-coreos
[15:53:56] <jlebon> added links to what happened in stable and testing yesterday
[15:56:04] *** gtema <gtema!~gtema@p5b072030.dip0.t-ipconnect.de> has quit IRC (Remote host closed the connection)
[16:02:46] *** suanand__ <suanand__!~suanand@182.68.101.185> has joined #fedora-coreos
[16:05:22] *** suanand_ <suanand_!~suanand@2401:4900:4455:c377:896a:2938:61fd:b32d> has quit IRC (Ping timeout: 260 seconds)
[16:06:35] *** paragan <paragan!~paragan@unaffiliated/paragan> has joined #fedora-coreos
[16:11:12] *** paragan <paragan!~paragan@unaffiliated/paragan> has quit IRC (Ping timeout: 260 seconds)
[16:19:26] <dustymabe> kaeso[m]: ok you should be able to start the pipeline runs now
[16:20:20] <kaeso[m]> dustymabe: ack, I'll start with `stable` first now, reusing the https://github.com/coreos/fedora-coreos-streams/issues/79 ticket
[16:20:56] <dustymabe> k
[16:21:27] <dustymabe> I think you already know this, but I definitely recommend you run them in parallel :)
[16:26:52] <kaeso[m]> dustymabe: eh yes, the non-parallel bottleneck is me clicking stuff :)
[16:27:21] <dustymabe> +1
[16:28:08] *** fldu1 <fldu1!6881c26d@104.129.194.109> has quit IRC (Remote host closed the connection)
[16:28:32] <kaeso[m]> from early jenkins logs, this one has brotli from f31 and no other unexpected f32 bits
[16:30:20] *** suanand__ <suanand__!~suanand@182.68.101.185> has quit IRC (Remote host closed the connection)
[16:31:11] *** pej <pej!~pej@eth0.kube-worker003.gv2.exoscale.net> has quit IRC (Quit: Ciao)
[16:36:33] <dustymabe> kaeso[m]: +1
[16:47:39] *** yuvalt <yuvalt!~yturgema@93-173-229-182.bb.netvision.net.il> has joined #fedora-coreos
[16:55:32] *** yuvalt <yuvalt!~yturgema@93-173-229-182.bb.netvision.net.il> has quit IRC (Quit: Leaving.)
[17:04:20] *** paragan <paragan!~paragan@unaffiliated/paragan> has joined #fedora-coreos
[17:18:37] *** cosmicsound <cosmicsound!vcm@unaffiliated/cosmicsound> has joined #fedora-coreos
[17:18:52] <cosmicsound> good day
[17:19:31] *** kelleyk <kelleyk!~kelleyk@47-208-193-232.trckcmtc01.res.dyn.suddenlink.net> has joined #fedora-coreos
[17:22:35] *** bersace <bersace!~bersace@mantienne.cae.li> has quit IRC (Ping timeout: 250 seconds)
[17:44:21] <dustymabe> jlebon: kaeso[m]: the upgrade test starting failing consistently on [testing-devel] - 31.20200317.20.2 and everything moving forward
[17:44:45] <jlebon> dustymabe: you already went through the log bisect?
[17:44:59] <dustymabe> jlebon: yep, the blue ocean view worked pretty well
[17:45:03] <jlebon> dustymabe: +1
[17:45:27] <jlebon> diff:
[17:45:31] <jlebon> Upgraded:
[17:45:32] <jlebon> fuse-overlayfs 0.7.5-2.fc31.x86_64 → 0.7.6-2.fc31.x86_64
[17:45:32] <jlebon> git-core 2.24.1-1.fc31.x86_64 → 2.25.1-2.fc31.x86_64
[17:45:32] <jlebon> libpcap 14:1.9.1-1.fc31.x86_64 → 14:1.9.1-2.fc31.x86_64
[17:45:33] <jlebon> pcre 8.43-3.fc31.x86_64 → 8.44-1.fc31.x86_64
[17:45:58] <jlebon> though we should also diff cosa and fcos
[17:46:03] <jlebon> fcos config*
[17:47:38] <dustymabe> 0317.20.1, 0317.20.2, and 0317.20.3 I don't see any package diffs when I look at the builds browser
[17:48:06] <jlebon> bahhh, was looking at 20.0
[17:48:24] <jlebon> right, so it's likely an fcos or cosa change
[17:48:28] <dustymabe> right
[17:48:33] <dustymabe> which one do you want to look at ?
[17:48:44] <dustymabe> i'll take the other
[17:49:35] <jlebon> fcos
[17:49:40] <dustymabe> ok i'll take cosa
[17:50:01] <jlebon> one commit:
[17:50:17] <jlebon> https://github.com/coreos/fedora-coreos-config/pull/308
[17:51:02] <dustymabe> hmm
[17:51:06] * dustymabe still looks at cosa
[17:51:07] <jlebon> that... shouldn't affect it though
[17:51:41] <jlebon> we were still including dhcp-client in testing-devel back then. it was just to fix bodhi-updates which had the new dracut
[17:53:13] <jlebon> dustymabe: what kola-aws build number was it btw?
[17:54:38] <dustymabe> jlebon: here is the set of commits to COSA
[17:54:43] <dustymabe> https://paste.centos.org/view/56a52fcb
[17:55:10] <dustymabe> jlebon: https://jenkins-fedora-coreos.apps.ci.centos.org/blue/organizations/jenkins/fedora-coreos%2Ffedora-coreos-fedora-coreos-pipeline-kola-aws/activity/
[17:55:19] <dustymabe> keep clicking on "show more" at the bottom of the page
[17:55:46] <dustymabe> then start clicking on testing devels around 0317.20.1 - you'll notice the ones before all pass
[17:55:49] <dustymabe> the ones after all fail
[17:56:05] <dustymabe> sorry - I mean 0317.20.2
[17:56:24] <jlebon> ahh so 31.20200317.20.1 is the first one, no?
[17:57:10] <dustymabe> oh actually yeah you are right.
[17:57:20] <dustymabe> looks like 31.20200317.20.1
[17:57:22] <jlebon> ahh kk, let me check fcos again
[17:58:45] <jlebon> one commit again:
[17:58:46] <jlebon> https://github.com/coreos/fedora-coreos-config/pull/307
[18:00:37] <jlebon> which... also should be fine since we don't have root= kargs in FCOS
[18:00:57] <jlebon> (but still, i did just also sanity check that growfs does run in the latest FCOS build i have locally here)
[18:01:45] <dustymabe> jlebon: cosa changes: https://paste.centos.org/view/654000a6
[18:03:00] <jlebon> hmm, so just https://github.com/coreos/coreos-assembler/pull/1253 looks like
[18:04:05] <jlebon> which seems very unlikely
[18:04:10] <jlebon> ouhhh so one confounding factor here is that the test did sometimes pass. so e.g. 20.0 could also have had the change but it just didn't flake
[18:04:14] <dustymabe> yeah but I wonder if the kola test are running with the exact same version of COSA as the build was run with
[18:04:34] <dustymabe> well I checked a lot of previous ones and that's where it started to fail consistently
[18:05:12] <dustymabe> see if you can find others before that on the testing-devel chain that were failing
[18:05:13] <jlebon> how about we just rerun the tests for the last e.g. 5 amis before and including 20.1 ?
[18:05:33] <dustymabe> WFM - though note the test is going to run with the latest cosa image
[18:05:38] <dustymabe> right?
[18:05:40] <jlebon> with latest cosa. if it still switches from red to green then we know it's not cosa
[18:05:43] <jlebon> right
[18:05:47] <dustymabe> well actually
[18:05:55] <dustymabe> we already *know* it's not COSA I think
[18:05:56] <jlebon> it's not the cosa *test
[18:06:01] <jlebon> right
[18:06:06] <dustymabe> because the stable builds pass
[18:06:17] <dustymabe> so I can stop looking at COSA I assume
[18:07:33] <jlebon> ok cool, let me start those tests
[18:07:43] <kaeso[m]> I was just checking my canary on `testing` on AWS
[18:08:38] <kaeso[m]> when it was at `31.20200407.2.0`, it correctly managed to update, and staging the commit took ~13s
[18:09:35] <dustymabe> jlebon: I have a new lead
[18:10:01] <jlebon> ok started kola aws tests from ​31.20200314.20.0 to ​31.20200317.20.1
[18:10:17] <kaeso[m]> the exact same version on aws in the pipeline failed due to timeouts
[18:11:12] <jlebon> kaeso[m]: did zincati succeed on first try in your canary?
[18:13:06] <kaeso[m]> jlebon: yes, it tried to stage and finalize that version exactly once, and it worked
[18:13:21] <dustymabe> kaeso[m]: jlebon: https://paste.centos.org/view/68c58cf4
[18:13:22] <jlebon> ohh fun, kola-aws hitting some schema validation failure
[18:13:52] <dustymabe> ostree/rpm-ostree upgrade perhaps?
[18:14:27] <jlebon> yeah might be. though 31.20200317.20.0 did pass
[18:14:27] <dustymabe> we can probably confirm by cross referencing with the bodhi updates kola aws tests
[18:14:55] <dustymabe> right but won't the
[18:15:07] <dustymabe> "problem" start showing up in the "N+1" test ?
[18:15:55] <jlebon> right, definitely possible. i mean, the test does try to test the new update as well as a test, but it's not as realistic as the first old to new update
[18:16:25] <jlebon> it basically tests both N-1 -> N and N -> "hypothetical N+1"
[18:16:52] <jlebon> anyway, worth digging into that diff still
[18:17:18] <jlebon> might be related to the readonly work that happened around that time
[18:17:22] <dustymabe> ok yeah bodhi updates started failing around the same time
[18:17:55] <dustymabe> 31.20200316.94.0 and 31.20200316.94.1 pass
[18:18:07] <dustymabe> 31.20200317.94.0+ fail
[18:18:51] <kaeso[m]> jlebon: I see some kernel xfs message in the middle of the transaction, is that "readonly work" related?
[18:19:19] <jlebon> kaeso[m]: can you pastebin it?
[18:19:44] <dustymabe> and 94.1 is where the new ostree/rpm-ostree are introduced
[18:20:39] *** agates[m] <agates[m]!agatesmatr@gateway/shell/matrix.org/x-fpcrkirfswqdgomi> has joined #fedora-coreos
[18:21:20] <kaeso[m]> jlebon: https://paste.centos.org/view/48c40797
[18:21:57] <jlebon> kaeso[m]: ahhh yup, that's related
[18:22:59] <kaeso[m]> jlebon: ie. a mount() blocking the thread were the dbus-handler is?
[18:23:04] <jlebon> hmm, something like the /sysroot remount on aws holding up the main thread?
[18:23:06] *** RaphGro <RaphGro!~raphgro@fedora/raphgro> has quit IRC (Quit: Please remember your own message. It'll be read as soon as possible.)
[18:23:45] <jlebon> kaeso[m]: possibly...
[18:24:24] <dustymabe> kaeso[m]: is that zincati error from your node that isn't running in the pipeline ?
[18:25:07] <kaeso[m]> dustymabe: no, that's a random pipeline log I had open from this morning
[18:25:13] <dustymabe> ahh +1
[18:25:50] <dustymabe> jlebon: did you say we were running on m4.large ?
[18:26:01] <jlebon> yup
[18:26:05] <dustymabe> i notice it only has 2 vCPUs
[18:26:09] <kaeso[m]> my canary has a small&fast disk (8GiB, NVMe), so that would explain why it has no troubles
[18:26:23] <dustymabe> I wonder if that means it effectively on gets one CPU with out SMT mitigations turned on
[18:26:34] <dustymabe> with our*
[18:26:39] <jlebon> dustymabe: that's correct
[18:27:26] <jlebon> what's that website again for instance types?
[18:27:31] <dustymabe> https://ec2instances.info/
[18:27:52] <jlebon> meanwhile i'm looking into the ostree/rpm-ostree diff there more closely
[18:27:56] <dustymabe> if we re-run the test locally in kola/qemu and provide it only 1 cpu I wonder if we would see the failure
[18:28:11] <dustymabe> jlebon: hate to do this since we're in the heat of chase
[18:28:20] <dustymabe> reminder: community meeting in 2 minutes
[18:28:46] <jlebon> hmm, that table doesn't show the thread breakdown
[18:28:54] <jlebon> yup :)
[18:30:27] <jlebon> darkmuggle davdunc dustymabe jdoss jlebon lorbus miabbott skunkerk
[18:30:27] <jlebon> FCOS community meeting in #fedora-meeting-1
[18:30:27] <jlebon> If you don't want to be pinged remove your name from this file: https://github.com/coreos/fedora-coreos-tracker/blob/master/meeting-people.txt
[18:31:09] <dustymabe> jlebon: you can ignore the Fedora Council ticket. We'll talk about that in the 04/22 meeting
[18:31:39] <jlebon> dustymabe: ack
[18:33:16] *** tmartin <tmartin!~TMA68@aorleans-657-1-19-154.w90-19.abo.wanadoo.fr> has quit IRC (Quit: Leaving)
[18:33:23] *** bgilbert <bgilbert!~bgilbert@cpe-98-14-226-113.nyc.res.rr.com> has joined #fedora-coreos
[18:36:45] *** DanyC <DanyC!~DanyC@cpc72331-sotn15-2-0-cust18.15-1.cable.virginm.net> has quit IRC (Remote host closed the connection)
[18:42:10] <kaeso[m]> dustymabe, jlebon: stable kola-upgrade-aws indeed just failed
[18:44:02] <dustymabe> kaeso[m]: i.e. not as part of a pipeline run?
[18:46:04] <kaeso[m]> dustymabe: nono, the pipeline run for the ongoing stable release
[18:52:00] *** DanyC <DanyC!~DanyC@cpc72331-sotn15-2-0-cust18.15-1.cable.virginm.net> has joined #fedora-coreos
top

   April 8, 2020
< | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30