[elbe-devel] [PATCH 1/2] Add optional mirror for host architecture

Fri Jun 5 13:10:50 CEST 2020

Hello Torben,

On 05/06/2020 09:41, Köry Maincent wrote:
> Hello Torben,
> 
> On 04/06/2020 18:13, Torben Hohn wrote:
>> On Thu, Jun 04, 2020 at 10:54:47AM +0200, Köry Maincent wrote:
>>> Hello Torben,
>>>
>>> Thanks for the review.
>>
>> No Problem, sorry for being so slow, but this pylint patchqueue creates
>> conflicts all over the place and needs to go in quickly.
> 
> No problem, pylint seems an important patch to add, good luck with it.
> 
>>
>>>
>>>
>>> On 04/06/2020 09:14, Torben Hohn wrote:
>>>> On Wed, May 20, 2020 at 11:53:03AM +0200, Kory Maincent wrote:
>>>>> If the primary mirror does not contain the packages for both target 
>>>>> and host
>>>>> architecture the build_host_sysroot process fails.
>>>>>
>>>>> This patch adds support for describing in the XML configuration an 
>>>>> additional
>>>>> optional mirror containing packages for the host architecture, like:
>>>>> <mirror>
>>>>> <host>http://archive.ubuntu.com/ubuntu bionic main universe</host>
>>>>> </mirror>
>>>>
>>>> Do we have to add "bionic main universe" here ?
>>>> The primary mirrors are only needed for debootstrap.
>>>>
>>>> The other mirror entries support [arch=amd64] and we should be able to
>>>> have all other mirror urls in the urllist without having problems.
>>>> If a mirror only has amd64 packages, it needs to be marked that way.
>>>
>>> If we add [arch=amd64] in the mirror urls of the urllist there will be a
>>> conflict with noauth.
>>
>> Then we have to fix noauth.
>> BTW you could also not use noauth, and add trusted=yes directly into the
>> mirror url. That way it wouldnt stop checking all signatures, and only
>> check the relevant signatures.
> 
> Yes, we could indeed remove the noauth from Elbe code.

Do you prefer to remove noauth or to add a specific architecture 
attribute to the url mirror?

Take account that noauth is also used by:
- pbuilder debootstrap config with no-check-gpg parameter
- pbuilder src package, with allow-unauthenticate parameter pass to 
apt-get source.

> 
> 
>>
>> Is noauth reuired to build ubuntu ?
> 
> Elbe seemed to not have trouble to generate an Ubuntu image without 
> noauth, but when I want to work with pbuilder I got an issue with the 
> keyring:
> 
> W: GPG error: 
> http://127.0.0.1:8080/var/cache/elbe/c26b1694-ff47-4efc-9346-13b3b0811c47/repo 
> bionic InRelease: The following signatures couldn't be verified because 
> the public key is not available: NO_PUBKEY 4A4B63D0DEC56F34
> E: The repository 
> 'http://127.0.0.1:8080/var/cache/elbe/c26b1694-ff47-4efc-9346-13b3b0811c47/repo 
> bionic InRelease' is not signed.
> W: Aborting with an error
> 
> Strangely it is related to the local repository but with noauth the 
> issue disappear.
> 
> 
>> We should make it possible to install the proper ubuntu key then.
>> Maybe we just need to install
>>
>> https://packages.debian.org/buster/ubuntu-keyring
> 
> Yes I did it in my initvm.
> 
>>
>>
>>
>>> apt can not support two separated options like this:
>>> deb [trusted=yes] [arch=armhf] http://...
>>> It need to be written like this:
>>> deb [trusted=yes arch=armhf] http://...
>>>
>>>>
>>>>
>>>>>
>>>>> It also adds the architecture type in the sources.list file to 
>>>>> avoid issues
>>>>> with apt-get update.
>>>>>
>>>>> Signed-off-by: Kory Maincent <kory.maincent at bootlin.com>
>>>>> ---
>>>>>    elbepack/elbexml.py | 40 ++++++++++++++++++++++++++++++----------
>>>>>    elbepack/rfs.py     | 23 +++++++++++++----------
>>>>>    schema/dbsfed.xsd   |  7 +++++++
>>>>>    3 files changed, 50 insertions(+), 20 deletions(-)
>>>>>
>>>>> diff --git a/elbepack/elbexml.py b/elbepack/elbexml.py
>>>>> index fa8158392..ece069706 100644
>>>>> --- a/elbepack/elbexml.py
>>>>> +++ b/elbepack/elbexml.py
>>>>> @@ -132,6 +132,13 @@ class ElbeXML(object):
>>>>>            return mirror.replace("LOCALMACHINE", "10.0.2.2")
>>>>> +    def get_host_mirror(self):
>>>>> +        if self.prj.has("mirror/host"):
>>>>> +            m = self.prj.node("mirror")
>>>>> +            mirror = m.text("host").strip() + "\n"
>>>>> +
>>>>> +        return mirror
>>>>
>>>> errm... this fails when <host> does not exist.
>>>> lets integrate this in get_primary_mirror()
>>>> and have a proper fallback.
>>>>
>>>>
>>>>
>>>>
>>>>> +
>>>>>        def get_primary_mirror(self, cdrompath, initvm=True):
>>>>>            if self.prj.has("mirror/primary_host"):
>>>>>                m = self.prj.node("mirror")
>>>>> @@ -146,7 +153,7 @@ class ElbeXML(object):
>>>>>            return replace_localmachine(mirror, initvm)
>>>>>        # XXX: maybe add cdrom path param ?
>>>>> -    def create_apt_sources_list(self, build_sources=False, 
>>>>> initvm=True):
>>>>> +    def create_apt_sources_list(self, build_sources=False, 
>>>>> initvm=True, arch="default"):
>>>>>            if self.prj is None:
>>>>>                return "# No Project"
>>>>> @@ -155,24 +162,37 @@ class ElbeXML(object):
>>>>>            noauth = ""
>>>>>            if self.prj.has("noauth"):
>>>>> -            noauth = "[trusted=yes] "
>>>>> +            noauth = "trusted=yes "
>>>>> +
>>>>> +        if arch == "default":
>>>>> +            arch = self.text("project/buildimage/arch", key="arch")
>>>>> +
>>>>> +        arch_prj = self.text("project/buildimage/arch", key="arch")
>>>>>            mirror = ""
>>>>> +        option = "[" + noauth + " arch=" + arch +"] "
>>>>>            if self.prj.has("mirror/primary_host"):
>>>>> -            mirror += "deb " + noauth + self.get_primary_mirror(None)
>>>>> -            mirror += " " + self.prj.text("suite") + " main\n"
>>>>> +            if (arch != arch_prj) and self.prj.has("mirror/host"):
>>>>> +                mirror += "deb " + option + self.get_host_mirror()
>>>>> +            else:
>>>>> +                mirror += "deb " + option + 
>>>>> self.get_primary_mirror(None)
>>>>> +                mirror += " " + self.prj.text("suite") + " main\n"
>>>>> +
>>>>
>>>> i find this logic too complex.
>>>> We really want the host and target suite to be identical, or we would
>>>> end up with different gcc versions.
>>>>
>>>> I also think its too complex to decide, whether we are building a host
>>>> sysroot, based on architectures.
>>>>
>>>> Please add a bool parameter "hostsdk" to get_primary_mirror() which 
>>>> will
>>>> make it look for <host> and fallback in the case its not there.
>>>>
>>>> it just needs to be passed down from build_host_sysroot() via
>>>> BuildEnv.__init__() and BuildEnv.debootstrap()
>>>>
>>>> this will simplify this patch a lot.
>>>
>>>
>>> Ok.
>>>
>>>
>>>
>>> -- 
>>> Köry Maincent, Bootlin
>>> Embedded Linux and kernel engineering
>>> https://bootlin.com
>>
> 

-- 
Köry Maincent, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com