[elbe-devel] [PATCH v2 1/3] init: add <max-cpus> field and make Makefile.mako obey the <mem> field

[Manuel Traut]

On Fri, Sep 28, 2018 at 10:34:18AM +0200, Torben Hohn wrote:
> On machines with many cores, 1GiB of memory is not very much. Especially,
> when g++ is used in a pbuilder with -jauto.
> 
> Add <max-cpus> field, that clamps the number of cpus for the initvm.
> Make it default to 8. Also let the Makefile.mako obey the memory settings.
> 
> Signed-off-by: Torben Hohn <torben.hohn at linutronix.de>
> ---
>  elbepack/init/Makefile.mako    | 13 +++++++++++--
>  elbepack/init/libvirt.xml.mako |  3 ++-
>  elbepack/xmldefaults.py        |  3 ++-
>  schema/dbsfed.xsd              | 11 +++++++++++
>  4 files changed, 26 insertions(+), 4 deletions(-)
> 
> diff --git a/elbepack/init/Makefile.mako b/elbepack/init/Makefile.mako
> index 4c2f323b..6e55056b 100644
> --- a/elbepack/init/Makefile.mako
> +++ b/elbepack/init/Makefile.mako
> @@ -5,8 +5,17 @@
>  ##
>  ## SPDX-License-Identifier: GPL-3.0-or-later
>  ##
> -MEMSIZE?=1024
> -SMP?=`nproc`
> +<%
> +import multiprocessing
> +from elbepack.filesystem import size_to_int
> +
> +cpus = int(prj.text('max-cpus', default=defs, key='max-cpus'))
> +cpus = min(multiprocessing.cpu_count(), cpus)
> +memory = size_to_int(prj.text('mem', default=defs, key='mem')) / 1024 / 1024
> +%>
> +
> +MEMSIZE?=${memory}
> +SMP?=${cpus}
>  INTERPRETER?=${prj.text('interpreter', default=defs, key='interpreter')}

A big adavantage of the Makefile to control the initvm is that it can be
moved easily around and that we can detect the runtime environment during
call-time. This is lost now :(

I'd like to have it implemented directly in the Makefile

The rest of the patch looks OK.

>  # this is a workaround for
> diff --git a/elbepack/init/libvirt.xml.mako b/elbepack/init/libvirt.xml.mako
> index c2b3d077..1260c55f 100644
> --- a/elbepack/init/libvirt.xml.mako
> +++ b/elbepack/init/libvirt.xml.mako
> @@ -19,7 +19,8 @@ from elbepack.filesystem import size_to_int
>  uid = uuid.uuid4()
>  
>  name = cfg['initvm_domain']
> -cpus = multiprocessing.cpu_count()
> +cpus = int(prj.text('max-cpus', default=defs, key='max-cpus'))
> +cpus = min(multiprocessing.cpu_count(), cpus)
>  memory = size_to_int(prj.text('mem', default=defs, key='mem')) / 1024
>  
>  imagetype = prj.text('img', default=defs, key='img')
> diff --git a/elbepack/xmldefaults.py b/elbepack/xmldefaults.py
> index 3a9e8684..dbc93db1 100644
> --- a/elbepack/xmldefaults.py
> +++ b/elbepack/xmldefaults.py
> @@ -142,7 +142,8 @@ archindep_defaults = {
>      "size": "20G",
>      "img": "qcow2",
>      "mem": "1GiB",
> -    "swap-size": "0"
> +    "swap-size": "0",
> +    "max-cpus": "8"
>  }
>  
>  defaults = {"armel": armel_defaults,
> diff --git a/schema/dbsfed.xsd b/schema/dbsfed.xsd
> index eaf707f1..731439db 100644
> --- a/schema/dbsfed.xsd
> +++ b/schema/dbsfed.xsd
> @@ -358,6 +358,17 @@
>            </documentation>
>          </annotation>
>        </element>
> +      <element name="max-cpus" type="integer" minOccurs="0" maxOccurs="1">
> +        <annotation>
> +          <documentation>
> +	    The number of cpus used by the initvm is clamped to this value.
> +	    This shall protect the initvm from running out of memory, when
> +	    running on machines with large numbers of cores.
> +	    When increasing this value, be sure to also increase mem and
> +	    maybe add some swap.
> +          </documentation>
> +        </annotation>
> +      </element>
>        <element name="img" type="rfs:string" minOccurs="0" maxOccurs="1">
>          <annotation>
>            <documentation>
> -- 
> 2.11.0
> 
> 
> _______________________________________________
> elbe-devel mailing list
> elbe-devel at linutronix.de
> https://lists.linutronix.de/mailman/listinfo/elbe-devel