[elbe-devel] [PATCH v2 1/3] init: add <max-cpus> field and make Makefile.mako obey the <mem> field
Torben Hohn
torben.hohn at linutronix.de
Thu Nov 22 10:23:35 CET 2018
On Fri, Nov 16, 2018 at 09:43:52AM +0100, Manuel Traut wrote:
> On Mon, Nov 12, 2018 at 11:59:50AM +0100, Torben Hohn wrote:
> > On Fri, Oct 19, 2018 at 04:26:35PM +0200, Manuel Traut wrote:
> > > On Fri, Sep 28, 2018 at 10:34:18AM +0200, Torben Hohn wrote:
> > > > On machines with many cores, 1GiB of memory is not very much. Especially,
> > > > when g++ is used in a pbuilder with -jauto.
> > > >
> > > > Add <max-cpus> field, that clamps the number of cpus for the initvm.
> > > > Make it default to 8. Also let the Makefile.mako obey the memory settings.
> > > >
> > > > Signed-off-by: Torben Hohn <torben.hohn at linutronix.de>
> > > > ---
> > > > elbepack/init/Makefile.mako | 13 +++++++++++--
> > > > elbepack/init/libvirt.xml.mako | 3 ++-
> > > > elbepack/xmldefaults.py | 3 ++-
> > > > schema/dbsfed.xsd | 11 +++++++++++
> > > > 4 files changed, 26 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/elbepack/init/Makefile.mako b/elbepack/init/Makefile.mako
> > > > index 4c2f323b..6e55056b 100644
> > > > --- a/elbepack/init/Makefile.mako
> > > > +++ b/elbepack/init/Makefile.mako
> > > > @@ -5,8 +5,17 @@
> > > > ##
> > > > ## SPDX-License-Identifier: GPL-3.0-or-later
> > > > ##
> > > > -MEMSIZE?=1024
> > > > -SMP?=`nproc`
> > > > +<%
> > > > +import multiprocessing
> > > > +from elbepack.filesystem import size_to_int
> > > > +
> > > > +cpus = int(prj.text('max-cpus', default=defs, key='max-cpus'))
> > > > +cpus = min(multiprocessing.cpu_count(), cpus)
> > > > +memory = size_to_int(prj.text('mem', default=defs, key='mem')) / 1024 / 1024
> > > > +%>
> > > > +
> > > > +MEMSIZE?=${memory}
> > > > +SMP?=${cpus}
> > > > INTERPRETER?=${prj.text('interpreter', default=defs, key='interpreter')}
> > >
> > > A big adavantage of the Makefile to control the initvm is that it can be
> > > moved easily around and that we can detect the runtime environment during
> > > call-time. This is lost now :(
> >
> > During normal operation, the only usage of this Makefile is to install
> > the initvm.
> >
> > This does not have to move around.
> >
> > Normal starting of the initvm is done via libvirt.
> > The number of CPUs might be increased using virt-manager.
>
> libvirt is nice, but currently no hard dependency. Generating the
> initvm is posible without libvirt. And using it also. 'make run'
> etc. can be still used. And it's sometimes nice, e.g. inside
> containers without an init system. I don't want to drop this.
i am not advicating to drop this.
I am only saying, that this is a case for experienced users, and
these people might as well write:
make run SMP=`nproc`
anyways... i found a way to do the calculation... expect a patch.
>
> And moving an existing initvm from one machine to another, or
> if running inside a VM reconfiguring the number of CPUs, etc.
> is done in real life. So why behave worse than necessary?
>
> Another thing is, that i don't like the heuristics at all.
which heuristics ?
there are no heuristics.
its just a default clamping value...
> But if we need them, we should handle them as transparent as
> possible for a user. A user might look at the Makefile, but probably
> not into the mako template.
>
> For libvirt i don't see another option than calculating it in
> the template. So i need to accept it ;) But not for the Makefile.
>
> Manu
>
>
> > >
> > > I'd like to have it implemented directly in the Makefile
> > >
> > > The rest of the patch looks OK.
> > >
> > > > # this is a workaround for
> > > > diff --git a/elbepack/init/libvirt.xml.mako b/elbepack/init/libvirt.xml.mako
> > > > index c2b3d077..1260c55f 100644
> > > > --- a/elbepack/init/libvirt.xml.mako
> > > > +++ b/elbepack/init/libvirt.xml.mako
> > > > @@ -19,7 +19,8 @@ from elbepack.filesystem import size_to_int
> > > > uid = uuid.uuid4()
> > > >
> > > > name = cfg['initvm_domain']
> > > > -cpus = multiprocessing.cpu_count()
> > > > +cpus = int(prj.text('max-cpus', default=defs, key='max-cpus'))
> > > > +cpus = min(multiprocessing.cpu_count(), cpus)
> > > > memory = size_to_int(prj.text('mem', default=defs, key='mem')) / 1024
> > > >
> > > > imagetype = prj.text('img', default=defs, key='img')
> > > > diff --git a/elbepack/xmldefaults.py b/elbepack/xmldefaults.py
> > > > index 3a9e8684..dbc93db1 100644
> > > > --- a/elbepack/xmldefaults.py
> > > > +++ b/elbepack/xmldefaults.py
> > > > @@ -142,7 +142,8 @@ archindep_defaults = {
> > > > "size": "20G",
> > > > "img": "qcow2",
> > > > "mem": "1GiB",
> > > > - "swap-size": "0"
> > > > + "swap-size": "0",
> > > > + "max-cpus": "8"
> > > > }
> > > >
> > > > defaults = {"armel": armel_defaults,
> > > > diff --git a/schema/dbsfed.xsd b/schema/dbsfed.xsd
> > > > index eaf707f1..731439db 100644
> > > > --- a/schema/dbsfed.xsd
> > > > +++ b/schema/dbsfed.xsd
> > > > @@ -358,6 +358,17 @@
> > > > </documentation>
> > > > </annotation>
> > > > </element>
> > > > + <element name="max-cpus" type="integer" minOccurs="0" maxOccurs="1">
> > > > + <annotation>
> > > > + <documentation>
> > > > + The number of cpus used by the initvm is clamped to this value.
> > > > + This shall protect the initvm from running out of memory, when
> > > > + running on machines with large numbers of cores.
> > > > + When increasing this value, be sure to also increase mem and
> > > > + maybe add some swap.
> > > > + </documentation>
> > > > + </annotation>
> > > > + </element>
> > > > <element name="img" type="rfs:string" minOccurs="0" maxOccurs="1">
> > > > <annotation>
> > > > <documentation>
> > > > --
> > > > 2.11.0
> > > >
> > > >
> > > > _______________________________________________
> > > > elbe-devel mailing list
> > > > elbe-devel at linutronix.de
> > > > https://lists.linutronix.de/mailman/listinfo/elbe-devel
> >
> > --
> > Torben Hohn
> > Linutronix GmbH | Bahnhofstrasse 3 | D-88690 Uhldingen-Mühlhofen
> > Phone: +49 7556 25 999 18; Fax.: +49 7556 25 999 99
> >
> > Hinweise zum Datenschutz finden Sie hier (Informations on data privacy
> > can be found here): https://linutronix.de/kontakt/Datenschutz.php
> >
> > Linutronix GmbH | Firmensitz (Registered Office): Uhldingen-Mühlhofen |
> > Registergericht (Registration Court): Amtsgericht Freiburg i.Br., HRB700
> > 806 | Geschäftsführer (Managing Directors): Heinz Egger, Thomas Gleixner
>
>
--
Torben Hohn
Linutronix GmbH | Bahnhofstrasse 3 | D-88690 Uhldingen-Mühlhofen
Phone: +49 7556 25 999 18; Fax.: +49 7556 25 999 99
Hinweise zum Datenschutz finden Sie hier (Informations on data privacy
can be found here): https://linutronix.de/kontakt/Datenschutz.php
Linutronix GmbH | Firmensitz (Registered Office): Uhldingen-Mühlhofen |
Registergericht (Registration Court): Amtsgericht Freiburg i.Br., HRB700
806 | Geschäftsführer (Managing Directors): Heinz Egger, Thomas Gleixner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.linutronix.de/pipermail/elbe-devel/attachments/20181122/26b2e4bc/attachment.sig>
More information about the elbe-devel
mailing list