Number of components per init

[Roman Iten]

Hi,

I wrote a simple run script 'manytimer', based on 'timer':
https://github.com/rite/genode/tree/manytimer
When the number of components exceeds 19, I get a "Quota exceeded!"
warning. It seems that it doesn't matter if I use timer or other
components. It also doesn't matter how much RAM quota I configure per
timer.

- Is this behaviour intended?

- Whose quota exceeds (init, core, ...)?

- Can I resolve the warning by increasing its quota?

- In a scenario with 19 or more components (within one init), is it
still possible to "assign all remaining resources to the last child by
simply specifying an overly large quantum" as described in "Genode
16.05 Foundations" (see Chapter 6.2.2 "Resource quota saturation"). Or
would there be no more slack memory available for init and core
respectively?

Thanks, Roman

[Norman Feske]

Hi Roman,

https://github.com/rite/genode/tree/manytimerWhen the number of
components exceeds 19, I get a "Quota exceeded!" warning. It seems that
it doesn't matter if I use timer or other components. It also doesn't
matter how much RAM quota I configure per timer. - Is this behaviour
intended? - Whose quota exceeds (init, core, ...)? - Can I resolve the
warning by increasing its quota? - In a scenario with 19 or more
components (within one init), is it still possible to "assign all
remaining resources to the last child by simply specifying an overly
large quantum" as described in "Genode 16.05 Foundations" (see Chapter
6.2.2 "Resource quota saturation"). Or would there be no more slack
memory available for init and core respectively?

I encountered the same problem in the Turmvilla scenario. In the
presence of the "overly large quantum", init transfers all remaining
quota from itself to the corresponding child and preserves just a tiny
bit of quota for itself. This preserved quota is needed to accommodate
the few metadata allocations that cannot be easily allocated from a
specific child (i.e., child-thread meta data). If the number of children
becomes too large, this preservation does not suffice. But you can
increase the value as done by the following commit:


https://github.com/nfeske/genode/commit/619ce3ff2c81df8e24b10a675ccc78b83ee30e7f

Cheers
Norman

[Roman Iten]

Hi Norman

Thanks for the insights. I still don't understand why init's(?) quota
exceeds. I get the warning no matter of the presence of the "overly
large quantum". Is this behavior intended?

- How big is the initial quota of init?

- Can I change that quota - or only the preserved slack memory?

- If not - is there another way to prevent the "Quota exceeded!"
warning?

- Does the size of the metadata allocation for a child depends on
whether I'm using a 32 or 64 bit system?

- Is a scenario with 19 or more components within one init considered
'large'?

Thanks,
Roman


On Mit, Okt 5, 2016 at 9:05 , Norman Feske

Post by Norman Feske
Hi Roman,

https://github.com/rite/genode/tree/manytimer. When the number of
components exceeds 19, I get a "Quota exceeded!" warning. It seems that
it doesn't matter if I use timer or other components. It also doesn't
matter how much RAM quota I configure per timer. - Is this behaviour
intended? - Whose quota exceeds (init, core, ...)? - Can I resolve the
warning by increasing its quota? - In a scenario with 19 or more
components (within one init), is it still possible to "assign all
remaining resources to the last child by simply specifying an overly
large quantum" as described in "Genode 16.05 Foundations" (see Chapter
6.2.2 "Resource quota saturation"). Or would there be no more slack
memory available for init and core respectively?

I encountered the same problem in the Turmvilla scenario. In the
presence of the "overly large quantum", init transfers all remaining
quota from itself to the corresponding child and preserves just a tiny
bit of quota for itself. This preserved quota is needed to accommodate
the few metadata allocations that cannot be easily allocated from a
specific child (i.e., child-thread meta data). If the number of children
becomes too large, this preservation does not suffice. But you can
https://github.com/nfeske/genode/commit/619ce3ff2c81df8e24b10a675ccc78b83ee30e7f
Cheers
Norman
--
Dr.-Ing. Norman Feske
Genode Labs
http://www.genode-labs.com · http://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
GeschÀftsfÌhrer: Dr.-Ing. Norman Feske, Christian Helmuth
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
genode-main mailing list
https://lists.sourceforge.net/lists/listinfo/genode-main

[Norman Feske]

Hi Roman,

Post by Roman Iten
- How big is the initial quota of init?

init receives the quota for all the available physical memory from core.
The amount depends on the physical memory of the machine. It is printed
by core when init is started. Look for a message like this:

24 MiB RAM assigned to init

Post by Roman Iten
- Can I change that quota - or only the preserved slack memory?

If you are using Qemu, you may adjust the memory of the virtual machine
by putting the following line (for 256 MiB) in your run script:

append qemu_args " -m 256 "

In your run script, the value is 64 MiB, which leaves merely 24 MiB (on
64 bit) for init.

Post by Roman Iten
- If not - is there another way to prevent the "Quota exceeded!"
warning?

I looked a bit closer at your scenario. Here is what happens:

For each child started by init, init needs to create several
capabilities (e.g., the parent capability presented to the child, or the
capability for the local ROM session for providing the child's binary as
"binary" ROM module). The allocation of those capabilities is performed
via the 'Nova_native_pd::alloc_rpc_cap' RPC function. This function
naturally consumes session quota of the corresponding PD session (init's
PD session). At one point, the initial session quota (that was passed to
core when init's PD session was created) is depleted. In this case, core
prints the diagnostic message and returns an error to the client (init).
Init responds to this error by upgrading the session quota of its PD
session using the preserved slack memory. The session upgrading is
handled at 'base-nova/src/lib/base/rpc_cap_alloc.cc'.

In your case, the message you see is merely a diagnostic message. The
condition is handled properly. In cases where proper error handing of
the 'Out_of_metadata' condition is missing, the message used to be quite
valuable to spot the problem. So we decided to kept it.

Of course, you may opt to suppress the message by adjusting the code in
'base/include/base/allocator_guard.h'.

Post by Roman Iten
- Does the size of the metadata allocation for a child depends on
whether I'm using a 32 or 64 bit system?

Yes. I.e., your scenario produces the message only on 64 bit, not on 32 bit.

Post by Roman Iten
- Is a scenario with 19 or more components within one init considered
'large'?

The static part of the current scenarios (like the ones tested by our
autopilot) typically consists of fewer components. Personally, I only
hit the limit with the Turmvilla scenario.

That said, the limit turned out not to be a problem in your case. The
message is a false-positive warning. The default limit becomes a problem
not before the PD-session upgrade fails. I can trigger the problem with
your run script when configuring Qemu with 64 MiB of memory and starting
76 children. Sorry that my previous email pointed you to the wrong
direction.

Cheers
Norman

[Roman Iten]

Hi Norman

Post by Norman Feske

Post by Roman Iten
- How big is the initial quota of init?

init receives the quota for all the available physical memory from core.
The amount depends on the physical memory of the machine. It is printed
by core when init is started.

Thats what I thought. But I couldn't imagine how init's quota could
possibly exceed on a x86-64 machine with several GiB physical memory :)

Post by Norman Feske
For each child started by init, init needs to create several
capabilities (e.g., the parent capability presented to the child, or the
capability for the local ROM session for providing the child's binary as
"binary" ROM module). The allocation of those capabilities is
performed
via the 'Nova_native_pd::alloc_rpc_cap' RPC function. This function
naturally consumes session quota of the corresponding PD session (init's
PD session). At one point, the initial session quota (that was passed to
core when init's PD session was created) is depleted. In this case, core
prints the diagnostic message and returns an error to the client (init).
Init responds to this error by upgrading the session quota of its PD
session using the preserved slack memory. The session upgrading is
handled at 'base-nova/src/lib/base/rpc_cap_alloc.cc'.
In your case, the message you see is merely a diagnostic message. The
condition is handled properly. In cases where proper error handing of
the 'Out_of_metadata' condition is missing, the message used to be quite
valuable to spot the problem. So we decided to kept it.

How can I distinguish if the condition is handled properly or not? Are
there any preceding or following log messages in either case?

Post by Norman Feske

Post by Roman Iten
- Does the size of the metadata allocation for a child depends on
whether I'm using a 32 or 64 bit system?

Yes. I.e., your scenario produces the message only on 64 bit, not on 32 bit.

Is it worth thinking about calculating the slack memory size based on
Genode::addr_t? Or make the value even configurable?

Post by Norman Feske
That said, the limit turned out not to be a problem in your case. The
message is a false-positive warning. The default limit becomes a problem
not before the PD-session upgrade fails. I can trigger the problem with
your run script when configuring Qemu with 64 MiB of memory and starting
76 children. Sorry that my previous email pointed you to the wrong
direction.

It didn't. I'm trying to improve my understanding of memory
configuration and allocation in Genode. So every hint helps ;)

Thanks,
Roman

[Nobody III]

I've had problems with init failing when a component was configured to use
all remaining available RAM. I've had to alter some of the standard run
scripts to make them work. I don't remember which ones right now, but I
think arora was one of them. Maybe init should have an XML option to set
its reserved RAM quota, or it could instead automatically calculate the
required quota based on the number of components it starts. Also, it would
be nice to have both reserved and maximum quotas for components in init's
configuration.

Post by Roman Iten
Hi Norman
- How big is the initial quota of init?
init receives the quota for all the available physical memory from core.
The amount depends on the physical memory of the machine. It is printed by
core when init is started.
Thats what I thought. But I couldn't imagine how init's quota could
possibly exceed on a x86-64 machine with several GiB physical memory :)
For each child started by init, init needs to create several capabilities
(e.g., the parent capability presented to the child, or the capability for
the local ROM session for providing the child's binary as "binary" ROM
module). The allocation of those capabilities is performed via the
'Nova_native_pd::alloc_rpc_cap' RPC function. This function naturally
consumes session quota of the corresponding PD session (init's PD session).
At one point, the initial session quota (that was passed to core when
init's PD session was created) is depleted. In this case, core prints the
diagnostic message and returns an error to the client (init). Init responds
to this error by upgrading the session quota of its PD session using the
preserved slack memory. The session upgrading is handled at
'base-nova/src/lib/base/rpc_cap_alloc.cc'.
In your case, the message you see is merely a diagnostic message. The
condition is handled properly. In cases where proper error handing of the
'Out_of_metadata' condition is missing, the message used to be quite
valuable to spot the problem. So we decided to kept it.
How can I distinguish if the condition is handled properly or not? Are
there any preceding or following log messages in either case?
- Does the size of the metadata allocation for a child depends on whether
I'm using a 32 or 64 bit system?
Yes. I.e., your scenario produces the message only on 64 bit, not on 32 bit.
Is it worth thinking about calculating the slack memory size based on
Genode::addr_t? Or make the value even configurable?
That said, the limit turned out not to be a problem in your case. The
message is a false-positive warning. The default limit becomes a problem
not before the PD-session upgrade fails. I can trigger the problem with
your run script when configuring Qemu with 64 MiB of memory and starting 76
children. Sorry that my previous email pointed you to the wrong direction.
It didn't. I'm trying to improve my understanding of memory configuration
and allocation in Genode. So every hint helps ;)
Thanks,
Roman
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
genode-main mailing list
https://lists.sourceforge.net/lists/listinfo/genode-main

[Norman Feske]

Hi Ben,

thanks for chiming in!

Post by Nobody III
Maybe init should have an XML option
to set its reserved RAM quota, or it could instead automatically
calculate the required quota based on the number of components it
starts.

The automatic calculation would work if init was solely used in a static
way. But init can actually respond to configuration changes at runtime.
Hence, the number of children may vary over the lifetime of init, which
makes it impossible to calculate the value up-front.

To address the problem in a fundamental way, init should allocate all
child-specific capabilities from the PD session of the respective child.
However, this is not a simple change. E.g., it would require us to pass
the PD session as argument to 'Entrypoint::manage' and to keep the
association of each RPC object with the PD session where its capability
was allocated from. Since this is a rare problem (actually only for init
at this point), I am not convinced to make the API more complicated for
everyone. However, I have no strong opinion yet.

In the shorter term, it would be sensible to make the preservation
configurable via init's configuration instead of hard-coding the value.
I plan to rework init soon and will keep this suggestion in mind.

Post by Nobody III
Also, it would be nice to have both reserved and maximum quotas
for components in init's configuration.

I agree. I will consider this while reworking init.

Cheers
Norman

[Roman Iten]

Hi

Thank you very much for this very good explanation.

Post by Norman Feske
In the shorter term, it would be sensible to make the preservation
configurable via init's configuration instead of hard-coding the value.
I plan to rework init soon and will keep this suggestion in mind.

I agree. Making the preservation configurable makes the memory
reservations more transparent, i.e. everything that requires memory is
visible in init's configuration - well, at least if init is used in a
static way. Or are there still other memory 'pools' one might not be
aware of?

Post by Norman Feske

Post by Nobody III
Also, it would be nice to have both reserved and maximum quotas
for components in init's configuration.

I agree. I will consider this while reworking init.

Great, I like the idea of having both, reserved/soft and a maximum/hard
quota.

One last question: how do I calculate the required memory preservation
for init on nova_x86_64, based on the number of children?

Thanks,
Roman


PS: sorry for screwing up the quoting in my previous mail. Tried another
mail client - I'm back to thunderbird now :)

[Norman Feske]

Hi Roman,

Post by Roman Iten
I agree. Making the preservation configurable makes the memory
reservations more transparent, i.e. everything that requires memory is
visible in init's configuration - well, at least if init is used in a
static way. Or are there still other memory 'pools' one might not be
aware of?

on NOVA, the most important one is the kernel's memory pool, which has a
fixed size that is defined in the kernel's linker script. The linker
script is located in nova/src/hypervisor.ld (look for '_mempool_f').

Another limited resource is core's capability space, in particular the
meta data required to manage the lifetime of capabilities. The details
differ from kernel to kernel. On most base platforms, those information
are kept in statically allocated arrays, which are dimensioned to
accommodate the current scenarios. Core is in a special position because
it has to keep track of all capabilities in the system (capabilities are
allocated via core's PD service). Since the capability space of core is
limited, we should apply Genode's resource-trading concept to
capabilities too. In fact, I plan to implement this idea sometime next
year. Until then, we have to life with the situation that capability
allocations are not properly accounted (which is a potential
denial-of-service issue).

Post by Roman Iten
One last question: how do I calculate the required memory preservation
for init on nova_x86_64, based on the number of children?

I cannot give you a precise formula. My previously reported experiment
where I started over 70 children with the default preservation of 128
KiB suggests that 2 KiB per child should suffice. Make it 16 KiB per
child and you should be really fine. ;-)

Cheers
Norman

[Norman Feske]

Hi Roman,

Post by Roman Iten
How can I distinguish if the condition is handled properly or not? Are
there any preceding or following log messages in either case?

I remember the following situations where the messages were somehow
useful to diagnose a problem:

- If the scenario got stuck with the this being the last message.
This happened in the phase when we introduced the dynamic resource
handling originally (around version 13.11). In the meanwhile, we
have equipped all places where clients may encounter the out-of-
metadata condition with corresponding handlers (i.e., in the form
of the 'base/src/include/base/internal/expanding_*_client.h' classes).
Therefore, I haven't observed this effect for a long time now.

- If a component aborts right after a quota-exceeded message, this is
a strong hint that an exception handler is missing. An example is the
issue recently reported by Denis (and fixed with [1]) where core
missed to reflect an 'Allocator::Out_of_memory' condition as an
'Rm_session::Out_of_metadata' exception. That said, in this case,
the message does not add too much information because the abort
message already prints the exception type, which should suffice
to diagnose the problem.

- If an unusual amount of quota-exceeded messages appears, this hints
at a resource leak or at least an opportunity for optimization. The
initial session quotas and the quota upgrades are dimensioned such
that the quota updates are needed only sporadically. If the log is
flooded by quota-exceeded messages, a superficial number of quota
upgrades is performed, which should better be investigated.

The more I think about it, the better I like the idea to remove the
message. In the log, it gives a wrong impression of presumed problems
where none exists because the quota upgrading is a normal part of life
of Genode components. The diagnostic value added by the messages is
indeed quite low.

[1]
https://github.com/genodelabs/genode/commit/fac69eff48b3d70a9d37ebc5d3ae67465b612150

Post by Roman Iten

Post by Roman Iten
- Does the size of the metadata allocation for a child depends on
whether I'm using a 32 or 64 bit system?
Yes. I.e., your scenario produces the message only on 64 bit, not on 32 bit.

Is it worth thinking about calculating the slack memory size based on
Genode::addr_t? Or make the value even configurable?

Making the value configurable seems to be the most sensible idea. I will
respond to Ben's (Nobody III) posting with an explanation why.

Cheers
Norman