Ideas about interactive GUI

[Boris Mulder]

Dear Genode Community,

We have been brewing on the best way to implement an interactive menu interface. We would like to hear your opinions and ideas on this. Currently, only launcher provides this functionality. However, we had the following objectives in mind:

- Decouple the menu state and the functionality. We think the menu should only be responsible for rendering the dialogs and registering input events. Whatever functionality is to be invoked based on this input should be delegated to other components.

- Have a fully customizable menu. The model of what buttons and (nested) submenus exist should be configurable, and should be separated from the layout (such as how a button looks like) and from the current state (which menus are collapsed, etc).

In order to shape this, we have the following ideas:

- The configuration of the menu model can be specified in XML in its config, and be reconfigurable at runtime. Parts of this could be defined by other components as well. Examples of how this could look like is described in the attached xml file.

- Events related to the menu either trigger menu actions (in which case the config dictates the proper response, such as showing or hiding a submenu) or trigger external actions. In the second case, the menu generates a report that for instance states a certain button has been clicked. Again, see the attachment.

- Any (custom) component can read these reports and take certain actions. For example, in case programs are to be started with the menu, we can add an init component that takes its config from another rom module. A subsystem management component containing a more extensive state model would read the events from the menu, change init's config based on this, and monitor the state of started subsystems and perhaps add extra context menus for these subsystems.
Another example is a shutdown button that talks to a component that can write the system state, and which could show a dialog with the options to power off or reset a systems. Perhaps it can try to gracefully stop running programs with disk state (such as virtualbox) by talking with the subsystem manager as explained above.
In any case, the menu should be able to support any functionality by providing a generic interface to components such as the examples above.

- The layout of the menu can be specified in the config as well, perhaps using (a subset of) CSS, or as it is now, with the styles database that contains pictures for each element. (buttons, bars, etc.)

While looking at ways to implement this, we also encountered the following problems:

-As it is currently with launcher, a lot of subsystem functionality is
intertwined with the interactive menu. This makes implementing such a
modification a bit more difficult than replacing the subsystem
management part with a reporting part. One of the issues we face is that
showing and hiding windows of other components is done through the
session_control function in the nitpicker session interface, but this is
limited to the session of the components and its children. Children of
an init component therefore cannot be affected.

One way to fix this is to add a policy in the window manager or nitpicker that allows
the menu or some other component to execute session_control on certain
or all sessions. Implementing session_control functionality in init
itself does not look like a clean solution.

Another issue is that init does not keep track of exited children. Launcher neatly closes the corresponding menu entries by installing a signal when the child exits, but init does not have such functionality now. Perhaps init could be modified such that it generates a report when a child exits? For now, if we would quit a child with another method than through the menu, we have no way of knowing if a child exited.

I'd very much like to hear your comments on these ideas and on the best way to
realize it (perhaps together), and for instance how you think we could resolve the above issues.

[Norman Feske]

Hi Boris,

thanks for sharing your ideas. In fact, they are very much in line
with my current ambitions.

First, given that we have the dynamic init now, I wholeheartedly agree
to relieve the existing interactive application starters (launcher,
CLI monitor) from the technicalities of hosting subsystems. They
should instead merely generate and update init configurations, and
consume init-state reports.

Second, I definitely aim to cultivate the clean separation of the
presentation and the application logic. The existing launcher already
anticipates this direction. The complex rendering of dialogs or visual
gimmicks like fading are implemented in the form of sandboxed
menu_view or nit_fader components. The launcher merely ties those
components together but does not perform any complex rendering or
parsing operations. That said, in contrast to the menu_view, the
current launcher is not meant to stay. As of now, it is more of an
experimentation ground.

Post by Boris Mulder
- The configuration of the menu model can be specified in XML in its
config, and be reconfigurable at runtime. Parts of this could be
defined by other components as well. Examples of how this could look
like is described in the attached xml file.

This is actually very similar to what the existing menu_view already
gives you. It consumes a description of a dialog, and produces reports
about the currently hovered item. It does not implement any program
logic.

As far as I see, your menu differs from the existing menu_view in two
ways:

* It introduces a form of abstraction because its dialog description
talks about semantics of a menu rather than dumb UI elements (frame,
label, button, hbox).

* It handles multiple dialogs (sub menus).

Post by Boris Mulder
- Events related to the menu either trigger menu actions (in which
case the config dictates the proper response, such as showing or
hiding a submenu) or trigger external actions. In the second case, the
menu generates a report that for instance states a certain button has
been clicked. Again, see the attachment.

I'd advise not to use the report/ROM interface to propagate events, but
only state. Even though one (myself included) is sometimes intuitively
tempted to mix up events and states, it calls for trouble.

In contrast to intermediate states, which are uninteresting (because
superseded by more current states) and thereby can be dropped, events
must never be dropped. So reporting a mouse click in the form of a
report is not the right approach. In contrast, reporting the currently
hovered element as a report is fine. For this reason, the existing
launcher intercepts the user input of the menu view to get hold of the
important input events.

This also applies for other operations like hiding/showing windows.
Instead of communicating commands like (hide this, show that) between
components, we should communicate states (the set of hidden windows).

The design of the window manager pretty much adheres to this principle.
E.g., if a window is moved, we do not propagate the event (window has
moved) but the new state (the new window layout). In contrast,
nitpicker's 'session_control' interface does not (see below).

Post by Boris Mulder
- Any (custom) component can read these reports and take certain
actions. For example, in case programs are to be started with the
menu, we can add an init component that takes its config from another
rom module. A subsystem management component containing a more
extensive state model would read the events from the menu, change
init's config based on this, and monitor the state of started
subsystems and perhaps add extra context menus for these subsystems.

I see the same picture. There will be one component (you may call it
subsystem manager) that will actually hold all the important strings,
including the configuration of the dynamic init instance that hosts the
dynamically created subsystems.

This raises the question of where does the responsibility of the menu
end, and where do the responsibilities of the subsystem manager start?
If we need a subsystem manager anyway, couldn't it use the menu_view(s)
directly to implement menus?

The answer may depend on the complexity of the subsystem manager, which
is obviously critical. Does the separation of the menu from the
subsystem manager contribute to significantly lower the TCB of the
subsystem manager?

When following this line of thinking, we might even end up integrating
the window layouter + menu + subsystem manager in a single "desktop"
component.

Post by Boris Mulder
Another example is a shutdown button that talks to a component that
can write the system state, and which could show a dialog with the
options to power off or reset a systems. Perhaps it can try to
gracefully stop running programs with disk state (such as virtualbox)
by talking with the subsystem manager as explained above. In any case,
the menu should be able to support any functionality by providing a
generic interface to components such as the examples above.

These points are also closely related to the "subsystem manager", which
controls the life cycle of subsystems or even the entire system. As the
menu and subsystem manager are so closely related, I would intuitively
tend to integrate both into a single component.

Post by Boris Mulder
-As it is currently with launcher, a lot of subsystem functionality is
intertwined with the interactive menu. This makes implementing such a
modification a bit more difficult than replacing the subsystem
management part with a reporting part. One of the issues we face is that
showing and hiding windows of other components is done through the
session_control function in the nitpicker session interface, but this is
limited to the session of the components and its children. Children of
an init component therefore cannot be affected.

The launcher in its current form is not here to stay. Some parts of its
implementation may be worth reusing (e.g., the fading dialog). But its
role of a runtime environment for subsystems is no more needed. Also the
use of nitpicker's session-control interface should be eliminated.

When I created the launcher, I foresaw a mix of nitpicker-based (like
rdesktop) and wm-based applications. The only way to manage the focus of
applications of both categories was to use a nitpicker mechanism. In the
meanwhile, I reached the conclusion that all regular applications of a
desktop scenario should talk to the wm. Thereby, the input focus,
visibility, and layout of the applications are subjected to the window
layouter's policy. The show/hide toggling of applications via the menu
should result in a hidden-applications report to be consumed by the
layouter.

This leaves the question about the input focus. When using the wm, the
focus is already defined by the window layouter via the 'focus' ROM. The
window layouter, in turn, already consumes a 'focus_request' ROM to
respond to externally triggered focus changes. This would be the right
hook for the menu / subsystem manager.

Post by Boris Mulder
One way to fix this is to add a policy in the window manager or nitpicker that allows
the menu or some other component to execute session_control on certain
or all sessions. Implementing session_control functionality in init
itself does not look like a clean solution.

By following the approach outlined above, nitpicker's session-control
interface becomes a mere mechanism used by the window manager but not by
anyone else.

Post by Boris Mulder
Another issue is that init does not keep track of exited children.
Launcher neatly closes the corresponding menu entries by installing a
signal when the child exits, but init does not have such functionality
now. Perhaps init could be modified such that it generates a report
when a child exits? For now, if we would quit a child with another
method than through the menu, we have no way of knowing if a child
exited.

The state report generated by init already contains a 'state' attribute
for each child. Right now, the attribute is present only if the child is
"incomplete". We could add a value "exited" that would be presented as
soon as the child issued a 'Parent::exit' call. Also the exit code could
be added to the report, if needed. The subsystem manager could then
inspect the state reports for children with this attribute value and
remove them from the init config.

Cheers
Norman

[Boris Mulder]

Hi,

Good to hear that you are also going to do this! For us, there is a
clear need for a GUI system that is both dynamic and simple in use.

Post by Norman Feske
This is actually very similar to what the existing menu_view already
gives you.

This configurablility would indeed be similar to menu_view.
Another main difference is that it does not aim to replace menu_view,
but is an overlapping component that could also handle input, and take
actions, such as collapsing and expanding submenus. It could still use
menu_view, fading_dialog and such parts. So instead it aims to replace
the menu part of launcher, except that it is dynamic and generic.
What we would like is such a menu component that could be used at many
places; not only for a desktop environment, but also inside any
application, or settings menu, or subsystem.

It would be complete with elements such as buttons, submenus and
dialogs, and the notion of clicking on these elements. These events
could somehow be communicated to other elements such that the
corresponding actions are taken.

Post by Norman Feske
I'd advise not to use the report/ROM interface to propagate events, but
only state. Even though one (myself included) is sometimes intuitively
tempted to mix up events and states, it calls for trouble.
[...]

Makes sense, you are right. However, for clicking buttons in this context, it makes sense to communicate events instead of states. Perhaps using another way than reports could be useful here, such as some list of events like in Input, or a packet-stream like functionality.

Post by Norman Feske
This raises the question of where does the responsibility of the menu
end, and where do the responsibilities of the subsystem manager start?
If we need a subsystem manager anyway, couldn't it use the menu_view(s)
directly to implement menus?

We would like to separate the menu with all its buttons from the subsystem state logic.

Whenever we want an application that uses the GUI that does different
things, we would have to import the menu management
code into these applications. This could be done as a library with
functions for creating buttons and menus, or as a separate component
that can be configured and uses some IPC mechanism. In the latter case,
more than one component can talk with the same menu, depending on the
application programmer's needs.
This can be made as complex or simple as is desirable. However now it is
already quite complicated to create only a basic menu as all the
features are very intertwined, which makes the code a bit hard to read
and modify.

Post by Norman Feske
The answer may depend on the complexity of the subsystem manager, which
is obviously critical. Does the separation of the menu from the
subsystem manager contribute to significantly lower the TCB of the
subsystem manager?

It does not necessarily change the TCB, but it is done for the sake of
reusability and code readability/ease of programming. From that
perspective we think it would be nice if these things were separated,
and for instance the menu part could be used in another application as well.

Post by Norman Feske
This leaves the question about the input focus. When using the wm, the
focus is already defined by the window layouter via the 'focus' ROM. The
window layouter, in turn, already consumes a 'focus_request' ROM to
respond to externally triggered focus changes. This would be the right
hook for the menu / subsystem manager.

I've been looking into this mechanism, but so far it's not entirely
clear to me. Do you mean that the subsystem manager could write to this
focus_request report and the layouter would for example show/hide the
appropriate windows automatically? If so, what about the focus_requests
that come from the window manager?
In short: how would that work?

Post by Norman Feske
The state report generated by init already contains a 'state' attribute
for each child. Right now, the attribute is present only if the child is
"incomplete". We could add a value "exited" that would be presented as
soon as the child issued a 'Parent::exit' call. Also the exit code could
be added to the report, if needed. The subsystem manager could then
inspect the state reports for children with this attribute value and
remove them from the init config.

This is exactly what I'm looking for in that case.

Thanks,

[Nobody III]

Given that we have qt5 ported, my plans with implementing GUIs have
generally involved using a qt-based component as the frontend, and using
report_ROM or a custom RPC interface to communicate with the backend. Is
there anything wrong with this approach? I get that it's not enough when
untrusted data (e.g. icons) needs to be displayed, but shouldn't this work
for most situations, provided untrusted data is processed by other
components?

Post by Boris Mulder
Hi,
Good to hear that you are also going to do this! For us, there is a
clear need for a GUI system that is both dynamic and simple in use.
This is actually very similar to what the existing menu_view already
gives you.
This configurablility would indeed be similar to menu_view.
Another main difference is that it does not aim to replace menu_view,
but is an overlapping component that could also handle input, and take
actions, such as collapsing and expanding submenus. It could still use
menu_view, fading_dialog and such parts. So instead it aims to replace
the menu part of launcher, except that it is dynamic and generic.
What we would like is such a menu component that could be used at many
places; not only for a desktop environment, but also inside any
application, or settings menu, or subsystem.
It would be complete with elements such as buttons, submenus and
dialogs, and the notion of clicking on these elements. These events
could somehow be communicated to other elements such that the
corresponding actions are taken.
I'd advise not to use the report/ROM interface to propagate events, but
only state. Even though one (myself included) is sometimes intuitively
tempted to mix up events and states, it calls for trouble.
[...]
Makes sense, you are right. However, for clicking buttons in this context, it makes sense to communicate events instead of states. Perhaps using another way than reports could be useful here, such as some list of events like in Input, or a packet-stream like functionality.
This raises the question of where does the responsibility of the menu
end, and where do the responsibilities of the subsystem manager start?
If we need a subsystem manager anyway, couldn't it use the menu_view(s)
directly to implement menus?
We would like to separate the menu with all its buttons from the subsystem state logic.
Whenever we want an application that uses the GUI that does different
things, we would have to import the menu management
code into these applications. This could be done as a library with
functions for creating buttons and menus, or as a separate component
that can be configured and uses some IPC mechanism. In the latter case,
more than one component can talk with the same menu, depending on the
application programmer's needs.
This can be made as complex or simple as is desirable. However now it is
already quite complicated to create only a basic menu as all the
features are very intertwined, which makes the code a bit hard to read
and modify.
The answer may depend on the complexity of the subsystem manager, which
is obviously critical. Does the separation of the menu from the
subsystem manager contribute to significantly lower the TCB of the
subsystem manager?
It does not necessarily change the TCB, but it is done for the sake of
reusability and code readability/ease of programming. From that
perspective we think it would be nice if these things were separated,
and for instance the menu part could be used in another application as well.
This leaves the question about the input focus. When using the wm, the
focus is already defined by the window layouter via the 'focus' ROM. The
window layouter, in turn, already consumes a 'focus_request' ROM to
respond to externally triggered focus changes. This would be the right
hook for the menu / subsystem manager.
I've been looking into this mechanism, but so far it's not entirely
clear to me. Do you mean that the subsystem manager could write to this
focus_request report and the layouter would for example show/hide the
appropriate windows automatically? If so, what about the focus_requests
that come from the window manager?
In short: how would that work?
The state report generated by init already contains a 'state' attribute
for each child. Right now, the attribute is present only if the child is
"incomplete". We could add a value "exited" that would be presented as
soon as the child issued a 'Parent::exit' call. Also the exit code could
be added to the report, if needed. The subsystem manager could then
inspect the state reports for children with this attribute value and
remove them from the init config.
This is exactly what I'm looking for in that case.
Thanks,
--
Met vriendelijke groet / kind regards,
Boris Mulder
Cyber Security Labs B.V. | Gooimeer 6-31 | 1411 DD Naarden | The Netherlands+31 35 631 3253 <+31%2035%20631%203253> (office)
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
genode-main mailing list
https://lists.sourceforge.net/lists/listinfo/genode-main

[Norman Feske]

Hi Boris,

Post by Boris Mulder

Post by Norman Feske
The answer may depend on the complexity of the subsystem manager, which
is obviously critical. Does the separation of the menu from the
subsystem manager contribute to significantly lower the TCB of the
subsystem manager?

It does not necessarily change the TCB, but it is done for the sake of
reusability and code readability/ease of programming. From that
perspective we think it would be nice if these things were separated,
and for instance the menu part could be used in another application as well.

Please don't get me wrong. I do not oppose your idea. In contrary, I am
all for the creation of re-usable components! It is just that I feel
unable to fully grasp the requirements on a generic menu component. I
encourage you to pursue your approach and let us see where it takes us.

Also, however the menu will emerge, by using (and possibly improving)
the menu_view as the basis for its presentation, we will both walk on
the same ground. :-)

Post by Boris Mulder

Post by Norman Feske
This leaves the question about the input focus. When using the wm, the
focus is already defined by the window layouter via the 'focus' ROM. The
window layouter, in turn, already consumes a 'focus_request' ROM to
respond to externally triggered focus changes. This would be the right
hook for the menu / subsystem manager.

I've been looking into this mechanism, but so far it's not entirely
clear to me. Do you mean that the subsystem manager could write to this
focus_request report and the layouter would for example show/hide the
appropriate windows automatically? If so, what about the focus_requests
that come from the window manager?
In short: how would that work?

It does not work out of the box but I think that a minor extension of
the window layouter will do the trick.

First, the 'focus_request' ROM is merely a vehicle to bridge the gap
between nitpicker's session_control interface and the aspired
report-ROM-based focus management. In the longer term I would like to
manage the nitpicker focus solely by the report-ROM mechanism. But this
is not easy because nitpicker changes the focus autonomously whenever
the user clicks on an unfocused view. So in order to streamline the
focus handling, we need to possibly redefine the role of nitpicker as
the sole arbitrator of the focus. Frankly speaking, there are still
loose ends. I hope to have the design clarified in summer when I will be
able to turn my attention to the tiled window manager.

Until then, I recommend you to follow the pattern of the 'focus_request'
in the window layouter. The most pragmatic way would be to add another
'external_focus_request' ROM and let the layouter respond to both.

The story about the hidden applications is similar. You will need to
extend the window layouter to request a 'hidden' ROM, which tells the
layouter which windows to suppress. The menu (or subsystem manager?)
would solely generate / update the 'hidden' reports.

Do you think this approach is viable for your line of work?

Cheers
Norman

[Boris Mulder]

That focus_request sounds useful indeed. We are looking into it!

For now we are starting experiments with a component that uses init as a
way to start subsystems.

Perhaps it is easier to have the menu and subsystem state management
combined for now, and use qt5 for bigger applications that require more
flexibility.

We will try to separate these two in code a bit instead of splitting it
into two components.


Boris

Post by Norman Feske
Hi Boris,

Post by Boris Mulder

Post by Norman Feske
The answer may depend on the complexity of the subsystem manager, which
is obviously critical. Does the separation of the menu from the
subsystem manager contribute to significantly lower the TCB of the
subsystem manager?

It does not necessarily change the TCB, but it is done for the sake of
reusability and code readability/ease of programming. From that
perspective we think it would be nice if these things were separated,
and for instance the menu part could be used in another application as well.

Please don't get me wrong. I do not oppose your idea. In contrary, I am
all for the creation of re-usable components! It is just that I feel
unable to fully grasp the requirements on a generic menu component. I
encourage you to pursue your approach and let us see where it takes us.
Also, however the menu will emerge, by using (and possibly improving)
the menu_view as the basis for its presentation, we will both walk on
the same ground. :-)

Post by Boris Mulder

Post by Norman Feske
This leaves the question about the input focus. When using the wm, the
focus is already defined by the window layouter via the 'focus' ROM. The
window layouter, in turn, already consumes a 'focus_request' ROM to
respond to externally triggered focus changes. This would be the right
hook for the menu / subsystem manager.

I've been looking into this mechanism, but so far it's not entirely
clear to me. Do you mean that the subsystem manager could write to this
focus_request report and the layouter would for example show/hide the
appropriate windows automatically? If so, what about the focus_requests
that come from the window manager?
In short: how would that work?

It does not work out of the box but I think that a minor extension of
the window layouter will do the trick.
First, the 'focus_request' ROM is merely a vehicle to bridge the gap
between nitpicker's session_control interface and the aspired
report-ROM-based focus management. In the longer term I would like to
manage the nitpicker focus solely by the report-ROM mechanism. But this
is not easy because nitpicker changes the focus autonomously whenever
the user clicks on an unfocused view. So in order to streamline the
focus handling, we need to possibly redefine the role of nitpicker as
the sole arbitrator of the focus. Frankly speaking, there are still
loose ends. I hope to have the design clarified in summer when I will be
able to turn my attention to the tiled window manager.
Until then, I recommend you to follow the pattern of the 'focus_request'
in the window layouter. The most pragmatic way would be to add another
'external_focus_request' ROM and let the layouter respond to both.
The story about the hidden applications is similar. You will need to
extend the window layouter to request a 'hidden' ROM, which tells the
layouter which windows to suppress. The menu (or subsystem manager?)
would solely generate / update the 'hidden' reports.
Do you think this approach is viable for your line of work?
Cheers
Norman