Porting applications strategy

[Tomasz Gajewski]

I made an attempt to port libunistring (needed for guile) trying to
follow porting guide. I didn't go far but feel that what I've done did
not go in the right direction and would like to ask for opinions about
my feelings.


Short story

I would like to be able to do:

./configure --host=genode-x86 && make


Longer story

List of compiled sources in libunistring depends on results of running
configure and there is more than 500 source files. Even if I would
prepare list of files to compile probably it would be different in next
revision so it would have to be at least updated during upgrade. I feel
it would be waste of resources.

From different perspective as on of the goals of Genode for this year is

to have more software packages ported I think the procedure to port
should be as straightforward as possible. Probably most of the ported
software will be based on libc (I think that packages that will not be
linked against libc mostly will constitute base of genode not userspace
programs) and to be able to quickly port libc based libraries and
applications build system should not replaced but reused.

My feeling is that current Genode build system is great for Genode
specific components but for porting general purpose requires too much
maintenance.


I don't know detailes underneath current makefiles based build system
but I imagine steps to compile using configure could look somehow like
this:

1. Prepare spec with dependencies containing libc - just like it is
right now

2. make deps - build dependencies and install headers somewhere - this
is probably done right now in current build system

3. Run configure in this prepared environment. Running it without
prepared environment fails when testing if genode-x86-gcc can create
binaries - so quite quickly :-)

4. make - real porting work - hopefully not much

5. make install - to some prepared place?

6. from installed files take headers in case of libraries and binaries -
this somehow works right now


Firstly I'd like to hear what you think about the idea in
general.

Eventually could you provide some technical informations where are the
most important parts of build system where I should look to be able to
make such changes?

[Nobody III]

I don't know exactly how to best port a program or library to Genode, so I
don't really have the best answer, but I suggest that you look at the
makefiles in libports. You can also look at the ones in ports, but the
noux-pkg makefiles won't be particularly helpful, except when porting
utilities to noux. The makefiles don't all list the source files
individually.

Post by Tomasz Gajewski
I made an attempt to port libunistring (needed for guile) trying to
follow porting guide. I didn't go far but feel that what I've done did
not go in the right direction and would like to ask for opinions about
my feelings.
Short story
./configure --host=genode-x86 && make
Longer story
List of compiled sources in libunistring depends on results of running
configure and there is more than 500 source files. Even if I would
prepare list of files to compile probably it would be different in next
revision so it would have to be at least updated during upgrade. I feel
it would be waste of resources.

From different perspective as on of the goals of Genode for this year is

to have more software packages ported I think the procedure to port
should be as straightforward as possible. Probably most of the ported
software will be based on libc (I think that packages that will not be
linked against libc mostly will constitute base of genode not userspace
programs) and to be able to quickly port libc based libraries and
applications build system should not replaced but reused.
My feeling is that current Genode build system is great for Genode
specific components but for porting general purpose requires too much
maintenance.
I don't know detailes underneath current makefiles based build system
but I imagine steps to compile using configure could look somehow like
1. Prepare spec with dependencies containing libc - just like it is
right now
2. make deps - build dependencies and install headers somewhere - this
is probably done right now in current build system
3. Run configure in this prepared environment. Running it without
prepared environment fails when testing if genode-x86-gcc can create
binaries - so quite quickly :-)
4. make - real porting work - hopefully not much
5. make install - to some prepared place?
6. from installed files take headers in case of libraries and binaries -
this somehow works right now
Firstly I'd like to hear what you think about the idea in
general.
Eventually could you provide some technical informations where are the
most important parts of build system where I should look to be able to
make such changes?
--
Tomasz Gajewski
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
genode-main mailing list
https://lists.sourceforge.net/lists/listinfo/genode-main

[Martin Stein]

Hi Tomasz,

Post by Tomasz Gajewski
./configure --host=genode-x86 && make

As far as I know, we currently do not issue 'configure' through the
ports mechanism. Instead the customized config.* files of the ports are
provided. An example for this is libav:

libports/src/lib/libav/config.h
libports/src/lib/libav/config.mak

Then the Genode sided directory is added to the include paths, so the
config header applies when building the port and the config makefile is
included and applied in the Genode config:

libports/lib/mk/av.inc:
...
INC_DIR += $(REP_DIR)/src/lib/libav
...
include $(REP_DIR)/src/lib/libav/config.mak
SRC_C += $(OBJS:.o=.c)
SRC_C += $(OBJS-yes:.o=.c)
...

Cheers,
Martin

[Josef Söntgen]

Hello Tomasz,

Post by Tomasz Gajewski
./configure --host=genode-x86 && make

At the first glance that would be convient indeed but on the second
not so much — mostly because it involves having a build environment
ready, which means having autotools, pkg-config and the like available.
That only works well if your host system is quite POSIXish and if the
distributed autotools files of the program/library in question are
aware of the host target.

Post by Tomasz Gajewski
Longer story
List of compiled sources in libunistring depends on results of running
configure and there is more than 500 source files. Even if I would
prepare list of files to compile probably it would be different in next
revision so it would have to be at least updated during upgrade. I feel
it would be waste of resources.
From different perspective as on of the goals of Genode for this year is
to have more software packages ported I think the procedure to port
should be as straightforward as possible. Probably most of the ported
software will be based on libc (I think that packages that will not be
linked against libc mostly will constitute base of genode not userspace
programs) and to be able to quickly port libc based libraries and
applications build system should not replaced but reused.
My feeling is that current Genode build system is great for Genode
specific components but for porting general purpose requires too much
maintenance.

You have to maintain the build system either way. Since all of the
projects using autotools are not aware of Genode you have to patch
the various scripts, makefiles or m4 macros files to recognize it or
rather you have to execute autoreconf — if the project even ships its
.ac files — or replace the distributed files directly. Futhermore,
nowadays pkg-config is used to query certain dependencies, so you have
to provide the pkg-config database and to manage the .pc files as well.

Post by Tomasz Gajewski
I don't know detailes underneath current makefiles based build system
but I imagine steps to compile using configure could look somehow like
1. Prepare spec with dependencies containing libc - just like it is
right now
2. make deps - build dependencies and install headers somewhere - this
is probably done right now in current build system
3. Run configure in this prepared environment. Running it without
prepared environment fails when testing if genode-x86-gcc can create
binaries - so quite quickly :-)
4. make - real porting work - hopefully not much
5. make install - to some prepared place?
6. from installed files take headers in case of libraries and binaries -
this somehow works right now

I am not sure if that really saves you much time in the long run.
Providing a mechanism that integrates well with the Genode build system
is one thing. Making this mechanism useable by a majority of GNU build
tool based build systems so that you can conviently just execute
'configure', is something completely different.

Post by Tomasz Gajewski
Firstly I'd like to hear what you think about the idea in
general.

To be honest I personally do not think that its such a good idea but
that is mostly based on my past experience in dealing with autotools.
Been there done that while using the OpenBSD ports system and when
porting programs to Haiku — everytime I had to use autotools on a
system that was unknown or slightly different than the status quo,
e.g. GNU/Linux, it was uncomfortable to say the least(*). And I do not
expect it be any different in this case.

I would rather spend my time doing the actual porting work than messing
around with short comings of the original build system or breaking
things for one particular program while fixing the build tools another
one.

The other thing that annoys me about autotools is the fact, that
you would have to run it _every_ time you want compile the source,
although the target system has not changed at all. Sure, we could cache
the result for later use, but… In short, I would rather avoid the
checking and bookkeeping that comes free when using it altogether.


(*) One can argue that this is not the fault of the GNU build system
itself but rather of the way it certainly is used in the wild.

Post by Tomasz Gajewski
Eventually could you provide some technical informations where are the
most important parts of build system where I should look to be able to
make such changes?

There is _repos/ports/mk/noux.mk_ which basically is a wrapper around
the autotools suite for building noux-pkgs of programs that use it from
within the Genode build system. This wrapper creates an environment
that contains the important compiler options and flags to build the
program for running in Noux. If you start from there and make your way
along to _repos/base/mk_ you will hit the most important parts.


Regards
Josef

[Tomasz Gajewski]

Post by Josef Söntgen
You have to maintain the build system either way. Since all of the
projects using autotools are not aware of Genode you have to patch
the various scripts, makefiles or m4 macros files to recognize it or
rather you have to execute autoreconf — if the project even ships its
.ac files — or replace the distributed files directly. Futhermore,
nowadays pkg-config is used to query certain dependencies, so you have
to provide the pkg-config database and to manage the .pc files as well.

In theory autoconf tests should test for features not for systems
although reality is probably different.

Post by Josef Söntgen
I am not sure if that really saves you much time in the long run.
Providing a mechanism that integrates well with the Genode build
system is one thing. Making this mechanism useable by a majority of
GNU build tool based build systems so that you can conviently just
execute 'configure', is something completely different.

I understand your points but I can't imagine another way than
incrementally adding support for different build systems/engines. Maybe
I'm too much influenced by existing linux package systems though, where
they use "native" packages' build systems.

Post by Josef Söntgen
To be honest I personally do not think that its such a good idea but
that is mostly based on my past experience in dealing with autotools.
Been there done that while using the OpenBSD ports system and when
porting programs to Haiku — everytime I had to use autotools on a
system that was unknown or slightly different than the status quo,
e.g. GNU/Linux, it was uncomfortable to say the least(*). And I do not
expect it be any different in this case.

I know that plenty of packages will not build on such system without
modifications but what is the alterntaive? I believe that even
maintaining lists of source files for some greater number of packges is
infeasible. And maybe I'm naive but fixes for autoconf tests could be
merged into source packages but Genode specific patches definitely not.

Post by Josef Söntgen
There is _repos/ports/mk/noux.mk_ which basically is a wrapper around
the autotools suite for building noux-pkgs of programs that use it
from within the Genode build system. This wrapper creates an
environment that contains the important compiler options and flags to
build the program for running in Noux. If you start from there and
make your way along to _repos/base/mk_ you will hit the most important
parts.

That was really the information that I needed. Thank you.

After fighting with my lack of knowledge about advanced make usage I've
managed to prepare a version of noux.mk for a library (without noux
dependency) and build libunistring - without any source changes. I
haven't tested it yet though.


What I think I understand about current Genode build system it consists
of two layers:

* a package manager like layer that maintains dependencies between
programs/libraries

* build rules for building concrete programs/libraries

Both are implemented using make. I believe (although I haven't verified
it) that work of Emery to port nix to Genode is an attempt to replace
the former layer with nix.


I think that using guix and guile could be a better approach due to
using general purpose language (although not liked too much :-) ) and
Genode specific requirements, which could be harder to achieve in a
language dedicated specifically to build packages. But I have to check
it myself first. If I'm able to have a proof of concept and know more
about Emery's work I'll try to make some conclusions share them with you
here.


libunistring that I compiled is a dependency for guile and compiling it
is first step in my attempt to check if guix can become package manager
for Genode.

[e***@vfemail.net]

My take on this is that in the future it will be possible to construct
Noux environments such that autotools can be run in Noux to produce a
native Noux build (and maybe cross-compiling), but the practical thing
to do will be to try to and fool existing autotools tests, and only
inside Noux.

I do think that Nix can trick autotools into building on Noux, but first
Noux needs to be faster and I want to find out how far merging file
systems can go before hitting the point of diminishing returns.

Post by Tomasz Gajewski
What I think I understand about current Genode build system it consists
* a package manager like layer that maintains dependencies between
programs/libraries
* build rules for building concrete programs/libraries
Both are implemented using make. I believe (although I haven't verified
it) that work of Emery to port nix to Genode is an attempt to replace
the former layer with nix.

Yes, I want to create an alternate layer for dependencies and locating
the required makefiles. When the package output specification is ready
then both build systems should produce cross-compatible packages.

Post by Tomasz Gajewski
I think that using guix and guile could be a better approach due to
using general purpose language (although not liked too much :-) ) and
Genode specific requirements, which could be harder to achieve in a
language dedicated specifically to build packages. But I have to check
it myself first. If I'm able to have a proof of concept and know more
about Emery's work I'll try to make some conclusions share them with you
here.

You might be right, sometimes I think that Nix is a bit heavyweight for
simply generating text files like it often does, but I don't know much
about Guile.

As far as porting Guix, it might look like a good idea to try and
make it work on top of Noux, but my experience with Nix was that to
fix every required unix-ism will take more time and in the end you
will loose some important potential features. It may be possible to
reuse my Nix libraries for Guix, but I made some changes that break
compabality with upstream Nix for the sake of security and content
addressing, which you might not agree with.

The build controller component I have actually contains no 'Nix' code
or libraries, so if Guix can produce the some low-level build recipe
files, that would deduplicate a lot of work.

Cheers,
Emery

[Jookia]

Post by e***@vfemail.net
My take on this is that in the future it will be possible to construct
Noux environments such that autotools can be run in Noux to produce a
native Noux build (and maybe cross-compiling), but the practical thing
to do will be to try to and fool existing autotools tests, and only
inside Noux.

I think this is a good assumption

Post by e***@vfemail.net
I do think that Nix can trick autotools into building on Noux, but first
Noux needs to be faster and I want to find out how far merging file
systems can go before hitting the point of diminishing returns.

Post by Tomasz Gajewski
What I think I understand about current Genode build system it consists
* a package manager like layer that maintains dependencies between
programs/libraries
* build rules for building concrete programs/libraries
Both are implemented using make. I believe (although I haven't verified
it) that work of Emery to port nix to Genode is an attempt to replace
the former layer with nix.

Yes, I want to create an alternate layer for dependencies and locating
the required makefiles. When the package output specification is ready
then both build systems should produce cross-compatible packages.
You might be right, sometimes I think that Nix is a bit heavyweight for
simply generating text files like it often does, but I don't know much
about Guile.

Guile's a bit more heavyweight I think. However, Guix has the ability to do a
lot of really cool stuff when it comes to build environments- I'm not sure if
Nix has the same support. One interesting thing I've been toying with recently
is the 'build a VM', which has some Guile code to take a derivation and run it
in a VM. I imagine we could have something like that in Noux, then take the
output and use that in Genode. This might be already how you do it though.

Post by e***@vfemail.net
As far as porting Guix, it might look like a good idea to try and
make it work on top of Noux, but my experience with Nix was that to
fix every required unix-ism will take more time and in the end you
will loose some important potential features. It may be possible to
reuse my Nix libraries for Guix, but I made some changes that break
compabality with upstream Nix for the sake of security and content
addressing, which you might not agree with.

Guix is based on POSIX-isms which Noux could certainly help out with, currently
there's work to bring HURD support in which is testing Linux-specific
assumptions. It'd be very interesting to survey the assumptions Guix makes for
it's host-side code.

Post by e***@vfemail.net
The build controller component I have actually contains no 'Nix' code
or libraries, so if Guix can produce the some low-level build recipe
files, that would deduplicate a lot of work.

Assuming you mean the build controller just runs Bash code, then I imagine it'd
be a problem given instead of using shells to build, Guix uses a well-defined
subset of build-side modules that can run derivations.

Post by e***@vfemail.net
Cheers,
Emery

[e***@vfemail.net]

Now that I look, a Guix port could use much of the same underside of Nix, the
derivation recipe format is the same:
http://git.savannah.gnu.org/cgit/guix.git/tree/README#n128

The problem is that I interpret the format differently than on unix, which is
not so bad, because Genode derivations are always marked as such. I use the
environment variables section to rewrite requests for ROMs and File_system.
It was a better solution than assuming that derivations must be built with
Noux, and its less work to write a few variations of a Noux wrapper at the
higher level than it is to make Noux work everytime at the C++ level.

I'm still not that interested in Guix, but, if someone could port the
Guile/Guix evaluator, I would help hook it to my store and builder
components. What would be really helpful is if someone could come up with
a good system of distributing binary packages that would be compatible
between the two.

Emery

[Jookia]

Post by e***@vfemail.net
I'm still not that interested in Guix, but, if someone could port the
Guile/Guix evaluator, I would help hook it to my store and builder
components. What would be really helpful is if someone could come up with
a good system of distributing binary packages that would be compatible
between the two.

I don't think you'll get binary compatibility between Nix and Guix.

Post by e***@vfemail.net
Emery

Jookia.

[e***@vfemail.net]

Post by Jookia

Post by e***@vfemail.net
I'm still not that interested in Guix, but, if someone could port the
Guile/Guix evaluator, I would help hook it to my store and builder
components. What would be really helpful is if someone could come up with
a good system of distributing binary packages that would be compatible
between the two.

I don't think you'll get binary compatibility between Nix and Guix.

Sorry, I just meant compatible infrastructure that can serve both
variants over a network.

Emery

[Norman Feske]

Hello Tomasz,

Post by Tomasz Gajewski
After fighting with my lack of knowledge about advanced make usage I've
managed to prepare a version of noux.mk for a library (without noux
dependency) and build libunistring - without any source changes. I
haven't tested it yet though.
What I think I understand about current Genode build system it consists
* a package manager like layer that maintains dependencies between
programs/libraries
* build rules for building concrete programs/libraries
Both are implemented using make. I believe (although I haven't verified
it) that work of Emery to port nix to Genode is an attempt to replace
the former layer with nix.

it is not as simple as that. Actually, it is far too easy to mix things
up. For a structured discussion, I propose to keep different topics
cleanly separated:

1. Integration of 3rd-party code with Genode's genuine code
(Genode's ports mechanism)
2. Building Genode components (the Genode build system)
3. Introducing the notion of "source code packages" (does not exist
yet). A source-code package may be structurally similar to what a
Genode source repository is today, but it would contain 3rd-party
source code also.
4. Introducing the notion of "binary packages" (does not exist yet).
A binary package may be similar to what a build directory is
today, but specific for a certain source-code package and its
dependencies.
5. Building binary packages out of source packages
6. Deployment of binary packages on top of Genode system
7. Distribution of source and binary packages
8. Updating packages

Apparently, Emery's Nix line of work is primarily concerned about the
last three points. The discussion about the GNU build system vs. the
Genode build system is just a detail of point 5. Of course, it is
possible to wrap the GNU build system as I did with the (arguably ugly)
noux.mk wrapper. But for low-level libraries in particular, it is
sensible to realize their build procedure directly with the Genode build
system to benefit from Genode's inner- and inter-library dependency
tracking and to accommodate work flows where no "packages" are used,
like the run-script-based work flow we use today.

Until May, I'll try to develop a tangible notion of what we understand
as a "source package". Once this is in place, we can move on with the
points 6 to 8. Does this make sense to you? ;-)

Btw, it might be insightful to revisit the previous discussion on the
subject at our issue tracker:

https://github.com/genodelabs/genode/issues/1082

Post by Tomasz Gajewski
I think that using guix and guile could be a better approach due to
using general purpose language (although not liked too much :-) ) and
Genode specific requirements, which could be harder to achieve in a
language dedicated specifically to build packages. But I have to check
it myself first. If I'm able to have a proof of concept and know more
about Emery's work I'll try to make some conclusions share them with you
here.
libunistring that I compiled is a dependency for guile and compiling it
is first step in my attempt to check if guix can become package manager
for Genode.

Interesting. Personally, I do not think that there needs to be one
single way of package management for Genode. What I like about Emery's
work is that he managed to keep even the existence of the Nix mechanism
hidden from the "user". The more we experiment, the better our
understanding of the problem will get. So your initiative is very much
appreciated.

Cheers
Norman