[GSoC] UEFI framebuffer task

[Philipp Kerling]

Hi,

I'm interested in participating in the Genode project as GSoC student,
specifically the UEFI framebuffer task, so I am writing to you to
briefly introduce myself and discuss the project idea. I am currently a
student of Computer Engineering (MSc) at Ilmenau University of
Technology in Germany in my second master semester. I've chosen
computer engineering over computer science out of interest for low-
level topics, which I think fits nicely with Genode. I worked on a toy
x86 operating system with a friend some years ago, which was a good
learning experience. It naturally did not go very far, but it did at
least have basic multitasking and virtual memory :-)

So, about the UEFI framebuffer proposal: Is the overall goal to have
Genode/Nova running as bare UEFI application (without any OS Loader) or
to have it loaded by GRUB (or similar) but not rely on outdated BIOS
features (such as VBE) any more? Or support both like e.g. the Linux
kernel does?
This heavily influences the options available for framebuffer output,
since the UEFI GOP (Graphics Output Protocol) that I assume you are
referring to as one of the alternative/modern ways to access the
framebuffer is only available as UEFI boot service. When being booted
from an OS loader, usually only UEFI run-time services are available.
In that case, the OS loader has to convey some information (mainly
memory address, width/stride, height, depth) about the linear GOP
framebuffer that the loader (or the firmware, for that matter)
initialized to the kernel. multiboot2 does this for example with the
framebuffer boot info tag. So to use this it would be sufficient to
write a generic framebuffer driver operating on that information that
is not specific to UEFI at all. It could probably also use the VBE
framebuffer provided the OS loader initialized it. The "problem" with
this is that it is probably not enough work to justify a 3-month GSoC
proposal.

I hope that the questions at least touch the right topics but do feel
free to say so if you had something different in mind altogether.

Best regards,
Philipp

[Sebastian Sumpf]

Hi Philipp,

Post by Philipp Kerling
Hi,
I'm interested in participating in the Genode project as GSoC student,
specifically the UEFI framebuffer task, so I am writing to you to
briefly introduce myself and discuss the project idea. I am currently a
student of Computer Engineering (MSc) at Ilmenau University of
Technology in Germany in my second master semester. I've chosen
computer engineering over computer science out of interest for low-
level topics, which I think fits nicely with Genode. I worked on a toy
x86 operating system with a friend some years ago, which was a good
learning experience. It naturally did not go very far, but it did at
least have basic multitasking and virtual memory :-)

Thanks for your introduction!

Post by Philipp Kerling
So, about the UEFI framebuffer proposal: Is the overall goal to have
Genode/Nova running as bare UEFI application (without any OS Loader) or
to have it loaded by GRUB (or similar) but not rely on outdated BIOS
features (such as VBE) any more? Or support both like e.g. the Linux
kernel does?

Loading Genode with GRUB2 is sufficient. The main goal is, as you said,
to not rely on outdated BIOS features. There is already a work in
progress on the NOVA hypervisor
(https://github.com/genodelabs/genode/issues/2242), which now supports
legacy boot (mutliboot) and UEFI boot (multiboot2).

Post by Philipp Kerling
This heavily influences the options available for framebuffer output,
since the UEFI GOP (Graphics Output Protocol) that I assume you are
referring to as one of the alternative/modern ways to access the
framebuffer is only available as UEFI boot service. When being booted
from an OS loader, usually only UEFI run-time services are available.
In that case, the OS loader has to convey some information (mainly
memory address, width/stride, height, depth) about the linear GOP
framebuffer that the loader (or the firmware, for that matter)
initialized to the kernel. multiboot2 does this for example with the
framebuffer boot info tag. So to use this it would be sufficient to
write a generic framebuffer driver operating on that information that
is not specific to UEFI at all. It could probably also use the VBE
framebuffer provided the OS loader initialized it. The "problem" with
this is that it is probably not enough work to justify a 3-month GSoC
proposal.

Sounds very good. VBE is not necessary as Genode already offers a VESA
driver. Additionally, multiboot2 support should be added to the SeL4,
base-hw, and Fiasco.OC kernels. Some experiments with Genode and
coreboot are also welcome.
I have to admit that this challenge is meant for students with little
systems programming knowledge an, so feel free to look into more
challenging topics here (http://genode.org/about/challenges). Also,
these are just suggestions and you are welcome to propose topics. We
also have topics not on the challenges list, if you let us know your
interests (like kernel hacking, device drivers, porting ...), we may be
able to suggest something more fitting for you.

Cheers,

Sebastian

Post by Philipp Kerling
I hope that the questions at least touch the right topics but do feel
free to say so if you had something different in mind altogether.
Best regards,
Philipp
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
genode-main mailing list
https://lists.sourceforge.net/lists/listinfo/genode-main

[Philipp Kerling]

Hi Sebastian,

Thanks for looking through my findings.

Post by Sebastian Sumpf
Loading Genode with GRUB2 is sufficient. The main goal is, as you said,
to not rely on outdated BIOS features. There is already a work in
progress on the NOVA hypervisor
(https://github.com/genodelabs/genode/issues/2242), which now
supports
legacy boot (mutliboot) and UEFI boot (multiboot2).

Yes, I had taken a look at that. I've investigated a bit, but apart
from VBE and getting rid of the BDA for communicating the serial port
address there doesn't seem to be that much left driver-wise to do at
first glance since Genode didn't use many of the AT BIOS features in
the first place. RTC driver using EFI runtime services could be added,
though.

Post by Sebastian Sumpf
Sounds very good. VBE is not necessary as Genode already offers a VESA
driver.

Yep, that was just to illustrate that it's not specifically an UEFI
issue.

Post by Sebastian Sumpf
I have to admit that this challenge is meant for students with little
systems programming knowledge an, so feel free to look into more
challenging topics here (http://genode.org/about/challenges). Also,
these are just suggestions and you are welcome to propose topics. We
also have topics not on the challenges list, if you let us know your
interests (like kernel hacking, device drivers, porting ...), we may be
able to suggest something more fitting for you.

I'd like to try my hand at device drivers, which was why I looked into
the UEFI thing in the first place. Do you have any suggestions there
that are not on the list? Isochronous USB is an option, but it looks
like it is more about hooking up/passing through the Linux driver code
correctly than the actual USB driver stuff.

Regards,
Philipp

[Sebastian Sumpf]

Hey Philipp,

Post by Philipp Kerling
Hi Sebastian,
Thanks for looking through my findings.

Post by Sebastian Sumpf
Loading Genode with GRUB2 is sufficient. The main goal is, as you said,
to not rely on outdated BIOS features. There is already a work in
progress on the NOVA hypervisor
(https://github.com/genodelabs/genode/issues/2242), which now
supports
legacy boot (mutliboot) and UEFI boot (multiboot2).

Yes, I had taken a look at that. I've investigated a bit, but apart
from VBE and getting rid of the BDA for communicating the serial port
address there doesn't seem to be that much left driver-wise to do at
first glance since Genode didn't use many of the AT BIOS features in
the first place. RTC driver using EFI runtime services could be added,
though.

Post by Sebastian Sumpf
Sounds very good. VBE is not necessary as Genode already offers a VESA
driver.

Yep, that was just to illustrate that it's not specifically an UEFI
issue.

Post by Sebastian Sumpf
I have to admit that this challenge is meant for students with little
systems programming knowledge an, so feel free to look into more
challenging topics here (http://genode.org/about/challenges). Also,
these are just suggestions and you are welcome to propose topics. We
also have topics not on the challenges list, if you let us know your
interests (like kernel hacking, device drivers, porting ...), we may be
able to suggest something more fitting for you.

I'd like to try my hand at device drivers, which was why I looked into
the UEFI thing in the first place. Do you have any suggestions there
that are not on the list? Isochronous USB is an option, but it looks
like it is more about hooking up/passing through the Linux driver code
correctly than the actual USB driver stuff.

Genode has two approaches with regard to drivers: First, we write our
own (e.g., AHCI, SD, or ACPI) and second, we port them if we think
writing one exceeds our available resources. That being said, the USB
driver is ported from Linux and supports UHCI/EHCI/XHCI on x86 and
diverse ARM platforms (omap4, exynos5, rpi). USB device driver vise
there is hid/nic/storage support. We also have our own storage driver
(usb_block) that connects to the USB driver (i.e., the host controller)
through our USB session interface. For isochronous USB this sessions the
USB host controller drivers would have be to extended to support
isochronous transfers. A USB device driver (e.g, audio or video) should
be written or ported. In the future we want to have all USB device
drivers outside of the current USB implementation.

As for additional driver stuff I can think of iMX6 USB and NIC support.
AHCI is slowly replaced by NVMe so a NVMHCI implementation would be very
welcome and if you are up for a really though one: We are currently
implementing our own Intel GPU multiplexer on Broadwell while also
having a running port with 3D support enabled. Additionally we are
looking into GPU virtualization. In this area there are a lot of things
to do or investigate.

@staff: Do you have any additional suggestions?

Regards,

Sebastian

[Philipp Kerling]

Hi,

Post by Sebastian Sumpf
Genode has two approaches with regard to drivers: First, we write our
own (e.g., AHCI, SD, or ACPI) and second, we port them if we think
writing one exceeds our available resources. That being said, the USB
driver is ported from Linux and supports UHCI/EHCI/XHCI on x86 and
diverse ARM platforms (omap4, exynos5, rpi). USB device driver vise
there is hid/nic/storage support. We also have our own storage driver
(usb_block) that connects to the USB driver (i.e., the host
controller)
through our USB session interface. For isochronous USB this sessions the
USB host controller drivers would have be to extended to support
isochronous transfers.

OK, that confirms my understanding of the task.

Post by Sebastian Sumpf
A USB device driver (e.g, audio or video) should
be written or ported. In the future we want to have all USB device
drivers outside of the current USB implementation.

I'm not quite sure how to interpret the last sentence. Does this mean
that you want to separate e.g. the HID driver that is currently inside
the Linux kernel and have it attach to the Genode USB session
interface?

Post by Sebastian Sumpf
As for additional driver stuff I can think of iMX6 USB and NIC
support.
AHCI is slowly replaced by NVMe so a NVMHCI implementation would be very
welcome and if you are up for a really though one: We are currently
implementing our own Intel GPU multiplexer on Broadwell while also
having a running port with 3D support enabled. Additionally we are
looking into GPU virtualization. In this area there are a lot of things
to do or investigate.

The GPU stuff does seem interesting (and challenging), but it also
sounds quite hard to quickly get into well enough for writing a solid
project proposal if you don't have any prior exposure (like me).
NVMHCI sounds like it could be what I was looking for though. I guess
the expectation is to have a driver that implements the Block interface
similar to the current AHCI driver? Do you have any specific feature
requests besides simple read/write? NVMe supports a lot of enhanced
stuff (like atomic compare/write and priorities) that cannot be exposed
over the current Block interface - question is if there's a need for
it.

Regards,
Philipp

[Sebastian Sumpf]

Hi Philipp,

Post by Philipp Kerling
Hi,

Post by Sebastian Sumpf
Genode has two approaches with regard to drivers: First, we write our
own (e.g., AHCI, SD, or ACPI) and second, we port them if we think
writing one exceeds our available resources. That being said, the USB
driver is ported from Linux and supports UHCI/EHCI/XHCI on x86 and
diverse ARM platforms (omap4, exynos5, rpi). USB device driver vise
there is hid/nic/storage support. We also have our own storage driver
(usb_block) that connects to the USB driver (i.e., the host
controller)
through our USB session interface. For isochronous USB this sessions the
USB host controller drivers would have be to extended to support
isochronous transfers.

OK, that confirms my understanding of the task.

Post by Sebastian Sumpf
A USB device driver (e.g, audio or video) should
be written or ported. In the future we want to have all USB device
drivers outside of the current USB implementation.

I'm not quite sure how to interpret the last sentence. Does this mean
that you want to separate e.g. the HID driver that is currently inside
the Linux kernel and have it attach to the Genode USB session
interface?

Exactly.

Post by Philipp Kerling

Post by Sebastian Sumpf
As for additional driver stuff I can think of iMX6 USB and NIC support.
AHCI is slowly replaced by NVMe so a NVMHCI implementation would be very
welcome and if you are up for a really though one: We are currently
implementing our own Intel GPU multiplexer on Broadwell while also
having a running port with 3D support enabled. Additionally we are
looking into GPU virtualization. In this area there are a lot of things
to do or investigate.

The GPU stuff does seem interesting (and challenging), but it also
sounds quite hard to quickly get into well enough for writing a solid
project proposal if you don't have any prior exposure (like me).
NVMHCI sounds like it could be what I was looking for though. I guess
the expectation is to have a driver that implements the Block interface
similar to the current AHCI driver?

Yes.

Do you have any specific feature

Post by Philipp Kerling
requests besides simple read/write?

Read/write with native command queuing support would suffice.

NVMe supports a lot of enhanced

Post by Philipp Kerling
stuff (like atomic compare/write and priorities) that cannot be exposed
over the current Block interface - question is if there's a need for
it.

If the above is working, it would be interesting to investigate if there
are features Genode could benefit from and maybe implement some.
Additionally power management is always a topic.


Regards,

Sebastian

[Philipp Kerling]

Hi,

Post by Sebastian Sumpf

Post by Philipp Kerling
Do you have any specific feature
requests besides simple read/write? 

Read/write with native command queuing support would suffice.

NCQ isn't directly applicable to NVMe, at least not in the SATA sense.
NVMe has mandatory support for 65535 distinct submission queues into
which you can each put 65536 commands that are executed in no
particular order (i.e.  possibly in parallel and as the controller sees
fit, similarly to NCQ).
For Genode, using one queue would probably be sufficient.
This does mean that the NVMe driver has to make sure that if you e.g.
get a write and then a read of the same address queued from an
application, the read would have to be postponed until the write is
completed since the NVMe controller does not guarantee in-order
execution. Now that I think about it though, this also applies to SATA
NCQ and is what I guess Ata_driver::overlap_check() is for in the AHCI
driver.

Post by Sebastian Sumpf
If the above is working, it would be interesting to investigate if there
are features Genode could benefit from and maybe implement some.
Additionally power management is always a topic.

Ok thanks, noted. Does Genode have support for ACPI device power state
switching, particularly Runtime D3?
NVMe also supports elaborate PM, some of which (e.g. automatic PM)
could easily be integrated. The more involved stuff depends a lot on
knowing the workload and requirements of the system (i.e. burstiness,
priority of latency or throughput, ...), but that could be made into
parameters provided to the driver.

Regards,
Philipp

[Sebastian Sumpf]

Hi Philipp,

Post by Philipp Kerling
Hi,

Post by Sebastian Sumpf

Post by Philipp Kerling
Do you have any specific feature
requests besides simple read/write?

Read/write with native command queuing support would suffice.

NCQ isn't directly applicable to NVMe, at least not in the SATA sense.
NVMe has mandatory support for 65535 distinct submission queues into
which you can each put 65536 commands that are executed in no
particular order (i.e. possibly in parallel and as the controller sees
fit, similarly to NCQ).
For Genode, using one queue would probably be sufficient.
This does mean that the NVMe driver has to make sure that if you e.g.
get a write and then a read of the same address queued from an
application, the read would have to be postponed until the write is
completed since the NVMe controller does not guarantee in-order
execution. Now that I think about it though, this also applies to SATA
NCQ and is what I guess Ata_driver::overlap_check() is for in the AHCI
driver.

Good research from your side here.

Post by Philipp Kerling

Post by Sebastian Sumpf
If the above is working, it would be interesting to investigate if there
are features Genode could benefit from and maybe implement some.
Additionally power management is always a topic.

Ok thanks, noted. Does Genode have support for ACPI device power state
switching, particularly Runtime D3?

No, power management is still in its infancy. But this is a topic the
Genode staff will have to work on.

Post by Philipp Kerling
NVMe also supports elaborate PM, some of which (e.g. automatic PM)
could easily be integrated. The more involved stuff depends a lot on
knowing the workload and requirements of the system (i.e. burstiness,
priority of latency or throughput, ...), but that could be made into
parameters provided to the driver.

Sounds good, looking forward to your proposal.

Sebastian

[Philipp Kerling]

Hi Sebastian,

Post by Sebastian Sumpf
Sounds good, looking forward to your proposal.

Thanks - I've submitted the draft. I'd be happy to hear any comments or
suggestions you and the other team members might have.
On another note, the microkernel devroom GSoC page recommends doing a
small qualification task for the project. Do you have any suggestions
here?

Best regards,
Philipp

[Sebastian Sumpf]

Hi Philipp,

Post by Philipp Kerling

Post by Sebastian Sumpf
Sounds good, looking forward to your proposal.

Thanks - I've submitted the draft. I'd be happy to hear any comments or
suggestions you and the other team members might have.

Okay, I gonna have a look at it tomorrow.

Post by Philipp Kerling
On another note, the microkernel devroom GSoC page recommends doing a
small qualification task for the project. Do you have any suggestions
here?

Maybe you could get started by accessing a register of the NVMHCI (like
a version register or similar). For this I would recommend to perform
the following steps:

1. Get Genode master
2. Start QEMU with NVME enabled (I would recommend NOVA as kernel)
3. Find and map the PCI I/O memory resource of the controller
4. Take advantage of Genode's MMIO framework and read a register of your
choice

Please feel free to ask questions,

Sebastian

[Christian Helmuth]

Hey Phillip,

I assume Sebastian will not mind if I give a small hint according to
item 3 on the list. ;-)

Post by Sebastian Sumpf
3. Find and map the PCI I/O memory resource of the controller

Have a look at os/run/pci.run and os/src/test/pci/test.cc to convince
the platform_drv to hand out PCI devices to your driver component.

Happy hacking

[Philipp Kerling]

Hi,

Post by Christian Helmuth
Hey Phillip,
I assume Sebastian will not mind if I give a small hint according to
item 3 on the list. ;-)

Post by Sebastian Sumpf
3. Find and map the PCI I/O memory resource of the controller

Have a look at os/run/pci.run and os/src/test/pci/test.cc to convince
the platform_drv to hand out PCI devices to your driver component.

Thanks - copying the policy from pci.run did help quite a bit :-)

You can now review the code at 
https://github.com/pkerling/genode/commit/4130907
I have to admit that it's mostly copypaste from the AHCI driver.
I tested it successfully with both base-hw and NOVA. My output is:

$ make run/nvme_test
[...]
Genode 17.02-148-g41309074e <local changes>
58 MiB RAM assigned to init
[...]
[init -> nvme_drv] NVMe demo
[init -> nvme_drv] NVMe found (vendor: 0x8086 device: 0x5845 class: 0x10802)
[init -> nvme_drv] BAR0: 0xfebf0000, size: 0x2000
[init -> nvme_drv] Controller supports NVMe version 1.1.0
[init -> nvme_drv] Memory pages from 4096 to 65536 bytes supported
[init -> nvme_drv] 2048 maximum entries per queue
[init -> nvme_drv] --- NVMe demo finished ---


Regards,
Philipp

[Sebastian Sumpf]

Hi Philipp,

Post by Philipp Kerling
Hi,
You can now review the code at
https://github.com/pkerling/genode/commit/4130907
I have to admit that it's mostly copypaste from the AHCI driver.
$ make run/nvme_test
[...]
Genode 17.02-148-g41309074e <local changes>
58 MiB RAM assigned to init
[...]
[init -> nvme_drv] NVMe demo
[init -> nvme_drv] NVMe found (vendor: 0x8086 device: 0x5845 class: 0x10802)
[init -> nvme_drv] BAR0: 0xfebf0000, size: 0x2000
[init -> nvme_drv] Controller supports NVMe version 1.1.0
[init -> nvme_drv] Memory pages from 4096 to 65536 bytes supported
[init -> nvme_drv] 2048 maximum entries per queue
[init -> nvme_drv] --- NVMe demo finished ---

I have had a look at your code and tested in on NOVA as well. No
objections from my side - good job.

Next I would suggest to finish your proposal (item 15) and mark it as
final until Monday.

Cheers,

Sebastian