I should change communication region to execute rtems with jailhouse.

I should change communication region to execute rtems with jailhouse, because patched rtems use communication region after 0x900000. jailhouse use 0x100000.

I change communication region base to 0x900000
at inmate.h :

And I reserve 0x900000 for communication region at each example demo file for test.
apic-demo.c :

pci-demo.c :

It is executed successfully. So I checked patched rtems whether it reserves 0x900000 for communication region.

but I cannot find that. Gedare Bloom said :

I believe RTEMS stays in real mode, and we do not have any MMU support
code in place. One thing to check is the "memory map" of the linked
executable in startup/linkcmds, which specifies how the ELF file is
loaded into memory. The mentioned address should be reserved for
something, or else it would possibly get assigned to another area and
you may have some data corruption problem.

I should check linkcmds to reserve communication region.

Final report of GSOC 2016 : RTEMS improvemt for Jailhouse hypervisor

Introduction and Project idea

 Jailhouse and RTEMS are open source project.

Jailhouse(https://github.com/siemens/jailhouse) is Linux-based partitioning hypervisor. 

initial version of Jailhouse developed x86 BSP. RTEMS also support x86, RTEMS can be easily as Jailhouse “inmate”. But Jailhouse needs required step for porting. The project aims to Integration, Merging, Fixup of existing support for Jailhouse and port support for other architectures (ARM). Other specifics of the project can be obtained from my proposal.

Status of Work Done

the work is divided into 2 part of bsp and starting jailhouse. And the individual bsps have 4 steps for being patched.

1) x86 : It is completed about 50 %

 (a) booting rtems for jailhouse.

   jailhouse use 0x000f0000 for start16addr ( it should move to jailhouse.cfg )

   pc386.cfg :

   jailhouse doesn't support video.

   start.S :

  

  jailhouse doesn't use local segment descriptor and A20

  start16.S :

 

 jailhouse starts inmate at 0x10000c to start of 32 bit code. and stack pointer address is 0xe0000

  start16.S :

 

   when rtems bsp reset on jailhouse, rtems just halt.

 jailhouse allocate 8Mb memory for rtems inmate cell.

 bspgetworkarea.c :

 

 (b) change pci port

   jailhouse only uses polled serial 1 and use 115200 baud rate.

   conscfg.c :

   last of bsp_console_select, end of libbsp/shared/console_select.c :

   

 (c) change lagacy pic(8259a) to apic(x2apic) to support timer and network.

 

Jailhouse non-root cells must use the xAPIC as minimum but should use
the x2APIC for best performance. When wanting to use x2apic, you need to enable it in the APIC BASE MSR. The BIOS may or may not have done this already on real hardware. Jailhouse behaves like a BIOS here in that it does enable x2apic when Linux was using it as well. So you just leave the APIC BASE MSR untouched and can even access the APIC in both ways then (memory mapped or via rdmsr/wrmsr) - Jailhouse maps one on the other.

in irq.c : you should remove all lagacy pic control things and add ioapic like

  for IOAPIC, bsae address is 0xfec00000

  irq.h : ( it should move to ioapic.h )

 

  in irq_asm.S, ack x2apic irq

 

 

in ldsgs.S, 8259a must not be used.

 

Work left: 

 (a) I should change config of jailhouse to execute rtems on jailhouse to test.

 (b) in patch files, rtems was patched to use early printk.

 (c)  In my cace, I use x408 port for pm timer but it can be changed by cases. I should check more.

 (d) rtems use e100, I have only e1000 device. It may change e100 to e1000.

2) arm : I couldn't start this part.

The workflow

The workflow in this project is important because you allow sufficient hardware support to execute jailhouse on qemu. When it comes to hardware spec to execute jailhouse successfully, you should use latest linux on native not vmware or qemu and latest version of qemu. For porting apic to RTEMS, latest qemu is required. A detailed explanation of this follows 

STEP-1:  install jailhouse and start example non-root cell on jailhouse.

STEP-2:  patch rtems-4.11 with origin patch files to test files.

STEP-3:  patch master rtems with origin patch.

in my proposal, I would use rtems-4.11 to execute patched rtems on jailhouse. bacause master rtems has lots of different with rtems-4.11, I change my workflow to patch master rtems.

there are many changes. rtems-4.11 uses 8259s pci but master rtems uses 8259a. cpu.h is removed. clock timer functions are changed. 

in pcibios.c of patch files, many functions are changed but master rtems use already same codes alread changed.

Explanation:

STEP-1: 

I wrote a instruction at this blog before.

http://wonjunhwang.blogspot.kr/2016/06/how-to-build-jailhouse-on-intel-host.html

jailhouse is still being developed. when you execute qemu with kvm options, there are different between when I executed jailhouse 2 month before and now. you can use iommu, intremap=on by using qemu 2.7.0-rc3. then you can see all of specifications is satisfied distincted from not using iommu,intremap=on.

STEP 2:

I wrote a instruction at this blog before.

http://wonjunhwang.blogspot.kr/2016/06/how-to-build-patched-rtems-for-jailhouse.html

STEP 3:

I worte a the strategy of patching rtems at this blog before

http://wonjunhwang.blogspot.kr/2016/08/refactoring-patch-files-and-rtems.html

rtems-jailhouse-wonjun is my repository to patch rtems for jailhouse.

https://github.com/hwj1593/rtems-jailhouse-wonjun/commits/master

I am still working to execute rtems as inmate on jailhouse.

I am modifying jailhouse inmate/lib files to change communication address for rtems.

( 0x100000 to 0x900000 )

jailhouse-wonjun is my repository to execute rtems as inmate

https://github.com/hwj1593/jailhouse-wonjun/commits/master

Thank you for patient reading.

refactoring patch files and where rtems should be modified for jailhouse.

I got 5 patch files from Jan.
0001-RTEMS-Jailhouse-port-1st-batch-including.patch
0002-Add-PCI-management-using-port-I-O.patch
0003-Fix-build-problem-missing-defines-in-cpu.h.patch
0004-Add-IOAPIC-for-PCI-interrupts-and-networking-with-e1.patch
0005-Generalize-some-HW-interfaces-for-Qemu-and-real-HW-e.patch

I should patch 4 points, booting rtems, timer ( select timer and calibrate ), pci ( uart ), networking.

And I should write test code to test whether patched rtems works successfully.

1. booting rtems.

I add jailhouse BSP variant. instead of modifying the make/custom/pc386.cfg file (patch 0001), a new jailhouse.cfg would be added that sets the build rules as needed for jailhouse.

for example

jailhouse.cfg :
   start16addr=0x00097c00 -> 0x000f0000



2. pci

in 0001 patch, patched rtems supports early printk() with no malloc
bsp_start_default() function is modified ( i386/pc386/startup/bspstart.c )

patched rtems is removed _IBMPC_initVideo because jailhouse dosn't support video  (start.S)


3. timer ( select timer and calibrate )
I also add local apic timer option for clock driver. There appears to be conflicting definitions for some of the APIC macros between libbsp/i386/shared/irq/apic.h and the changes made for jailhouse.

because jailhouse non-root cells must use the xAPIC as minimum but should use the x2APIC for best performance, I add local apic option to use x2apic.

pc386/configure.ac :

#define CLOCK_DRIVER_USE_LAPIC $CLOCK_DRIVER_USE_LAPIC
RTEMS_BSPOPTS_SET([CLOCK_DRIVER_USE_LAPIC,[*],[0])
RTEMS_BSPOPTS_HELP([CLOCK_DRIVER_USE_LAPIC],
[ if enabled, the clock driver will be set for jailhouse and use x2apic,
  it is not completed yet.
]

jailhouse use APIC Timers per tick and TSC/APIC Timer ratio to calibrate timer.
I patch ckinit.c file

add function

static void calibrate_pmtimer(void)
> calibrate apic timer / tsc cycle per tick using pm timer


set address and port address for jailhouse

Comm-region/PM-Timer port. ( 0x90000/0x608 )


4. networking
 ....


5. test

I am writing test code divided into three stages.

1) pci - check whether printk function works successfully or not.
2) timer - check ioapic and lapic
3) networking

Loading problem when I load rtems to cell with a loader file

when I started to load rtems to cell, I asked about loading rtems to Jan and I got a email.

you need to load the .bin of your RTEMS build into the inmate memory 
apparently, there is the need for a loader in addition 
if running over KVM, find the binary attached (still looking for the sources)

The email had a loader file for loading rtems to jailhouse.

I should select a address for rtems when I load a rtems file to cell.

First time, I used 0xf0000 for the loader file and 0x100000 for rtems.

I got a error when I load rtems.

 JAILHOUSE_CELL_LOAD: Invalid argument 

I found that I get a error when I load wrong address.

I have tried various address 0x0 to 0x3b000000.

>> Initializing Jailhouse hypervisor v0.5 (262-g425f61a-dirty) on CPU 0
>> Code location: 0xfffffffff0000030
>> Using x2APIC
>> Page pool usage after early setup: mem 40/1498, remap 64/131072
>> Initializing processors:
>> CPU 0... (APIC ID 0) OK
>> CPU 2... (APIC ID 2) OK
>> CPU 1... (APIC ID 1) OK
>> CPU 3... (APIC ID 3) OK
>> WARNING: No VT-d support found!
>> Adding PCI device 00:01.0 to cell "QEMU-VM"
>> Adding PCI device 00:02.0 to cell "QEMU-VM"
>> Adding PCI device 00:1b.0 to cell "QEMU-VM"
>> Adding PCI device 00:1f.0 to cell "QEMU-VM"
>> Adding PCI device 00:1f.2 to cell "QEMU-VM"
>> Adding PCI device 00:1f.3 to cell "QEMU-VM"
>> Adding PCI device 00:1f.7 to cell "QEMU-VM"
>> Adding virtual PCI device 00:0f.0 to cell "QEMU-VM"
>> Page pool usage after late setup: mem 179/1498, remap 65606/131072
>> Activating hypervisor
>> Virtual PCI connection established "linux-x86-demo" <--> "QEMU-VM"
>> Adding virtual PCI device 00:0f.0 to cell "linux-x86-demo"
>> Created cell "linux-x86-demo"
>> Page pool usage after cell creation: mem 226/1498, remap 65606/131072
>> Cell "linux-x86-demo" can be loaded
>> CPU 1 received SIPI, vector 100
>> CPU 2 received SIPI, vector 100
>> CPU 3 received SIPI, vector 100
>> Started cell "linux-x86-demo"
>> FATAL: Unsupported paging mode
>> FATAL: Invalid MMIO/RAM read, addr: 0x000000000010000c
>> RIP: 0x000000000010000c RSP: 0x00000000000e0000 FLAGS: 10006
>> RAX: 0x000000000010000c RBX: 0x0000000000000000 RCX: 0x0000000000000000
>> RDX: 0x0000000000000000 RSI: 0x0000000000000000 RDI: 0x0000000000000000
>> CS: 8 BASE: 0x0000000000000000 AR-BYTES: c09f EFER.LMA 0
>> CR0: 0x0000000000000031 CR3: 0x0000000000000000 CR4: 0x0000000000002000
>> EFER: 0x0000000000000000
>> Parking CPU 1 (Cell: "linux-x86-demo")

>> #jailhouse cell load 1 hello.bin -a 0xe0000
>>
>> JAILHOUSE_CELL_LOAD: Invalid argument

So I also asked Maxim who wrote a jailhouse paper.

He said 

Hello!

Usually, this error appeared when binary which you load into cell is bigger than the memory region in your cell's configuration. So, firstly, try to check if memory regions in your cell configuration are big enough. Binary must fit into one of your memory regions (considering the offset which you specied with -a option; so, offset+size of the binary must be smaller than the memregion). Jailhouse does not do more sothisticated checks than this size check.

When I ported the Fiasco, it was needed to tune memory regions in the fiasco's cell (and also check if nonroot cell's memory does not interfere with root cell's memregs).

I hope this simple idea will help. If you have some other questions I would be happy to help you.

As his advice, I will try to change a config file memory setting.

About mid term evaulation of GSoC

1. Planned progress vs. real progress according to the original project plan.

> planned progress for mid term is to show demo executing rtems as jailhouse "inmate" on x86 with appropriate tutorial to guide users.

> real progress
write tutorials how to build jailhouse, how to execute jailhouse cell and how to build patched rtems for jailhouse.
Next step is to execute rtems as jailhouse "imate". I should write a config file for rtems about executing jailhouse cell.
  
2. In case there is a divergence from planned/real, then what are the reasons?
A config file is needed to execute rtems as jailhouse "inmate".
When I started the project, mentor said that he was collecting materials about existed rtems demo. But after building rtems and jailhouse, I need to write a config file for rtems demo.

3. Updated plan for the second half of the GSoC period.
There is little difference but planned progress would be completed at the end.

4. Code produced so far.
Not thing. But I completed patching rtems by mentor's patch files.

5. What are the current blocking points?
I am studying about how rtems boots for writing a config file.

6. What is realistic to achieve finally?
As I suggested, I will also show RTEMS demo as jailhouse "inmate" on arm and write a documentation for users.

7. Which external help do you expect.
To execute jailhouse on arm, I need a bananaPi board. could you support about buying bananaPi board?

How to build patched RTEMS for Jailhouse

If you want to make a rtems cell for jailhouse, you should patch rtems.

I got five patch files from Jan Kiszka. ( jan.kiszka@siemens.com )
The patch files are involved with apic, ioapic, pic, memory etc for jailhouse to support rtems.

I got rtems and rtems-source-builder from git.
I checkout rtems and rtems-source-builder.

rtems baseline is commit f334847.
rtems-source-builder baseline is commit 04aadb6.
I matched rtems and rtems-source-builder by date of commit.

If you use rtems-source-builder master, you would get some error when you install i386-rtems by sb-set-builder.

Joel Sherrill will clone RTEMS for this old version issue.

After patching rtems, I installed rtems with command for jailhouse.
# ../rtems_git_dir/configure USE_COM1_AS_CONSOLE=1 CLOCK_DRIVER_USE_TSC=1 \
>    CLOCK_DRIVER_USE_8254=0 BSP_PRESS_KEY_FOR_RESET=1 --target=i386-rtems4.11 \
>    --enable-networking --enable-tests=samples --disable-posix --disable-multiprocessing \
>    --disable-cxx --enable-rtemsbsp=pc386 --prefix=~/rtems_build_dir \
>     BSP_PRINT_EXCEPTION_CONTEXT=1

And then, you can use hello.bin for jailhouse in testsuites.

Now I am writing config file for jailhouse to create cell for rtems.

How to build Jailhouse on Intel host

Until I set up Jailhouse starting cell, I underwent trial and error.
Actually, There are many instructions at Jailhouse Github, articles and google group. If you use AMD host, check instruction at jailhouse github.

https://github.com/siemens/jailhouse

http://www.linuxjournal.com/content/jailhouse

https://groups.google.com/forum/#!forum/jailhouse-dev/join

In my case, It cost lots of time to build Jailhouse.
I hope that everyone who read this build Jailhouse easily.

Check your host that support VT-d and VT-x and 4 cores.

http://virt-tools.org/learning/check-hardware-virt/

It requires latest Linux distro and QEMU, kernel to patch guest and Jailhouse.

I use Intel i5-2500 for host.

I use Ubuntu 16.04 LTS for host and guest ( I recommend same linux for host and guest ), QEMU 2.6.0 version, kernel 4.7-rc2 and Jailhouse master.
-- you can get a config file at google group -- 14th June 2016

First, install Ubuntu 16.04 LTS for host.

change your CMDLINE_LINUX at /etc/default/grub.

CMDLINE_LINUX="kvm-intel.nested=1 intel_iommu=off intremap=off"

#update-grub
#reboot
#modprobe kvm
#modprobe kvm-intel
#cat /sys/module/kvm_intel/parameters/nested
Y

Second, install QEMU version 2.6.0 with default configuration.

qemu-system-x86_64 -machine q35 -m 1G -enable-kvm -smp 4 \
    -cpu kvm64,-kvm_pv_eoi,-kvm_steal_time,-kvm_asyncpf,-kvmclock,+vmx \
    -drive file=yourlinux.img,format=raw,id=disk,if=none \
    -device ide-hd,drive=disk -serial stdio -serial vc \
    -device intel-hda,addr=1b.0 -device hda-duplex

Never change anything.
yourlinux.img is created by qemu-img create yourlinux.img ubuntu.iso 20G.

after starting QEMU and installing Ubuntu for QEMU guest, change your CMDLINE_LINUX at /etc/default/grub.

CMDLINE_LINUX="memmap=66M$0x3b000000"

#update-grub
#reboot

Check cat /proc/cmdline -> memmap=66M$0x3b000000

and download jailhouse.


#apt-get update
#apt-get install git python-mako 
#git clone https://github.com/siemens/jailhouse
#make
#make firmware_install
#make install
#insmod driver/jailhouse.ko
#jailhouse enable configs/qemu-vm.cell
#jailhouse cell create configs/apic-demo.cell
#jailhouse cell load apic-demo inmates/demos/x86/apic-demo.bin -a 0xf0000
#jailhouse start apic-demo

If you get error, join Jailhouse google group.
There are talks about error.
If you don't get error, you can explore Jailhouse.