Final Report on Porting Rock to RTEMS

GSOC : Final Report

This post would serve as my final report on my project as well as it would act as a reference to anyone who wishes to continue with my project. 

Introduction and Project idea

Rock(http://rock-robotics.org/stable/) is an open source software framework for development of robotic systems. Rock along with RTEMS has been used in ESA for quite a long time and that is why there had been several porting attempts of Rock. However due to some or the other reason none of the attempts could be merged before.And though scripts are present without maintenance in a year or so they lose their value.

This GSoC project aimed at gathering all the scripts updating them and finally preparing a usable port of Rock for RTEMS. Other specifics of the project can be obtained from my proposal

Status of Work Done

As per the  originally planned approach from the proposal,the work is done up to  around 80% of Phase-1 and around 10% of Phase-3.The details are as follows (please refer my proposal for a clearer idea):

(a) Phase-1 work

The phase 1 work was divided into two parts:

       (i) Updating the Porting scripts of all the external                                                                                         dependencies.The work was done through my repository. All the scripts were tested to build               on Linux Mint 17.2. They install fine and work with the RTT fork. However specific                           applications to test them on QEMU is still a work in progress. 

      (ii) Updating the Porting scripts of the Rock core dependencies.For this the RTT was updated but             subsequent tests on the changes made were failing.The work ahead should not begin without               testing the RTT.The commits were made on a RTT fork.

Work left: Typelib, Orogen, Rock base Packages could not be ported as I was stuck with RTT.Example applications for this is also a work in progress(see this).Then an Autoproj script for automating the entire port had to be written which could not be done. After that Rock examples and packages useful to RTEMS should have been selected and ported.However this could not be done at all.

(b)Phase-2 work

Work left: The phase-2 part of porting examples and packages could not be done at all.

(c) Phase-3 work

The phase-3 work of the project was preparing a RSB recipe for the port.The entire port was not over,so as soon as a part was over its build recipe was provided. I  started with a libxml recipe. The work was done through my  RSB fork.

Work left: The total port had to be automated by RTEMS Source Builder. All dependencies external as well as internal dependencies (RTT,Typelib,Orogen) had to be provided RSB recipes for building which is left.

The workflow

The workflow in this project is important because Rock is an amalgamation of a large number of software frameworks.For porting Rock to RTEMS, RTEMS support needs to be added to all these softwares. When it comes to Rock, the core dependencies should be ported first rather than the base dependencies.A detailed explanation of this follows (my initial workflow as mentioned in the proposal needed to be changed with time.)

STEP-1: Update all the external dependencies of Rock. i.e. 1) Boost (v1.54 recommended)

                                                                                                2) Eigen

                                                                                                3) Libxml

                                                                                                4) OmniORB

STEP-2: Make RSB recipes for them as soon as they were successfully built and tested.

STEP-3:  Start porting the Rock core dependencies followed by the Rock Base dependencies.

Explanation:

STEP-1: 

1)Boost(v1.54) is an open source peer-reviewed library(http://www.boost.org/).Boost is essentially used for getting the serialization and the threading libraries for RTT.It is highly recommended that v 1.54 be used so that we don't get compatibility errors.Only Boost v1.54 contains the specific templates that the whole of RTT is designed for. Changing the codes in so many places will be really cumbersome. But check the latest versions once more to make sure. For cross-compiling for RTEMS what you have to do is just enable static libraries that is done through the flags(see Phase 1 commits).

2)Eigen is a library for linear algebra ,matrix and vector operations.It is esssential for the Rock Base packages.The patches were just to make sure latest versions get installed.

3)Libxml is an XML C parser that aids in the serialization process. Libxml is however not used anymore. Its successor libxml2 is used. For the changes in cross compilation see Phase 1 commits.It was however mentioned that in later stages of the port libxml2 needs to be replaced  by castxml2 or llvm front end.However I could not reach that stage so I am avoiding a comment on it.

4)OmniORB is an Object Request Broker and it helps in invoking operations on an object on an object on a computer across a network, on different Operating System, On diffrent CPU systems or even implemented with diffrent programming languages.The tests with the OmniORB for the patches worked well.For testing the instructions in the OmniORB Documentation canbe followed using QEMU to run RTEMS.For understanding its functionality.It takes a little effort to understand CORBA.Please refer previous posts on this blog.

This part of my project is complete and following are the instructions to build the external Rock dependencies.

1> Prepare the following directory structure Port/x86/ and then in that /Build and /Install.Then in the / /Build directory prepare /Build/pkgs.

       

2> Go to Port/x86/Build and clone my repository

      $git clone https://github.com/Sambeet161616/Rock-Port_ana.git

3> Now go to the cloned repository and change the user name as per your system in the script env.sh.

     
      $ cd Rock-Port_ana/
      $  gedit env.sh

       Then in  gedit edit the underlined in the script env.sh as per your system

      export BUILD_PREFIX=/home/sambeet/Port/x86/Build
      export INSTALL_PREFIX=/home/sambeet/Port/x86/Install

4> Now give the following command  to set the environment variables
   
     $ source env.sh

5> Now run each of the scripts to build the dependencies

    $ bash libxml.sh 

     

     Replace libxml.sh with boost_cross_compiler.sh, eigen_cross_compiler.sh,                                      xerces_c_cross_compiler.sh, prepare_RTEMS_mod.sh.

Note: Please do ensure proper internet connection while running the scripts.

6> After the scripts are run successfully, we can see the boost, eigen, libxml, omniORB, rtems and xerces folders in the Port/x86/Install folder.

STEP 2:

To make RSB recipes for the above builds, RSB documentation provides easy to understand instructions. The work on this is on going  on my repository.

STEP 3:

The Rock core dependencies needs to be ported in a particular order i.e rtt,typelib,then orogen followed by base packages (base/orogen,base/cmake,base/types).

To add RTEMS support to RTT and test it is where I am stuck now, so I would like to write to give a rather good explanation of the problems I encountered.

Rock software package fundamentally is the usage of Orocos Real Time Toolkit Libraries to build and control real time components in a very convenient and user friendly way. Rock uses Orogen,the code generator to generate the components and a ruby-scripting for controlling the components.(N.B. This is not the only way to run Orocos components.There are two other ways to complete this: (a)Using the Orocos Component Library http://www.orocos.org/stable/documentation/ocl/v2.x/doc-xml/orocos-deployment.html (b)Manually write and compile the component using Cmake. Have a look at this) Very important in this regard is the RTT Cheat Sheet. 

Adding RTEMS support to RTT.The detailed approach to adding RTEMS support is mentioned in Orocos Component Builder's Manual. Apart from adapting the fosi.h file for RTEMS which is very similar to GNU Linux since RTEMS supports POSIX standard, we should ensure a static build of RTT libraries as the Dynamic Loader is a relatively new concept for RTEMS as compared to static. My patches on RTT build fine.But I still could not build a RTT component and all tests regarding this have been failing. For testing the RTT on could utilise the manually written component. And the following command from the terminal could be given (could give errors especially linking errors with Boost,need work!)

sambeet@Holmes ~/NewRockPort/x86/Build/rock/rtt $ sudo /home/sambeet/NewRockPort/x86/Install/rtems/4.11.0-rc3/bin/i386-rtems4.11-g++ -B /home/sambeet/NewRockPort/x86/Install/rtems/4.11.0-rc3/i386-rtems4.11/pc486/lib/ -specs bsp_specs -qrtems HelloWorld.cpp /usr/local/lib/liborocos-rtt-rtems.a /usr/local/lib/orocos/rtems/types/librtt-typekit-rtems.a -I ./ -I /usr/local/boost_1_54_0/include/ -L /usr/local/boost_1_54_0/lib/ -I /home/sambeet/NewRockPort/x86/Install/rtems/4.11.0-rc3/i386-rtems4.11/pc486/lib/include/ -L /home/sambeet/NewRockPort/x86/Install/rtems/4.11.0-rc3/i386-rtems4.11/pc486/lib/ -static -DOROCOS_TARGET=rtems /usr/local/boost_1_54_0/lib/libboost_filesystem.a /usr/local/boost_1_54_0/lib/libboost_serialization.a 2>&1 | tee testout.txt

However the tests are still not working i.e I cannot still guarantee that RTT is ported successfully.The important issues in this regard:

1) Make sure that Boost 1.54 is used as mentioned before.

2) Before attempting a static build on RTEMS , one should attempt a static build on GNU Linux.

3) For the testing do not use OCL (Orocos Components Library) Deployer as Rock does not use it. Remove any references to OCL.

I would also like to share my correspondences with my Rock co-mentors to give a better idea to anybody who would like to go into the details of the errors I faced.Here is the link. 

That will be all the work that is done on this project. I hope this blog post can be enough for anyone who wants to start on this project.I will summarise by providing a Getting Started on the porting Rock to RTEMS:

1) Before starting complete the RTEMS Hello World Examples.

2) Complete the Rock Basic tutorials to get a basic idea about Rock.

3) Next try to build a manual(handwritten) component as well as an Orogen component by studying RTT Manuals.

4) Go through this blog post to get a basic idea of what is the general workflow and work according to progress of the project (this is the current status, I will be updating this blog as I progress).

Please note, there are two other porting Rock to RTEMS. One is the original ESA project whose scripts I have  provided here.I have started my work on this. There is however another project by Matthias Goldhoorn.However many of the codes are in private repositories,and moreover Matthias Goldhoorn is not a part of Rock anymore since we spoke last.I did not understand the changes in buildconf.I welcome others to give a try by this approach.

5)Now you can start the port preferably from external dependencies.Try replicating Phase-1 commits and then proceed to RTT port (or the latest work).

Thank you for patient reading :)

Tests on RTT

There has been a lot of delay between this post and the last one.It is mostly because I don't have enough results to showcase. The patches for the updated scripts can be found in my github repository

https://github.com/Sambeet161616/Rock-Port_ana

By my last post I hoped to complete tests on RTT. For the purpose of this I created a hello world component.However the major problem with RTT is, RTT  is too complex to be used by itself,so it uses various code generators like Orogen,which should be ported at a later stage.The Rock porting should take place as per the dependency order rtt,typelib,then orogen followed by base packages (base/orogen,base/cmake,base/types).

I am in the very preliminary stage of Rock porting i.e RTT. Testing is what caused a lot of problems for me.The patches in RTT to enable static building caused linking errors which could not be resolved till now.

Presently I am focused on creating RSB recipes for all the builds which I created.

To start working on this port:
1) Run the following bash scripts boost_cross_compiler.sh, eigen_cross_compiler.sh, env.sh, libxml.sh, OmniORB_cross_compiler.sh, prepare_rtems_mod.sh,xerces_c_cross_compiler.sh (don't run prepare_rock.sh,it is not yet ready).

2) Now you will be ready to port RTT to RTEMS. Clone the following
https://github.com/orocos-toolchain/rtt and apply the patches from

https://github.com/Sambeet161616/Rock-Port_ana/tree/master/files/rock/UpdatedPatches

Patch RTT and write a HelloWorld component to test RTT such as the given

https://github.com/Sambeet161616/Rock-Port_ana/tree/master/Tests

If it works RTT is successfully ported.However this is where I am facing Linking failures.

I am still in the process of debugging.Still it is the general method upto now.I will make updates as required.

Progress report for Mid term evaluations

This is a report on the mid term progress for my project.For reading through the report,it will be helpful if I provide a link to my original proposal

https://docs.google.com/document/d/1Ntfj3X4Tplmpxg7yGkojk5YSuNOjZB7q_2I3tSuf7r4/edit?usp=sharing

1)Planned goals vs Real Progress

(a) Planned goals as per my proposal:

By this time phase 1 of my project should have been completed.Phase 1 included the following things:

• Providing RTEMS support in the Orocos-toolchain/Real-time toolkit (rtt) (i386 and SPARC
• Creating a Orocos component using the Orogen template and adding RTEMS support there for i386 and SPARC architecture-
• Build the component using autoproj making necessary changes and patching typelib and rtt-
• Cross-compiling other necessary libraries like OmniOrb(for CORBA support),Xerces(Parser) and linking them to the component designed.

(b) Real Goals Attained

While framing the proposal there was less clarity about the  Orocos toolchain. The first three parts are essentially one big chunk and the next one is other. I have prepared the the first part i.e added rtems support to Orocos toolchain and the third part i.e preparing rock dependencies. However I have still not built up the component.I have tested the dependencies initially but further modifications needs to be carried out which would be clear as I build the component.  

2) Reasons for divergence

The main reason for divergence is it took me time to understand the RTT build system.There were references from the previous port that were greatly helpful.But the RTT part of the port took some time.With help from the mentors, I finally made the patch required.It is a delay now but I think gaining a complete understanding now would help me work faster for the remaining parts. Clearing this one big lag will help clearing many more small lags in the future.

3) Updated Plan

I think the workflow I gave in my proposal is definitely achievable. I will proceed as per my plans. I am a little slow now but I am positive that I could make up for the delay,

4) Code produced so far

The dependency scripts from the previous port have been modified to build but.The updated patch on RTT is also ready. I am waiting for the building up the component so that I can be sure the changes I made work.I will then write a blog post.

5) Current Blocking Points

I have worked with QEMU in the past to simulate for RTEMS. I am now finding a way out to build the Rock component I created for i386 on QEMU.

6) Realistic Goals to be achieved

Based on the work up to now,I think I don't need to make changes with the goals I had set. It might take more or less time ,but definitely can be done.

7) External Help expected

The three organisations OmniORB, RTEMS and Rock communities have been very helpful.Within the scope of the project I don't think I need more help.

  

OmniORB finally worked

It took me nearly a week to get a basic idea of CORBA.

The Definition

In summary of it CORBA stands for Common ORB Architecture.Common Architecture means a technical standard. ORB stands for object request broker.It is an object oriented version of Remote Procedure Call.An ORB is basically a mechanism for invoking operations on an object in a different process. Basically CORBA is a technical standard that governs the behavior of an ORB - how it should go about doing its work.

The ORB definition looks pretty normal - different process- no big deal. But what make it special is that the different process can reside in an entirely different address space ,on a different network on a different operating system and even in a different language too.This is where distributed computing comes into play,i.e. two entirely different systems (different processor,different memory) working to solve the same problem communicating with each other via Inter Process Communication channels.

For CORBA to come into play, there needs to be a server, a process which contains objects and client, a process which wants objects ( Do not confuse this with the regular definitions of server and client!). The objects involved in the process are written in a different language called as IDL (Interface Definition Language). It is  another object oriented language very similar to C++ and JAVA. And coding with IDL should not be difficult for anyone who has a good hand over any object oriented language.

How does CORBA work?
 CORBA objects are written in IDL.The IDL code is however not used by itself. The IDL code needs to be passed through an IDL compiler to generate the language mapping to any of the popular object oriented languages of your choice. It basically depends on the vendor od the ORB. But language mapping are there for most popular languages e.g. C++, Python, JAVA,COBOL etc.

CORBA generates two major codes a stub code for the client and a skeleton code for the server. The client process invokes operations on the stub code which gives output to it. The stub code has the mechanism to marshal the details of the invocation via a stringified object reference (IOR - Inter Operable Object Reference)and communicate via IPC channels with the skeleton code. The skeleton code also does not contain the real language implementation of the object. The skeleton code unmarshals the details of invocation and passes it to the server mainline code which produces the real implementation of the object known as servants and thus the operation is invoked.

 This is a very high level overview of CORBA. For more details please refer to

http://www.ciaranmchale.com/corba-explained-simply

And for an even better understanding please do download any ORB and run a few tutorial example. For C++ programmers, Omni ORB is god to start. It has got a very easy to understand documentation and a tutorial example called echo.I highly recommend this for beginners who do not have any experience with CORBA.

http://www.omniorb-support.com/

I have setup an Omni ORB environment and tested the echo example to verify its working correctly.

My next step would be to directly analyze the scripts from the previous port and make the necessary changes. I would go over RTT in detail if required later. For now it is really important to maintain focus on the real work at hand. I have done the Rock tutorials which serves as a good enough introduction to RTT. So i don't think I should face much problems in the work ahead.

Progress with OmniORB

Its been a week since the last blog post. But there is very less progress.I have not been able to set up the naming service still.I have not yet ready to build CORBA examples.

I have spent this week learning various aspects of CORBA from the popular

http://www.ciaranmchale.com/corba-explained-simply

I have learned about how does CORBA work about interfaces and modules and IDL coding language.It is written very simply and I have found it very helpful.
Before adding RTEMS support to Orocos I have to port OmniORB to RTEMS. Again there are scripts and instructions for it but the port is not maintained since a long time.

I will proceed to it once I set up the environment and tried out some CORBA examples.

Omni ORB creating problems

Omni ORB is the Rock's Object Request Broker. What it basically does among many other things is to allow objects to interact in different address spaces irrespective of architecture,operating systems or even in the network. Rock relies on Omni ORB for its naming service and there are no other options available.Further Omni ORB is relatively easy to port and has already been tested with RTEMS.That makes my job even easier.  :)

My problem statement now is to understand the naming service of Omni ORB. Now I have installed Omni ORB but I could not configure the naming service. I would be spending a lot of time on this tomorrow.

As per my plan after this I would start of phase-1 i.e adding RTEMS support to Orocos RTT. There's still time in the community bonding period. So I am on schedule :).

libxml and gccxml-testing the functionality

Rock uses a lot of dependencies.Of them the following were used in the old port:

1. Boost
2. Eigen
3. libxml
4. omniORB
5. Xerces

Role of each dependency should be understood for easier building.Boost is popular C++ library offering various functions integral to Rock.Eigen is used to implement matrices in C.

Xerces(C++) and libxml(C) are parsers for xml using C/C++. CastXML and GCCXML converts C code to XML. Both of the above parse th xml file so produced.So I tried to understand their functionality taking a few examples.The details are in the link below:

https://github.com/Sambeet161616/Rock-on-RTEMS-/tree/master/libxml_test

Remember the process is easy just compiling the libxml file will give problems.however the issue is solved.

https://bugs.launchpad.net/ubuntu/+source/libxml2/+bug/1032639

The readme in the git repo describes all the steps.
This is done. I successfully parsed a xml document.

Next in the line is OmniORB and and RTT.