CSV export of compounds that contain containers (might be) broken

[AlexanderFabisch]

Hi,

I am trying to convert pocolog files to some other format (probably something like MessagePack). In order to do that I read how the CSV output is generated in typelib. I might be wrong but I think there is a bug:

I am trying to read a RigidBodyState from memory in typelib's seralization format. A RigidBodyState consists of a timestamp and two strings at the beginning. Strings are containers of 1 byte signed integers. The following lines are extracted from the XML type definition in the logfile:

  <compound name="/base/samples/RigidBodyState_m" size="416" >
    <field name="time" type="/base/Time" offset="0">...</field>
    <field name="sourceFrame" type="/std/string" offset="8">...</field>
    <field name="targetFrame" type="/std/string" offset="16">...</field>
  ...
  </compound>

It seems like offset should be an indicator of where the serialized field is stored in the serialized compound. This is not possible, of course, because strings can have any length, even in the same logfile. That alone would be OK, if the real length of a string would be used to compute where the next field starts in the serialized representation of the RigidBodyState. Hoever, this information is not used, instead, the offset will be used directly:

        virtual bool visit_ (Compound const& type, Field const& field)
        {
            m_stack.push_back( m_stack.back() + field.getOffset() );
            bool ret = m_visitor.visit_(Value(m_stack.back(), field.getType()), type, field);
            m_stack.pop_back();
            return ret;
        }

I don't know how that should work. Do you have any idea? Is there a good way to work around this issue? Is there a way to fix this? I would be happy to contribute.

[AlexanderFabisch]

@doudou maybe you have an idea?

[doudou]

@doudou maybe you have an idea?

The serialization format and the XML representations differ. What one can do is infer the latter from the former. Moreover, there is no current API that allow you to compute the offset of a field in the serialized data (it is possible, but not there).

The authoritative serialization information is the memory layout. You can visualize it from within the Ruby bindings with

Orocos.load
Orocos.load_typekit 'base'
puts Types.base.samples.RigidBodyState.memory_layout

which gives for RigidBodyState_m

FLAG_MEMCPY 8
FLAG_CONTAINER #<Typelib::ContainerType: /std/string>
  FLAG_MEMCPY 1
FLAG_END
FLAG_CONTAINER #<Typelib::ContainerType: /std/string>
  FLAG_MEMCPY 1
FLAG_END
FLAG_MEMCPY 392

Containers in the memory layout are represented as a type code and the memory layout of the container element. In the marshalled stream, it translates to the container size (64 bits) and then each marshalled container elements. Anything else is copied as-is. Alignment bytes are kept when marshalled into pocolog.

So you have two choice if you want to convert typelib-marshalled data into msgpack:

• make something that interprets the type info, memory layout and marshalled data and does the conversion. The good way to do that is to follow the memory layout example: create a stream that describes the operations that must be done and then interpret that stream. Interpreting the type model using a visitor for each value that you want to convert is slow as hell (this is why the whole memory layout thing exists in the first place)
• deserialize the data first and create a msgpack marshaller (using the same "op stream" method)

[AlexanderFabisch]

That was fast, thanks!