Semantic Annotations (Views)

Views are semantic annotations to world entities. For instance, they can be used to say that a certain body should be interpreted as a handle or that a combination of bodies should be interpreted as a drawer. Ontologies inspire views. The semantic world overcomes the technical limitations of ontologies by representing semantic annotations as python classes and use the typing system of python together with EQL for reasoning. This tutorial shows you how to apply views to the world and how to create your own views.

Used Concepts:

• Factories

• Creating Custom Bodies

• World Structure Manipulation

• Entity Query Language

First, let’s create a world containing a drawer.

from dataclasses import dataclass
from typing import List

from entity_query_language import entity, an, let, symbolic_mode

from semantic_world.datastructures.prefixed_name import PrefixedName
from semantic_world.spatial_types.spatial_types import TransformationMatrix
from semantic_world.views.factories import (
    DrawerFactory,
    ContainerFactory,
    HandleFactory,
    Direction,
)
from semantic_world.views.views import Drawer, Handle, Container
from semantic_world.world import World
from semantic_world.world_description.connections import Connection6DoF
from semantic_world.world_description.geometry import Sphere, Scale
from semantic_world.world_description.world_entity import View, Body
from semantic_world.spatial_computations.raytracer import RayTracer


world = DrawerFactory(
    name=PrefixedName("drawer"),
    container_factory=ContainerFactory(name=PrefixedName("container"), direction=Direction.Z),
    handle_factory=HandleFactory(name=PrefixedName("handle")),
).create()

print(*world.views, sep="\n")
rt = RayTracer(world)
rt.update_scene()
rt.scene.show("jupyter")

Container(name=PrefixedName(name='container', prefix=None), body=Body(name=PrefixedName(name='container', prefix=None), index=0, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None))
Handle(name=PrefixedName(name='handle', prefix=None), body=Body(name=PrefixedName(name='handle', prefix=None), index=1, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None))
Drawer(name=PrefixedName(name='drawer', prefix=None), container=Container(name=PrefixedName(name='container', prefix=None), body=Body(name=PrefixedName(name='container', prefix=None), index=0, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None)), handle=Handle(name=PrefixedName(name='handle', prefix=None), body=Body(name=PrefixedName(name='handle', prefix=None), index=1, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None)))

The annotations now proof useful when an agent needs to infer information from the world. For instance, an agent might want to open a drawer. Opening a drawer is done by grasping the drawer handle. Due to the semantic structure, it is easily possible to access this information for the agent by formulating a query like this

with symbolic_mode():
    handles = an(entity(let(Handle, world.views)))
print(list(handles.evaluate()))

[Handle(name=PrefixedName(name='handle', prefix=None), body=Body(name=PrefixedName(name='handle', prefix=None), index=1, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None))]

The interlinking of the semantic annotations is done via classes. This is very useful, to filter for more context in the world. For instance, consider a world that has a second handle that is attached to nothing and hence shouldn’t be used by the agent to open a drawer.

useless_handle = HandleFactory(name=PrefixedName("useless handle")).create()
rt = RayTracer(useless_handle)
rt.update_scene()
rt.scene.show("jupyter")
print(useless_handle.views)

with world.modify_world():
    world.merge_world_at_pose(
        useless_handle, TransformationMatrix.from_xyz_rpy(x=1.0, y=1.0)
    )
rt = RayTracer(world)
rt.update_scene()
rt.scene.show("jupyter")

[Handle(name=PrefixedName(name='useless handle', prefix=None), body=Body(name=PrefixedName(name='useless handle', prefix=None), index=0, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None))]

If we now evaluate the handle query, we see that multiple options exist.

print(*handles.evaluate(), sep="\n")

Handle(name=PrefixedName(name='handle', prefix=None), body=Body(name=PrefixedName(name='handle', prefix=None), index=1, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None))

We can refine the handle the agent wants by saying it must belong to a drawer

with symbolic_mode():
    drawer = let(Drawer, world.views)
    handle = let(Handle, world.views)
    result = an(entity(handle, drawer.handle == handle))
print(*result.evaluate(), sep="\n")

Handle(name=PrefixedName(name='handle', prefix=None), body=Body(name=PrefixedName(name='handle', prefix=None), index=1, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None))

Now we will shift the focus to creating new views. Views define what they are (is-a) through inheritance. Make sure you follow the (Liskov substitution principle)[https://en.wikipedia.org/wiki/Liskov_substitution_principle] when creating new views that inherit from existing views. Views define what they relate to via their attribute types. For instance, we can make an Apple view that classifies a body as an apple.

@dataclass
class Apple(View):
    """A simple custom view declaring that a Body is an Apple."""

    body: Body

    def __post_init__(self):
        # Give the view a default name if none was specified
        if self.name is None:
            self.name = PrefixedName(str(self.body.name), self.__class__.__name__)


world = World()
with world.modify_world():

    root = Body(name=PrefixedName("root"))

    # Create a body with spherical geometry
    apple_body = Body(name=PrefixedName("apple_body"))
    sphere = Sphere(
        radius=0.15, origin=TransformationMatrix(reference_frame=apple_body)
    )
    apple_body.collision = [sphere]
    apple_body.visual = [sphere]

    root_to_apple = Connection6DoF(parent=root, child=apple_body, _world=world)
    world.add_connection(root_to_apple)

    # Declare the body as an Apple view and add it to the world
    apple_view = Apple(body=apple_body, name=PrefixedName("apple"))
    world.add_view(apple_view)

print(world.views)
rt = RayTracer(world)
rt.update_scene()
rt.scene.show("jupyter")

[Apple(name=PrefixedName(name='apple', prefix=None), body=Body(name=PrefixedName(name='apple_body', prefix=None), index=1, collision_config=CollisionCheckingConfig(buffer_zone_distance=None, violated_distance=0.0, disabled=None, max_avoided_bodies=1), temp_collision_config=None))]

Views can become arbitrary complex. For instance, we can make a box of fruits.

@dataclass
class FruitBox(View):
    box: Container
    fruits: List[Apple]

    def __post_init__(self):
        if self.name is None:
            self.name = PrefixedName(str(self.box.name), self.__class__.__name__)


apple_body_2 = Body(name=PrefixedName("apple_body_2"))
sphere = Sphere(radius=0.15, origin=TransformationMatrix(reference_frame=apple_body_2))
apple_body_2.collision = [sphere]
apple_body_2.visual = [sphere]


with world.modify_world():
    bowl_world = ContainerFactory(
        name=PrefixedName("box"), direction=Direction.Z, scale=Scale(1.0, 1.0, 0.3)
    ).create()
    world.merge_world_at_pose(
        bowl_world,
        TransformationMatrix(),
    )

    root_to_apple2 = Connection6DoF(
        parent=root,
        child=apple_body_2,
        _world=world,
    )
    world.add_connection(root_to_apple2)
    world.state[root_to_apple2.x.name].position = 0.3

world.add_view(Apple(body=apple_body_2, name=PrefixedName("apple2")))
fruit_box = FruitBox(
    box=world.get_views_by_type(Container)[0], fruits=world.get_views_by_type(Apple)
)
world.add_view(fruit_box)
print(f"Fruit box with {len(fruit_box.fruits)} fruits")
rt = RayTracer(world)
rt.update_scene()
rt.scene.show("jupyter")

Fruit box with 2 fruits

One great quality of this is that every other agent who imports your definitions of views in the world is now able to understand what you mean and how you define it. Since everything is python, the processes won’t have any compatibility issues.