Persistence of annotated worlds

Persistence of annotated worlds#

The semantic digital twin comes with an ORM attached to it that is derived from the python datastructures. The ORM can be used to serialize entire worlds into an SQL database and retrieve them later. The semantic annotations are stored alongside the kinematic information. The queried worlds are full objects that can be reconstructed into the original objects without any problems. The resulting SQL databases are perfect entry points for machine learning.

Concepts used:

Let’s go into an example where we create a world, store it, retrieve and reconstruct it.

First, let’s load a world from a URDF file.

import logging
import os
from importlib.resources import files
from pathlib import Path

from sqlalchemy import select
from sqlalchemy.orm import Session

from krrood.ormatic.utils import create_engine
from krrood.ormatic.data_access_objects.helper import to_dao

from semantic_digital_twin.adapters.urdf import URDFParser
from semantic_digital_twin.orm.ormatic_interface import *
from semantic_digital_twin.semantic_annotations.semantic_annotations import Table


logging.disable(logging.CRITICAL)
# set up an in memory database
engine = create_engine('sqlite:///:memory:')
session = Session(engine)
Base.metadata.create_all(bind=session.bind)

# load the table world from urdf
urdf_dir = os.path.join(Path(files("semantic_digital_twin")).parent.parent, "resources", "urdf")
table = os.path.join(urdf_dir, "table.urdf")
world = URDFParser.from_file(table).parse()

Sapien library is required for Partnet Mobility dataset loading. Please install it using 'pip install -U 'sapien>=3.0.0b1'

robocasa/robosuite are required for RoboCasa dataset loading. Install robosuite from git ('pip install git+https://github.com/ARISE-Initiative/robosuite.git') and then robocasa ('pip install -e .' from a clone of https://github.com/robocasa/robocasa), then run 'python -m robocasa.scripts.download_kitchen_assets' to fetch the fixture/object assets.

Next, we create a semantic annotation that describes the table.

table_semantic_annotation = Table(root=[b for b in world.bodies if "top" in str(b.name)][0])
with world.modify_world():
    world.add_semantic_annotation(table_semantic_annotation)
print(table_semantic_annotation)

Table(name=PrefixedName('None/Table'), id=UUID('87228a45-7c09-4e1c-8fe2-34daf17db390'), root=Body(name=PrefixedName('table/top'), id=UUID('d9e5bdb8-2739-43dd-ac01-467bbec9ad83'), index=5), objects=[], supporting_surface=None)

Now, let’s store the world to a database. For that, we need to convert it to its data access object which than can be stored in the database.

dao = to_dao(world)
session.add(dao)
session.commit()

We can now query the database about the world and reconstruct it to the original instance. As you can see the semantic annotations are also available and fully working.

queried_world = session.scalars(select(WorldMappingDAO)).one()
reconstructed_world = queried_world.from_dao()
table = [semantic_annotation for semantic_annotation in reconstructed_world.semantic_annotations if isinstance(semantic_annotation, Table)][0]
print(table)
print(table.sample_points_from_surface(amount=2))

Table(name=PrefixedName('None/Table'), id=UUID('87228a45-7c09-4e1c-8fe2-34daf17db390'), root=Body(name=PrefixedName('table/top'), id=UUID('d9e5bdb8-2739-43dd-ac01-467bbec9ad83'), index=5), objects=[], supporting_surface=None)
[Point3([-0.415287, -0.492845, 0.0025, 1]), Point3([-0.386093, -0.462309, 0.0025, 1])]

Maintaining the ORM 🧰#

You can maintain the ORM by maintaining the generate_orm.py. In there you have to list all the classes you want to generate mappings for and perhaps some type decorators for advanced use cases. Whenever you write a new dataclass that should appear or has semantic meaningful content make sure it appears in the set of classes. Pay attention to the logger during generation and see if it understands your datastructures correctly.

The sharp bits 🔪#

The world class manages the dependencies of the bodies in the world. Whenever you retrieve a body or connection, it comes as a data access object that is disconnected from the world itself. The relationships to the world exist and can be joined. However, when you reconstruct something else but the world, the reconstructed object does not have a world available. You can always reconstruct the entire world by querying for the objects world instead.

Accessing a permanent database#

This tutorial used an in memory database for the purpose of demonstration. If you want to permanently store worlds, you have to

  • Install an RDBMS that is supported by SQLAlchemy. (I recommend PostgreSQL)

  • Create a user and database in your RDBMS, for instance with this script. The script contains the documentation on how to run itself.

  • Set the environment variable SEMANTIC_DIGITAL_TWIN_DATABASE_URI to the connection string of your RDBMS, for instance by adding export SEMANTIC_DIGITAL_TWIN_DATABASE_URI=postgresql://semantic_digital_twin:a_very_strong_password_here@localhost:5432/semantic_digital_twin to your bashrc.

  • Create a session for database interaction, for instance with semantic_digital_twin_sessionmaker()()