Finding the parts of an object

Finding the parts of an object#

An object often comes as a single mesh, while a robot needs its parts: the handle of a mug to grasp it, the legs of a table to avoid them. AnnotationPartFinder gives an annotation the parts its part-whole relationships call for, by finding them in the shape of its root body and splitting each one off into a body and annotation of its own.

from semantic_digital_twin.part_finding.annotation_part_finder import AnnotationPartFinder

parts = AnnotationPartFinder().find_parts(mug)
mug.handle  # the handle that was found, or still None

In a pipeline, the step FindParts does the same for every annotation of a type, such as FindParts(whole_type=Mug).

Only relationships the annotation has no part in yet are looked at. An optional part, such as a mug’s handle, and each of any number of parts, such as a table’s legs, is added only when the finder is at least minimum_confidence sure of it.

A relationship that holds a single part, such as a mug’s handle, gets the piece the finder is most sure of. Annotations that may have several parts of a kind hold them in a list, such as a pot’s handles, and get each piece found as a part of its own.

Collision shapes#

The parts are cut out of the visual shapes, and the pieces collide as they look. A body whose collision shapes differ from its visual shapes, such as the convex decompositions most RoboCasa objects come with, is not split, since that would throw its collision shapes away. Either find the parts before decomposing, so that the decomposition runs on each piece:

from semantic_digital_twin.pipeline.mesh_decomposition.coacd import COACDMeshDecomposer

AnnotationPartFinder().find_parts(mug)
COACDMeshDecomposer().apply(mug._world)

or let the finder decompose the pieces again:

AnnotationPartFinder(decomposer=COACDMeshDecomposer()).find_parts(mug)

How the parts are found#

The parts are found by a combination of Qwen3-VL and SAM 2.1. In a benchmark on 93 kitchen objects this found 78 of 87 handles and marked nothing on any of six cups without one, at about 5 seconds per object on an A40.

Every setting is a field of VisionLanguagePartLocator and the classes it is made of.

Where the models run#

By default AnnotationPartFinder asks the part finding server that runs on an internal server of the institute, which needs nothing beyond this package. Set the environment variable SEMANTIC_DIGITAL_TWIN_PART_FINDING_SERVER_URL to its address, which you get from the institute, or give a server directly:

from semantic_digital_twin.adapters.part_finding.client import RemotePartLocator

AnnotationPartFinder(locator=RemotePartLocator(base_url="http://localhost:8790"))

The server keeps every answer, so the same question about the same shape is answered from its cache.

To run the models on your own GPU instead, install the vision extra, after installing the torch build that matches your driver’s CUDA version (nvidia-smi shows it):

uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
uv pip install -e "semantic_digital_twin[vision]"

from semantic_digital_twin.adapters.part_finding.vision_language_models import (
    QwenBoxDetector, QwenModel, Quantization, Sam2BoxSegmenter,
)
from semantic_digital_twin.part_finding.vision_language import (
    VisionLanguagePartLocator,
)

locator = VisionLanguagePartLocator(
    detector=QwenBoxDetector(model=QwenModel.QWEN3_VL_4B, quantization=Quantization.FOUR_BIT),
    segmenter=Sam2BoxSegmenter(),
)
AnnotationPartFinder(locator=locator)

The 8B model needs about 19 GB of GPU memory; the 4B model in 4 bits fits into 8 GB and found 73 of the 87 handles in the benchmark. On a CPU (device=Device.CPU) a single object takes minutes.

Running the server#

The server is the same locator behind http. Start it with semantic-digital-twin-part-finding-server (see --help for the model, quantization, device, port and cache), or build the image in semantic_digital_twin/docker/part_finding. It checks no credentials, so run it only inside a trusted network.