Tabletop scenes and instructions#

A one-dimensional tabletop and a tiny instruction language for the one-motor VLA.

Classes

Scene

One episode: an image strip, an instruction, and the ground truth behind them.

Functions

`cell_centres`
`encode`
`sample_scene`	Draw a scene with 2-3 objects, an instruction, and the intended target.
`clarification`	The person's answer to "which one?": a word that singles out the intended target.

class Scene(image, text, cells, colors, valid, target, words)[source]#

Bases: object

One episode: an image strip, an instruction, and the ground truth behind them.

Variables:

image ((n_cells, 3) array) – Noisy one-hot colour strip; each object occupies one cell.
text ((len(WORDS),) array) – Multi-hot bag of instruction words.
colors (cells,) – Cell index and colour index of every object.
valid (list of int) – Cells the instruction is true of. More than one means the instruction is ambiguous.
target (int) – The cell the person actually means (drawn uniformly from valid).

Parameters:

image (ndarray)
text (ndarray)
cells (list)
colors (list)
valid (list)
target (int)
words (tuple)

cell_centres(n_cells)[source]#

encode(words)[source]#

sample_scene(rng, n_cells=16, pixel_noise=0.1, kinds=('color', 'leftmost', 'rightmost', 'block'))[source]#

Draw a scene with 2-3 objects, an instruction, and the intended target.

Scenes in which a middle object could not be named afterwards (same colour as another object, neither extreme) are redrawn, so a clarifying answer always exists.

clarification(scene, rng=None, n_cells=16, pixel_noise=0.1)[source]#: The person’s answer to “which one?”: a word that singles out the intended target.