Multimodal fusion#

Multimodal fusion for robots: from detector scores, speech and gaze to one belief over intents.

A request such as “bring me that cup” is resolved from several sources: an object detector’s scores, the speech recogniser’s n-best list, where the person looks and where they point. This module turns each source into a likelihood over the same hypotheses and offers three ways to combine them:

bayes_fusion(): independent cues multiply (order-free);
dempster_shafer_fusion(): Dempster’s rule with discounted sources, which keeps an explicit “don’t know” mass when sources conflict;
QuantumIntentResolver: quantum-like fusion in which incompatible cues are applied as rotated Kraus operators, so cue order matters (it reduces to Bayes when every incompatibility angle is zero); fit_incompatibility() estimates the angles from logged choices, and compare_fusion() compares the three rules on held-out data.

Adapters: detector_likelihood(), asr_likelihood(), direction_likelihood() (gaze or pointing).

Examples

>>> import numpy as np
>>> from quantum_mind.applications.fusion import detector_likelihood, direction_likelihood, bayes_fusion
>>> objects = ['red cup', 'blue cup', 'bowl']
>>> vision = detector_likelihood({'red cup': 0.6, 'blue cup': 0.5, 'bowl': 0.2}, objects)
>>> gaze = direction_likelihood([1, 0, 0], {'red cup': [0.9, 0.1, 0], 'blue cup': [0, 1, 0],
...                                         'bowl': [-1, 0, 0]}, objects, kappa=4)
>>> post = bayes_fusion([vision, gaze])
>>> objects[int(np.argmax(post))]
'red cup'

Functions

`detector_likelihood`	Likelihood over hypotheses from an object detector's scores.
`asr_likelihood`	Likelihood over hypotheses from a speech recogniser's n-best list.
`direction_likelihood`	Likelihood over hypotheses from a gaze or pointing direction (von Mises-Fisher).
`bayes_fusion`	Posterior of independent cues: prior times the product of the likelihoods.
`dempster_shafer_fusion`	Dempster's rule for singleton masses with discounting.
`fit_incompatibility`	Fit the incompatibility angle of each cue of a quantum-like resolver from logged choices.
`compare_fusion`	Compare Bayesian, Dempster-Shafer and fitted quantum-like fusion on held-out trials.

detector_likelihood(scores, hypotheses, temperature=1.0, floor=0.001)[source]#

Likelihood over hypotheses from an object detector’s scores.

Parameters:

scores (dict or array_like) – {hypothesis: score} (missing hypotheses score 0) or scores in the order of hypotheses. Scores are confidences in [0, 1] or logits.
hypotheses (sequence) – Hypotheses (object or intent names).
temperature (float, optional) – Softens (> 1) or sharpens (< 1) the scores before normalising.
floor (float, optional) – Added to every entry so that no hypothesis is ruled out by one cue.

Returns:

numpy.ndarray – Normalised likelihood, one entry per hypothesis.

asr_likelihood(nbest, keywords, hypotheses, floor=0.02)[source]#

Likelihood over hypotheses from a speech recogniser’s n-best list.

Parameters:

nbest (sequence of tuple) – [(transcript, confidence), ...].
keywords (dict) – {hypothesis: [words that refer to it]}.
hypotheses (sequence)
floor (float, optional)

Returns:

numpy.ndarray – Each transcript’s confidence is shared among the hypotheses it mentions (or spread over all hypotheses when it mentions none).

direction_likelihood(direction, targets, hypotheses, kappa=5.0, floor=0.001)[source]#

Likelihood over hypotheses from a gaze or pointing direction (von Mises-Fisher).

Parameters:

direction (array_like) – Direction vector (2D or 3D) from the person’s head or hand.
targets (dict) – {hypothesis: direction to that object} from the same origin.
hypotheses (sequence)
kappa (float, optional) – Concentration: larger values trust the direction more.
floor (float, optional)

Returns:

numpy.ndarray – \(\propto \exp(\kappa \cos \angle(d, t_h))\).

bayes_fusion(likelihoods, prior=None)[source]#

Posterior of independent cues: prior times the product of the likelihoods.

Parameters:

likelihoods (sequence of array_like) – One likelihood vector per cue.
prior (array_like, optional) – Prior over hypotheses (uniform by default).

Returns:

numpy.ndarray

dempster_shafer_fusion(likelihoods, reliabilities=None)[source]#

Dempster’s rule for singleton masses with discounting.

Each cue becomes a mass function on the single hypotheses, discounted by its reliability r: a fraction 1 - r of its mass goes to “any hypothesis” (ignorance). Conflict between cues is renormalised away by Dempster’s rule.

Parameters:

likelihoods (sequence of array_like) – One normalised likelihood per cue.
reliabilities (sequence of float, optional) – Reliability of each cue in [0, 1] (default 0.9).

Returns:

belief (numpy.ndarray) – Pignistic probability over hypotheses (ignorance shared equally).
ignorance (float) – Remaining mass on “any hypothesis”.

fit_incompatibility(resolver_cls, intents, cue_likelihoods, trials, prior=None, bounds=(0.0, 1.5707963267948966))[source]#

Fit the incompatibility angle of each cue of a quantum-like resolver from logged choices.

Parameters:

resolver_cls (type) – QuantumIntentResolver.
intents (sequence of str) – Intent names.
cue_likelihoods (dict) – {cue name: likelihood over intents}.
trials (sequence of tuple) – [(cue order, chosen intent), ...]: the order in which cues arrived and what the person actually wanted (or chose).
prior (array_like, optional) – Prior over intents.
bounds (tuple, optional) – Range of the angles.

Returns:

resolver (QuantumIntentResolver) – Resolver with the fitted angles.
angles (dict) – {cue name: fitted angle}.

compare_fusion(intents, cue_likelihoods, train, test, prior=None)[source]#

Compare Bayesian, Dempster-Shafer and fitted quantum-like fusion on held-out trials.

Parameters:

intents (sequence of str)
cue_likelihoods (dict) – {cue name: likelihood over intents}.
train (sequence of tuple) – [(cue order, chosen intent), ...]; the quantum-like angles are fitted on train.
test (sequence of tuple) – [(cue order, chosen intent), ...]; the quantum-like angles are fitted on train.
prior (array_like, optional)

Returns:

dict – {rule: {'log_loss': ..., 'accuracy': ...}} on test (lower log loss is better), plus 'angles'.