Environments with simulated people#

Human-in-the-loop environments for reinforcement learning (Gymnasium API).

The simulated people follow the package’s quantum-like models, so an agent trained here faces the effects that classical simulators leave out: answers that depend on the order of questions, and trust that changes when it is asked about. All people are simulated; the parameters are illustrative.

Every environment follows the Gymnasium interface: reset(seed=None) -> (observation, info) and step(action) -> (observation, reward, terminated, truncated, info).

Examples

>>> from quantum_mind.envs import ClarificationEnv
>>> env = ClarificationEnv()
>>> obs, info = env.reset(seed=0)
>>> obs.shape, int(env.action_space.n)
((3,), 6)
>>> obs, reward, terminated, truncated, info = env.step(0)      # ask question 0
>>> reward, terminated
(-0.3, False)

Classes

`ClarificationEnv`	Resolve an ambiguous request by asking questions, then act.
`TrustHandoverEnv`	Hand objects to a simulated person whose trust follows the trust qubit.

Functions

`register_envs`	Register the environments with Gymnasium.
`sample_people`	Draw simulated people whose parameters reflect what is known, and not known, about real people.

class ClarificationEnv(model=None, ask_cost=0.3, success_reward=1.0, error_cost=5.0, max_questions=6, people=None)[source]#

Bases: Env

Resolve an ambiguous request by asking questions, then act.

Actions 0 .. Q-1 ask a question; actions Q .. Q+H-1 act on a hypothesis (ending the episode). The observation has one entry per question: 0 if not asked yet, +1 for “yes”, -1 for “no”. The person’s answers follow a ProjectiveAnswerModel, so they depend on the questions asked before.

Parameters:

model (ProjectiveAnswerModel or IndependentAnswerModel, optional) – How the simulated person answers (default: default_domain()).
ask_cost (float, optional) – Reward penalty per question.
success_reward (float, optional) – Reward for acting on the right hypothesis.
error_cost (float, optional) – Penalty for acting on a wrong hypothesis.
max_questions (int, optional) – After this many questions the episode is truncated with the error cost.
people (list or callable, optional) – A population of answer models (same questions and hypotheses) or a function rng -> model; a new person is drawn at every reset().

reset(seed=None, options=None)[source]#

Start an episode with a new hidden intent.

Parameters:

seed (int, optional)
options (dict, optional) – {'target': hypothesis} fixes the hidden intent.

Returns:

observation (numpy.ndarray)
info (dict) – target (for evaluation only).

step(action)[source]#

Ask a question or act.

Parameters:: action (int)
Returns:: observation, reward, terminated, truncated, info

class TrustHandoverEnv(model=None, n_steps=20, fail_cost=10.0, slow_cost=1.0, ask_cost=0.5, wait_cost=0.3, slow_factor=0.5, people=None)[source]#

Bases: Env

Hand objects to a simulated person whose trust follows the trust qubit.

Actions: 0 hand over, 1 hand over slowly, 2 ask about trust, 3 wait. Observation: the last hand-over outcome (+1 success, -1 failure, 0 none), the last answer to the trust question (+1, -1, 0) and the fraction of the episode elapsed. A hand-over fails with probability 1 - P(trust); asking collapses the person’s trust as in OpenSystemBelief.

Parameters:

model (OpenSystemBelief, optional) – Trust model of the simulated person.
n_steps (int, optional) – Episode length.
fail_cost (float, optional) – Costs (rewards are their negatives); a successful hand-over gives +1.
slow_cost (float, optional) – Costs (rewards are their negatives); a successful hand-over gives +1.
ask_cost (float, optional) – Costs (rewards are their negatives); a successful hand-over gives +1.
wait_cost (float, optional) – Costs (rewards are their negatives); a successful hand-over gives +1.
slow_factor (float, optional) – Failure probability of a slow hand-over relative to a normal one.
people (list or callable, optional) – A population of trust models (for example from sample_people()) or a function rng -> model; a new person is drawn at every reset().

property p_trust#

the simulated person’s current P(trust).

Type:: float

reset(seed=None, options=None)[source]#

Start an episode at the model’s initial trust.

Parameters:

seed (int, optional)
options (dict, optional)

Returns:

observation (numpy.ndarray)
info (dict)

step(action)[source]#

Take one action.

Parameters:: action (int) – 0 hand over, 1 slow hand-over, 2 ask, 3 wait.
Returns:: observation, reward, terminated, truncated, info

register_envs()[source]#

Register the environments with Gymnasium.

Returns:: list of str – The registered ids (empty when Gymnasium is not installed).

sample_people(source, n, rng=None)[source]#

Draw simulated people whose parameters reflect what is known, and not known, about real people.

An agent trained against one simulated person learns that person. Drawing a new person for every episode from the uncertainty of the fitted human model (domain randomisation over people) trains a policy that holds up across the people the model is unsure about.

Parameters:

source (BootstrapResult, OnlinePersonModel, PopulationPrior or Model) – BootstrapResult (refits of a fitted model), OnlinePersonModel (particles weighted by the posterior), PopulationPrior (Gaussian on the free parameters), or a single model (returned n times).
n (int) – Number of people.
rng (numpy.random.Generator, optional)

Returns:

list of Model