Skip to main content

Overview

Once you know which actions are available, you can submit them to the environment using the step() method on the EnvironmentWrapper. This method also supports including reasoning logs with each action.

Submitting a simple action

Use the step() method to submit an action:

Submitting a complex action

Complex actions require additional data, such as x,y coordinates:

Including reasoning logs

The step() method accepts an optional reasoning parameter that allows you to include reasoning metadata with each action. This is useful for tracking your agent’s thought process in recordings.
The reasoning dictionary can contain any key-value pairs you want to track. Common fields include:
  • thought: A description of why the action was chosen
  • confidence: A confidence score for the action
  • alternatives_considered: Other actions that were considered
  • reasoning_tokens: Token count from reasoning models

Checking the result

After submitting an action, check the returned FrameDataRaw object:

Next steps