nicovision

The nicovision package contains classes to control the audio interface of the NICO.

VideoRecorder

The VideoRecorder class manages the caption of videos from the cameras of the NICO robot. To choose between different video codecs one can use the VideoCodec class

class nicovision.VideoRecorder.VideoCodec

Bases: object

The VideoCodec class represents different video codecs.

DIVX = 3

DivX codec (Also known as MPEG-4)

H264 = 2

H.264 codec

MPEG1 = 1

MPEG-1 codec

XVID = 4

XVID codec

class nicovision.VideoRecorder.VideoRecorder(device='', colorspace=1, framerate=20, width=640, height=480, zoom=None, pan=None, tilt=None, settings_file=None, setting='standard', videoformat=1)

Bases: object

_callback(rval, frame)
get_color_space()

Returns the currently used colorspace

Returns

Colorspace

Return type

vision.Colorspace

get_frame_rate()

Returns the current frame rate

Returns

Framerate (frames per second)

Return type

int

get_resolution()

Returns the current resolution

Returns

(width, height)

Return type

tuple

get_video_format()

Returns the current video format

Returns

Video format

Return type

str

is_recording()

Returns true if the VideoRecorder is recording

Returns

True if recording

Return type

bool

load_settings(file_path, setting='standard')

Loads a settings json file and applies the given setting to all cameras

Parameters

  • file_path (str) – the settings file

  • setting (str) – name of the setting that should be applied

pan(value)

Sets pan (x-axis) value if camera supports it. Requires v4l-utils.

Parameters

value (int) – pan value between -648000 and 648000, step 3600

set_color_Space(colorspace)

Sets the current color space

Parameters

colorspace (vision.Colorspace) – Colorspace

set_frame_rate(framerate)

Sets the current framerate

Parameters

framerate (int) – Framerate (frames per second)

set_resolution(width, height)

Sets the current resolution

Parameters

  • width (int) – Width

  • height (int) – Height

set_video_format(format)

Sets the current video format

Parameters

format (VideoCodec) – video format

start_recording(folder, file, overwrite=True)

Starts the recording into folder/file. File has to have the extension ‘.avi’. Extension will be attached automatically.

Parameters

  • folder (str) – Target folder. Will be created if none existent and overwrite is set to true

  • file (str) – Target file name

  • overwrite (bool) – If set to False no files will be overwritten

stop_recording()

Stops the current recording

tilt(value)

Sets tilt (y-axis) value if camera supports it. Requires v4l-utils.

Parameters

value (int) – tilt value between -648000 and 648000, step 3600

zoom(value)

Sets zoom value if camera supports it. Requires v4l-utils.

Parameters

value (int) – zoom value between 100 and 800

nicovision.VideoRecorder.get_devices()

Returns a list containing the possible path of all video capturing devices

Returns

Video devices

Return type

list

ImageRecorder

The ImageRecorder class enables the capturing of single images from a camera.

class nicovision.ImageRecorder.ImageRecorder(device='', width=640, height=480, framerate=20, zoom=None, pan=None, tilt=None, settings_file=None, setting='standard', writer_threads=2, compressed=True, pixel_format='MJPG', calibration_file=None)

Bases: object

The ImageRecorder class enables the capturing of images from a camera.

_callback(rval, frame)

Internal callback

Parameters

  • rval – rval

  • frame – frame

camera_value(value_name, value)

Sets zoom value if camera supports it. Requires v4l-utils.

Parameters

value (int) – zoom value between 100 and 800

custom_callback(iso_time, frame)
enable_write(state=True)

Sets the writing to disk state

Parameters

state – Write enabled

load_settings(file_path, setting='standard')

Loads a settings json file and applies the given setting to all cameras

Parameters

  • file_path (str) – the settings file

  • setting (str) – name of the setting that should be applied

pan(value)

Sets pan (x-axis) value if camera supports it. Requires v4l-utils.

Parameters

value (int) – pan value between -648000 and 648000, step 3600

save_image_to(path)

Saves an Image to a given file

Parameters

path – Target file

Returns

True if successful

Return type

bool

save_one_image()

Saves an image to a unique file

Returns

Path (or ‘’ if an error occured

Return type

str

start_recording(path='picture-{}.png')
stop_recording(wait_for_writer=True)

Saves an Image to a given file

Parameters

wait_for_writer (bool) – Whether execution should be blocked until all images are written (WARNING if this is set to False wait_for_writer() needs to be called to avoid memory leaks)

tilt(value)

Sets tilt (y-axis) value if camera supports it. Requires v4l-utils.

Parameters

value (int) – tilt value between -648000 and 648000, step 3600

wait_for_writer()
zoom(value)

Sets zoom value if camera supports it. Requires v4l-utils.

Parameters

value (int) – zoom value between 100 and 800

nicovision.ImageRecorder.get_devices()

Returns a list containing the possible path of all video capturing devices

Returns

Video devices

Return type

list

MultiCamRecorder

The MultiCamRecorder class enables synchronized capturing of single images from multiple cameras at once.

class nicovision.MultiCamRecorder.MultiCamRecorder(devices=[], width=640, height=480, framerate=20, zoom=None, pan=None, tilt=None, settings_file=None, setting='standard', writer_threads=4, pixel_format='MJPG', calibration_file=None, undistortion_mode='mono')

Bases: object

The MultiCamRecorder class enables simultanious capturing of images from multiple cameras.

_callback(rval, frame, id)

Internal callback

Parameters

  • rval – rval

  • frame – frame

_eventloop(id)

Internal event loop

This will call all registered callbacks for each frame

_singleImageCallback(rval, frame, id)

Internal callback for taking a single image

Parameters

  • rval – rval

  • frame – frame

add_callback(function)

Adds a callback for the event loop

Whenever a new frame arrives, all registered callbacks are called.

The callback must take exactly 2 arguments: rval and frame. frame can be None if any error occures or no image could be grabbed.

Parameters

function (function) – Function to add as callback

camera_value(value_name, value)

Sets the a camera value over the v4l-utils. Run ‘v4l2-ctl -l’ for full list of controlls and values. Requires v4l-utils.

Parameters

  • value_name (str) – name of the value to set

  • value (int) – value to set

clean_callbacks()

Removes all saved callbacks

custom_callback(iso_time, frame, id)
enable_write(state=True)

Sets the writing to disk state

Parameters

state – Write enabled

get_zoom()

Returns zoom value of all cameras that support it. Requires v4l-utils.

Returns

value of zoom_absolute

Return type

int

load_callibration(file_path, undistortion_mode='mono')

Loads a calibration json file and prepares undistortion for all cameras

Parameters

  • file_path (str) – the calibration file

  • undistortion_mode (str) – mono or stereo undistortion

load_settings(file_path, setting='standard')

Loads a settings json file and applies the given setting to all cameras

Parameters

  • file_path (str) – the settings file

  • setting (str) – name of the setting that should be applied

open()
pan(value)

Sets pan (x-axis) value of all cameras that support it. Requires v4l-utils.

Parameters

value (int) – pan value between -648000 and 648000, step 3600

Returns

success

Return type

bool

start_recording(path='camera{}/picture-{}.png')

Starts continuous image recording.

Parameters

path (str) – path of the image. First {} is replaced with camera id, the second one with a timestamp.

stop_recording()

Stops the recording and closes the cameras

takeSingleImage(path='camera{}/picture-{}.png')

Takes a single image with each camera. For performance reasons this does not close the camera. Please call stop_recording() once if you don’t take any more images or want to delete the object.

Parameters

path (str) – path of the image. First {} is replaced with camera id, the second one with a timestamp.

tilt(value)

Sets tilt (y-axis) value of all cameras that support it. Requires v4l-utils.

Parameters

value (int) – tilt value between -648000 and 648000, step 3600

Returns

success

Return type

bool

undistort(frame, id)

Undistorts the frame with the loaded calibration

Parameters

  • frame (cv2 image) – frame to undistort

  • id (int) – camera id

Returns

undistorted frame

Return type

cv2 image

zoom(value)

Sets zoom value of all cameras that support it. Requires v4l-utils.

Parameters

value (int) – zoom value between 100 and 800

Returns

success

Return type

bool

nicovision.MultiCamRecorder.autodetect_nicoeyes()

Returns a tuple containing the detected path of left and right NICO eye camera

Returns

Devicenames as (left, right) tuple (None if not found)

Return type

tuple

nicovision.MultiCamRecorder.get_devices()

Returns a list containing the possible path of all video capturing devices

Returns

Video devices

Return type

list

PyrepRecorder

The PyrepRecorder class allows recording images and videos from CoppeliaSim vision sensors using pyrep.

class nicovision.PyrepRecorder.PyrepRecorder(sensor_names=[], width=640, height=480, near_clipping_plane=0.01, far_clipping_plane=10.0, view_angle=60.0)

Bases: object

PyrepRecorder allows recording images and videos from CoppeliaSim vision sensors using pyrep.

static autodetect_nicoeyes()

Returns a tuple containing the detected names of left and right eye sensor

Returns

Devicenames as (left, right) tuple (None if not found)

Return type

tuple

create_image_callback(sensor_name, filename, timestamp_format='%Y-%m-%d_%H%M%S%f')

Creates a callback to save each frame as an image.

Parameters

  • sensor_name (str) – sensor_name that is inserted into the filename

  • filename (str) – filename with optional ‘sensor_name’ and ‘timestamp’ placeholders.

  • timestamp_format (str) – formatstring for the timestamp

Returns

the callback function

Return type

function

create_video_callback(writer)

Creates a callback to add each frame to a video.

Parameters

writer – videowriter to write the frames

Returns

the callback function

Return type

function

preprocess_image(frame)

Can be overriden to preprocess each image.

Parameters

frame (cv2.image) – the frame to preprocess.

Returns

the preprocessed frame

Return type

cv2.image

property sensor_names
Returns

names of the recorded sensors

Return type

str

start_image_recording(filename='{sensor_name}_image_{timestamp}.png', timestamp_format='%Y-%m-%d_%H%M%S%f')

Starts recording which saves a png image at each simulation step.

Parameters

  • filename (str) – filename with optional ‘sensor_name’ and ‘timestamp’ placeholders.

  • timestamp_format (str) – formatstring for the timestamp

start_video_recording(filename='{sensor_name}_video_{timestamp}.avi', fourcc='DIVX', timestamp_format='%Y-%m-%d_%H%M%S%f')

Starts video recording that records each simulation step.

Parameters

  • filename (str) – filename with optional ‘sensor_name’ and ‘timestamp’ placeholders.

  • fourcc (str) – The fourcc of the video codec.

  • timestamp_format (str) – formatstring for the timestamp

step()

Triggers step method of all recorded sensors

stop_image_recording()

Stops recording of images.

stop_video_recording()

Stops the video recording.

take_one_image(filename='{sensor_name}_image_{timestamp}.png', timestamp_format='%Y-%m-%d_%H%M%S%f')

Takes a single image at the current simulation step.

Parameters

  • filename (str) – filename with optional ‘sensor_name’ and ‘timestamp’ placeholders.

  • timestamp_format (str) – formatstring for the timestamp

CameraCalibrator

CameraCalibrator allows to calibrate the cameras with a chessboard pattern

class nicovision.CameraCalibrator.CameraCalibrator(devices=[], cam_width=640, cam_height=480, window_width_per_cam=640, window_height=480, framerate=20, zoom=None, pan=None, tilt=None, settings_file=None, setting='standard', pixel_format='MJPG')

Bases: object

CameraCalibrator allows to calibrate the cameras with a chessboard pattern

_calibrate_mono(id)

Generates calibration parameters from image points recorded with the device at the given index in device ids (NOT the actual device id)

Parameters

id (int) – position of device in list of deviceIds (NOT the actual device id)

_calibrate_stereo()

Generates calibration parameters from image points recorded with stereo cameras

_callback(rval, frame, id)
_display_thread(stop_event, window_width_per_cam, window_height)
start_calibration(chessboard=(11, 8), duration=30, number_of_samples=100, stereo=True, calibration_file='/home/nico/.NICO-python3/lib/python3.8/site-packages/nicovision/../../../../../json/nico_vision_calibration_params.json', overwrite=False, term_criteria=(3, 30, 0.1), calibration_flags=14)

Records images for given duration to calibrate cameras.

Calibration related code was partially taken from: https://medium.com/@kennethjiang/calibrate-fisheye-lens-using-opencv-333b05afa0b0 and https://github.com/sourishg/fisheye-stereo-calibration/blob/master/calibrate.cpp (stereo term_criteria and fov_scale)

Parameters

  • chessboard (tuple(int)) – Dimensions of the chessboard pattern (inner corners)

  • duration (int) – Duration of the recording

  • number_of_samples (int) – Subset of images used for calibration (to reduce computing time)

  • stereo – Whether cameras should be calibrated individually or as stereo cameras

  • calibration_file (str) – json file path to save calibration parameters

  • overwrite (bool) – Whether preexisting parameters in the file should be overwritten

  • term_criteria (list) – cv2 term_criteria for calibration

  • calibration_flags (list) – cv2 calibration_flags

nicovision.CameraCalibrator.takespread(sequence, num)

Display

The Display class displays multiple cameras in one window

class nicovision.Display.Display(devices=[], cam_width=640, cam_height=480, window_width_per_cam=640, window_height=480, framerate=20, zoom=None, pan=None, tilt=None, settings_file=None, setting='standard', pixel_format='MJPG', calibration_file=None, undistortion_mode='mono')

Bases: object

The Display Class displays multiple cameras in one window

_callback(rval, frame, id)
_display_thread(stop_event, window_width_per_cam=640, window_height=480)
close()

Closes the display

open(window_width_per_cam=640, window_height=480)

Opens the display :param window_width_per_cam: Adjusted width of the displayed image(s) :type window_width_per_cam: float :param window_height: Adjusted height of the displayed image(s) :type window_height: float

Helper classes

Barrier

The Barrier class provides a Python 2 compatible barrier for thread synchronization

class nicovision.Barrier.Barrier(parties)

Bases: object

Barrier for thread synchronization in python 2 similar to https://docs.python.org/3/library/threading.html#barrier-objects partially taken from https://greenteapress.com/wp/semaphores/ p.44.

abort()

Breaks the barrier and resumes all waiting threads. This leaves the barrier in a broken state, future calls will raise a BrokenBarrierError.

phase1()

First phase of the barrier which blocks incoming calls until all threads have arrived.

phase2()

Second barrier to ensure that no thread can reenter phase 1 until all other threads have left it.

reset()

Aborts and resets the barrier to its intitial state. This will resume all waiting threads.

wait()

Blocks until all threads have called this method.

exception nicovision.Barrier.BrokenBarrierError

Bases: RuntimeError

Colorspace

The Colorspace class represents different colorspaces.

class nicovision.Colorspace.Colorspace

Bases: object

The colorspace class represents different color spaces

RGB = 1

ImageWriter

The ImageWriter class saves image with multiple worker threads to allow capturing high resolution images at a high frequency

class nicovision.ImageWriter.ImageWriter(workers=2, write_enabled=True)

Bases: object

Multithreaded image writer for high resolution image processing

_worker_thread()
close()

Finishes writing process and stops all worker threads

enable_write(state=True)
open()

Initializes and starts all worker threads

write_image(path, image)

Adds image to queue to be written to path by a worker thread

PyrepDevice

The PyrepDevice class handles communication with CoppeliaSim Vision Sensors via Pyrep. It allows registering callback functions which are executed at each simulation step.

class nicovision.PyrepDevice.PyrepDevice(sensor_name, width=640, height=480, near_clipping_plane=0.01, far_clipping_plane=10.0, view_angle=60.0)

Bases: object

PyrepDevice handles communication with CoppeliaSim Vision Sensors via Pyrep. Retrieved images are converted to cv2 format and passed to registered callback functions.

add_callback(function)

Adds a callback for the event loop

Whenever step is called, all registered callbacks are executed.

The callback must take exactly 1 argument through which it receives the frame.

Parameters

function (function) – Function to add as callback

Returns

id of the added callback

Return type

int

static autodetect_nicoeyes()

Returns a tuple containing the detected names of left and right eye sensor

Returns

Devicenames as (left, right) tuple (None if not found)

Return type

tuple

clean_callbacks()

Removes all saved callbacks

property far_clipping_plane
Returns

the maximum distance from which the sensor will be able to detect.

Return type

float

get_image()

Captures an image for the current simulation step from the vision sensor and converts it to cv2 format

Returns

the image

Return type

cv2 image

property near_clipping_plane
Returns

the minimum distance from which the sensor will be able to detect.

Return type

float

remove_callback(id)

Removes callback with given id

Parameters

id (int) – id of the callback

property resolution
Returns

camera resolution (width, height)

Return type

tuple(int, int)

property sensor_name
Returns

name of the sensor in the scene

Return type

str

step()

Executes all registered callbacks with the sensor image for the current simulation step.

property view_angle
Returns

field of view of the camera in degree.

Return type

float

class nicovision.PyrepDevice.StepListener

Bases: object

StepListener is a static class that allows to register objects as listeners whose step functions are executed after each simulation step.

_subscribers = []
static register(obj_wref)

Registers an weak-referenced object as a listener. The object needs to have a step() method, which is called each time a simulation step is executed.

Parameters

obj_wref (weakref.ref) – weakref to register as listener. Needs to have a step method.

static step()

Executes the step() method of all registered objects.

static unregister(obj_wref)

Removes a registered object from listeners

Parameters

obj_wref (weakref.ref) – weakref to remove as listener

nicovision.PyrepDevice.step_with_listeners(self)

Triggers StepListener step function after executing simulation step

VideoDevice

The VideoDevice class handles low-level communication with the video capture devices.

class nicovision.VideoDevice.VideoDevice(id, framerate=20, width=640, height=480, zoom=None, pan=None, tilt=None, settings_file=None, setting='standard', compressed=True, pixel_format='MJPG', calibration_file=None)

Bases: object

The VideoDevice class handles low-level communication with the video devices

_VIDEO_DEVICE_PATH = '/dev/v4l/by-id/'

This variable holds the base path for the ids of the capturing devices

_eventloop()

Internal event loop

This will call all registered callbacks for each frame

add_callback(function)

Adds a callback for the event loop

Whenever a new frame arrives, all registered callbacks are called.

The callback must take exactly 2 arguments: rval and frame. frame can be None if any error occures or no image could be grabbed.

Parameters

function (function) – Function to add as callback

static autodetect_nicoeyes()

Returns a tuple containing the detected path of left and right NICO eye camera

Returns

Devicenames as (left, right) tuple (None if not found)

Return type

tuple

camera_value(value_name, value)

Sets the a camera value over the v4l-utils. Run ‘v4l2-ctl -l’ for full list of controlls and values. Requires v4l-utils.

Parameters

  • value_name (str) – name of the value to set

  • value (int) – value to set

clean_callbacks()

Removes all saved callbacks

close()

Stopps the eventloop and closes the device

static from_device(device, framerate=20, width=640, height=480, zoom=None, pan=None, tilt=None, settings_file=None, setting='standard', compressed=True, pixel_format='MJPG', calibration_file=None)

Convenience method for creating a VideoDevice from a device

Parameters

device (str) – device

Returns

VideoDevice or None if path is not valid / ambiguous

Return type

VideoDevice or None

static get_all_devices()

Returns a list containing the possible path of all video capturing devices

Returns

Paths of video devices

Return type

list

get_framerate()

Returns the current frame rate

Returns

Framerate (frames per second)

Return type

int

get_resolution()

Returns the current resolution

Returns

(width, height)

Return type

tuple

get_zoom()

Returns zoom value if camera supports it. Requires v4l-utils.

Returns

value of zoom_absolute

Return type

int

is_open()

Checks if the VideoDevice is open

Returns

True if VideoRecorder is open

Return type

bool

load_callibration(file_path)

Loads a calibration json file and prepares undistortion for all cameras

Parameters

file_path (str) – the calibration file

load_settings(file_path, setting='standard')

Loads a settings json file and applies the given setting to all cameras

Parameters

  • file_path (str) – the settings file

  • setting (str) – name of the setting that should be applied

open()

Opens the device and starts the eventloop

pan(value)

Sets pan (x-axis) value if camera supports it. Requires v4l-utils.

Parameters

value (int) – pan value between -648000 and 648000, step 3600

static resolve_device(device)

Returns the id of a device

Parameters

device (str) – device

Returns

device id (-1 = error)

Return type

int

set_framerate(framerate)

Sets the current framerate

Parameters

framerate (int) – Framerate (frames per second)

set_resolution(width, height)

Sets the current resolution

Parameters

  • width (int) – Width

  • height (int) – Height

tilt(value)

Sets tilt (y-axis) value if camera supports it. Requires v4l-utils.

Parameters

value (int) – tilt value between -648000 and 648000, step 3600

undistort(frame)

Undistorts the frame with the loaded calibration

Parameters

frame (cv2 image) – frame to undistort

Returns

undistorted frame

Return type

cv2 image

zoom(value)

Sets zoom value if camera supports it. Requires v4l-utils.

Parameters

value (int) – zoom value between 100 and 800