moler.io.raw package¶

Submodules¶

moler.io.raw.memory module¶

External-IO connections based on memory buffer.

The only 3 requirements for these connections are: (1) store Moler’s connection inside self.moler_connection attribute (2) plugin into Moler’s connection the way IO outputs data to external world:

self.moler_connection.how2send = self.send

forward IO received data into self.moler_connection.data_received(data)

Logging inside ext-IO is mainly focused on connection establishment/close/drop. Data transfer aspects of connection are logged by embedded Moler’s connection.

class moler.io.raw.memory.FifoBuffer(moler_connection, echo=True, name=None, logger_name='')¶

Bases: moler.io.io_connection.IOConnection

FIFO-in-memory.:

inject             |\
         +---------+ \  read
write    +---------+ /
                   |/

Usable for unit tests (manually inject what is expected).

inject(input_bytes, delay=0.0)¶

Add bytes to FIFO with injection-delay before each data

Parameters:	input_bytes – iterable of bytes to inject delay – delay before each inject
Returns:	None

inject_response(input_bytes, delay=0.0)¶

Injection is activated by nearest write()

Parameters:	input_bytes – iterable of bytes to inject delay – delay before each inject
Returns:	None

name¶: Get name of connection

read(bufsize=None)¶: Remove bytes from front of buffer

receive(bufsize=None)¶: Remove bytes from front of buffer

send(input_bytes)¶: What is written to connection comes back on read() only if we simulate echo service of remote end.

write(input_bytes)¶: What is written to connection comes back on read() only if we simulate echo service of remote end.

class moler.io.raw.memory.ThreadedFifoBuffer(moler_connection, echo=True, name=None, logger_name='')¶

Bases: moler.io.raw.memory.FifoBuffer

FIFO-in-memory connection inside dedicated thread.

This is external-IO usable for Moler since it has it’s own runner (thread) that can work in background and pull data from FIFO-mem connection. Usable for integration tests.

close()¶: Stop pulling thread.

inject(input_bytes, delay=0.0)¶

Add bytes to end of buffer

Parameters:	input_bytes – iterable of bytes to inject delay – delay before each inject
Returns:	None

open()¶: Start thread pulling data from FIFO buffer.

pull_data(pulling_done)¶: Pull data from FIFO buffer.

moler.io.raw.sshshell module¶

External-IO connections based on Paramiko.

The only 3 requirements for these connections are: (1) store Moler’s connection inside self.moler_connection attribute (2) plugin into Moler’s connection the way IO outputs data to external world:

self.moler_connection.how2send = self.send

forward IO received data into self.moler_connection.data_received(data)

class moler.io.raw.sshshell.SshShell(host, port=22, username=None, login=None, password=None, receive_buffer_size=262144, logger=None, existing_client=None)¶

Bases: object

Implementation of ‘remote shell over Ssh’ connection using python Paramiko module

This connection is not intended for one-shot actions like execute_command of paramiko. It’s purpose is to provide continuous stream of bytes from remote shell. Moreover, it works with Pty assigned to remote shell to enable interactive dialog like asking for login or password.

close()¶

Close SshShell connection. Close channel of that connection.

If SshShell was created with “reused ssh transport” then closing will close only ssh channel of remote shell. Ssh transport will be closed after it’s last channel is closed.

classmethod from_sshshell(sshshell, logger=None)¶

Build new sshshell based on existing one - it will reuse its transport

No need to provide host, port and login credentials - they will be reused. You should use this constructor if you are connecting towards same host/port using same credentials.

Parameters:	sshshell – existing connection to reuse it’s ssh transport logger – new logger for new connection
Returns:	instance of new sshshell connection with reused ssh transport

open()¶

Open Ssh channel to remote shell.

If SshShell was created with “reused ssh transport” then no new transport is created - just shell channel. (such connection establishment is quicker) Else - before creating channel we create ssh transport and perform full login with provided credentials.

May be used as context manager: with connection.open():

receive(timeout=30.0)¶

Receive data.

Parameters:	timeout (float) – max time to await for data, default 30 sec

send(data, timeout=1)¶

Send data via SshShell connection.

Parameters:	data (bytes) – data timeout (float) – max time to spend on sending all data, default 1 sec

class moler.io.raw.sshshell.ThreadedSshShell(moler_connection, host, port=22, username=None, login=None, password=None, receive_buffer_size=262144, name=None, logger_name='', existing_client=None)¶

Bases: moler.io.io_connection.IOConnection

SshShell connection feeding Moler’s connection inside dedicated thread.

This is external-IO usable for Moler since it has it’s own runner (thread) that can work in background and pull data from SshShell connection.

close()¶

Close SshShell connection. Close channel of that connection & stop pulling thread.

If SshShell was created with “reused ssh transport” then closing will close only ssh channel of remote shell. Ssh transport will be closed after it’s last channel is closed.

classmethod from_sshshell(moler_connection, sshshell, name=None, logger_name='')¶

Build new sshshell based on existing one - it will reuse its transport

No need to provide host, port and login credentials - they will be reused. You should use this constructor if you are connecting towards same host/port using same credentials.

Parameters:	moler_connection – moler-connection may not be reused; we need fresh one sshshell – existing connection to reuse it’s ssh transport logger_name – new logger for new connection
Returns:	instance of new sshshell connection with reused ssh transport

name¶: Get name of connection

open()¶

Open Ssh channel to remote shell & start thread pulling data from it.

If SshShell was created with “reused ssh transport” then no new transport is created - just shell channel. (such connection establishment is quicker) Else - before creating channel we create ssh transport and perform full login with provided credentials.

May be used as context manager: with connection.open():

receive()¶

Pull data bytes from external-IO:

data = io_connection.receive()

data is intended to forward into Moler’s connection:

self.moler_connection.data_received(data)

send(data, timeout=1)¶

Send data via SshShell connection.

Parameters:	data (bytes) – data timeout (float) – max time to spend on sending all data, default 1 sec

moler.io.raw.subprocess module¶

External-IO connections based on pyhon subprocess module.

class moler.io.raw.subprocess.Subprocess¶

Bases: object

data_received(data, recv_time)¶

read_subprocess_output()¶

receive(timeout=30)¶

send(data)¶

start()¶

stop()¶

class moler.io.raw.subprocess.ThreadedSubprocess¶

Bases: moler.io.raw.subprocess.Subprocess

close()¶

open()¶

pull_data(pulling_done)¶

moler.io.raw.tcp module¶

External-IO connections based on raw sockets.

The only 3 requirements for these connections are: (1) store Moler’s connection inside self.moler_connection attribute (2) plugin into Moler’s connection the way IO outputs data to external world:

self.moler_connection.how2send = self.send

forward IO received data into self.moler_connection.data_received(data)

class moler.io.raw.tcp.Tcp(port, host='localhost', receive_buffer_size=262144, logger=None)¶

Bases: object

Implementation of TCP connection using python builtin modules.:

socket.send    /|           |\
              / +-----------+ \
             /    host:port    \  TCP server
socket.recv  \                 /
              \ +-----------+ /
               \|           |/

close()¶

Close TCP connection.

Connection should allow for calling close on closed/not-open connection.

open()¶

Open TCP connection.

Should allow for using as context manager: with connection.open():

receive(timeout=30)¶

Receive data.

Parameters:	timeout (float) – time-out, default 30 sec

send(data)¶

Send data via TCP service.

Parameters:	data (str) – data

class moler.io.raw.tcp.ThreadedTcp(moler_connection, port, host='localhost', receive_buffer_size=262144, logger=None)¶

Bases: moler.io.raw.tcp.Tcp

TCP connection feeding Moler’s connection inside dedicated thread.

This is external-IO usable for Moler since it has it’s own runner (thread) that can work in background and pull data from TCP connection.

close()¶: Close TCP connection & stop pulling thread.

open()¶: Open TCP connection & start thread pulling data from it.

pull_data(pulling_done)¶: Pull data from TCP connection.

moler.io.raw.tcpserverpiped module¶

TCP server based on raw sockets and multiprocessing with services usable for integration tests.

It has backdoor connection over pipe that forms control-service which can: - inject data that should be server’s response for client’s request - send asynchronous data towards client (without awaiting client’s request) - close client connection - return history of server activity (client connected, data received, …) - shutdown server

class moler.io.raw.tcpserverpiped.TcpServerPiped(host, port, pipe_in, buffer_size=1024, delay=0, use_stderr_logger=False)¶

Bases: multiprocessing.context.Process

check_and_handle_client_socket()¶: Handle data coming from client connection

check_and_handle_server_socket()¶: Check if client is knocking :-)

configure_stderr_logger()¶: Configure logging output on stderr

do_close_connection(**kwargs)¶: Handles following pipe message (“close connection”, {}) Force server to close connection to client

do_get_history(**kwargs)¶: Handles following pipe message (“get history”, {}) Retrieve server’s history

do_inject_response(**kwargs)¶: Handles following pipe message (“inject response”, {‘req’: rpc_startup_msg, ‘resp’: response_for_rpc_startup_msg}) It defines response (‘resp’) that should be send by server in reaction to received client data (‘req’).

do_send_async_msg(**kwargs)¶: Handles following pipe message (“send async msg”, {‘msg’: msg_payload}) It injects message to be send by server ‘just now’ (without awaiting any data from client)

do_set_delay(**kwargs)¶: Handles following pipe message (“set delay”, {‘delay’: 2.5}) It defines how much we should delay sending response

do_shutdown(**kwargs)¶: Handles following pipe message (“shutdown”, {}) Force server to shut down

handle_controlling_action()¶: Handle actions coming from server-controlling pipe

handle_data(data)¶: Handle data that came from client

handle_incomming_client()¶: Accept incoming client - make socket connection for it

handle_leaving_client()¶: Handle client leaving server

interpret_pipe_msg(msg)¶: Interpret message that came from controlling pipe

prepare_server_socket()¶: Create, configure and start server-listening socket

run()¶: Run main loop of server process

moler.io.raw.tcpserverpiped.tcp_server_piped(port=19543, use_stderr_logger=False)¶

TCP server context-manager used for integration tests.

It starts server on localhost/given-port during context-manager start() and performs server-cleanup during context-manager stop()

Parameters:	port – where to start server at use_stderr_logger – configure logging to output into stderr?
Returns:	pair (server, inter-process-pipe used to control server)

moler.io.raw.terminal module¶

class moler.io.raw.terminal.ThreadedTerminal(moler_connection, cmd='/bin/bash', select_timeout=0.002, read_buffer_size=4096, first_prompt='[%$#]+', target_prompt='moler_bash#', set_prompt_cmd='export PS1="moler_bash# "n', dimensions=(100, 300), terminal_delayafterclose=0.2)¶

Bases: moler.io.io_connection.IOConnection

Works on Unix (like Linux) systems only!

ThreadedTerminal is shell working under Pty

close()¶: Close ThreadedTerminal connection & stop pulling thread.

open()¶: Open ThreadedTerminal connection & start thread pulling data from it.

pull_data(pulling_done)¶: Pull data from ThreadedTerminal connection.

send(data)¶: Write data into ThreadedTerminal connection.

Module contents¶

External-IO connections based on raw libraries like: sockets, subprocess, …

The only 3 requirements for these connections is: (1) store Moler’s connection inside self.moler_connection attribute (2) plugin into Moler’s connection the way IO outputs data to external world:

self.moler_connection.how2send = self.send

forward IO received data into self.moler_connection.data_received(data)

class moler.io.raw.TillDoneThread(done_event, target=None, name=None, kwargs=None)¶

Bases: threading.Thread

join(timeout=None)¶: Wait until the thread terminates. Set event indicating “I’m done” before awaiting.