Skip to content

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

Inclusion of Stem library for TOR communication.

Steve Micallef32f085048c59
diff --git a/ext/stem/__init__.py b/ext/stem/__init__.py
new file mode 100644
index 00000000..9ff12636
--- /dev/null
+++ b/ext/stem/__init__.py
@@ -0,0 +1,833 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Library for working with the tor process.
+
+**Module Overview:**
+
+::
+
+ ControllerError - Base exception raised when using the controller.
+ |- ProtocolError - Malformed socket data.
+ |- OperationFailed - Tor was unable to successfully complete the operation.
+ | |- UnsatisfiableRequest - Tor was unable to satisfy a valid request.
+ | | +- CircuitExtensionFailed - Attempt to make or extend a circuit failed.
+ | |- DescriptorUnavailable - The given relay descriptor is unavailable.
+ | +- InvalidRequest - Invalid request.
+ | +- InvalidArguments - Invalid request parameters.
+ +- SocketError - Communication with the socket failed.
+ +- SocketClosed - Socket has been shut down.
+
+.. data:: Runlevel (enum)
+
+ Rating of importance used for event logging.
+
+ =========== ===========
+ Runlevel Description
+ =========== ===========
+ **ERR** critical issues that impair tor's ability to function
+ **WARN** non-critical issues the user should be aware of
+ **NOTICE** information that may be helpful to the user
+ **INFO** high level runtime information
+ **DEBUG** low level runtime information
+ =========== ===========
+
+.. data:: Signal (enum)
+
+ Signals that the tor process will accept.
+
+ ========================= ===========
+ Signal Description
+ ========================= ===========
+ **RELOAD** or **HUP** reloads our torrc
+ **SHUTDOWN** or **INT** shut down, waiting ShutdownWaitLength first if we're a relay
+ **DUMP** or **USR1** dumps information about open connections and circuits to our log
+ **DEBUG** or **USR2** switch our logging to the DEBUG runlevel
+ **HALT** or **TERM** exit tor immediately
+ **NEWNYM** switch to new circuits, so new application requests don't share any circuits with old ones (this also clears our DNS cache)
+ **CLEARDNSCACHE** clears cached DNS results
+ **HEARTBEAT** trigger a heartbeat log message
+ ========================= ===========
+
+.. data:: Flag (enum)
+
+ Flag assigned to tor relays by the authorities to indicate various
+ characteristics.
+
+ **Note:** The BADDIRECTORY flag was `removed from tor <https://gitweb.torproject.org/torspec.git/commit/dir-spec.txt?id=2f012f1>`_.
+
+ ================= ===========
+ Flag Description
+ ================= ===========
+ **AUTHORITY** relay is a directory authority
+ **BADEXIT** relay shouldn't be used as an exit due to being either problematic or malicious (`wiki <https://trac.torproject.org/projects/tor/wiki/doc/badRelays>`_)
+ **BADDIRECTORY** relay shouldn't be used for directory information
+ **EXIT** relay's exit policy makes it more useful as an exit rather than middle hop
+ **FAST** relay's suitable for high-bandwidth circuits
+ **GUARD** relay's suitable for being an entry guard (first hop)
+ **HSDIR** relay is being used as a v2 hidden service directory
+ **NAMED** relay can be referred to by its nickname
+ **RUNNING** relay is currently usable
+ **STABLE** relay's suitable for long-lived circuits
+ **UNNAMED** relay isn't currently bound to a nickname
+ **V2DIR** relay supports the v2 directory protocol
+ **VALID** relay has been validated
+ ================= ===========
+
+.. data:: CircStatus (enum)
+
+ Statuses that a circuit can be in. Tor may provide statuses not in this enum.
+
+ ============ ===========
+ CircStatus Description
+ ============ ===========
+ **LAUNCHED** new circuit was created
+ **BUILT** circuit finished being created and can accept traffic
+ **EXTENDED** circuit has been extended by a hop
+ **FAILED** circuit construction failed
+ **CLOSED** circuit has been closed
+ ============ ===========
+
+.. data:: CircBuildFlag (enum)
+
+ Attributes about how a circuit is built. These were introduced in tor version
+ 0.2.3.11. Tor may provide flags not in this enum.
+
+ ================= ===========
+ CircBuildFlag Description
+ ================= ===========
+ **ONEHOP_TUNNEL** single hop circuit to fetch directory information
+ **IS_INTERNAL** circuit that won't be used for client traffic
+ **NEED_CAPACITY** circuit only includes high capacity relays
+ **NEED_UPTIME** circuit only includes relays with a high uptime
+ ================= ===========
+
+.. data:: CircPurpose (enum)
+
+ Description of what a circuit is intended for. These were introduced in tor
+ version 0.2.1.6. Tor may provide purposes not in this enum.
+
+ ==================== ===========
+ CircPurpose Description
+ ==================== ===========
+ **GENERAL** client traffic or fetching directory information
+ **HS_CLIENT_INTRO** client side introduction point for a hidden service circuit
+ **HS_CLIENT_REND** client side hidden service rendezvous circuit
+ **HS_SERVICE_INTRO** server side introduction point for a hidden service circuit
+ **HS_SERVICE_REND** server side hidden service rendezvous circuit
+ **TESTING** testing to see if we're reachable, so we can be used as a relay
+ **CONTROLLER** circuit that was built by a controller
+ **MEASURE_TIMEOUT** circuit being kept around to see how long it takes
+ ==================== ===========
+
+.. data:: CircClosureReason (enum)
+
+ Reason that a circuit is being closed or failed to be established. Tor may
+ provide reasons not in this enum.
+
+ ========================= ===========
+ CircClosureReason Description
+ ========================= ===========
+ **NONE** no reason given
+ **TORPROTOCOL** violation in the tor protocol
+ **INTERNAL** internal error
+ **REQUESTED** requested by the client via a TRUNCATE command
+ **HIBERNATING** relay is currently hibernating
+ **RESOURCELIMIT** relay is out of memory, sockets, or circuit IDs
+ **CONNECTFAILED** unable to contact the relay
+ **OR_IDENTITY** relay had the wrong OR identification
+ **OR_CONN_CLOSED** connection failed after being established
+ **FINISHED** circuit has expired (see tor's MaxCircuitDirtiness config option)
+ **TIMEOUT** circuit construction timed out
+ **DESTROYED** circuit unexpectedly closed
+ **NOPATH** not enough relays to make a circuit
+ **NOSUCHSERVICE** requested hidden service does not exist
+ **MEASUREMENT_EXPIRED** same as **TIMEOUT** except that it was left open for measurement purposes
+ ========================= ===========
+
+.. data:: CircEvent (enum)
+
+ Type of change reflected in a circuit by a CIRC_MINOR event. Tor may provide
+ event types not in this enum.
+
+ ===================== ===========
+ CircEvent Description
+ ===================== ===========
+ **PURPOSE_CHANGED** circuit purpose or hidden service state has changed
+ **CANNIBALIZED** circuit connections are being reused for a different circuit
+ ===================== ===========
+
+.. data:: HiddenServiceState (enum)
+
+ State that a hidden service circuit can have. These were introduced in tor
+ version 0.2.3.11. Tor may provide states not in this enum.
+
+ Enumerations fall into four groups based on their prefix...
+
+ ======= ===========
+ Prefix Description
+ ======= ===========
+ HSCI_* client-side introduction-point
+ HSCR_* client-side rendezvous-point
+ HSSI_* service-side introduction-point
+ HSSR_* service-side rendezvous-point
+ ======= ===========
+
+ ============================= ===========
+ HiddenServiceState Description
+ ============================= ===========
+ **HSCI_CONNECTING** connecting to the introductory point
+ **HSCI_INTRO_SENT** sent INTRODUCE1 and awaiting a reply
+ **HSCI_DONE** received a reply, circuit is closing
+ **HSCR_CONNECTING** connecting to the introductory point
+ **HSCR_ESTABLISHED_IDLE** rendezvous-point established, awaiting an introduction
+ **HSCR_ESTABLISHED_WAITING** introduction received, awaiting a rend
+ **HSCR_JOINED** connected to the hidden service
+ **HSSI_CONNECTING** connecting to the introductory point
+ **HSSI_ESTABLISHED** established introductory point
+ **HSSR_CONNECTING** connecting to the introductory point
+ **HSSR_JOINED** connected to the rendezvous-point
+ ============================= ===========
+
+.. data:: RelayEndReason (enum)
+
+ Reasons why the stream is to be closed.
+
+ =================== ===========
+ RelayEndReason Description
+ =================== ===========
+ **MISC** none of the following reasons
+ **RESOLVEFAILED** unable to resolve the hostname
+ **CONNECTREFUSED** remote host refused the connection
+ **EXITPOLICY** OR refuses to connect to the destination
+ **DESTROY** circuit is being shut down
+ **DONE** connection has been closed
+ **TIMEOUT** connection timed out
+ **NOROUTE** routing error while contacting the destination
+ **HIBERNATING** relay is temporarily hibernating
+ **INTERNAL** internal error at the relay
+ **RESOURCELIMIT** relay has insufficient resources to service the request
+ **CONNRESET** connection was unexpectedly reset
+ **TORPROTOCOL** violation in the tor protocol
+ **NOTDIRECTORY** directory information requested from a relay that isn't mirroring it
+ =================== ===========
+
+.. data:: StreamStatus (enum)
+
+ State that a stream going through tor can have. Tor may provide states not in
+ this enum.
+
+ ================= ===========
+ StreamStatus Description
+ ================= ===========
+ **NEW** request for a new connection
+ **NEWRESOLVE** request to resolve an address
+ **REMAP** address is being re-mapped to another
+ **SENTCONNECT** sent a connect cell along a circuit
+ **SENTRESOLVE** sent a resolve cell along a circuit
+ **SUCCEEDED** stream has been established
+ **FAILED** stream is detached, and won't be re-established
+ **DETACHED** stream is detached, but might be re-established
+ **CLOSED** stream has closed
+ ================= ===========
+
+.. data:: StreamClosureReason (enum)
+
+ Reason that a stream is being closed or failed to be established. This
+ includes all values in the :data:`~stem.RelayEndReason` enumeration as
+ well as the following. Tor may provide reasons not in this enum.
+
+ ===================== ===========
+ StreamClosureReason Description
+ ===================== ===========
+ **END** endpoint has sent a RELAY_END cell
+ **PRIVATE_ADDR** endpoint was a private address (127.0.0.1, 10.0.0.1, etc)
+ ===================== ===========
+
+.. data:: StreamSource (enum)
+
+ Cause of a stream being remapped to another address. Tor may provide sources
+ not in this enum.
+
+ ============= ===========
+ StreamSource Description
+ ============= ===========
+ **CACHE** tor is remapping because of a cached answer
+ **EXIT** exit relay requested the remap
+ ============= ===========
+
+.. data:: StreamPurpose (enum)
+
+ Purpsoe of the stream. This is only provided with new streams and tor may
+ provide purposes not in this enum.
+
+ ================= ===========
+ StreamPurpose Description
+ ================= ===========
+ **DIR_FETCH** fetching directory information (descriptors, consensus, etc)
+ **DIR_UPLOAD** uploading our descriptor to an authority
+ **DNS_REQUEST** user initiated DNS request
+ **DIRPORT_TEST** checking that our directory port is reachable externally
+ **USER** either relaying user traffic or not one of the above categories
+ ================= ===========
+
+.. data:: ORStatus (enum)
+
+ State that an OR connection can have. Tor may provide states not in this
+ enum.
+
+ =============== ===========
+ ORStatus Description
+ =============== ===========
+ **NEW** received OR connection, starting server-side handshake
+ **LAUNCHED** launched outbound OR connection, starting client-side handshake
+ **CONNECTED** OR connection has been established
+ **FAILED** attempt to establish OR connection failed
+ **CLOSED** OR connection has been closed
+ =============== ===========
+
+.. data:: ORClosureReason (enum)
+
+ Reason that an OR connection is being closed or failed to be established. Tor
+ may provide reasons not in this enum.
+
+ =================== ===========
+ ORClosureReason Description
+ =================== ===========
+ **DONE** OR connection shut down cleanly
+ **CONNECTREFUSED** got a ECONNREFUSED when connecting to the relay
+ **IDENTITY** identity of the relay wasn't what we expected
+ **CONNECTRESET** got a ECONNRESET or similar error from relay
+ **TIMEOUT** got a ETIMEOUT or similar error from relay
+ **NOROUTE** got a ENOTCONN, ENETUNREACH, ENETDOWN, EHOSTUNREACH, or similar error from relay
+ **IOERROR** got a different kind of error from relay
+ **RESOURCELIMIT** relay has insufficient resources to service the request
+ **MISC** connection refused for another reason
+ **PT_MISSING** no pluggable transport was available
+ =================== ===========
+
+.. data:: AuthDescriptorAction (enum)
+
+ Actions that directory authorities might take with relay descriptors. Tor may
+ provide reasons not in this enum.
+
+ ===================== ===========
+ AuthDescriptorAction Description
+ ===================== ===========
+ **ACCEPTED** accepting the descriptor as the newest version
+ **DROPPED** descriptor rejected without notifying the relay
+ **REJECTED** relay notified that its descriptor has been rejected
+ ===================== ===========
+
+.. data:: StatusType (enum)
+
+ Sources for tor status events. Tor may provide types not in this enum.
+
+ ============= ===========
+ StatusType Description
+ ============= ===========
+ **GENERAL** general tor activity, not specifically as a client or relay
+ **CLIENT** related to our activity as a tor client
+ **SERVER** related to our activity as a tor relay
+ ============= ===========
+
+.. data:: GuardType (enum)
+
+ Use a guard relay can be for. Tor may provide types not in this enum.
+
+ =========== ===========
+ GuardType Description
+ =========== ===========
+ **ENTRY** used to connect to the tor network
+ =========== ===========
+
+.. data:: GuardStatus (enum)
+
+ Status a guard relay can have. Tor may provide types not in this enum.
+
+ ============= ===========
+ GuardStatus Description
+ ============= ===========
+ **NEW** new guard that we weren't previously using
+ **DROPPED** removed from use as one of our guards
+ **UP** guard is now reachable
+ **DOWN** guard is now unreachable
+ **BAD** consensus or relay considers this relay to be unusable as a guard
+ **GOOD** consensus or relay considers this relay to be usable as a guard
+ ============= ===========
+
+.. data:: TimeoutSetType (enum)
+
+ Way in which the timeout value of a circuit is changing. Tor may provide
+ types not in this enum.
+
+ =============== ===========
+ TimeoutSetType Description
+ =============== ===========
+ **COMPUTED** tor has computed a new timeout based on prior circuits
+ **RESET** timeout reverted to its default
+ **SUSPENDED** timeout reverted to its default until network connectivity has recovered
+ **DISCARD** throwing out timeout value from when the network was down
+ **RESUME** resumed calculations to determine the proper timeout
+ =============== ===========
+
+.. data:: ConnectionType (enum)
+
+ Purpose for a tor connection. Tor may provide types not in this enum.
+
+ The meaning behind these values is a bit unclear, pending :trac:`10086`.
+
+ =============== ===========
+ ConnectionType Description
+ =============== ===========
+ **OR** carrying traffic within the tor network
+ **DIR** fetching or sending tor descriptor data
+ **EXIT** carrying traffic between the tor network and an external destination
+ =============== ===========
+
+.. data:: TokenBucket (enum)
+
+ Bucket categories of TB_EMPTY events.
+
+ =============== ===========
+ TokenBucket Description
+ =============== ===========
+ **GLOBAL** global token bucket
+ **RELAY** relay token bucket
+ **ORCONN** bucket used for OR connections
+ =============== ===========
+
+.. data:: HSDescAction (enum)
+
+ Action beeing taken in a HS_DESC event.
+
+ =============== ===========
+ HSDescAction Description
+ =============== ===========
+ **REQUESTED** uncached hidden service descriptor is being requested
+ **UPLOAD** descriptor is being uploaded with HSPOST
+ **RECEIVED** hidden service descriptor has been retrieved
+ **UPLOADED** descriptor was uploaded with HSPOST
+ **IGNORE** fetched descriptor was ignored because we already have its v0 descriptor
+ **FAILED** we were unable to retrieve the descriptor
+ =============== ===========
+
+.. data:: HSDescReason (enum)
+
+ Reason for the hidden service descriptor to fail to be fetched.
+
+ =================== ===========
+ HSDescReason Description
+ =================== ===========
+ **BAD_DESC** descriptor was unparseable
+ **QUERY_REJECTED** hidden service directory refused to provide the descriptor
+ **UPLOAD_REJECTED** descriptor was rejected by the hidden service directory
+ **NOT_FOUND** descriptor with the given identifier wasn't found
+ **UNEXPECTED** failure type is unknown
+ =================== ===========
+
+.. data:: HSAuth (enum)
+
+ Type of authentication being used for a HS_DESC event.
+
+ ================= ===========
+ HSAuth Description
+ ================= ===========
+ **NO_AUTH** no authentication
+ **BASIC_AUTH** general hidden service authentication
+ **STEALTH_AUTH** authentication method that hides service activity from unauthorized clients
+ **UNKNOWN** unrecognized method of authentication
+ ================= ===========
+"""
+
+__version__ = '1.4.1'
+__author__ = 'Damian Johnson'
+__contact__ = '[email protected]'
+__url__ = 'https://stem.torproject.org/'
+__license__ = 'LGPLv3'
+
+__all__ = [
+ 'descriptor',
+ 'response',
+ 'util',
+ 'connection',
+ 'control',
+ 'exit_policy',
+ 'prereq',
+ 'process',
+ 'socket',
+ 'version',
+ 'ControllerError',
+ 'ProtocolError',
+ 'OperationFailed',
+ 'UnsatisfiableRequest',
+ 'CircuitExtensionFailed',
+ 'DescriptorUnavailable',
+ 'InvalidRequest',
+ 'InvalidArguments',
+ 'SocketError',
+ 'SocketClosed',
+ 'Runlevel',
+ 'Signal',
+ 'Flag',
+ 'CircStatus',
+ 'CircBuildFlag',
+ 'CircPurpose',
+ 'CircClosureReason',
+ 'CircEvent',
+ 'HiddenServiceState',
+ 'HSAuth',
+ 'HSDescAction',
+ 'HSDescReason',
+ 'RelayEndReason',
+ 'StreamStatus',
+ 'StreamClosureReason',
+ 'StreamSource',
+ 'StreamPurpose',
+ 'ORStatus',
+ 'ORClosureReason',
+ 'AuthDescriptorAction',
+ 'StatusType',
+ 'GuardType',
+ 'GuardStatus',
+ 'TimeoutSetType',
+]
+
+import stem.prereq
+
+if stem.prereq.is_python_3():
+ str_type = str
+ int_type = int
+else:
+ str_type = unicode
+ int_type = long
+
+import stem.util.enum
+
+# Constant to indicate an undefined argument default. Usually we'd use None for
+# this, but users will commonly provide None as the argument so need something
+# else fairly unique...
+
+UNDEFINED = '<Undefined_ >'
+
+
+class ControllerError(Exception):
+ 'Base error for controller communication issues.'
+
+
+class ProtocolError(ControllerError):
+ 'Malformed content from the control socket.'
+
+
+class OperationFailed(ControllerError):
+ """
+ Base exception class for failed operations that return an error code
+
+ :var str code: error code returned by Tor
+ :var str message: error message returned by Tor or a human readable error
+ message
+ """
+
+ def __init__(self, code = None, message = None):
+ super(ControllerError, self).__init__(message)
+ self.code = code
+ self.message = message
+
+
+class UnsatisfiableRequest(OperationFailed):
+ """
+ Exception raised if Tor was unable to process our request.
+ """
+
+
+class CircuitExtensionFailed(UnsatisfiableRequest):
+ """
+ An attempt to create or extend a circuit failed.
+
+ :var stem.response.CircuitEvent circ: response notifying us of the failure
+ """
+
+ def __init__(self, message, circ = None):
+ super(CircuitExtensionFailed, self).__init__(message = message)
+ self.circ = circ
+
+
+class DescriptorUnavailable(OperationFailed):
+ """
+ Tor was unable to provide a descriptor for the given relay.
+ """
+
+ def __init__(self, message):
+ super(DescriptorUnavailable, self).__init__(message = message)
+
+
+class InvalidRequest(OperationFailed):
+ """
+ Exception raised when the request was invalid or malformed.
+ """
+
+
+class InvalidArguments(InvalidRequest):
+ """
+ Exception class for requests which had invalid arguments.
+
+ :var str code: error code returned by Tor
+ :var str message: error message returned by Tor or a human readable error
+ message
+ :var list arguments: a list of arguments which were invalid
+ """
+
+ def __init__(self, code = None, message = None, arguments = None):
+ super(InvalidArguments, self).__init__(code, message)
+ self.arguments = arguments
+
+
+class SocketError(ControllerError):
+ 'Error arose while communicating with the control socket.'
+
+
+class SocketClosed(SocketError):
+ 'Control socket was closed before completing the message.'
+
+Runlevel = stem.util.enum.UppercaseEnum(
+ 'DEBUG',
+ 'INFO',
+ 'NOTICE',
+ 'WARN',
+ 'ERR',
+)
+
+Flag = stem.util.enum.Enum(
+ ('AUTHORITY', 'Authority'),
+ ('BADEXIT', 'BadExit'),
+ ('BADDIRECTORY', 'BadDirectory'),
+ ('EXIT', 'Exit'),
+ ('FAST', 'Fast'),
+ ('GUARD', 'Guard'),
+ ('HSDIR', 'HSDir'),
+ ('NAMED', 'Named'),
+ ('RUNNING', 'Running'),
+ ('STABLE', 'Stable'),
+ ('UNNAMED', 'Unnamed'),
+ ('V2DIR', 'V2Dir'),
+ ('V3DIR', 'V3Dir'),
+ ('VALID', 'Valid'),
+)
+
+Signal = stem.util.enum.UppercaseEnum(
+ 'RELOAD',
+ 'HUP',
+ 'SHUTDOWN',
+ 'INT',
+ 'DUMP',
+ 'USR1',
+ 'DEBUG',
+ 'USR2',
+ 'HALT',
+ 'TERM',
+ 'NEWNYM',
+ 'CLEARDNSCACHE',
+ 'HEARTBEAT',
+)
+
+CircStatus = stem.util.enum.UppercaseEnum(
+ 'LAUNCHED',
+ 'BUILT',
+ 'EXTENDED',
+ 'FAILED',
+ 'CLOSED',
+)
+
+CircBuildFlag = stem.util.enum.UppercaseEnum(
+ 'ONEHOP_TUNNEL',
+ 'IS_INTERNAL',
+ 'NEED_CAPACITY',
+ 'NEED_UPTIME',
+)
+
+CircPurpose = stem.util.enum.UppercaseEnum(
+ 'GENERAL',
+ 'HS_CLIENT_INTRO',
+ 'HS_CLIENT_REND',
+ 'HS_SERVICE_INTRO',
+ 'HS_SERVICE_REND',
+ 'TESTING',
+ 'CONTROLLER',
+ 'MEASURE_TIMEOUT',
+)
+
+CircClosureReason = stem.util.enum.UppercaseEnum(
+ 'NONE',
+ 'TORPROTOCOL',
+ 'INTERNAL',
+ 'REQUESTED',
+ 'HIBERNATING',
+ 'RESOURCELIMIT',
+ 'CONNECTFAILED',
+ 'OR_IDENTITY',
+ 'OR_CONN_CLOSED',
+ 'FINISHED',
+ 'TIMEOUT',
+ 'DESTROYED',
+ 'NOPATH',
+ 'NOSUCHSERVICE',
+ 'MEASUREMENT_EXPIRED',
+)
+
+CircEvent = stem.util.enum.UppercaseEnum(
+ 'PURPOSE_CHANGED',
+ 'CANNIBALIZED',
+)
+
+HiddenServiceState = stem.util.enum.UppercaseEnum(
+ 'HSCI_CONNECTING',
+ 'HSCI_INTRO_SENT',
+ 'HSCI_DONE',
+ 'HSCR_CONNECTING',
+ 'HSCR_ESTABLISHED_IDLE',
+ 'HSCR_ESTABLISHED_WAITING',
+ 'HSCR_JOINED',
+ 'HSSI_CONNECTING',
+ 'HSSI_ESTABLISHED',
+ 'HSSR_CONNECTING',
+ 'HSSR_JOINED',
+)
+
+RelayEndReason = stem.util.enum.UppercaseEnum(
+ 'MISC',
+ 'RESOLVEFAILED',
+ 'CONNECTREFUSED',
+ 'EXITPOLICY',
+ 'DESTROY',
+ 'DONE',
+ 'TIMEOUT',
+ 'NOROUTE',
+ 'HIBERNATING',
+ 'INTERNAL',
+ 'RESOURCELIMIT',
+ 'CONNRESET',
+ 'TORPROTOCOL',
+ 'NOTDIRECTORY',
+)
+
+StreamStatus = stem.util.enum.UppercaseEnum(
+ 'NEW',
+ 'NEWRESOLVE',
+ 'REMAP',
+ 'SENTCONNECT',
+ 'SENTRESOLVE',
+ 'SUCCEEDED',
+ 'FAILED',
+ 'DETACHED',
+ 'CLOSED',
+)
+
+# StreamClosureReason is a superset of RelayEndReason
+StreamClosureReason = stem.util.enum.UppercaseEnum(*(RelayEndReason.keys() + [
+ 'END',
+ 'PRIVATE_ADDR',
+]))
+
+StreamSource = stem.util.enum.UppercaseEnum(
+ 'CACHE',
+ 'EXIT',
+)
+
+StreamPurpose = stem.util.enum.UppercaseEnum(
+ 'DIR_FETCH',
+ 'DIR_UPLOAD',
+ 'DNS_REQUEST',
+ 'DIRPORT_TEST',
+ 'USER',
+)
+
+ORStatus = stem.util.enum.UppercaseEnum(
+ 'NEW',
+ 'LAUNCHED',
+ 'CONNECTED',
+ 'FAILED',
+ 'CLOSED',
+)
+
+ORClosureReason = stem.util.enum.UppercaseEnum(
+ 'DONE',
+ 'CONNECTREFUSED',
+ 'IDENTITY',
+ 'CONNECTRESET',
+ 'TIMEOUT',
+ 'NOROUTE',
+ 'IOERROR',
+ 'RESOURCELIMIT',
+ 'MISC',
+ 'PT_MISSING',
+)
+
+AuthDescriptorAction = stem.util.enum.UppercaseEnum(
+ 'ACCEPTED',
+ 'DROPPED',
+ 'REJECTED',
+)
+
+StatusType = stem.util.enum.UppercaseEnum(
+ 'GENERAL',
+ 'CLIENT',
+ 'SERVER',
+)
+
+GuardType = stem.util.enum.UppercaseEnum(
+ 'ENTRY',
+)
+
+GuardStatus = stem.util.enum.UppercaseEnum(
+ 'NEW',
+ 'UP',
+ 'DOWN',
+ 'BAD',
+ 'GOOD',
+ 'DROPPED',
+)
+
+TimeoutSetType = stem.util.enum.UppercaseEnum(
+ 'COMPUTED',
+ 'RESET',
+ 'SUSPENDED',
+ 'DISCARD',
+ 'RESUME',
+)
+
+ConnectionType = stem.util.enum.UppercaseEnum(
+ 'OR',
+ 'DIR',
+ 'EXIT',
+)
+
+TokenBucket = stem.util.enum.UppercaseEnum(
+ 'GLOBAL',
+ 'RELAY',
+ 'ORCONN',
+)
+
+HSDescAction = stem.util.enum.UppercaseEnum(
+ 'REQUESTED',
+ 'UPLOAD',
+ 'RECEIVED',
+ 'UPLOADED',
+ 'IGNORE',
+ 'FAILED',
+)
+
+HSDescReason = stem.util.enum.UppercaseEnum(
+ 'BAD_DESC',
+ 'QUERY_REJECTED',
+ 'UPLOAD_REJECTED',
+ 'NOT_FOUND',
+ 'UNEXPECTED',
+)
+
+HSAuth = stem.util.enum.UppercaseEnum(
+ 'NO_AUTH',
+ 'BASIC_AUTH',
+ 'STEALTH_AUTH',
+ 'UNKNOWN',
+)
diff --git a/ext/stem/connection.py b/ext/stem/connection.py
new file mode 100644
index 00000000..fb85225e
--- /dev/null
+++ b/ext/stem/connection.py
@@ -0,0 +1,1284 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Functions for connecting and authenticating to the tor process.
+
+The :func:`~stem.connection.connect` function give an easy, one line
+method for getting an authenticated control connection. This is handy for CLI
+applications and the python interactive interpreter, but does several things
+that makes it undesirable for applications (uses stdin/stdout, suppresses
+exceptions, etc).
+
+::
+
+ import sys
+
+ from stem.connection import connect
+
+ if __name__ == '__main__':
+ controller = connect()
+
+ if not controller:
+ sys.exit(1) # unable to get a connection
+
+ print 'Tor is running version %s' % controller.get_version()
+ controller.close()
+
+::
+
+ % python example.py
+ Tor is running version 0.2.4.10-alpha-dev (git-8be6058d8f31e578)
+
+... or if Tor isn't running...
+
+::
+
+ % python example.py
+ [Errno 111] Connection refused
+
+The :func:`~stem.connection.authenticate` function, however, gives easy but
+fine-grained control over the authentication process. For instance...
+
+::
+
+ import sys
+ import getpass
+ import stem.connection
+ import stem.socket
+
+ try:
+ control_socket = stem.socket.ControlPort(port = 9051)
+ except stem.SocketError as exc:
+ print 'Unable to connect to port 9051 (%s)' % exc
+ sys.exit(1)
+
+ try:
+ stem.connection.authenticate(control_socket)
+ except stem.connection.IncorrectSocketType:
+ print 'Please check in your torrc that 9051 is the ControlPort.'
+ print 'Maybe you configured it to be the ORPort or SocksPort instead?'
+ sys.exit(1)
+ except stem.connection.MissingPassword:
+ controller_password = getpass.getpass('Controller password: ')
+
+ try:
+ stem.connection.authenticate_password(control_socket, controller_password)
+ except stem.connection.PasswordAuthFailed:
+ print 'Unable to authenticate, password is incorrect'
+ sys.exit(1)
+ except stem.connection.AuthenticationFailure as exc:
+ print 'Unable to authenticate: %s' % exc
+ sys.exit(1)
+
+**Module Overview:**
+
+::
+
+ connect - Simple method for getting authenticated control connection
+
+ authenticate - Main method for authenticating to a control socket
+ authenticate_none - Authenticates to an open control socket
+ authenticate_password - Authenticates to a socket supporting password auth
+ authenticate_cookie - Authenticates to a socket supporting cookie auth
+ authenticate_safecookie - Authenticates to a socket supporting safecookie auth
+
+ get_protocolinfo - Issues a PROTOCOLINFO query
+
+ AuthenticationFailure - Base exception raised for authentication failures
+ |- UnrecognizedAuthMethods - Authentication methods are unsupported
+ |- IncorrectSocketType - Socket does not speak the tor control protocol
+ |
+ |- OpenAuthFailed - Failure when authenticating by an open socket
+ | +- OpenAuthRejected - Tor rejected this method of authentication
+ |
+ |- PasswordAuthFailed - Failure when authenticating by a password
+ | |- PasswordAuthRejected - Tor rejected this method of authentication
+ | |- IncorrectPassword - Password was rejected
+ | +- MissingPassword - Socket supports password auth but wasn't attempted
+ |
+ |- CookieAuthFailed - Failure when authenticating by a cookie
+ | |- CookieAuthRejected - Tor rejected this method of authentication
+ | |- IncorrectCookieValue - Authentication cookie was rejected
+ | |- IncorrectCookieSize - Size of the cookie file is incorrect
+ | |- UnreadableCookieFile - Unable to read the contents of the auth cookie
+ | +- AuthChallengeFailed - Failure completing the authchallenge request
+ | |- AuthChallengeUnsupported - Tor doesn't recognize the AUTHCHALLENGE command
+ | |- AuthSecurityFailure - Server provided the wrong nonce credentials
+ | |- InvalidClientNonce - The client nonce is invalid
+ | +- UnrecognizedAuthChallengeMethod - AUTHCHALLENGE does not support the given methods.
+ |
+ +- MissingAuthInfo - Unexpected PROTOCOLINFO response, missing auth info
+ |- NoAuthMethods - Missing any methods for authenticating
+ +- NoAuthCookie - Supports cookie auth but doesn't have its path
+
+.. data:: AuthMethod (enum)
+
+ Enumeration of PROTOCOLINFO responses for supported authentication methods.
+
+ ============== ===========
+ AuthMethod Description
+ ============== ===========
+ **NONE** No authentication required.
+ **PASSWORD** Password required, see tor's HashedControlPassword option.
+ **COOKIE** Contents of the cookie file required, see tor's CookieAuthentication option.
+ **SAFECOOKIE** Need to reply to a hmac challenge using the contents of the cookie file.
+ **UNKNOWN** Tor provided one or more authentication methods that we don't recognize, probably something new.
+ ============== ===========
+"""
+
+import binascii
+import getpass
+import os
+
+import stem.control
+import stem.response
+import stem.socket
+import stem.util.connection
+import stem.util.enum
+import stem.util.str_tools
+import stem.util.system
+import stem.version
+
+from stem.util import log
+
+AuthMethod = stem.util.enum.Enum('NONE', 'PASSWORD', 'COOKIE', 'SAFECOOKIE', 'UNKNOWN')
+
+CLIENT_HASH_CONSTANT = b'Tor safe cookie authentication controller-to-server hash'
+SERVER_HASH_CONSTANT = b'Tor safe cookie authentication server-to-controller hash'
+
+MISSING_PASSWORD_BUG_MSG = """
+BUG: You provided a password but despite this stem reported that it was
+missing. This shouldn't happen - please let us know about it!
+
+ http://bugs.torproject.org
+"""
+
+UNRECOGNIZED_AUTH_TYPE_MSG = """
+Tor is using a type of authentication we do not recognize...
+
+ {auth_methods}
+
+Please check that stem is up to date and if there is an existing issue on
+'http://bugs.torproject.org'. If there isn't one then let us know!
+"""
+
+
+UNREADABLE_COOKIE_FILE_MSG = """
+We were unable to read tor's authentication cookie...
+
+ Path: {path}
+ Issue: {issue}
+"""
+
+WRONG_PORT_TYPE_MSG = """
+Please check in your torrc that {port} is the ControlPort. Maybe you
+configured it to be the ORPort or SocksPort instead?
+"""
+
+WRONG_SOCKET_TYPE_MSG = """
+Unable to connect to tor. Are you sure the interface you specified belongs to
+tor?
+"""
+
+CONNECT_MESSAGES = {
+ 'general_auth_failure': 'Unable to authenticate: {error}',
+ 'incorrect_password': 'Incorrect password',
+ 'no_control_port': "Unable to connect to tor. Maybe it's running without a ControlPort?",
+ 'password_prompt': 'Tor controller password:',
+ 'needs_password': 'Tor requires a password to authenticate',
+ 'socket_doesnt_exist': "The socket file you specified ({path}) doesn't exist",
+ 'tor_isnt_running': "Unable to connect to tor. Are you sure it's running?",
+ 'unable_to_use_port': 'Unable to connect to {address}:{port}: {error}',
+ 'unable_to_use_socket': "Unable to connect to '{path}': {error}",
+ 'missing_password_bug': MISSING_PASSWORD_BUG_MSG.strip(),
+ 'uncrcognized_auth_type': UNRECOGNIZED_AUTH_TYPE_MSG.strip(),
+ 'unreadable_cookie_file': UNREADABLE_COOKIE_FILE_MSG.strip(),
+ 'wrong_port_type': WRONG_PORT_TYPE_MSG.strip(),
+ 'wrong_socket_type': WRONG_SOCKET_TYPE_MSG.strip(),
+}
+
+
+def connect(control_port = ('127.0.0.1', 9051), control_socket = '/var/run/tor/control', password = None, password_prompt = False, chroot_path = None, controller = stem.control.Controller):
+ """
+ Convenience function for quickly getting a control connection. This is very
+ handy for debugging or CLI setup, handling setup and prompting for a password
+ if necessary (and none is provided). If any issues arise this prints a
+ description of the problem and returns **None**.
+
+ If both a **control_port** and **control_socket** are provided then the
+ **control_socket** is tried first, and this provides a generic error message
+ if they're both unavailable.
+
+ In much the same vein as git porcelain commands, users should not rely on
+ details of how this works. Messages and details of this function's behavior
+ could change in the future.
+
+ .. versionadded:: 1.2.0
+
+ :param tuple contol_port: address and port tuple, for instance **('127.0.0.1', 9051)**
+ :param str path: path where the control socket is located
+ :param str password: passphrase to authenticate to the socket
+ :param bool password_prompt: prompt for the controller password if it wasn't
+ supplied
+ :param str chroot_path: path prefix if in a chroot environment
+ :param Class controller: :class:`~stem.control.BaseController` subclass to be
+ returned, this provides a :class:`~stem.socket.ControlSocket` if **None**
+
+ :returns: authenticated control connection, the type based on the controller argument
+
+ :raises: **ValueError** if given an invalid control_port, or both
+ **control_port** and **control_socket** are **None**
+ """
+
+ if control_port is None and control_socket is None:
+ raise ValueError('Neither a control port nor control socket were provided. Nothing to connect to.')
+ elif control_port:
+ if len(control_port) != 2:
+ raise ValueError('The control_port argument for connect() should be an (address, port) tuple.')
+ elif not stem.util.connection.is_valid_ipv4_address(control_port[0]):
+ raise ValueError("'%s' isn't a vaid IPv4 address" % control_port[0])
+ elif not stem.util.connection.is_valid_port(control_port[1]):
+ raise ValueError("'%s' isn't a valid port" % control_port[1])
+
+ control_connection, error_msg = None, ''
+
+ if control_socket:
+ if os.path.exists(control_socket):
+ try:
+ control_connection = stem.socket.ControlSocketFile(control_socket)
+ except stem.SocketError as exc:
+ error_msg = CONNECT_MESSAGES['unable_to_use_socket'].format(path = control_socket, error = exc)
+ else:
+ error_msg = CONNECT_MESSAGES['socket_doesnt_exist'].format(path = control_socket)
+
+ if control_port and not control_connection:
+ address, port = control_port
+
+ try:
+ control_connection = stem.socket.ControlPort(address, port)
+ except stem.SocketError as exc:
+ error_msg = CONNECT_MESSAGES['unable_to_use_port'].format(address = address, port = port, error = exc)
+
+ # If unable to connect to either a control socket or port then finally fail
+ # out. If we only attempted to connect to one of them then provide the error
+ # output from that. Otherwise we provide a more generic error message.
+ #
+ # We check for a 'tor.real' process name because that's what TBB uses.
+
+ if not control_connection:
+ if control_socket and control_port:
+ is_tor_running = stem.util.system.is_running('tor') or stem.util.system.is_running('tor.real')
+ error_msg = CONNECT_MESSAGES['no_control_port'] if is_tor_running else CONNECT_MESSAGES['tor_isnt_running']
+
+ print(error_msg)
+ return None
+
+ return _connect_auth(control_connection, password, password_prompt, chroot_path, controller)
+
+
+def connect_port(address = '127.0.0.1', port = 9051, password = None, chroot_path = None, controller = stem.control.Controller):
+ """
+ Convenience function for quickly getting a control connection. This is very
+ handy for debugging or CLI setup, handling setup and prompting for a password
+ if necessary (and none is provided). If any issues arise this prints a
+ description of the problem and returns **None**.
+
+ .. deprecated:: 1.2.0
+ Use :func:`~stem.connection.connect` instead.
+
+ :param str address: ip address of the controller
+ :param int port: port number of the controller
+ :param str password: passphrase to authenticate to the socket
+ :param str chroot_path: path prefix if in a chroot environment
+ :param Class controller: :class:`~stem.control.BaseController` subclass to be
+ returned, this provides a :class:`~stem.socket.ControlSocket` if **None**
+
+ :returns: authenticated control connection, the type based on the controller argument
+ """
+
+ try:
+ control_port = stem.socket.ControlPort(address, port)
+ except stem.SocketError as exc:
+ print(exc)
+ return None
+
+ return _connect_auth(control_port, password, True, chroot_path, controller)
+
+
+def connect_socket_file(path = '/var/run/tor/control', password = None, chroot_path = None, controller = stem.control.Controller):
+ """
+ Convenience function for quickly getting a control connection. For more
+ information see the :func:`~stem.connection.connect_port` function.
+
+ In much the same vein as git porcelain commands, users should not rely on
+ details of how this works. Messages or details of this function's behavior
+ might change in the future.
+
+ .. deprecated:: 1.2.0
+ Use :func:`~stem.connection.connect` instead.
+
+ :param str path: path where the control socket is located
+ :param str password: passphrase to authenticate to the socket
+ :param str chroot_path: path prefix if in a chroot environment
+ :param Class controller: :class:`~stem.control.BaseController` subclass to be
+ returned, this provides a :class:`~stem.socket.ControlSocket` if **None**
+
+ :returns: authenticated control connection, the type based on the controller argument
+ """
+
+ try:
+ control_socket = stem.socket.ControlSocketFile(path)
+ except stem.SocketError as exc:
+ print(exc)
+ return None
+
+ return _connect_auth(control_socket, password, True, chroot_path, controller)
+
+
+def _connect_auth(control_socket, password, password_prompt, chroot_path, controller):
+ """
+ Helper for the connect_* functions that authenticates the socket and
+ constructs the controller.
+
+ :param stem.socket.ControlSocket control_socket: socket being authenticated to
+ :param str password: passphrase to authenticate to the socket
+ :param bool password_prompt: prompt for the controller password if it wasn't
+ supplied
+ :param str chroot_path: path prefix if in a chroot environment
+ :param Class controller: :class:`~stem.control.BaseController` subclass to be
+ returned, this provides a :class:`~stem.socket.ControlSocket` if **None**
+
+ :returns: authenticated control connection, the type based on the controller argument
+ """
+
+ try:
+ authenticate(control_socket, password, chroot_path)
+
+ if controller is None:
+ return control_socket
+ else:
+ return controller(control_socket, is_authenticated = True)
+ except IncorrectSocketType:
+ if isinstance(control_socket, stem.socket.ControlPort):
+ print(CONNECT_MESSAGES['wrong_port_type'].format(port = control_socket.get_port()))
+ else:
+ print(CONNECT_MESSAGES['wrong_socket_type'])
+
+ control_socket.close()
+ return None
+ except UnrecognizedAuthMethods as exc:
+ print(CONNECT_MESSAGES['uncrcognized_auth_type'].format(auth_methods = ', '.join(exc.unknown_auth_methods)))
+ control_socket.close()
+ return None
+ except IncorrectPassword:
+ print(CONNECT_MESSAGES['incorrect_password'])
+ control_socket.close()
+ return None
+ except MissingPassword:
+ if password is not None:
+ control_socket.close()
+ raise ValueError(CONNECT_MESSAGES['missing_password_bug'])
+
+ if password_prompt:
+ try:
+ password = getpass.getpass(CONNECT_MESSAGES['password_prompt'] + ' ')
+ except KeyboardInterrupt:
+ control_socket.close()
+ return None
+
+ return _connect_auth(control_socket, password, password_prompt, chroot_path, controller)
+ else:
+ print(CONNECT_MESSAGES['needs_password'])
+ control_socket.close()
+ return None
+ except UnreadableCookieFile as exc:
+ print(CONNECT_MESSAGES['unreadable_cookie_file'].format(path = exc.cookie_path, issue = str(exc)))
+ control_socket.close()
+ return None
+ except AuthenticationFailure as exc:
+ print(CONNECT_MESSAGES['general_auth_failure'].format(error = exc))
+ control_socket.close()
+ return None
+
+
+def authenticate(controller, password = None, chroot_path = None, protocolinfo_response = None):
+ """
+ Authenticates to a control socket using the information provided by a
+ PROTOCOLINFO response. In practice this will often be all we need to
+ authenticate, raising an exception if all attempts to authenticate fail.
+
+ All exceptions are subclasses of AuthenticationFailure so, in practice,
+ callers should catch the types of authentication failure that they care
+ about, then have a :class:`~stem.connection.AuthenticationFailure` catch-all
+ at the end.
+
+ This can authenticate to either a :class:`~stem.control.BaseController` or
+ :class:`~stem.socket.ControlSocket`.
+
+ :param controller: tor controller or socket to be authenticated
+ :param str password: passphrase to present to the socket if it uses password
+ authentication (skips password auth if **None**)
+ :param str chroot_path: path prefix if in a chroot environment
+ :param stem.response.protocolinfo.ProtocolInfoResponse protocolinfo_response:
+ tor protocolinfo response, this is retrieved on our own if **None**
+
+ :raises: If all attempts to authenticate fails then this will raise a
+ :class:`~stem.connection.AuthenticationFailure` subclass. Since this may
+ try multiple authentication methods it may encounter multiple exceptions.
+ If so then the exception this raises is prioritized as follows...
+
+ * :class:`stem.connection.IncorrectSocketType`
+
+ The controller does not speak the tor control protocol. Most often this
+ happened because the user confused the SocksPort or ORPort with the
+ ControlPort.
+
+ * :class:`stem.connection.UnrecognizedAuthMethods`
+
+ All of the authentication methods tor will accept are new and
+ unrecognized. Please upgrade stem and, if that doesn't work, file a
+ ticket on 'trac.torproject.org' and I'd be happy to add support.
+
+ * :class:`stem.connection.MissingPassword`
+
+ We were unable to authenticate but didn't attempt password authentication
+ because none was provided. You should prompt the user for a password and
+ try again via 'authenticate_password'.
+
+ * :class:`stem.connection.IncorrectPassword`
+
+ We were provided with a password but it was incorrect.
+
+ * :class:`stem.connection.IncorrectCookieSize`
+
+ Tor allows for authentication by reading it a cookie file, but that file
+ is the wrong size to be an authentication cookie.
+
+ * :class:`stem.connection.UnreadableCookieFile`
+
+ Tor allows for authentication by reading it a cookie file, but we can't
+ read that file (probably due to permissions).
+
+ * **\***:class:`stem.connection.IncorrectCookieValue`
+
+ Tor allows for authentication by reading it a cookie file, but rejected
+ the contents of that file.
+
+ * **\***:class:`stem.connection.AuthChallengeUnsupported`
+
+ Tor doesn't recognize the AUTHCHALLENGE command. This is probably a Tor
+ version prior to SAFECOOKIE being implement, but this exception shouldn't
+ arise because we won't attempt SAFECOOKIE auth unless Tor claims to
+ support it.
+
+ * **\***:class:`stem.connection.UnrecognizedAuthChallengeMethod`
+
+ Tor couldn't recognize the AUTHCHALLENGE method Stem sent to it. This
+ shouldn't happen at all.
+
+ * **\***:class:`stem.connection.InvalidClientNonce`
+
+ Tor says that the client nonce provided by Stem during the AUTHCHALLENGE
+ process is invalid.
+
+ * **\***:class:`stem.connection.AuthSecurityFailure`
+
+ Nonce value provided by the server was invalid.
+
+ * **\***:class:`stem.connection.OpenAuthRejected`
+
+ Tor says that it allows for authentication without any credentials, but
+ then rejected our authentication attempt.
+
+ * **\***:class:`stem.connection.MissingAuthInfo`
+
+ Tor provided us with a PROTOCOLINFO reply that is technically valid, but
+ missing the information we need to authenticate.
+
+ * **\***:class:`stem.connection.AuthenticationFailure`
+
+ There are numerous other ways that authentication could have failed
+ including socket failures, malformed controller responses, etc. These
+ mostly constitute transient failures or bugs.
+
+ **\*** In practice it is highly unusual for this to occur, being more of a
+ theoretical possibility rather than something you should expect. It's fine
+ to treat these as errors. If you have a use case where this commonly
+ happens, please file a ticket on 'trac.torproject.org'.
+
+ In the future new :class:`~stem.connection.AuthenticationFailure`
+ subclasses may be added to allow for better error handling.
+ """
+
+ if not protocolinfo_response:
+ try:
+ protocolinfo_response = get_protocolinfo(controller)
+ except stem.ProtocolError:
+ raise IncorrectSocketType('unable to use the control socket')
+ except stem.SocketError as exc:
+ raise AuthenticationFailure('socket connection failed (%s)' % exc)
+
+ auth_methods = list(protocolinfo_response.auth_methods)
+ auth_exceptions = []
+
+ if len(auth_methods) == 0:
+ raise NoAuthMethods('our PROTOCOLINFO response did not have any methods for authenticating')
+
+ # remove authentication methods that are either unknown or for which we don't
+ # have an input
+ if AuthMethod.UNKNOWN in auth_methods:
+ auth_methods.remove(AuthMethod.UNKNOWN)
+
+ unknown_methods = protocolinfo_response.unknown_auth_methods
+ plural_label = 's' if len(unknown_methods) > 1 else ''
+ methods_label = ', '.join(unknown_methods)
+
+ # we... er, can't do anything with only unrecognized auth types
+ if not auth_methods:
+ exc_msg = 'unrecognized authentication method%s (%s)' % (plural_label, methods_label)
+ auth_exceptions.append(UnrecognizedAuthMethods(exc_msg, unknown_methods))
+ else:
+ log.debug('Authenticating to a socket with unrecognized auth method%s, ignoring them: %s' % (plural_label, methods_label))
+
+ if protocolinfo_response.cookie_path is None:
+ for cookie_auth_method in (AuthMethod.COOKIE, AuthMethod.SAFECOOKIE):
+ if cookie_auth_method in auth_methods:
+ auth_methods.remove(cookie_auth_method)
+
+ exc_msg = 'our PROTOCOLINFO response did not have the location of our authentication cookie'
+ auth_exceptions.append(NoAuthCookie(exc_msg, cookie_auth_method == AuthMethod.SAFECOOKIE))
+
+ if AuthMethod.PASSWORD in auth_methods and password is None:
+ auth_methods.remove(AuthMethod.PASSWORD)
+ auth_exceptions.append(MissingPassword('no passphrase provided'))
+
+ # iterating over AuthMethods so we can try them in this order
+ for auth_type in (AuthMethod.NONE, AuthMethod.PASSWORD, AuthMethod.SAFECOOKIE, AuthMethod.COOKIE):
+ if auth_type not in auth_methods:
+ continue
+
+ try:
+ if auth_type == AuthMethod.NONE:
+ authenticate_none(controller, False)
+ elif auth_type == AuthMethod.PASSWORD:
+ authenticate_password(controller, password, False)
+ elif auth_type in (AuthMethod.COOKIE, AuthMethod.SAFECOOKIE):
+ cookie_path = protocolinfo_response.cookie_path
+
+ if chroot_path:
+ cookie_path = os.path.join(chroot_path, cookie_path.lstrip(os.path.sep))
+
+ if auth_type == AuthMethod.SAFECOOKIE:
+ authenticate_safecookie(controller, cookie_path, False)
+ else:
+ authenticate_cookie(controller, cookie_path, False)
+
+ return # success!
+ except OpenAuthRejected as exc:
+ auth_exceptions.append(exc)
+ except IncorrectPassword as exc:
+ auth_exceptions.append(exc)
+ except PasswordAuthRejected as exc:
+ # Since the PROTOCOLINFO says password auth is available we can assume
+ # that if PasswordAuthRejected is raised it's being raised in error.
+ log.debug('The authenticate_password method raised a PasswordAuthRejected when password auth should be available. Stem may need to be corrected to recognize this response: %s' % exc)
+ auth_exceptions.append(IncorrectPassword(str(exc)))
+ except AuthSecurityFailure as exc:
+ log.info('Tor failed to provide the nonce expected for safecookie authentication. (%s)' % exc)
+ auth_exceptions.append(exc)
+ except (InvalidClientNonce, UnrecognizedAuthChallengeMethod, AuthChallengeFailed) as exc:
+ auth_exceptions.append(exc)
+ except (IncorrectCookieSize, UnreadableCookieFile, IncorrectCookieValue) as exc:
+ auth_exceptions.append(exc)
+ except CookieAuthRejected as exc:
+ auth_func = 'authenticate_safecookie' if exc.is_safecookie else 'authenticate_cookie'
+
+ log.debug('The %s method raised a CookieAuthRejected when cookie auth should be available. Stem may need to be corrected to recognize this response: %s' % (auth_func, exc))
+ auth_exceptions.append(IncorrectCookieValue(str(exc), exc.cookie_path, exc.is_safecookie))
+ except stem.ControllerError as exc:
+ auth_exceptions.append(AuthenticationFailure(str(exc)))
+
+ # All authentication attempts failed. Raise the exception that takes priority
+ # according to our pydocs.
+
+ for exc_type in AUTHENTICATE_EXCEPTIONS:
+ for auth_exc in auth_exceptions:
+ if isinstance(auth_exc, exc_type):
+ raise auth_exc
+
+ # We really, really shouldn't get here. It means that auth_exceptions is
+ # either empty or contains something that isn't an AuthenticationFailure.
+
+ raise AssertionError('BUG: Authentication failed without providing a recognized exception: %s' % str(auth_exceptions))
+
+
+def authenticate_none(controller, suppress_ctl_errors = True):
+ """
+ Authenticates to an open control socket. All control connections need to
+ authenticate before they can be used, even if tor hasn't been configured to
+ use any authentication.
+
+ If authentication fails tor will disconnect and we'll make a best effort
+ attempt to re-establish the connection. This may not succeed, so check
+ :func:`~stem.socket.ControlSocket.is_alive` before using the socket further.
+
+ This can authenticate to either a :class:`~stem.control.BaseController` or
+ :class:`~stem.socket.ControlSocket`.
+
+ For general usage use the :func:`~stem.connection.authenticate` function
+ instead.
+
+ :param controller: tor controller or socket to be authenticated
+ :param bool suppress_ctl_errors: reports raised
+ :class:`~stem.ControllerError` as authentication rejection if
+ **True**, otherwise they're re-raised
+
+ :raises: :class:`stem.connection.OpenAuthRejected` if the empty authentication credentials aren't accepted
+ """
+
+ try:
+ auth_response = _msg(controller, 'AUTHENTICATE')
+
+ # if we got anything but an OK response then error
+ if str(auth_response) != 'OK':
+ try:
+ controller.connect()
+ except:
+ pass
+
+ raise OpenAuthRejected(str(auth_response), auth_response)
+ except stem.ControllerError as exc:
+ try:
+ controller.connect()
+ except:
+ pass
+
+ if not suppress_ctl_errors:
+ raise exc
+ else:
+ raise OpenAuthRejected('Socket failed (%s)' % exc)
+
+
+def authenticate_password(controller, password, suppress_ctl_errors = True):
+ """
+ Authenticates to a control socket that uses a password (via the
+ HashedControlPassword torrc option). Quotes in the password are escaped.
+
+ If authentication fails tor will disconnect and we'll make a best effort
+ attempt to re-establish the connection. This may not succeed, so check
+ :func:`~stem.socket.ControlSocket.is_alive` before using the socket further.
+
+ If you use this function directly, rather than
+ :func:`~stem.connection.authenticate`, we may mistakenly raise a
+ PasswordAuthRejected rather than IncorrectPassword. This is because we rely
+ on tor's error messaging which is liable to change in future versions
+ (:trac:`4817`).
+
+ This can authenticate to either a :class:`~stem.control.BaseController` or
+ :class:`~stem.socket.ControlSocket`.
+
+ For general usage use the :func:`~stem.connection.authenticate` function
+ instead.
+
+ :param controller: tor controller or socket to be authenticated
+ :param str password: passphrase to present to the socket
+ :param bool suppress_ctl_errors: reports raised
+ :class:`~stem.ControllerError` as authentication rejection if
+ **True**, otherwise they're re-raised
+
+ :raises:
+ * :class:`stem.connection.PasswordAuthRejected` if the socket doesn't
+ accept password authentication
+ * :class:`stem.connection.IncorrectPassword` if the authentication
+ credentials aren't accepted
+ """
+
+ # Escapes quotes. Tor can include those in the password hash, in which case
+ # it expects escaped quotes from the controller. For more information see...
+ # https://trac.torproject.org/projects/tor/ticket/4600
+
+ password = password.replace('"', '\\"')
+
+ try:
+ auth_response = _msg(controller, 'AUTHENTICATE "%s"' % password)
+
+ # if we got anything but an OK response then error
+ if str(auth_response) != 'OK':
+ try:
+ controller.connect()
+ except:
+ pass
+
+ # all we have to go on is the error message from tor...
+ # Password did not match HashedControlPassword value value from configuration...
+ # Password did not match HashedControlPassword *or*...
+
+ if 'Password did not match HashedControlPassword' in str(auth_response):
+ raise IncorrectPassword(str(auth_response), auth_response)
+ else:
+ raise PasswordAuthRejected(str(auth_response), auth_response)
+ except stem.ControllerError as exc:
+ try:
+ controller.connect()
+ except:
+ pass
+
+ if not suppress_ctl_errors:
+ raise exc
+ else:
+ raise PasswordAuthRejected('Socket failed (%s)' % exc)
+
+
+def authenticate_cookie(controller, cookie_path, suppress_ctl_errors = True):
+ """
+ Authenticates to a control socket that uses the contents of an authentication
+ cookie (generated via the CookieAuthentication torrc option). This does basic
+ validation that this is a cookie before presenting the contents to the
+ socket.
+
+ The :class:`~stem.connection.IncorrectCookieSize` and
+ :class:`~stem.connection.UnreadableCookieFile` exceptions take precedence
+ over the other types.
+
+ If authentication fails tor will disconnect and we'll make a best effort
+ attempt to re-establish the connection. This may not succeed, so check
+ :func:`~stem.socket.ControlSocket.is_alive` before using the socket further.
+
+ If you use this function directly, rather than
+ :func:`~stem.connection.authenticate`, we may mistakenly raise a
+ :class:`~stem.connection.CookieAuthRejected` rather than
+ :class:`~stem.connection.IncorrectCookieValue`. This is because we rely on
+ tor's error messaging which is liable to change in future versions
+ (:trac:`4817`).
+
+ This can authenticate to either a :class:`~stem.control.BaseController` or
+ :class:`~stem.socket.ControlSocket`.
+
+ For general usage use the :func:`~stem.connection.authenticate` function
+ instead.
+
+ :param controller: tor controller or socket to be authenticated
+ :param str cookie_path: path of the authentication cookie to send to tor
+ :param bool suppress_ctl_errors: reports raised
+ :class:`~stem.ControllerError` as authentication rejection if
+ **True**, otherwise they're re-raised
+
+ :raises:
+ * :class:`stem.connection.IncorrectCookieSize` if the cookie file's size
+ is wrong
+ * :class:`stem.connection.UnreadableCookieFile` if the cookie file doesn't
+ exist or we're unable to read it
+ * :class:`stem.connection.CookieAuthRejected` if cookie authentication is
+ attempted but the socket doesn't accept it
+ * :class:`stem.connection.IncorrectCookieValue` if the cookie file's value
+ is rejected
+ """
+
+ cookie_data = _read_cookie(cookie_path, False)
+
+ try:
+ # binascii.b2a_hex() takes a byte string and returns one too. With python 3
+ # this is a problem because string formatting for byte strings includes the
+ # b'' wrapper...
+ #
+ # >>> "AUTHENTICATE %s" % b'content'
+ # "AUTHENTICATE b'content'"
+ #
+ # This seems dumb but oh well. Converting the result to unicode so it won't
+ # misbehave.
+
+ auth_token_hex = binascii.b2a_hex(stem.util.str_tools._to_bytes(cookie_data))
+ msg = 'AUTHENTICATE %s' % stem.util.str_tools._to_unicode(auth_token_hex)
+ auth_response = _msg(controller, msg)
+
+ # if we got anything but an OK response then error
+ if str(auth_response) != 'OK':
+ try:
+ controller.connect()
+ except:
+ pass
+
+ # all we have to go on is the error message from tor...
+ # ... Authentication cookie did not match expected value.
+ # ... *or* authentication cookie.
+
+ if '*or* authentication cookie.' in str(auth_response) or \
+ 'Authentication cookie did not match expected value.' in str(auth_response):
+ raise IncorrectCookieValue(str(auth_response), cookie_path, False, auth_response)
+ else:
+ raise CookieAuthRejected(str(auth_response), cookie_path, False, auth_response)
+ except stem.ControllerError as exc:
+ try:
+ controller.connect()
+ except:
+ pass
+
+ if not suppress_ctl_errors:
+ raise exc
+ else:
+ raise CookieAuthRejected('Socket failed (%s)' % exc, cookie_path, False)
+
+
+def authenticate_safecookie(controller, cookie_path, suppress_ctl_errors = True):
+ """
+ Authenticates to a control socket using the safe cookie method, which is
+ enabled by setting the CookieAuthentication torrc option on Tor client's which
+ support it.
+
+ Authentication with this is a two-step process...
+
+ 1. send a nonce to the server and receives a challenge from the server for
+ the cookie's contents
+ 2. generate a hash digest using the challenge received in the first step, and
+ use it to authenticate the controller
+
+ The :class:`~stem.connection.IncorrectCookieSize` and
+ :class:`~stem.connection.UnreadableCookieFile` exceptions take precedence
+ over the other exception types.
+
+ The :class:`~stem.connection.AuthChallengeUnsupported`,
+ :class:`~stem.connection.UnrecognizedAuthChallengeMethod`,
+ :class:`~stem.connection.InvalidClientNonce` and
+ :class:`~stem.connection.CookieAuthRejected` exceptions are next in the order
+ of precedence. Depending on the reason, one of these is raised if the first
+ (AUTHCHALLENGE) step fails.
+
+ In the second (AUTHENTICATE) step,
+ :class:`~stem.connection.IncorrectCookieValue` or
+ :class:`~stem.connection.CookieAuthRejected` maybe raised.
+
+ If authentication fails tor will disconnect and we'll make a best effort
+ attempt to re-establish the connection. This may not succeed, so check
+ :func:`~stem.socket.ControlSocket.is_alive` before using the socket further.
+
+ For general usage use the :func:`~stem.connection.authenticate` function
+ instead.
+
+ :param controller: tor controller or socket to be authenticated
+ :param str cookie_path: path of the authentication cookie to send to tor
+ :param bool suppress_ctl_errors: reports raised
+ :class:`~stem.ControllerError` as authentication rejection if
+ **True**, otherwise they're re-raised
+
+ :raises:
+ * :class:`stem.connection.IncorrectCookieSize` if the cookie file's size
+ is wrong
+ * :class:`stem.connection.UnreadableCookieFile` if the cookie file doesn't
+ exist or we're unable to read it
+ * :class:`stem.connection.CookieAuthRejected` if cookie authentication is
+ attempted but the socket doesn't accept it
+ * :class:`stem.connection.IncorrectCookieValue` if the cookie file's value
+ is rejected
+ * :class:`stem.connection.UnrecognizedAuthChallengeMethod` if the Tor
+ client fails to recognize the AuthChallenge method
+ * :class:`stem.connection.AuthChallengeUnsupported` if AUTHCHALLENGE is
+ unimplemented, or if unable to parse AUTHCHALLENGE response
+ * :class:`stem.connection.AuthSecurityFailure` if AUTHCHALLENGE's response
+ looks like a security attack
+ * :class:`stem.connection.InvalidClientNonce` if stem's AUTHCHALLENGE
+ client nonce is rejected for being invalid
+ """
+
+ cookie_data = _read_cookie(cookie_path, True)
+ client_nonce = os.urandom(32)
+
+ try:
+ client_nonce_hex = stem.util.str_tools._to_unicode(binascii.b2a_hex(client_nonce))
+ authchallenge_response = _msg(controller, 'AUTHCHALLENGE SAFECOOKIE %s' % client_nonce_hex)
+
+ if not authchallenge_response.is_ok():
+ try:
+ controller.connect()
+ except:
+ pass
+
+ authchallenge_response_str = str(authchallenge_response)
+
+ if 'Authentication required.' in authchallenge_response_str:
+ raise AuthChallengeUnsupported("SAFECOOKIE authentication isn't supported", cookie_path)
+ elif 'AUTHCHALLENGE only supports' in authchallenge_response_str:
+ raise UnrecognizedAuthChallengeMethod(authchallenge_response_str, cookie_path)
+ elif 'Invalid base16 client nonce' in authchallenge_response_str:
+ raise InvalidClientNonce(authchallenge_response_str, cookie_path)
+ elif 'Cookie authentication is disabled' in authchallenge_response_str:
+ raise CookieAuthRejected(authchallenge_response_str, cookie_path, True)
+ else:
+ raise AuthChallengeFailed(authchallenge_response, cookie_path)
+ except stem.ControllerError as exc:
+ try:
+ controller.connect()
+ except:
+ pass
+
+ if not suppress_ctl_errors:
+ raise exc
+ else:
+ raise AuthChallengeFailed('Socket failed (%s)' % exc, cookie_path, True)
+
+ try:
+ stem.response.convert('AUTHCHALLENGE', authchallenge_response)
+ except stem.ProtocolError as exc:
+ if not suppress_ctl_errors:
+ raise exc
+ else:
+ raise AuthChallengeFailed('Unable to parse AUTHCHALLENGE response: %s' % exc, cookie_path)
+
+ expected_server_hash = stem.util.connection._hmac_sha256(
+ SERVER_HASH_CONSTANT,
+ cookie_data + client_nonce + authchallenge_response.server_nonce)
+
+ if not stem.util.connection._cryptovariables_equal(authchallenge_response.server_hash, expected_server_hash):
+ raise AuthSecurityFailure('Tor provided the wrong server nonce', cookie_path)
+
+ try:
+ client_hash = stem.util.connection._hmac_sha256(
+ CLIENT_HASH_CONSTANT,
+ cookie_data + client_nonce + authchallenge_response.server_nonce)
+
+ auth_response = _msg(controller, 'AUTHENTICATE %s' % stem.util.str_tools._to_unicode(binascii.b2a_hex(client_hash)))
+ except stem.ControllerError as exc:
+ try:
+ controller.connect()
+ except:
+ pass
+
+ if not suppress_ctl_errors:
+ raise exc
+ else:
+ raise CookieAuthRejected('Socket failed (%s)' % exc, cookie_path, True, auth_response)
+
+ # if we got anything but an OK response then err
+ if not auth_response.is_ok():
+ try:
+ controller.connect()
+ except:
+ pass
+
+ # all we have to go on is the error message from tor...
+ # ... Safe cookie response did not match expected value
+ # ... *or* authentication cookie.
+
+ if '*or* authentication cookie.' in str(auth_response) or \
+ 'Safe cookie response did not match expected value' in str(auth_response):
+ raise IncorrectCookieValue(str(auth_response), cookie_path, True, auth_response)
+ else:
+ raise CookieAuthRejected(str(auth_response), cookie_path, True, auth_response)
+
+
+def get_protocolinfo(controller):
+ """
+ Issues a PROTOCOLINFO query to a control socket, getting information about
+ the tor process running on it. If the socket is already closed then it is
+ first reconnected.
+
+ According to the control spec the cookie_file is an absolute path. However,
+ this often is not the case (especially for the Tor Browser Bundle). If the
+ path is relative then we'll make an attempt (which may not work) to correct
+ this (:trac:`1101`).
+
+ This can authenticate to either a :class:`~stem.control.BaseController` or
+ :class:`~stem.socket.ControlSocket`.
+
+ :param controller: tor controller or socket to be queried
+
+ :returns: :class:`~stem.response.protocolinfo.ProtocolInfoResponse` provided by tor
+
+ :raises:
+ * :class:`stem.ProtocolError` if the PROTOCOLINFO response is
+ malformed
+ * :class:`stem.SocketError` if problems arise in establishing or
+ using the socket
+ """
+
+ try:
+ protocolinfo_response = _msg(controller, 'PROTOCOLINFO 1')
+ except:
+ protocolinfo_response = None
+
+ # Tor hangs up on sockets after receiving a PROTOCOLINFO query if it isn't
+ # next followed by authentication. Transparently reconnect if that happens.
+
+ if not protocolinfo_response or str(protocolinfo_response) == 'Authentication required.':
+ controller.connect()
+
+ try:
+ protocolinfo_response = _msg(controller, 'PROTOCOLINFO 1')
+ except stem.SocketClosed as exc:
+ raise stem.SocketError(exc)
+
+ stem.response.convert('PROTOCOLINFO', protocolinfo_response)
+
+ # attempt to expand relative cookie paths
+
+ if protocolinfo_response.cookie_path:
+ _expand_cookie_path(protocolinfo_response, stem.util.system.pid_by_name, 'tor')
+
+ # attempt to expand relative cookie paths via the control port or socket file
+
+ if isinstance(controller, stem.socket.ControlSocket):
+ control_socket = controller
+ else:
+ control_socket = controller.get_socket()
+
+ if isinstance(control_socket, stem.socket.ControlPort):
+ if control_socket.get_address() == '127.0.0.1':
+ pid_method = stem.util.system.pid_by_port
+ _expand_cookie_path(protocolinfo_response, pid_method, control_socket.get_port())
+ elif isinstance(control_socket, stem.socket.ControlSocketFile):
+ pid_method = stem.util.system.pid_by_open_file
+ _expand_cookie_path(protocolinfo_response, pid_method, control_socket.get_socket_path())
+
+ return protocolinfo_response
+
+
+def _msg(controller, message):
+ """
+ Sends and receives a message with either a
+ :class:`~stem.socket.ControlSocket` or :class:`~stem.control.BaseController`.
+ """
+
+ if isinstance(controller, stem.socket.ControlSocket):
+ controller.send(message)
+ return controller.recv()
+ else:
+ return controller.msg(message)
+
+
+def _read_cookie(cookie_path, is_safecookie):
+ """
+ Provides the contents of a given cookie file.
+
+ :param str cookie_path: absolute path of the cookie file
+ :param bool is_safecookie: **True** if this was for SAFECOOKIE
+ authentication, **False** if for COOKIE
+
+ :raises:
+ * :class:`stem.connection.UnreadableCookieFile` if the cookie file is
+ unreadable
+ * :class:`stem.connection.IncorrectCookieSize` if the cookie size is
+ incorrect (not 32 bytes)
+ """
+
+ if not os.path.exists(cookie_path):
+ exc_msg = "Authentication failed: '%s' doesn't exist" % cookie_path
+ raise UnreadableCookieFile(exc_msg, cookie_path, is_safecookie)
+
+ # Abort if the file isn't 32 bytes long. This is to avoid exposing arbitrary
+ # file content to the port.
+ #
+ # Without this a malicious socket could, for instance, claim that
+ # '~/.bash_history' or '~/.ssh/id_rsa' was its authentication cookie to trick
+ # us into reading it for them with our current permissions.
+ #
+ # https://trac.torproject.org/projects/tor/ticket/4303
+
+ auth_cookie_size = os.path.getsize(cookie_path)
+
+ if auth_cookie_size != 32:
+ exc_msg = "Authentication failed: authentication cookie '%s' is the wrong size (%i bytes instead of 32)" % (cookie_path, auth_cookie_size)
+ raise IncorrectCookieSize(exc_msg, cookie_path, is_safecookie)
+
+ try:
+ with open(cookie_path, 'rb', 0) as f:
+ return f.read()
+ except IOError as exc:
+ exc_msg = "Authentication failed: unable to read '%s' (%s)" % (cookie_path, exc)
+ raise UnreadableCookieFile(exc_msg, cookie_path, is_safecookie)
+
+
+def _expand_cookie_path(protocolinfo_response, pid_resolver, pid_resolution_arg):
+ """
+ Attempts to expand a relative cookie path with the given pid resolver. This
+ leaves the cookie_path alone if it's already absolute, **None**, or the
+ system calls fail.
+ """
+
+ cookie_path = protocolinfo_response.cookie_path
+ if cookie_path and not os.path.isabs(cookie_path):
+ try:
+ tor_pid = pid_resolver(pid_resolution_arg)
+
+ if not tor_pid:
+ raise IOError('pid lookup failed')
+
+ tor_cwd = stem.util.system.cwd(tor_pid)
+
+ if not tor_cwd:
+ raise IOError('cwd lookup failed')
+
+ cookie_path = stem.util.system.expand_path(cookie_path, tor_cwd)
+ except IOError as exc:
+ resolver_labels = {
+ stem.util.system.pid_by_name: ' by name',
+ stem.util.system.pid_by_port: ' by port',
+ stem.util.system.pid_by_open_file: ' by socket file',
+ }
+
+ pid_resolver_label = resolver_labels.get(pid_resolver, '')
+ log.debug('unable to expand relative tor cookie path%s: %s' % (pid_resolver_label, exc))
+
+ protocolinfo_response.cookie_path = cookie_path
+
+
+class AuthenticationFailure(Exception):
+ """
+ Base error for authentication failures.
+
+ :var stem.socket.ControlMessage auth_response: AUTHENTICATE response from the
+ control socket, **None** if one wasn't received
+ """
+
+ def __init__(self, message, auth_response = None):
+ super(AuthenticationFailure, self).__init__(message)
+ self.auth_response = auth_response
+
+
+class UnrecognizedAuthMethods(AuthenticationFailure):
+ """
+ All methods for authenticating aren't recognized.
+
+ :var list unknown_auth_methods: authentication methods that weren't recognized
+ """
+
+ def __init__(self, message, unknown_auth_methods):
+ super(UnrecognizedAuthMethods, self).__init__(message)
+ self.unknown_auth_methods = unknown_auth_methods
+
+
+class IncorrectSocketType(AuthenticationFailure):
+ 'Socket does not speak the control protocol.'
+
+
+class OpenAuthFailed(AuthenticationFailure):
+ 'Failure to authenticate to an open socket.'
+
+
+class OpenAuthRejected(OpenAuthFailed):
+ 'Attempt to connect to an open control socket was rejected.'
+
+
+class PasswordAuthFailed(AuthenticationFailure):
+ 'Failure to authenticate with a password.'
+
+
+class PasswordAuthRejected(PasswordAuthFailed):
+ 'Socket does not support password authentication.'
+
+
+class IncorrectPassword(PasswordAuthFailed):
+ 'Authentication password incorrect.'
+
+
+class MissingPassword(PasswordAuthFailed):
+ "Password authentication is supported but we weren't provided with one."
+
+
+class CookieAuthFailed(AuthenticationFailure):
+ """
+ Failure to authenticate with an authentication cookie.
+
+ :param str cookie_path: location of the authentication cookie we attempted
+ :param bool is_safecookie: **True** if this was for SAFECOOKIE
+ authentication, **False** if for COOKIE
+ :param stem.response.ControlMessage auth_response: reply to our
+ authentication attempt
+ """
+
+ def __init__(self, message, cookie_path, is_safecookie, auth_response = None):
+ super(CookieAuthFailed, self).__init__(message, auth_response)
+ self.is_safecookie = is_safecookie
+ self.cookie_path = cookie_path
+
+
+class CookieAuthRejected(CookieAuthFailed):
+ 'Socket does not support password authentication.'
+
+
+class IncorrectCookieValue(CookieAuthFailed):
+ 'Authentication cookie value was rejected.'
+
+
+class IncorrectCookieSize(CookieAuthFailed):
+ 'Aborted because the cookie file is the wrong size.'
+
+
+class UnreadableCookieFile(CookieAuthFailed):
+ 'Error arose in reading the authentication cookie.'
+
+
+class AuthChallengeFailed(CookieAuthFailed):
+ """
+ AUTHCHALLENGE command has failed.
+ """
+
+ def __init__(self, message, cookie_path):
+ super(AuthChallengeFailed, self).__init__(message, cookie_path, True)
+
+
+class AuthChallengeUnsupported(AuthChallengeFailed):
+ """
+ AUTHCHALLENGE isn't implemented.
+ """
+
+
+class UnrecognizedAuthChallengeMethod(AuthChallengeFailed):
+ """
+ Tor couldn't recognize our AUTHCHALLENGE method.
+
+ :var str authchallenge_method: AUTHCHALLENGE method that Tor couldn't recognize
+ """
+
+ def __init__(self, message, cookie_path, authchallenge_method):
+ super(UnrecognizedAuthChallengeMethod, self).__init__(message, cookie_path)
+ self.authchallenge_method = authchallenge_method
+
+
+class AuthSecurityFailure(AuthChallengeFailed):
+ 'AUTHCHALLENGE response is invalid.'
+
+
+class InvalidClientNonce(AuthChallengeFailed):
+ 'AUTHCHALLENGE request contains an invalid client nonce.'
+
+
+class MissingAuthInfo(AuthenticationFailure):
+ """
+ The PROTOCOLINFO response didn't have enough information to authenticate.
+ These are valid control responses but really shouldn't happen in practice.
+ """
+
+
+class NoAuthMethods(MissingAuthInfo):
+ "PROTOCOLINFO response didn't have any methods for authenticating."
+
+
+class NoAuthCookie(MissingAuthInfo):
+ """
+ PROTOCOLINFO response supports cookie auth but doesn't have its path.
+
+ :param bool is_safecookie: **True** if this was for SAFECOOKIE
+ authentication, **False** if for COOKIE
+ """
+
+ def __init__(self, message, is_safecookie):
+ super(NoAuthCookie, self).__init__(message)
+ self.is_safecookie = is_safecookie
+
+# authentication exceptions ordered as per the authenticate function's pydocs
+AUTHENTICATE_EXCEPTIONS = (
+ IncorrectSocketType,
+ UnrecognizedAuthMethods,
+ MissingPassword,
+ IncorrectPassword,
+ IncorrectCookieSize,
+ UnreadableCookieFile,
+ IncorrectCookieValue,
+ AuthChallengeUnsupported,
+ UnrecognizedAuthChallengeMethod,
+ InvalidClientNonce,
+ AuthSecurityFailure,
+ OpenAuthRejected,
+ MissingAuthInfo,
+ AuthenticationFailure
+)
diff --git a/ext/stem/control.py b/ext/stem/control.py
new file mode 100644
index 00000000..657f5596
--- /dev/null
+++ b/ext/stem/control.py
@@ -0,0 +1,3631 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Module for interacting with the Tor control socket. The
+:class:`~stem.control.Controller` is a wrapper around a
+:class:`~stem.socket.ControlSocket`, retaining many of its methods (connect,
+close, is_alive, etc) in addition to providing its own for working with the
+socket at a higher level.
+
+Stem has `several ways <../faq.html#how-do-i-connect-to-tor>`_ of getting a
+:class:`~stem.control.Controller`, but the most flexible are
+:func:`~stem.control.Controller.from_port` and
+:func:`~stem.control.Controller.from_socket_file`. These static
+:class:`~stem.control.Controller` methods give you an **unauthenticated**
+Controller you can then authenticate yourself using its
+:func:`~stem.control.Controller.authenticate` method. For example...
+
+::
+
+ import getpass
+ import sys
+
+ import stem
+ import stem.connection
+
+ from stem.control import Controller
+
+ if __name__ == '__main__':
+ try:
+ controller = Controller.from_port()
+ except stem.SocketError as exc:
+ print("Unable to connect to tor on port 9051: %s" % exc)
+ sys.exit(1)
+
+ try:
+ controller.authenticate()
+ except stem.connection.MissingPassword:
+ pw = getpass.getpass("Controller password: ")
+
+ try:
+ controller.authenticate(password = pw)
+ except stem.connection.PasswordAuthFailed:
+ print("Unable to authenticate, password is incorrect")
+ sys.exit(1)
+ except stem.connection.AuthenticationFailure as exc:
+ print("Unable to authenticate: %s" % exc)
+ sys.exit(1)
+
+ print("Tor is running version %s" % controller.get_version())
+ controller.close()
+
+If you're fine with allowing your script to raise exceptions then this can be more nicely done as...
+
+::
+
+ from stem.control import Controller
+
+ if __name__ == '__main__':
+ with Controller.from_port() as controller:
+ controller.authenticate()
+
+ print("Tor is running version %s" % controller.get_version())
+
+**Module Overview:**
+
+::
+
+ Controller - General controller class intended for direct use
+ | |- from_port - Provides a Controller based on a port connection.
+ | +- from_socket_file - Provides a Controller based on a socket file connection.
+ |
+ |- authenticate - authenticates this controller with tor
+ |
+ |- get_info - issues a GETINFO query for a parameter
+ |- get_version - provides our tor version
+ |- get_exit_policy - provides our exit policy
+ |- get_ports - provides the local ports where tor is listening for connections
+ |- get_listeners - provides the addresses and ports where tor is listening for connections
+ |- get_accounting_stats - provides stats related to relaying limits
+ |- get_protocolinfo - information about the controller interface
+ |- get_user - provides the user tor is running as
+ |- get_pid - provides the pid of our tor process
+ |
+ |- get_microdescriptor - querying the microdescriptor for a relay
+ |- get_microdescriptors - provides all currently available microdescriptors
+ |- get_server_descriptor - querying the server descriptor for a relay
+ |- get_server_descriptors - provides all currently available server descriptors
+ |- get_network_status - querying the router status entry for a relay
+ |- get_network_statuses - provides all preently available router status entries
+ |- get_hidden_service_descriptor - queries the given hidden service descriptor
+ |
+ |- get_conf - gets the value of a configuration option
+ |- get_conf_map - gets the values of multiple configuration options
+ |- set_conf - sets the value of a configuration option
+ |- reset_conf - reverts configuration options to their default values
+ |- set_options - sets or resets the values of multiple configuration options
+ |
+ |- get_hidden_service_conf - provides our hidden service configuration
+ |- set_hidden_service_conf - sets our hidden service configuration
+ |- create_hidden_service - creates a new hidden service or adds a new port
+ |- remove_hidden_service - removes a hidden service or drops a port
+ |
+ |- list_ephemeral_hidden_services - list ephemeral hidden serivces
+ |- create_ephemeral_hidden_service - create a new ephemeral hidden service
+ |- remove_ephemeral_hidden_service - removes an ephemeral hidden service
+ |
+ |- add_event_listener - attaches an event listener to be notified of tor events
+ |- remove_event_listener - removes a listener so it isn't notified of further events
+ |
+ |- is_caching_enabled - true if the controller has enabled caching
+ |- set_caching - enables or disables caching
+ |- clear_cache - clears any cached results
+ |
+ |- load_conf - loads configuration information as if it was in the torrc
+ |- save_conf - saves configuration information to the torrc
+ |
+ |- is_feature_enabled - checks if a given controller feature is enabled
+ |- enable_feature - enables a controller feature that has been disabled by default
+ |
+ |- get_circuit - provides an active circuit
+ |- get_circuits - provides a list of active circuits
+ |- new_circuit - create new circuits
+ |- extend_circuit - create new circuits and extend existing ones
+ |- repurpose_circuit - change a circuit's purpose
+ |- close_circuit - close a circuit
+ |
+ |- get_streams - provides a list of active streams
+ |- attach_stream - attach a stream to a circuit
+ |- close_stream - close a stream
+ |
+ |- signal - sends a signal to the tor client
+ |- is_newnym_available - true if tor would currently accept a NEWNYM signal
+ |- get_newnym_wait - seconds until tor would accept a NEWNYM signal
+ |- get_effective_rate - provides our effective relaying rate limit
+ |- is_geoip_unavailable - true if we've discovered our geoip db to be unavailable
+ |- map_address - maps one address to another such that connections to the original are replaced with the other
+ +- drop_guards - drops our set of guard relays and picks a new set
+
+ BaseController - Base controller class asynchronous message handling
+ |- msg - communicates with the tor process
+ |- is_alive - reports if our connection to tor is open or closed
+ |- is_localhost - returns if the connection is for the local system or not
+ |- connection_time - time when we last connected or disconnected
+ |- is_authenticated - checks if we're authenticated to tor
+ |- connect - connects or reconnects to tor
+ |- close - shuts down our connection to the tor process
+ |- get_socket - provides the socket used for control communication
+ |- get_latest_heartbeat - timestamp for when we last heard from tor
+ |- add_status_listener - notifies a callback of changes in our status
+ |- remove_status_listener - prevents further notification of status changes
+ +- __enter__ / __exit__ - manages socket connection
+
+.. data:: State (enum)
+
+ Enumeration for states that a controller can have.
+
+ ========== ===========
+ State Description
+ ========== ===========
+ **INIT** new control connection
+ **RESET** received a reset/sighup signal
+ **CLOSED** control connection closed
+ ========== ===========
+
+.. data:: EventType (enum)
+
+ Known types of events that the
+ :func:`~stem.control.Controller.add_event_listener` method of the
+ :class:`~stem.control.Controller` can listen for.
+
+ The most frequently listened for event types tend to be the logging events
+ (**DEBUG**, **INFO**, **NOTICE**, **WARN**, and **ERR**), bandwidth usage
+ (**BW**), and circuit or stream changes (**CIRC** and **STREAM**).
+
+ Enums are mapped to :class:`~stem.response.events.Event` subclasses as
+ follows...
+
+ ======================= ===========
+ EventType Event Class
+ ======================= ===========
+ **ADDRMAP** :class:`stem.response.events.AddrMapEvent`
+ **AUTHDIR_NEWDESCS** :class:`stem.response.events.AuthDirNewDescEvent`
+ **BUILDTIMEOUT_SET** :class:`stem.response.events.BuildTimeoutSetEvent`
+ **BW** :class:`stem.response.events.BandwidthEvent`
+ **CELL_STATS** :class:`stem.response.events.CellStatsEvent`
+ **CIRC** :class:`stem.response.events.CircuitEvent`
+ **CIRC_BW** :class:`stem.response.events.CircuitBandwidthEvent`
+ **CIRC_MINOR** :class:`stem.response.events.CircMinorEvent`
+ **CLIENTS_SEEN** :class:`stem.response.events.ClientsSeenEvent`
+ **CONF_CHANGED** :class:`stem.response.events.ConfChangedEvent`
+ **CONN_BW** :class:`stem.response.events.ConnectionBandwidthEvent`
+ **DEBUG** :class:`stem.response.events.LogEvent`
+ **DESCCHANGED** :class:`stem.response.events.DescChangedEvent`
+ **ERR** :class:`stem.response.events.LogEvent`
+ **GUARD** :class:`stem.response.events.GuardEvent`
+ **HS_DESC** :class:`stem.response.events.HSDescEvent`
+ **HS_DESC_CONTENT** :class:`stem.response.events.HSDescContentEvent`
+ **INFO** :class:`stem.response.events.LogEvent`
+ **NEWCONSENSUS** :class:`stem.response.events.NewConsensusEvent`
+ **NEWDESC** :class:`stem.response.events.NewDescEvent`
+ **NOTICE** :class:`stem.response.events.LogEvent`
+ **NS** :class:`stem.response.events.NetworkStatusEvent`
+ **ORCONN** :class:`stem.response.events.ORConnEvent`
+ **SIGNAL** :class:`stem.response.events.SignalEvent`
+ **STATUS_CLIENT** :class:`stem.response.events.StatusEvent`
+ **STATUS_GENERAL** :class:`stem.response.events.StatusEvent`
+ **STATUS_SERVER** :class:`stem.response.events.StatusEvent`
+ **STREAM** :class:`stem.response.events.StreamEvent`
+ **STREAM_BW** :class:`stem.response.events.StreamBwEvent`
+ **TB_EMPTY** :class:`stem.response.events.TokenBucketEmptyEvent`
+ **TRANSPORT_LAUNCHED** :class:`stem.response.events.TransportLaunchedEvent`
+ **WARN** :class:`stem.response.events.LogEvent`
+ ======================= ===========
+
+.. data:: Listener (enum)
+
+ Purposes for inbound connections that Tor handles.
+
+ ============= ===========
+ Listener Description
+ ============= ===========
+ **OR** traffic we're relaying as a member of the network (torrc's **ORPort** and **ORListenAddress**)
+ **DIR** mirroring for tor descriptor content (torrc's **DirPort** and **DirListenAddress**)
+ **SOCKS** client traffic we're sending over Tor (torrc's **SocksPort** and **SocksListenAddress**)
+ **TRANS** transparent proxy handling (torrc's **TransPort** and **TransListenAddress**)
+ **NATD** forwarding for ipfw NATD connections (torrc's **NatdPort** and **NatdListenAddress**)
+ **DNS** DNS lookups for our traffic (torrc's **DNSPort** and **DNSListenAddress**)
+ **CONTROL** controller applications (torrc's **ControlPort** and **ControlListenAddress**)
+ ============= ===========
+"""
+
+import calendar
+import collections
+import functools
+import inspect
+import io
+import os
+import threading
+import time
+
+try:
+ # Added in 2.7
+ from collections import OrderedDict
+except ImportError:
+ from stem.util.ordereddict import OrderedDict
+
+try:
+ import queue
+ from io import StringIO
+except ImportError:
+ import Queue as queue
+ from StringIO import StringIO
+
+import stem.descriptor.microdescriptor
+import stem.descriptor.reader
+import stem.descriptor.router_status_entry
+import stem.descriptor.server_descriptor
+import stem.exit_policy
+import stem.response
+import stem.response.events
+import stem.socket
+import stem.util.connection
+import stem.util.enum
+import stem.util.str_tools
+import stem.util.system
+import stem.util.tor_tools
+import stem.version
+
+from stem import UNDEFINED, CircStatus, Signal, str_type
+from stem.util import log
+
+# state changes a control socket can have
+
+State = stem.util.enum.Enum('INIT', 'RESET', 'CLOSED')
+
+EventType = stem.util.enum.UppercaseEnum(
+ 'ADDRMAP',
+ 'AUTHDIR_NEWDESCS',
+ 'BUILDTIMEOUT_SET',
+ 'BW',
+ 'CELL_STATS',
+ 'CIRC',
+ 'CIRC_BW',
+ 'CIRC_MINOR',
+ 'CONF_CHANGED',
+ 'CONN_BW',
+ 'CLIENTS_SEEN',
+ 'DEBUG',
+ 'DESCCHANGED',
+ 'ERR',
+ 'GUARD',
+ 'HS_DESC',
+ 'HS_DESC_CONTENT',
+ 'INFO',
+ 'NEWCONSENSUS',
+ 'NEWDESC',
+ 'NOTICE',
+ 'NS',
+ 'ORCONN',
+ 'SIGNAL',
+ 'STATUS_CLIENT',
+ 'STATUS_GENERAL',
+ 'STATUS_SERVER',
+ 'STREAM',
+ 'STREAM_BW',
+ 'TB_EMPTY',
+ 'TRANSPORT_LAUNCHED',
+ 'WARN',
+)
+
+Listener = stem.util.enum.UppercaseEnum(
+ 'OR',
+ 'DIR',
+ 'SOCKS',
+ 'TRANS',
+ 'NATD',
+ 'DNS',
+ 'CONTROL',
+)
+
+# Configuration options that are fetched by a special key. The keys are
+# lowercase to make case insensitive lookups easier.
+
+MAPPED_CONFIG_KEYS = {
+ 'hiddenservicedir': 'HiddenServiceOptions',
+ 'hiddenserviceport': 'HiddenServiceOptions',
+ 'hiddenserviceversion': 'HiddenServiceOptions',
+ 'hiddenserviceauthorizeclient': 'HiddenServiceOptions',
+ 'hiddenserviceoptions': 'HiddenServiceOptions',
+}
+
+# unchangeable GETINFO parameters
+
+CACHEABLE_GETINFO_PARAMS = (
+ 'version',
+ 'config-file',
+ 'exit-policy/default',
+ 'fingerprint',
+ 'config/names',
+ 'config/defaults',
+ 'info/names',
+ 'events/names',
+ 'features/names',
+ 'process/descriptor-limit',
+)
+
+# GETCONF parameters we shouldn't cache. This includes hidden service
+# perameters due to the funky way they're set and retrieved (for instance,
+# 'SETCONF HiddenServiceDir' effects 'GETCONF HiddenServiceOptions').
+
+UNCACHEABLE_GETCONF_PARAMS = (
+ 'hiddenserviceoptions',
+ 'hiddenservicedir',
+ 'hiddenserviceport',
+ 'hiddenserviceversion',
+ 'hiddenserviceauthorizeclient',
+)
+
+# number of sequential attempts before we decide that the Tor geoip database
+# is unavailable
+GEOIP_FAILURE_THRESHOLD = 5
+
+SERVER_DESCRIPTORS_UNSUPPORTED = "Tor is currently not configured to retrieve \
+server descriptors. As of Tor version 0.2.3.25 it downloads microdescriptors \
+instead unless you set 'UseMicrodescriptors 0' in your torrc."
+
+AccountingStats = collections.namedtuple('AccountingStats', [
+ 'retrieved',
+ 'status',
+ 'interval_end',
+ 'time_until_reset',
+ 'read_bytes',
+ 'read_bytes_left',
+ 'read_limit',
+ 'written_bytes',
+ 'write_bytes_left',
+ 'write_limit',
+])
+
+CreateHiddenServiceOutput = collections.namedtuple('CreateHiddenServiceOutput', [
+ 'path',
+ 'hostname',
+ 'hostname_for_client',
+ 'config',
+])
+
+
+def with_default(yields = False):
+ """
+ Provides a decorator to support having a default value. This should be
+ treated as private.
+ """
+
+ def decorator(func):
+ def get_default(func, args, kwargs):
+ arg_names = inspect.getargspec(func).args[1:] # drop 'self'
+ default_position = arg_names.index('default') if 'default' in arg_names else None
+
+ if default_position is not None and default_position < len(args):
+ return args[default_position]
+ else:
+ return kwargs.get('default', UNDEFINED)
+
+ if not yields:
+ @functools.wraps(func)
+ def wrapped(self, *args, **kwargs):
+ try:
+ return func(self, *args, **kwargs)
+ except Exception as exc:
+ default = get_default(func, args, kwargs)
+
+ if default == UNDEFINED:
+ raise exc
+ else:
+ return default
+ else:
+ @functools.wraps(func)
+ def wrapped(self, *args, **kwargs):
+ try:
+ for val in func(self, *args, **kwargs):
+ yield val
+ except Exception as exc:
+ default = get_default(func, args, kwargs)
+
+ if default == UNDEFINED:
+ raise exc
+ else:
+ if default is not None:
+ for val in default:
+ yield val
+
+ return wrapped
+
+ return decorator
+
+
+class BaseController(object):
+ """
+ Controller for the tor process. This is a minimal base class for other
+ controllers, providing basic process communication and event listing. Don't
+ use this directly - subclasses like the :class:`~stem.control.Controller`
+ provide higher level functionality.
+
+ It's highly suggested that you don't interact directly with the
+ :class:`~stem.socket.ControlSocket` that we're constructed from - use our
+ wrapper methods instead.
+
+ If the **control_socket** is already authenticated to Tor then the caller
+ should provide the **is_authenticated** flag. Otherwise, we will treat the
+ socket as though it hasn't yet been authenticated.
+ """
+
+ def __init__(self, control_socket, is_authenticated = False):
+ self._socket = control_socket
+ self._msg_lock = threading.RLock()
+
+ self._status_listeners = [] # tuples of the form (callback, spawn_thread)
+ self._status_listeners_lock = threading.RLock()
+
+ # queues where incoming messages are directed
+ self._reply_queue = queue.Queue()
+ self._event_queue = queue.Queue()
+
+ # thread to continually pull from the control socket
+ self._reader_thread = None
+
+ # thread to pull from the _event_queue and call handle_event
+ self._event_notice = threading.Event()
+ self._event_thread = None
+
+ # saves our socket's prior _connect() and _close() methods so they can be
+ # called along with ours
+
+ self._socket_connect = self._socket._connect
+ self._socket_close = self._socket._close
+
+ self._socket._connect = self._connect
+ self._socket._close = self._close
+
+ self._last_heartbeat = 0.0 # timestamp for when we last heard from tor
+ self._is_authenticated = False
+
+ self._state_change_threads = [] # threads we've spawned to notify of state changes
+
+ if self._socket.is_alive():
+ self._launch_threads()
+
+ if is_authenticated:
+ self._post_authentication()
+
+ def msg(self, message):
+ """
+ Sends a message to our control socket and provides back its reply.
+
+ :param str message: message to be formatted and sent to tor
+
+ :returns: :class:`~stem.response.ControlMessage` with the response
+
+ :raises:
+ * :class:`stem.ProtocolError` the content from the socket is
+ malformed
+ * :class:`stem.SocketError` if a problem arises in using the
+ socket
+ * :class:`stem.SocketClosed` if the socket is shut down
+ """
+
+ with self._msg_lock:
+ # If our _reply_queue isn't empty then one of a few things happened...
+ #
+ # - Our connection was closed and probably re-restablished. This was
+ # in reply to pulling for an asynchronous event and getting this is
+ # expected - ignore it.
+ #
+ # - Pulling for asynchronous events produced an error. If this was a
+ # ProtocolError then it's a tor bug, and if a non-closure SocketError
+ # then it was probably a socket glitch. Deserves an INFO level log
+ # message.
+ #
+ # - This is a leftover response for a msg() call. We can't tell who an
+ # exception was earmarked for, so we only know that this was the case
+ # if it's a ControlMessage.
+ #
+ # This is the most concerning situation since it indicates that one of
+ # our callers didn't get their reply. However, this is still a
+ # perfectly viable use case. For instance...
+ #
+ # 1. We send a request.
+ # 2. The reader thread encounters an exception, for instance a socket
+ # error. We enqueue the exception.
+ # 3. The reader thread receives the reply.
+ # 4. We raise the socket error, and have an undelivered message.
+ #
+ # Thankfully this only seems to arise in edge cases around rapidly
+ # closing/reconnecting the socket.
+
+ while not self._reply_queue.empty():
+ try:
+ response = self._reply_queue.get_nowait()
+
+ if isinstance(response, stem.SocketClosed):
+ pass # this is fine
+ elif isinstance(response, stem.ProtocolError):
+ log.info('Tor provided a malformed message (%s)' % response)
+ elif isinstance(response, stem.ControllerError):
+ log.info('Socket experienced a problem (%s)' % response)
+ elif isinstance(response, stem.response.ControlMessage):
+ log.info('Failed to deliver a response: %s' % response)
+ except queue.Empty:
+ # the empty() method is documented to not be fully reliable so this
+ # isn't entirely surprising
+
+ break
+
+ try:
+ self._socket.send(message)
+ response = self._reply_queue.get()
+
+ # If the message we received back had an exception then re-raise it to the
+ # caller. Otherwise return the response.
+
+ if isinstance(response, stem.ControllerError):
+ raise response
+ else:
+ # I really, really don't like putting hooks into this method, but
+ # this is the most reliable method I can think of for taking actions
+ # immediately after successfully authenticating to a connection.
+
+ if message.upper().startswith('AUTHENTICATE'):
+ self._post_authentication()
+
+ return response
+ except stem.SocketClosed as exc:
+ # If the recv() thread caused the SocketClosed then we could still be
+ # in the process of closing. Calling close() here so that we can
+ # provide an assurance to the caller that when we raise a SocketClosed
+ # exception we are shut down afterward for realz.
+
+ self.close()
+ raise exc
+
+ def is_alive(self):
+ """
+ Checks if our socket is currently connected. This is a pass-through for our
+ socket's :func:`~stem.socket.ControlSocket.is_alive` method.
+
+ :returns: **bool** that's **True** if our socket is connected and **False** otherwise
+ """
+
+ return self._socket.is_alive()
+
+ def is_localhost(self):
+ """
+ Returns if the connection is for the local system or not.
+
+ .. versionadded:: 1.3.0
+
+ :returns: **bool** that's **True** if the connection is for the local host and **False** otherwise
+ """
+
+ return self._socket.is_localhost()
+
+ def connection_time(self):
+ """
+ Provides the unix timestamp for when our socket was either connected or
+ disconnected. That is to say, the time we connected if we're currently
+ connected and the time we disconnected if we're not connected.
+
+ .. versionadded:: 1.3.0
+
+ :returns: **float** for when we last connected or disconnected, zero if
+ we've never connected
+ """
+
+ return self._socket.connection_time()
+
+ def is_authenticated(self):
+ """
+ Checks if our socket is both connected and authenticated.
+
+ :returns: **bool** that's **True** if our socket is authenticated to tor
+ and **False** otherwise
+ """
+
+ if self.is_alive():
+ return self._is_authenticated
+
+ return False
+
+ def connect(self):
+ """
+ Reconnects our control socket. This is a pass-through for our socket's
+ :func:`~stem.socket.ControlSocket.connect` method.
+
+ :raises: :class:`stem.SocketError` if unable to make a socket
+ """
+
+ self._socket.connect()
+
+ def close(self):
+ """
+ Closes our socket connection. This is a pass-through for our socket's
+ :func:`~stem.socket.ControlSocket.close` method.
+ """
+
+ self._socket.close()
+
+ # Join on any outstanding state change listeners. Closing is a state change
+ # of its own, so if we have any listeners it's quite likely there's some
+ # work in progress.
+ #
+ # It's important that we do this outside of our locks so those daemons have
+ # access to us. This is why we're doing this here rather than _close().
+
+ for t in self._state_change_threads:
+ if t.is_alive() and threading.current_thread() != t:
+ t.join()
+
+ def get_socket(self):
+ """
+ Provides the socket used to speak with the tor process. Communicating with
+ the socket directly isn't advised since it may confuse this controller.
+
+ :returns: :class:`~stem.socket.ControlSocket` we're communicating with
+ """
+
+ return self._socket
+
+ def get_latest_heartbeat(self):
+ """
+ Provides the unix timestamp for when we last heard from tor. This is zero
+ if we've never received a message.
+
+ :returns: float for the unix timestamp of when we last heard from tor
+ """
+
+ return self._last_heartbeat
+
+ def add_status_listener(self, callback, spawn = True):
+ """
+ Notifies a given function when the state of our socket changes. Functions
+ are expected to be of the form...
+
+ ::
+
+ my_function(controller, state, timestamp)
+
+ The state is a value from the :data:`stem.control.State` enum. Functions
+ **must** allow for new values. The timestamp is a float for the unix time
+ when the change occurred.
+
+ This class only provides **State.INIT** and **State.CLOSED** notifications.
+ Subclasses may provide others.
+
+ If spawn is **True** then the callback is notified via a new daemon thread.
+ If **False** then the notice is under our locks, within the thread where
+ the change occurred. In general this isn't advised, especially if your
+ callback could block for a while. If still outstanding these threads are
+ joined on as part of closing this controller.
+
+ :param function callback: function to be notified when our state changes
+ :param bool spawn: calls function via a new thread if **True**, otherwise
+ it's part of the connect/close method call
+ """
+
+ with self._status_listeners_lock:
+ self._status_listeners.append((callback, spawn))
+
+ def remove_status_listener(self, callback):
+ """
+ Stops listener from being notified of further events.
+
+ :param function callback: function to be removed from our listeners
+
+ :returns: **bool** that's **True** if we removed one or more occurrences of
+ the callback, **False** otherwise
+ """
+
+ with self._status_listeners_lock:
+ new_listeners, is_changed = [], False
+
+ for listener, spawn in self._status_listeners:
+ if listener != callback:
+ new_listeners.append((listener, spawn))
+ else:
+ is_changed = True
+
+ self._status_listeners = new_listeners
+ return is_changed
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exit_type, value, traceback):
+ self.close()
+
+ def _handle_event(self, event_message):
+ """
+ Callback to be overwritten by subclasses for event listening. This is
+ notified whenever we receive an event from the control socket.
+
+ :param stem.response.ControlMessage event_message: message received from
+ the control socket
+ """
+
+ pass
+
+ def _connect(self):
+ self._launch_threads()
+ self._notify_status_listeners(State.INIT)
+ self._socket_connect()
+ self._is_authenticated = False
+
+ def _close(self):
+ # Our is_alive() state is now false. Our reader thread should already be
+ # awake from recv() raising a closure exception. Wake up the event thread
+ # too so it can end.
+
+ self._event_notice.set()
+ self._is_authenticated = False
+
+ # joins on our threads if it's safe to do so
+
+ for t in (self._reader_thread, self._event_thread):
+ if t and t.is_alive() and threading.current_thread() != t:
+ t.join()
+
+ self._notify_status_listeners(State.CLOSED)
+
+ self._socket_close()
+
+ def _post_authentication(self):
+ # actions to be taken after we have a newly authenticated connection
+
+ self._is_authenticated = True
+
+ def _notify_status_listeners(self, state):
+ """
+ Informs our status listeners that a state change occurred.
+
+ :param stem.control.State state: state change that has occurred
+ """
+
+ # Any changes to our is_alive() state happen under the send lock, so we
+ # need to have it to ensure it doesn't change beneath us.
+
+ with self._socket._get_send_lock():
+ with self._status_listeners_lock:
+ # States imply that our socket is either alive or not, which may not
+ # hold true when multiple events occur in quick succession. For
+ # instance, a sighup could cause two events (State.RESET for the sighup
+ # and State.CLOSE if it causes tor to crash). However, there's no
+ # guarantee of the order in which they occur, and it would be bad if
+ # listeners got the State.RESET last, implying that we were alive.
+
+ expect_alive = None
+
+ if state in (State.INIT, State.RESET):
+ expect_alive = True
+ elif state == State.CLOSED:
+ expect_alive = False
+
+ change_timestamp = time.time()
+
+ if expect_alive is not None and expect_alive != self.is_alive():
+ return
+
+ self._state_change_threads = list(filter(lambda t: t.is_alive(), self._state_change_threads))
+
+ for listener, spawn in self._status_listeners:
+ if spawn:
+ name = '%s notification' % state
+ args = (self, state, change_timestamp)
+
+ notice_thread = threading.Thread(target = listener, args = args, name = name)
+ notice_thread.setDaemon(True)
+ notice_thread.start()
+ self._state_change_threads.append(notice_thread)
+ else:
+ listener(self, state, change_timestamp)
+
+ def _launch_threads(self):
+ """
+ Initializes daemon threads. Threads can't be reused so we need to recreate
+ them if we're restarted.
+ """
+
+ # In theory concurrent calls could result in multiple start() calls on a
+ # single thread, which would cause an unexpected exception. Best be safe.
+
+ with self._socket._get_send_lock():
+ if not self._reader_thread or not self._reader_thread.is_alive():
+ self._reader_thread = threading.Thread(target = self._reader_loop, name = 'Tor Listener')
+ self._reader_thread.setDaemon(True)
+ self._reader_thread.start()
+
+ if not self._event_thread or not self._event_thread.is_alive():
+ self._event_thread = threading.Thread(target = self._event_loop, name = 'Event Notifier')
+ self._event_thread.setDaemon(True)
+ self._event_thread.start()
+
+ def _reader_loop(self):
+ """
+ Continually pulls from the control socket, directing the messages into
+ queues based on their type. Controller messages come in two varieties...
+
+ * Responses to messages we've sent (GETINFO, SETCONF, etc).
+ * Asynchronous events, identified by a status code of 650.
+ """
+
+ while self.is_alive():
+ try:
+ control_message = self._socket.recv()
+ self._last_heartbeat = time.time()
+
+ if control_message.content()[-1][0] == '650':
+ # asynchronous message, adds to the event queue and wakes up its handler
+ self._event_queue.put(control_message)
+ self._event_notice.set()
+ else:
+ # response to a msg() call
+ self._reply_queue.put(control_message)
+ except stem.ControllerError as exc:
+ # Assume that all exceptions belong to the reader. This isn't always
+ # true, but the msg() call can do a better job of sorting it out.
+ #
+ # Be aware that the msg() method relies on this to unblock callers.
+
+ self._reply_queue.put(exc)
+
+ def _event_loop(self):
+ """
+ Continually pulls messages from the _event_queue and sends them to our
+ handle_event callback. This is done via its own thread so subclasses with a
+ lengthy handle_event implementation don't block further reading from the
+ socket.
+ """
+
+ while True:
+ try:
+ event_message = self._event_queue.get_nowait()
+ self._handle_event(event_message)
+ except queue.Empty:
+ if not self.is_alive():
+ break
+
+ self._event_notice.wait()
+ self._event_notice.clear()
+
+
+class Controller(BaseController):
+ """
+ Communicates with a control socket. This is built on top of the
+ BaseController and provides a more user friendly API for library users.
+ """
+
+ @staticmethod
+ def from_port(address = '127.0.0.1', port = 9051):
+ """
+ Constructs a :class:`~stem.socket.ControlPort` based Controller.
+
+ :param str address: ip address of the controller
+ :param int port: port number of the controller
+
+ :returns: :class:`~stem.control.Controller` attached to the given port
+
+ :raises: :class:`stem.SocketError` if we're unable to establish a connection
+ """
+
+ if not stem.util.connection.is_valid_ipv4_address(address):
+ raise ValueError('Invalid IP address: %s' % address)
+ elif not stem.util.connection.is_valid_port(port):
+ raise ValueError('Invalid port: %s' % port)
+
+ control_port = stem.socket.ControlPort(address, port)
+ return Controller(control_port)
+
+ @staticmethod
+ def from_socket_file(path = '/var/run/tor/control'):
+ """
+ Constructs a :class:`~stem.socket.ControlSocketFile` based Controller.
+
+ :param str path: path where the control socket is located
+
+ :returns: :class:`~stem.control.Controller` attached to the given socket file
+
+ :raises: :class:`stem.SocketError` if we're unable to establish a connection
+ """
+
+ control_socket = stem.socket.ControlSocketFile(path)
+ return Controller(control_socket)
+
+ def __init__(self, control_socket, is_authenticated = False):
+ self._is_caching_enabled = True
+ self._request_cache = {}
+ self._last_newnym = 0.0
+
+ self._cache_lock = threading.RLock()
+
+ # mapping of event types to their listeners
+
+ self._event_listeners = {}
+ self._event_listeners_lock = threading.RLock()
+
+ # number of sequential 'GETINFO ip-to-country/*' lookups that have failed
+
+ self._geoip_failure_count = 0
+ self._enabled_features = []
+
+ super(Controller, self).__init__(control_socket, is_authenticated)
+
+ def _sighup_listener(event):
+ if event.signal == Signal.RELOAD:
+ self.clear_cache()
+ self._notify_status_listeners(State.RESET)
+
+ self.add_event_listener(_sighup_listener, EventType.SIGNAL)
+
+ def _confchanged_listener(event):
+ if self.is_caching_enabled():
+ self._set_cache(dict((k, None) for k in event.config), 'getconf')
+
+ if 'exitpolicy' in event.config.keys():
+ self._set_cache({'exitpolicy': None})
+
+ self.add_event_listener(_confchanged_listener, EventType.CONF_CHANGED)
+
+ def connect(self):
+ super(Controller, self).connect()
+ self.clear_cache()
+
+ def close(self):
+ # making a best-effort attempt to quit before detaching the socket
+ if self.is_alive():
+ try:
+ self.msg('QUIT')
+ except:
+ pass
+
+ self.clear_cache()
+
+ super(Controller, self).close()
+
+ def authenticate(self, *args, **kwargs):
+ """
+ A convenience method to authenticate the controller. This is just a
+ pass-through to :func:`stem.connection.authenticate`.
+ """
+
+ import stem.connection
+ stem.connection.authenticate(self, *args, **kwargs)
+
+ @with_default()
+ def get_info(self, params, default = UNDEFINED, get_bytes = False):
+ """
+ get_info(params, default = UNDEFINED, get_bytes = False)
+
+ Queries the control socket for the given GETINFO option. If provided a
+ default then that's returned if the GETINFO option is undefined or the
+ call fails for any reason (error response, control port closed, initiated,
+ etc).
+
+ .. versionchanged:: 1.1.0
+ Added the get_bytes argument.
+
+ :param str,list params: GETINFO option or options to be queried
+ :param object default: response if the query fails
+ :param bool get_bytes: provides **bytes** values rather than a **str** under python 3.x
+
+ :returns:
+ Response depends upon how we were called as follows...
+
+ * **str** with the response if our param was a **str**
+ * **dict** with the 'param => response' mapping if our param was a **list**
+ * default if one was provided and our call failed
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails and we weren't
+ provided a default response
+ * :class:`stem.InvalidArguments` if the 'params' requested was
+ invalid
+ * :class:`stem.ProtocolError` if the geoip database is known to be
+ unavailable
+ """
+
+ start_time = time.time()
+ reply = {}
+
+ if isinstance(params, (bytes, str_type)):
+ is_multiple = False
+ params = set([params])
+ else:
+ if not params:
+ return {}
+
+ is_multiple = True
+ params = set(params)
+
+ # check for cached results
+
+ from_cache = [param.lower() for param in params]
+ cached_results = self._get_cache_map(from_cache, 'getinfo')
+
+ for key in cached_results:
+ user_expected_key = _case_insensitive_lookup(params, key)
+ reply[user_expected_key] = cached_results[key]
+ params.remove(user_expected_key)
+
+ for param in params:
+ if param.startswith('ip-to-country/') and self.is_geoip_unavailable():
+ # the geoip database already looks to be unavailable - abort the request
+
+ raise stem.ProtocolError('Tor geoip database is unavailable')
+
+ # if everything was cached then short circuit making the query
+ if not params:
+ log.trace('GETINFO %s (cache fetch)' % ' '.join(reply.keys()))
+
+ if is_multiple:
+ return reply
+ else:
+ return list(reply.values())[0]
+
+ try:
+ response = self.msg('GETINFO %s' % ' '.join(params))
+ stem.response.convert('GETINFO', response)
+ response._assert_matches(params)
+
+ # usually we want unicode values under python 3.x
+
+ if stem.prereq.is_python_3() and not get_bytes:
+ response.entries = dict((k, stem.util.str_tools._to_unicode(v)) for (k, v) in response.entries.items())
+
+ reply.update(response.entries)
+
+ if self.is_caching_enabled():
+ to_cache = {}
+
+ for key, value in response.entries.items():
+ key = key.lower() # make case insensitive
+
+ if key in CACHEABLE_GETINFO_PARAMS:
+ to_cache[key] = value
+ elif key.startswith('ip-to-country/'):
+ # both cache-able and means that we should reset the geoip failure count
+ to_cache[key] = value
+ self._geoip_failure_count = -1
+
+ self._set_cache(to_cache, 'getinfo')
+
+ log.debug('GETINFO %s (runtime: %0.4f)' % (' '.join(params), time.time() - start_time))
+
+ if is_multiple:
+ return reply
+ else:
+ return list(reply.values())[0]
+ except stem.ControllerError as exc:
+ # bump geoip failure count if...
+ # * we're caching results
+ # * this was soley a geoip lookup
+ # * we've never had a successful geoip lookup (failure count isn't -1)
+
+ is_geoip_request = len(params) == 1 and list(params)[0].startswith('ip-to-country/')
+
+ if is_geoip_request and self.is_caching_enabled() and self._geoip_failure_count != -1:
+ self._geoip_failure_count += 1
+
+ if self.is_geoip_unavailable():
+ log.warn("Tor's geoip database is unavailable.")
+
+ log.debug('GETINFO %s (failed: %s)' % (' '.join(params), exc))
+
+ raise exc
+
+ @with_default()
+ def get_version(self, default = UNDEFINED):
+ """
+ get_version(default = UNDEFINED)
+
+ A convenience method to get tor version that current controller is
+ connected to.
+
+ :param object default: response if the query fails
+
+ :returns: :class:`~stem.version.Version` of the tor instance that we're
+ connected to
+
+ :raises:
+ * :class:`stem.ControllerError` if unable to query the version
+ * **ValueError** if unable to parse the version
+
+ An exception is only raised if we weren't provided a default response.
+ """
+
+ version = self._get_cache('version')
+
+ if not version:
+ version = stem.version.Version(self.get_info('version'))
+ self._set_cache({'version': version})
+
+ return version
+
+ @with_default()
+ def get_exit_policy(self, default = UNDEFINED):
+ """
+ get_exit_policy(default = UNDEFINED)
+
+ Effective ExitPolicy for our relay. This accounts for
+ ExitPolicyRejectPrivate and default policies.
+
+ :param object default: response if the query fails
+
+ :returns: :class:`~stem.exit_policy.ExitPolicy` of the tor instance that
+ we're connected to
+
+ :raises:
+ * :class:`stem.ControllerError` if unable to query the policy
+ * **ValueError** if unable to parse the policy
+
+ An exception is only raised if we weren't provided a default response.
+ """
+
+ with self._msg_lock:
+ config_policy = self._get_cache('exit_policy')
+
+ if not config_policy:
+ policy = []
+
+ if self.get_conf('ExitPolicyRejectPrivate') == '1':
+ policy.append('reject private:*')
+
+ for policy_line in self.get_conf('ExitPolicy', multiple = True):
+ policy += policy_line.split(',')
+
+ policy += self.get_info('exit-policy/default').split(',')
+
+ config_policy = stem.exit_policy.get_config_policy(policy, self.get_info('address', None))
+ self._set_cache({'exit_policy': config_policy})
+
+ return config_policy
+
+ @with_default()
+ def get_ports(self, listener_type, default = UNDEFINED):
+ """
+ get_ports(listener_type, default = UNDEFINED)
+
+ Provides the local ports where tor is listening for the given type of
+ connections. This is similar to
+ :func:`~stem.control.Controller.get_listeners`, but doesn't provide
+ addresses nor include non-local endpoints.
+
+ .. versionadded:: 1.2.0
+
+ :param stem.control.Listener listener_type: connection type being handled
+ by the ports we return
+ :param object default: response if the query fails
+
+ :returns: **list** of **ints** for the local ports where tor handles
+ connections of the given type
+
+ :raises: :class:`stem.ControllerError` if unable to determine the ports
+ and no default was provided
+ """
+
+ return [port for (addr, port) in self.get_listeners(listener_type) if addr == '127.0.0.1']
+
+ @with_default()
+ def get_listeners(self, listener_type, default = UNDEFINED):
+ """
+ get_listeners(listener_type, default = UNDEFINED)
+
+ Provides the addresses and ports where tor is listening for connections of
+ the given type. This is similar to
+ :func:`~stem.control.Controller.get_ports` but includes listener addresses
+ and non-local endpoints.
+
+ .. versionadded:: 1.2.0
+
+ :param stem.control.Listener listener_type: connection type being handled
+ by the listeners we return
+ :param object default: response if the query fails
+
+ :returns: **list** of **(address, port)** tuples for the available
+ listeners
+
+ :raises: :class:`stem.ControllerError` if unable to determine the listeners
+ and no default was provided
+ """
+
+ proxy_addrs = []
+ query = 'net/listeners/%s' % listener_type.lower()
+
+ try:
+ for listener in self.get_info(query).split():
+ if not (listener.startswith('"') and listener.endswith('"')):
+ raise stem.ProtocolError("'GETINFO %s' responses are expected to be quoted: %s" % (query, listener))
+ elif ':' not in listener:
+ raise stem.ProtocolError("'GETINFO %s' had a listener without a colon: %s" % (query, listener))
+
+ listener = listener[1:-1] # strip quotes
+ addr, port = listener.split(':')
+
+ # Skip unix sockets, for instance...
+ #
+ # GETINFO net/listeners/control
+ # 250-net/listeners/control="unix:/tmp/tor/socket"
+ # 250 OK
+
+ if addr == 'unix':
+ continue
+
+ proxy_addrs.append((addr, port))
+ except stem.InvalidArguments:
+ # Tor version is old (pre-tor-0.2.2.26-beta), use get_conf() instead.
+ # Some options (like the ORPort) can have optional attributes after the
+ # actual port number.
+
+ port_option = {
+ Listener.OR: 'ORPort',
+ Listener.DIR: 'DirPort',
+ Listener.SOCKS: 'SocksPort',
+ Listener.TRANS: 'TransPort',
+ Listener.NATD: 'NatdPort',
+ Listener.DNS: 'DNSPort',
+ Listener.CONTROL: 'ControlPort',
+ }[listener_type]
+
+ listener_option = {
+ Listener.OR: 'ORListenAddress',
+ Listener.DIR: 'DirListenAddress',
+ Listener.SOCKS: 'SocksListenAddress',
+ Listener.TRANS: 'TransListenAddress',
+ Listener.NATD: 'NatdListenAddress',
+ Listener.DNS: 'DNSListenAddress',
+ Listener.CONTROL: 'ControlListenAddress',
+ }[listener_type]
+
+ port_value = self.get_conf(port_option).split()[0]
+
+ for listener in self.get_conf(listener_option, multiple = True):
+ if ':' in listener:
+ addr, port = listener.split(':')
+ proxy_addrs.append((addr, port))
+ else:
+ proxy_addrs.append((listener, port_value))
+
+ # validate that address/ports are valid, and convert ports to ints
+
+ for addr, port in proxy_addrs:
+ if not stem.util.connection.is_valid_ipv4_address(addr):
+ raise stem.ProtocolError('Invalid address for a %s listener: %s' % (listener_type, addr))
+ elif not stem.util.connection.is_valid_port(port):
+ raise stem.ProtocolError('Invalid port for a %s listener: %s' % (listener_type, port))
+
+ return [(addr, int(port)) for (addr, port) in proxy_addrs]
+
+ @with_default()
+ def get_accounting_stats(self, default = UNDEFINED):
+ """
+ get_accounting_stats(default = UNDEFINED)
+
+ Provides stats related to our relaying limitations if AccountingMax was set
+ in our torrc. This provides a **namedtuple** with the following
+ attributes...
+
+ * retrieved (float) - unix timestamp for when this was fetched
+ * status (str) - hibernation status of 'awake', 'soft', or 'hard'
+ * interval_end (datetime)
+ * time_until_reset (int) - seconds until our limits reset
+ * read_bytes (int)
+ * read_bytes_left (int)
+ * read_limit (int)
+ * written_bytes (int)
+ * write_bytes_left (int)
+ * write_limit (int)
+
+ .. versionadded:: 1.3.0
+
+ :param object default: response if the query fails
+
+ :returns: **namedtuple** with our accounting stats
+
+ :raises: :class:`stem.ControllerError` if unable to determine the listeners
+ and no default was provided
+ """
+
+ if self.get_info('accounting/enabled') != '1':
+ raise stem.ControllerError("Accounting isn't enabled")
+
+ retrieved = time.time()
+ status = self.get_info('accounting/hibernating')
+ interval_end = self.get_info('accounting/interval-end')
+ used = self.get_info('accounting/bytes')
+ left = self.get_info('accounting/bytes-left')
+
+ interval_end = stem.util.str_tools._parse_timestamp(interval_end)
+ used_read, used_written = [int(val) for val in used.split(' ', 1)]
+ left_read, left_written = [int(val) for val in left.split(' ', 1)]
+
+ return AccountingStats(
+ retrieved = retrieved,
+ status = status,
+ interval_end = interval_end,
+ time_until_reset = calendar.timegm(interval_end.timetuple()) - int(retrieved),
+ read_bytes = used_read,
+ read_bytes_left = left_read,
+ read_limit = used_read + left_read,
+ written_bytes = used_written,
+ write_bytes_left = left_written,
+ write_limit = used_written + left_written,
+ )
+
+ def get_socks_listeners(self, default = UNDEFINED):
+ """
+ Provides the SOCKS **(address, port)** tuples that tor has open.
+
+ .. deprecated:: 1.2.0
+ Use :func:`~stem.control.Controller.get_listeners` with
+ **Listener.SOCKS** instead.
+
+ :param object default: response if the query fails
+
+ :returns: list of **(address, port)** tuples for the available SOCKS
+ listeners
+
+ :raises: :class:`stem.ControllerError` if unable to determine the listeners
+ and no default was provided
+ """
+
+ return self.get_listeners(Listener.SOCKS, default)
+
+ @with_default()
+ def get_protocolinfo(self, default = UNDEFINED):
+ """
+ get_protocolinfo(default = UNDEFINED)
+
+ A convenience method to get the protocol info of the controller.
+
+ :param object default: response if the query fails
+
+ :returns: :class:`~stem.response.protocolinfo.ProtocolInfoResponse` provided by tor
+
+ :raises:
+ * :class:`stem.ProtocolError` if the PROTOCOLINFO response is
+ malformed
+ * :class:`stem.SocketError` if problems arise in establishing or
+ using the socket
+
+ An exception is only raised if we weren't provided a default response.
+ """
+
+ import stem.connection
+ return stem.connection.get_protocolinfo(self)
+
+ @with_default()
+ def get_user(self, default = UNDEFINED):
+ """
+ get_user(default = UNDEFINED)
+
+ Provides the user tor is running as. This often only works if tor is
+ running locally. Also, most of its checks are platform dependent, and hence
+ are not entirely reliable.
+
+ .. versionadded:: 1.1.0
+
+ :param object default: response if the query fails
+
+ :returns: str with the username tor is running as
+ """
+
+ user = self._get_cache('user')
+
+ if not user:
+ user = self.get_info('process/user', None)
+
+ if not user and self.is_localhost():
+ pid = self.get_pid(None)
+
+ if pid:
+ user = stem.util.system.user(pid)
+
+ if user:
+ self._set_cache({'user': user})
+ return user
+ else:
+ raise ValueError("Unable to resolve tor's user" if self.is_localhost() else "Tor isn't running locally")
+
+ @with_default()
+ def get_pid(self, default = UNDEFINED):
+ """
+ get_pid(default = UNDEFINED)
+
+ Provides the process id of tor. This often only works if tor is running
+ locally. Also, most of its checks are platform dependent, and hence are not
+ entirely reliable.
+
+ .. versionadded:: 1.1.0
+
+ :param object default: response if the query fails
+
+ :returns: **int** for tor's pid
+
+ :raises: **ValueError** if unable to determine the pid and no default was
+ provided
+ """
+
+ pid = self._get_cache('pid')
+
+ if not pid:
+ getinfo_pid = self.get_info('process/pid', None)
+
+ if getinfo_pid and getinfo_pid.isdigit():
+ pid = int(getinfo_pid)
+
+ if not pid and self.is_localhost():
+ pid_file_path = self.get_conf('PidFile', None)
+
+ if pid_file_path is not None:
+ with open(pid_file_path) as pid_file:
+ pid_file_contents = pid_file.read().strip()
+
+ if pid_file_contents.isdigit():
+ pid = int(pid_file_contents)
+
+ if not pid:
+ pid = stem.util.system.pid_by_name('tor')
+
+ if not pid:
+ control_socket = self.get_socket()
+
+ if isinstance(control_socket, stem.socket.ControlPort):
+ pid = stem.util.system.pid_by_port(control_socket.get_port())
+ elif isinstance(control_socket, stem.socket.ControlSocketFile):
+ pid = stem.util.system.pid_by_open_file(control_socket.get_socket_path())
+
+ if pid:
+ self._set_cache({'pid': pid})
+ return pid
+ else:
+ raise ValueError("Unable to resolve tor's pid" if self.is_localhost() else "Tor isn't running locally")
+
+ @with_default()
+ def get_microdescriptor(self, relay = None, default = UNDEFINED):
+ """
+ get_microdescriptor(relay = None, default = UNDEFINED)
+
+ Provides the microdescriptor for the relay with the given fingerprint or
+ nickname. If the relay identifier could be either a fingerprint *or*
+ nickname then it's queried as a fingerprint.
+
+ If no **relay** is provided then this defaults to ourselves. Remember that
+ this requires that we've retrieved our own descriptor from remote
+ authorities so this both won't be available for newly started relays and
+ may be up to around an hour out of date.
+
+ .. versionchanged:: 1.3.0
+ Changed so we'd fetch our own descriptor if no 'relay' is provided.
+
+ :param str relay: fingerprint or nickname of the relay to be queried
+ :param object default: response if the query fails
+
+ :returns: :class:`~stem.descriptor.microdescriptor.Microdescriptor` for the given relay
+
+ :raises:
+ * :class:`stem.DescriptorUnavailable` if unable to provide a descriptor
+ for the given relay
+ * :class:`stem.ControllerError` if unable to query the descriptor
+ * **ValueError** if **relay** doesn't conform with the pattern for being
+ a fingerprint or nickname
+
+ An exception is only raised if we weren't provided a default response.
+ """
+
+ if relay is None:
+ try:
+ relay = self.get_info('fingerprint')
+ except stem.ControllerError as exc:
+ raise stem.ControllerError('Unable to determine our own fingerprint: %s' % exc)
+
+ if stem.util.tor_tools.is_valid_fingerprint(relay):
+ query = 'md/id/%s' % relay
+ elif stem.util.tor_tools.is_valid_nickname(relay):
+ query = 'md/name/%s' % relay
+ else:
+ raise ValueError("'%s' isn't a valid fingerprint or nickname" % relay)
+
+ try:
+ desc_content = self.get_info(query, get_bytes = True)
+ except stem.InvalidArguments as exc:
+ if str(exc).startswith('GETINFO request contained unrecognized keywords:'):
+ raise stem.DescriptorUnavailable("Tor was unable to provide the descriptor for '%s'" % relay)
+ else:
+ raise exc
+
+ if not desc_content:
+ raise stem.DescriptorUnavailable('Descriptor information is unavailable, tor might still be downloading it')
+
+ return stem.descriptor.microdescriptor.Microdescriptor(desc_content)
+
+ @with_default(yields = True)
+ def get_microdescriptors(self, default = UNDEFINED):
+ """
+ get_microdescriptors(default = UNDEFINED)
+
+ Provides an iterator for all of the microdescriptors that tor currently
+ knows about.
+
+ **Tor does not expose this information via the control protocol**
+ (:trac:`8323`). Until it does this reads the microdescriptors from disk,
+ and hence won't work remotely or if we lack read permissions.
+
+ :param list default: items to provide if the query fails
+
+ :returns: iterates over
+ :class:`~stem.descriptor.microdescriptor.Microdescriptor` for relays in
+ the tor network
+
+ :raises: :class:`stem.ControllerError` if unable to query tor and no
+ default was provided
+ """
+
+ try:
+ data_directory = self.get_conf('DataDirectory')
+ except stem.ControllerError as exc:
+ raise stem.OperationFailed(message = 'Unable to determine the data directory (%s)' % exc)
+
+ cached_descriptor_path = os.path.join(data_directory, 'cached-microdescs')
+
+ if not os.path.exists(data_directory):
+ raise stem.OperationFailed(message = "Data directory reported by tor doesn't exist (%s)" % data_directory)
+ elif not os.path.exists(cached_descriptor_path):
+ raise stem.OperationFailed(message = "Data directory doens't contain cached microescriptors (%s)" % cached_descriptor_path)
+
+ with stem.descriptor.reader.DescriptorReader([cached_descriptor_path]) as reader:
+ for desc in reader:
+ # It shouldn't be possible for these to be something other than
+ # microdescriptors but as the saying goes: trust but verify.
+
+ if not isinstance(desc, stem.descriptor.microdescriptor.Microdescriptor):
+ raise stem.OperationFailed(message = 'BUG: Descriptor reader provided non-microdescriptor content (%s)' % type(desc))
+
+ yield desc
+
+ @with_default()
+ def get_server_descriptor(self, relay = None, default = UNDEFINED):
+ """
+ get_server_descriptor(relay = None, default = UNDEFINED)
+
+ Provides the server descriptor for the relay with the given fingerprint or
+ nickname. If the relay identifier could be either a fingerprint *or*
+ nickname then it's queried as a fingerprint.
+
+ If no **relay** is provided then this defaults to ourselves. Remember that
+ this requires that we've retrieved our own descriptor from remote
+ authorities so this both won't be available for newly started relays and
+ may be up to around an hour out of date.
+
+ **As of Tor version 0.2.3.25 relays no longer get server descriptors by
+ default.** It's advised that you use microdescriptors instead, but if you
+ really need server descriptors then you can get them by setting
+ 'UseMicrodescriptors 0'.
+
+ .. versionchanged:: 1.3.0
+ Changed so we'd fetch our own descriptor if no 'relay' is provided.
+
+ :param str relay: fingerprint or nickname of the relay to be queried
+ :param object default: response if the query fails
+
+ :returns: :class:`~stem.descriptor.server_descriptor.RelayDescriptor` for the given relay
+
+ :raises:
+ * :class:`stem.DescriptorUnavailable` if unable to provide a descriptor
+ for the given relay
+ * :class:`stem.ControllerError` if unable to query the descriptor
+ * **ValueError** if **relay** doesn't conform with the pattern for being
+ a fingerprint or nickname
+
+ An exception is only raised if we weren't provided a default response.
+ """
+
+ try:
+ if relay is None:
+ try:
+ relay = self.get_info('fingerprint')
+ except stem.ControllerError as exc:
+ raise stem.ControllerError('Unable to determine our own fingerprint: %s' % exc)
+
+ if stem.util.tor_tools.is_valid_fingerprint(relay):
+ query = 'desc/id/%s' % relay
+ elif stem.util.tor_tools.is_valid_nickname(relay):
+ query = 'desc/name/%s' % relay
+ else:
+ raise ValueError("'%s' isn't a valid fingerprint or nickname" % relay)
+
+ try:
+ desc_content = self.get_info(query, get_bytes = True)
+ except stem.InvalidArguments as exc:
+ if str(exc).startswith('GETINFO request contained unrecognized keywords:'):
+ raise stem.DescriptorUnavailable("Tor was unable to provide the descriptor for '%s'" % relay)
+ else:
+ raise exc
+
+ if not desc_content:
+ raise stem.DescriptorUnavailable('Descriptor information is unavailable, tor might still be downloading it')
+
+ return stem.descriptor.server_descriptor.RelayDescriptor(desc_content)
+ except Exception as exc:
+ if not self._is_server_descriptors_available():
+ raise ValueError(SERVER_DESCRIPTORS_UNSUPPORTED)
+
+ raise exc
+
+ @with_default(yields = True)
+ def get_server_descriptors(self, default = UNDEFINED):
+ """
+ get_server_descriptors(default = UNDEFINED)
+
+ Provides an iterator for all of the server descriptors that tor currently
+ knows about.
+
+ **As of Tor version 0.2.3.25 relays no longer get server descriptors by
+ default.** It's advised that you use microdescriptors instead, but if you
+ really need server descriptors then you can get them by setting
+ 'UseMicrodescriptors 0'.
+
+ :param list default: items to provide if the query fails
+
+ :returns: iterates over
+ :class:`~stem.descriptor.server_descriptor.RelayDescriptor` for relays in
+ the tor network
+
+ :raises: :class:`stem.ControllerError` if unable to query tor and no
+ default was provided
+ """
+
+ # TODO: We should iterate over the descriptors as they're read from the
+ # socket rather than reading the whole thing into memory.
+ #
+ # https://trac.torproject.org/8248
+
+ desc_content = self.get_info('desc/all-recent', get_bytes = True)
+
+ if not desc_content:
+ if not self._is_server_descriptors_available():
+ raise stem.ControllerError(SERVER_DESCRIPTORS_UNSUPPORTED)
+ else:
+ raise stem.DescriptorUnavailable('Descriptor information is unavailable, tor might still be downloading it')
+
+ for desc in stem.descriptor.server_descriptor._parse_file(io.BytesIO(desc_content)):
+ yield desc
+
+ def _is_server_descriptors_available(self):
+ """
+ Checks to see if tor server descriptors should be available or not.
+ """
+
+ return self.get_version() < stem.version.Requirement.MICRODESCRIPTOR_IS_DEFAULT or \
+ self.get_conf('UseMicrodescriptors', None) == '0'
+
+ @with_default()
+ def get_network_status(self, relay = None, default = UNDEFINED):
+ """
+ get_network_status(relay = None, default = UNDEFINED)
+
+ Provides the router status entry for the relay with the given fingerprint
+ or nickname. If the relay identifier could be either a fingerprint *or*
+ nickname then it's queried as a fingerprint.
+
+ This provides
+ :class:`~stem.descriptor.router_status_entry.RouterStatusEntryMicroV3`
+ instances if tor is using microdescriptors...
+
+ ::
+
+ controller.get_conf('UseMicrodescriptors', '0') == '1'
+
+ ... and :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3`
+ otherwise.
+
+ If no **relay** is provided then this defaults to ourselves. Remember that
+ this requires that we've retrieved our own descriptor from remote
+ authorities so this both won't be available for newly started relays and
+ may be up to around an hour out of date.
+
+ .. versionchanged:: 1.3.0
+ Changed so we'd fetch our own descriptor if no 'relay' is provided.
+
+ :param str relay: fingerprint or nickname of the relay to be queried
+ :param object default: response if the query fails
+
+ :returns: :class:`~stem.descriptor.router_status_entry.RouterStatusEntry`
+ for the given relay
+
+ :raises:
+ * :class:`stem.DescriptorUnavailable` if unable to provide a descriptor
+ for the given relay
+ * :class:`stem.ControllerError` if unable to query the descriptor
+ * **ValueError** if **relay** doesn't conform with the pattern for being
+ a fingerprint or nickname
+
+ An exception is only raised if we weren't provided a default response.
+ """
+
+ if relay is None:
+ try:
+ relay = self.get_info('fingerprint')
+ except stem.ControllerError as exc:
+ raise stem.ControllerError('Unable to determine our own fingerprint: %s' % exc)
+
+ if stem.util.tor_tools.is_valid_fingerprint(relay):
+ query = 'ns/id/%s' % relay
+ elif stem.util.tor_tools.is_valid_nickname(relay):
+ query = 'ns/name/%s' % relay
+ else:
+ raise ValueError("'%s' isn't a valid fingerprint or nickname" % relay)
+
+ try:
+ desc_content = self.get_info(query, get_bytes = True)
+ except stem.InvalidArguments as exc:
+ if str(exc).startswith('GETINFO request contained unrecognized keywords:'):
+ raise stem.DescriptorUnavailable("Tor was unable to provide the descriptor for '%s'" % relay)
+ else:
+ raise exc
+
+ if not desc_content:
+ raise stem.DescriptorUnavailable('Descriptor information is unavailable, tor might still be downloading it')
+
+ if self.get_conf('UseMicrodescriptors', '0') == '1':
+ return stem.descriptor.router_status_entry.RouterStatusEntryMicroV3(desc_content)
+ else:
+ return stem.descriptor.router_status_entry.RouterStatusEntryV3(desc_content)
+
+ @with_default(yields = True)
+ def get_network_statuses(self, default = UNDEFINED):
+ """
+ get_network_statuses(default = UNDEFINED)
+
+ Provides an iterator for all of the router status entries that tor
+ currently knows about.
+
+ This provides
+ :class:`~stem.descriptor.router_status_entry.RouterStatusEntryMicroV3`
+ instances if tor is using microdescriptors...
+
+ ::
+
+ controller.get_conf('UseMicrodescriptors', '0') == '1'
+
+ ... and :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3`
+ otherwise.
+
+ :param list default: items to provide if the query fails
+
+ :returns: iterates over
+ :class:`~stem.descriptor.router_status_entry.RouterStatusEntry` for
+ relays in the tor network
+
+ :raises: :class:`stem.ControllerError` if unable to query tor and no
+ default was provided
+ """
+
+ # TODO: We should iterate over the descriptors as they're read from the
+ # socket rather than reading the whole thing into memory.
+ #
+ # https://trac.torproject.org/8248
+
+ if self.get_conf('UseMicrodescriptors', '0') == '1':
+ desc_class = stem.descriptor.router_status_entry.RouterStatusEntryMicroV3
+ else:
+ desc_class = stem.descriptor.router_status_entry.RouterStatusEntryV3
+
+ desc_content = self.get_info('ns/all', get_bytes = True)
+
+ if not desc_content:
+ raise stem.DescriptorUnavailable('Descriptor information is unavailable, tor might still be downloading it')
+
+ desc_iterator = stem.descriptor.router_status_entry._parse_file(
+ io.BytesIO(desc_content),
+ True,
+ entry_class = desc_class,
+ )
+
+ for desc in desc_iterator:
+ yield desc
+
+ @with_default()
+ def get_hidden_service_descriptor(self, address, default = UNDEFINED, servers = None, await_result = True):
+ """
+ get_hidden_service_descriptor(address, default = UNDEFINED, servers = None, await_result = True)
+
+ Provides the descriptor for a hidden service. The **address** is the
+ '.onion' address of the hidden service (for instance 3g2upl4pq6kufc4m.onion
+ for DuckDuckGo).
+
+ If **await_result** is **True** then this blocks until we either receive
+ the descriptor or the request fails. If **False** this returns right away.
+
+ .. versionadded:: 1.4.0
+
+ :param str address: address of the hidden service descriptor, the '.onion' suffix is optional
+ :param object default: response if the query fails
+ :param list servers: requrest the descriptor from these specific servers
+
+ :returns: :class:`~stem.descriptor.hidden_service_descriptor.HiddenServiceDescriptor`
+ for the given service if **await_result** is **True**, or **None** otherwise
+
+ :raises:
+ * :class:`stem.DescriptorUnavailable` if **await_result** is **True** and
+ unable to provide a descriptor for the given service
+ * :class:`stem.ControllerError` if unable to query the descriptor
+ * **ValueError** if **address** doesn't conform with the pattern of a
+ hidden service address
+
+ An exception is only raised if we weren't provided a default response.
+ """
+
+ if address.endswith('.onion'):
+ address = address[:-6]
+
+ if not stem.util.tor_tools.is_valid_hidden_service_address(address):
+ raise ValueError("'%s.onion' isn't a valid hidden service address" % address)
+
+ if self.get_version() < stem.version.Requirement.HSFETCH:
+ raise stem.UnsatisfiableRequest(message = 'HSFETCH was added in tor version %s' % stem.version.Requirement.HSFETCH)
+
+ hs_desc_queue, hs_desc_listener = queue.Queue(), None
+ hs_desc_content_queue, hs_desc_content_listener = queue.Queue(), None
+
+ if await_result:
+ def hs_desc_listener(event):
+ hs_desc_queue.put(event)
+
+ def hs_desc_content_listener(event):
+ hs_desc_content_queue.put(event)
+
+ self.add_event_listener(hs_desc_listener, EventType.HS_DESC)
+ self.add_event_listener(hs_desc_content_listener, EventType.HS_DESC_CONTENT)
+
+ try:
+ request = 'HSFETCH %s' % address
+
+ if servers:
+ request += ' '.join(['SERVER=%s' % s for s in servers])
+
+ response = self.msg(request)
+ stem.response.convert('SINGLELINE', response)
+
+ if not response.is_ok():
+ raise stem.ProtocolError('HSFETCH returned unexpected response code: %s' % response.code)
+
+ if not await_result:
+ return None # not waiting, so nothing to provide back
+ else:
+ while True:
+ event = hs_desc_content_queue.get()
+
+ if event.address == address:
+ if event.descriptor:
+ return event.descriptor
+ else:
+ # no descriptor, looking through HS_DESC to figure out why
+
+ while True:
+ event = hs_desc_queue.get()
+
+ if event.address == address and event.action == stem.HSDescAction.FAILED:
+ if event.reason == stem.HSDescReason.NOT_FOUND:
+ raise stem.DescriptorUnavailable('No running hidden service at %s.onion' % address)
+ else:
+ raise stem.DescriptorUnavailable('Unable to retrieve the descriptor for %s.onion (retrieved from %s): %s' % (address, event.directory_fingerprint, event.reason))
+ finally:
+ if hs_desc_listener:
+ self.remove_event_listener(hs_desc_listener)
+
+ if hs_desc_content_listener:
+ self.remove_event_listener(hs_desc_content_listener)
+
+ def get_conf(self, param, default = UNDEFINED, multiple = False):
+ """
+ Queries the current value for a configuration option. Some configuration
+ options (like the ExitPolicy) can have multiple values. This provides a
+ **list** with all of the values if **multiple** is **True**. Otherwise this
+ will be a **str** with the first value.
+
+ If provided with a **default** then that is provided if the configuration
+ option was unset or the query fails (invalid configuration option, error
+ response, control port closed, initiated, etc).
+
+ If the configuration value is unset and no **default** was given then this
+ provides **None** if **multiple** was **False** and an empty list if it was
+ **True**.
+
+ :param str param: configuration option to be queried
+ :param object default: response if the option is unset or the query fails
+ :param bool multiple: if **True** then provides a list with all of the
+ present values (this is an empty list if the config option is unset)
+
+ :returns:
+ Response depends upon how we were called as follows...
+
+ * **str** with the configuration value if **multiple** was **False**,
+ **None** if it was unset
+ * **list** with the response strings if multiple was **True**
+ * default if one was provided and the configuration option was either
+ unset or our call failed
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails and we weren't
+ provided a default response
+ * :class:`stem.InvalidArguments` if the configuration option
+ requested was invalid
+ """
+
+ # Config options are case insensitive and don't contain whitespace. Using
+ # strip so the following check will catch whitespace-only params.
+
+ param = param.lower().strip()
+
+ if not param:
+ return default if default != UNDEFINED else None
+
+ entries = self.get_conf_map(param, default, multiple)
+ return _case_insensitive_lookup(entries, param, default)
+
+ def get_conf_map(self, params, default = UNDEFINED, multiple = True):
+ """
+ Similar to :func:`~stem.control.Controller.get_conf` but queries multiple
+ configuration options, providing back a mapping of those options to their
+ values.
+
+ There are three use cases for GETCONF:
+
+ 1. a single value is provided (e.g. **ControlPort**)
+ 2. multiple values are provided for the option (e.g. **ExitPolicy**)
+ 3. a set of options that weren't necessarily requested are returned (for
+ instance querying **HiddenServiceOptions** gives **HiddenServiceDir**,
+ **HiddenServicePort**, etc)
+
+ The vast majority of the options fall into the first two categories, in
+ which case calling :func:`~stem.control.Controller.get_conf` is sufficient.
+ However, for batch queries or the special options that give a set of values
+ this provides back the full response. As of tor version 0.2.1.25
+ **HiddenServiceOptions** was the only option that falls into the third
+ category.
+
+ :param str,list params: configuration option(s) to be queried
+ :param object default: value for the mappings if the configuration option
+ is either undefined or the query fails
+ :param bool multiple: if **True** then the values provided are lists with
+ all of the present values
+
+ :returns:
+ **dict** of the 'config key => value' mappings. The value is a...
+
+ * **str** if **multiple** is **False**, **None** if the configuration
+ option is unset
+ * **list** if **multiple** is **True**
+ * the **default** if it was set and the value was either undefined or our
+ lookup failed
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails and we weren't provided
+ a default response
+ * :class:`stem.InvalidArguments` if the configuration option requested
+ was invalid
+ """
+
+ start_time = time.time()
+ reply = {}
+
+ if isinstance(params, (bytes, str_type)):
+ params = [params]
+
+ # remove strings which contain only whitespace
+ params = [entry for entry in params if entry.strip()]
+
+ if params == []:
+ return {}
+
+ # translate context sensitive options
+ lookup_params = set([MAPPED_CONFIG_KEYS.get(entry, entry) for entry in params])
+
+ # check for cached results
+
+ from_cache = [param.lower() for param in lookup_params]
+ cached_results = self._get_cache_map(from_cache, 'getconf')
+
+ for key in cached_results:
+ user_expected_key = _case_insensitive_lookup(lookup_params, key)
+ reply[user_expected_key] = cached_results[key]
+ lookup_params.remove(user_expected_key)
+
+ # if everything was cached then short circuit making the query
+ if not lookup_params:
+ log.trace('GETCONF %s (cache fetch)' % ' '.join(reply.keys()))
+ return self._get_conf_dict_to_response(reply, default, multiple)
+
+ try:
+ response = self.msg('GETCONF %s' % ' '.join(lookup_params))
+ stem.response.convert('GETCONF', response)
+ reply.update(response.entries)
+
+ if self.is_caching_enabled():
+ to_cache = dict((k.lower(), v) for k, v in response.entries.items())
+
+ for key in UNCACHEABLE_GETCONF_PARAMS:
+ if key in to_cache:
+ del to_cache[key]
+
+ self._set_cache(to_cache, 'getconf')
+
+ # Maps the entries back to the parameters that the user requested so the
+ # capitalization matches (ie, if they request "exitpolicy" then that
+ # should be the key rather than "ExitPolicy"). When the same
+ # configuration key is provided multiple times this determines the case
+ # based on the first and ignores the rest.
+ #
+ # This retains the tor provided camel casing of MAPPED_CONFIG_KEYS
+ # entries since the user didn't request those by their key, so we can't
+ # be sure what they wanted.
+
+ for key in reply:
+ if not key.lower() in MAPPED_CONFIG_KEYS.values():
+ user_expected_key = _case_insensitive_lookup(params, key, key)
+
+ if key != user_expected_key:
+ reply[user_expected_key] = reply[key]
+ del reply[key]
+
+ log.debug('GETCONF %s (runtime: %0.4f)' % (' '.join(lookup_params), time.time() - start_time))
+ return self._get_conf_dict_to_response(reply, default, multiple)
+ except stem.ControllerError as exc:
+ log.debug('GETCONF %s (failed: %s)' % (' '.join(lookup_params), exc))
+
+ if default != UNDEFINED:
+ return dict((param, default) for param in params)
+ else:
+ raise exc
+
+ def _get_conf_dict_to_response(self, config_dict, default, multiple):
+ """
+ Translates a dictionary of 'config key => [value1, value2...]' into the
+ return value of :func:`~stem.control.Controller.get_conf_map`, taking into
+ account what the caller requested.
+ """
+
+ return_dict = {}
+
+ for key, values in list(config_dict.items()):
+ if values == []:
+ # config option was unset
+ if default != UNDEFINED:
+ return_dict[key] = default
+ else:
+ return_dict[key] = [] if multiple else None
+ else:
+ return_dict[key] = values if multiple else values[0]
+
+ return return_dict
+
+ def set_conf(self, param, value):
+ """
+ Changes the value of a tor configuration option. Our value can be any of
+ the following...
+
+ * a string to set a single value
+ * a list of strings to set a series of values (for instance the ExitPolicy)
+ * None to either set the value to 0/NULL
+
+ :param str param: configuration option to be set
+ :param str,list value: value to set the parameter to
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails
+ * :class:`stem.InvalidArguments` if configuration options
+ requested was invalid
+ * :class:`stem.InvalidRequest` if the configuration setting is
+ impossible or if there's a syntax error in the configuration values
+ """
+
+ self.set_options({param: value}, False)
+
+ def reset_conf(self, *params):
+ """
+ Reverts one or more parameters to their default values.
+
+ :param str params: configuration option to be reset
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails
+ * :class:`stem.InvalidArguments` if configuration options requested was invalid
+ * :class:`stem.InvalidRequest` if the configuration setting is
+ impossible or if there's a syntax error in the configuration values
+ """
+
+ self.set_options(dict([(entry, None) for entry in params]), True)
+
+ def set_options(self, params, reset = False):
+ """
+ Changes multiple tor configuration options via either a SETCONF or
+ RESETCONF query. Both behave identically unless our value is None, in which
+ case SETCONF sets the value to 0 or NULL, and RESETCONF returns it to its
+ default value. This accepts str, list, or None values in a similar fashion
+ to :func:`~stem.control.Controller.set_conf`. For example...
+
+ ::
+
+ my_controller.set_options({
+ 'Nickname': 'caerSidi',
+ 'ExitPolicy': ['accept *:80', 'accept *:443', 'reject *:*'],
+ 'ContactInfo': '[email protected]',
+ 'Log': None,
+ })
+
+ The params can optionally be a list of key/value tuples, though the only
+ reason this type of argument would be useful is for hidden service
+ configuration (those options are order dependent).
+
+ :param dict,list params: mapping of configuration options to the values
+ we're setting it to
+ :param bool reset: issues a RESETCONF, returning **None** values to their
+ defaults if **True**
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails
+ * :class:`stem.InvalidArguments` if configuration options
+ requested was invalid
+ * :class:`stem.InvalidRequest` if the configuration setting is
+ impossible or if there's a syntax error in the configuration values
+ """
+
+ start_time = time.time()
+
+ # constructs the SETCONF or RESETCONF query
+ query_comp = ['RESETCONF' if reset else 'SETCONF']
+
+ if isinstance(params, dict):
+ params = list(params.items())
+
+ for param, value in params:
+ if isinstance(value, str):
+ query_comp.append('%s="%s"' % (param, value.strip()))
+ elif value:
+ query_comp.extend(['%s="%s"' % (param, val.strip()) for val in value])
+ else:
+ query_comp.append(param)
+
+ query = ' '.join(query_comp)
+ response = self.msg(query)
+ stem.response.convert('SINGLELINE', response)
+
+ if response.is_ok():
+ log.debug('%s (runtime: %0.4f)' % (query, time.time() - start_time))
+
+ if self.is_caching_enabled():
+ to_cache = {}
+
+ for param, value in params:
+ param = param.lower()
+
+ if isinstance(value, (bytes, str_type)):
+ value = [value]
+
+ to_cache[param] = value
+
+ if param == 'exitpolicy':
+ self._set_cache({'exitpolicy': None})
+
+ self._set_cache(to_cache, 'getconf')
+ else:
+ log.debug('%s (failed, code: %s, message: %s)' % (query, response.code, response.message))
+
+ if response.code == '552':
+ if response.message.startswith("Unrecognized option: Unknown option '"):
+ key = response.message[37:response.message.find("'", 37)]
+ raise stem.InvalidArguments(response.code, response.message, [key])
+ raise stem.InvalidRequest(response.code, response.message)
+ elif response.code in ('513', '553'):
+ raise stem.InvalidRequest(response.code, response.message)
+ else:
+ raise stem.ProtocolError('Returned unexpected status code: %s' % response.code)
+
+ @with_default()
+ def get_hidden_service_conf(self, default = UNDEFINED):
+ """
+ get_hidden_service_conf(default = UNDEFINED)
+
+ This provides a mapping of hidden service directories to their
+ attribute's key/value pairs. All hidden services are assured to have a
+ 'HiddenServicePort', but other entries may or may not exist.
+
+ ::
+
+ {
+ "/var/lib/tor/hidden_service_empty/": {
+ "HiddenServicePort": [
+ ]
+ },
+ "/var/lib/tor/hidden_service_with_two_ports/": {
+ "HiddenServiceAuthorizeClient": "stealth a, b",
+ "HiddenServicePort": [
+ (8020, "127.0.0.1", 8020), # the ports order is kept
+ (8021, "127.0.0.1", 8021)
+ ],
+ "HiddenServiceVersion": "2"
+ },
+ }
+
+ .. versionadded:: 1.3.0
+
+ :param object default: response if the query fails
+
+ :returns: **dict** with the hidden service configuration
+
+ :raises: :class:`stem.ControllerError` if the call fails and we weren't
+ provided a default response
+ """
+
+ start_time = time.time()
+
+ try:
+ response = self.msg('GETCONF HiddenServiceOptions')
+ stem.response.convert('GETCONF', response)
+ log.debug('GETCONF HiddenServiceOptions (runtime: %0.4f)' %
+ (time.time() - start_time))
+ except stem.ControllerError as exc:
+ log.debug('GETCONF HiddenServiceOptions (failed: %s)' % exc)
+ raise exc
+
+ service_dir_map = OrderedDict()
+ directory = None
+
+ for status_code, divider, content in response.content():
+ if content == 'HiddenServiceOptions':
+ continue
+
+ if '=' not in content:
+ continue
+
+ k, v = content.split('=', 1)
+
+ if k == 'HiddenServiceDir':
+ directory = v
+ service_dir_map[directory] = {'HiddenServicePort': []}
+ elif k == 'HiddenServicePort':
+ port = target_port = v
+ target_address = '127.0.0.1'
+
+ if not v.isdigit():
+ port, target = v.split()
+
+ if target.isdigit():
+ target_port = target
+ else:
+ target_address, target_port = target.split(':')
+
+ if not stem.util.connection.is_valid_port(port):
+ raise stem.ProtocolError('GETCONF provided an invalid HiddenServicePort port (%s): %s' % (port, content))
+ elif not stem.util.connection.is_valid_ipv4_address(target_address):
+ raise stem.ProtocolError('GETCONF provided an invalid HiddenServicePort target address (%s): %s' % (target_address, content))
+ elif not stem.util.connection.is_valid_port(target_port):
+ raise stem.ProtocolError('GETCONF provided an invalid HiddenServicePort target port (%s): %s' % (target_port, content))
+
+ service_dir_map[directory]['HiddenServicePort'].append((int(port), target_address, int(target_port)))
+ else:
+ service_dir_map[directory][k] = v
+
+ return service_dir_map
+
+ def set_hidden_service_conf(self, conf):
+ """
+ Update all the configured hidden services from a dictionary having
+ the same format as
+ :func:`~stem.control.Controller.get_hidden_service_conf`.
+
+ For convenience the HiddenServicePort entries can be an integer, string, or
+ tuple. If an **int** then we treat it as just a port. If a **str** we pass
+ that directly as the HiddenServicePort. And finally, if a **tuple** then
+ it's expected to be the **(port, target_address, target_port)** as provided
+ by :func:`~stem.control.Controller.get_hidden_service_conf`.
+
+ This is to say the following three are equivalent...
+
+ ::
+
+ "HiddenServicePort": [
+ 80,
+ '80 127.0.0.1:80',
+ (80, '127.0.0.1', 80),
+ ]
+
+ .. versionadded:: 1.3.0
+
+ :param dict conf: configuration dictionary
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails
+ * :class:`stem.InvalidArguments` if configuration options
+ requested was invalid
+ * :class:`stem.InvalidRequest` if the configuration setting is
+ impossible or if there's a syntax error in the configuration values
+ """
+
+ # If we're not adding or updating any hidden services then call RESETCONF
+ # so we drop existing values. Otherwise calling SETCONF is a no-op.
+
+ if not conf:
+ self.reset_conf('HiddenServiceDir')
+ return
+
+ # Convert conf dictionary into a list of ordered config tuples
+
+ hidden_service_options = []
+
+ for directory in conf:
+ hidden_service_options.append(('HiddenServiceDir', directory))
+
+ for k, v in list(conf[directory].items()):
+ if k == 'HiddenServicePort':
+ for entry in v:
+ if isinstance(entry, int):
+ entry = '%s 127.0.0.1:%s' % (entry, entry)
+ elif isinstance(entry, str):
+ pass # just pass along what the user gave us
+ elif isinstance(entry, tuple):
+ port, target_address, target_port = entry
+ entry = '%s %s:%s' % (port, target_address, target_port)
+
+ hidden_service_options.append(('HiddenServicePort', entry))
+ else:
+ hidden_service_options.append((k, str(v)))
+
+ self.set_options(hidden_service_options)
+
+ def create_hidden_service(self, path, port, target_address = None, target_port = None, auth_type = None, client_names = None):
+ """
+ Create a new hidden service. If the directory is already present, a
+ new port is added. This provides a **namedtuple** of the following...
+
+ * path (str) - hidden service directory
+
+ * hostname (str) - Content of the hostname file, if no **client_names**
+ are provided this is the onion address of the service. This is only
+ retrieved if we can read the hidden service directory.
+
+ * hostname_for_client (dict) - mapping of client names to their onion
+ address, this is only set if the **client_names** was provided and we
+ can read the hidden service directory
+
+ * config (dict) - tor's new hidden service configuration
+
+ Our *.onion address is fetched by reading the hidden service directory.
+ However, this directory is only readable by the tor user, so if unavailable
+ the **hostname** will be **None**.
+
+ **As of Tor 0.2.7.1 there's two ways for creating hidden services. This is
+ no longer the recommended method.** Rather, try using
+ :func:`~stem.control.Controller.create_ephemeral_hidden_service` instead.
+
+ .. versionadded:: 1.3.0
+
+ .. versionchanged:: 1.4.0
+ Added the auth_type and client_names arguments.
+
+ :param str path: path for the hidden service's data directory
+ :param int port: hidden service port
+ :param str target_address: address of the service, by default 127.0.0.1
+ :param int target_port: port of the service, by default this is the same as
+ **port**
+ :param str auth_type: authentication type: basic, stealth or None to disable auth
+ :param list client_names: client names (1-16 characters "A-Za-z0-9+-_")
+
+ :returns: **CreateHiddenServiceOutput** if we create or update a hidden service, **None** otherwise
+
+ :raises: :class:`stem.ControllerError` if the call fails
+ """
+
+ if not stem.util.connection.is_valid_port(port):
+ raise ValueError("%s isn't a valid port number" % port)
+ elif target_address and not stem.util.connection.is_valid_ipv4_address(target_address):
+ raise ValueError("%s isn't a valid IPv4 address" % target_address)
+ elif target_port is not None and not stem.util.connection.is_valid_port(target_port):
+ raise ValueError("%s isn't a valid port number" % target_port)
+ elif auth_type not in (None, 'basic', 'stealth'):
+ raise ValueError("%s isn't a recognized type of authentication" % auth_type)
+
+ port = int(port)
+ target_address = target_address if target_address else '127.0.0.1'
+ target_port = port if target_port is None else int(target_port)
+
+ conf = self.get_hidden_service_conf()
+
+ if path in conf and (port, target_address, target_port) in conf[path]['HiddenServicePort']:
+ return None
+
+ conf.setdefault(path, OrderedDict()).setdefault('HiddenServicePort', []).append((port, target_address, target_port))
+
+ if auth_type and client_names:
+ hsac = "%s %s" % (auth_type, ','.join(client_names))
+ conf[path]['HiddenServiceAuthorizeClient'] = hsac
+
+ self.set_hidden_service_conf(conf)
+
+ hostname, hostname_for_client = None, {}
+
+ if self.is_localhost():
+ hostname_path = os.path.join(path, 'hostname')
+
+ if not os.path.isabs(hostname_path):
+ cwd = stem.util.system.cwd(self.get_pid(None))
+
+ if cwd:
+ hostname_path = stem.util.system.expand_path(hostname_path, cwd)
+
+ if os.path.isabs(hostname_path):
+ start_time = time.time()
+
+ while not os.path.exists(hostname_path):
+ wait_time = time.time() - start_time
+
+ if wait_time >= 3:
+ break
+ else:
+ time.sleep(0.05)
+
+ try:
+ with open(hostname_path) as hostname_file:
+ hostname = hostname_file.read().strip()
+
+ if client_names and '\n' in hostname:
+ # When there's multiple clients this looks like...
+ #
+ # ndisjxzkgcdhrwqf.onion sjUwjTSPznqWLdOPuwRUzg # client: c1
+ # ndisjxzkgcdhrwqf.onion sUu92axuL5bKnA76s2KRfw # client: c2
+
+ for line in hostname.splitlines():
+ if ' # client: ' in line:
+ address = line.split()[0]
+ client = line.split(' # client: ', 1)[1]
+
+ if len(address) == 22 and address.endswith('.onion'):
+ hostname_for_client[client] = address
+ except:
+ pass
+
+ return CreateHiddenServiceOutput(
+ path = path,
+ hostname = hostname,
+ hostname_for_client = hostname_for_client,
+ config = conf,
+ )
+
+ def remove_hidden_service(self, path, port = None):
+ """
+ Discontinues a given hidden service.
+
+ .. versionadded:: 1.3.0
+
+ :param str path: path for the hidden service's data directory
+ :param int port: hidden service port
+
+ :returns: **True** if the hidden service is discontinued, **False** if it
+ wasn't running in the first place
+
+ :raises: :class:`stem.ControllerError` if the call fails
+ """
+
+ if port and not stem.util.connection.is_valid_port(port):
+ raise ValueError("%s isn't a valid port number" % port)
+
+ port = int(port) if port else None
+ conf = self.get_hidden_service_conf()
+
+ if path not in conf:
+ return False
+
+ if not port:
+ del conf[path]
+ else:
+ to_remove = [entry for entry in conf[path]['HiddenServicePort'] if entry[0] == port]
+
+ if not to_remove:
+ return False
+
+ for entry in to_remove:
+ conf[path]['HiddenServicePort'].remove(entry)
+
+ if not conf[path]['HiddenServicePort']:
+ del conf[path] # no ports left
+
+ self.set_hidden_service_conf(conf)
+ return True
+
+ @with_default()
+ def list_ephemeral_hidden_services(self, default = UNDEFINED, our_services = True, detached = False):
+ """
+ list_ephemeral_hidden_services(default = UNDEFINED, our_services = True, detached = False)
+
+ Lists hidden service addresses created by
+ :func:`~stem.control.Controller.create_ephemeral_hidden_service`.
+
+ .. versionadded:: 1.4.0
+
+ :param object default: response if the query fails
+ :param bool our_services: include services created with this controller
+ that weren't flagged as 'detached'
+ :param bool detached: include services whos contiuation isn't tied to a
+ controller
+
+ :returns: **list** of hidden service addresses without their '.onion'
+ suffix
+
+ :raises: :class:`stem.ControllerError` if the call fails and we weren't
+ provided a default response
+ """
+
+ if self.get_version() < stem.version.Requirement.ADD_ONION:
+ raise stem.UnsatisfiableRequest(message = 'Ephemeral hidden services were added in tor version %s' % stem.version.Requirement.ADD_ONION)
+
+ result = []
+
+ if our_services:
+ try:
+ result += self.get_info('onions/current').split('\n')
+ except stem.ProtocolError as exc:
+ if 'No onion services of the specified type.' not in str(exc):
+ raise exc
+
+ if detached:
+ try:
+ result += self.get_info('onions/detached').split('\n')
+ except stem.ProtocolError as exc:
+ if 'No onion services of the specified type.' not in str(exc):
+ raise exc
+
+ return result
+
+ def create_ephemeral_hidden_service(self, ports, key_type = 'NEW', key_content = 'BEST', discard_key = False, detached = False, await_publication = False):
+ """
+ Creates a new hidden service. Unlike
+ :func:`~stem.control.Controller.create_hidden_service` this style of
+ hidden service doesn't touch disk, carrying with it a lot of advantages.
+ This is the suggested method for making hidden services.
+
+ Our **ports** argument can be a single port...
+
+ ::
+
+ create_ephemeral_hidden_service(80)
+
+ ... list of ports the service is available on...
+
+ ::
+
+ create_ephemeral_hidden_service([80, 443])
+
+ ... or a mapping of hidden service ports to their targets...
+
+ ::
+
+ create_ephemeral_hidden_service({80: 80, 443: '173.194.33.133:443'})
+
+ .. versionadded:: 1.4.0
+
+ :param int,list,dict ports: hidden service port(s) or mapping of hidden
+ service ports to their targets
+ :param str key_type: type of key being provided, generates a new key if
+ 'NEW' (options are: **NEW** and **RSA1024**)
+ :param str key_content: key for the service to use or type of key to be
+ generated (options when **key_type** is **NEW** are **BEST** and
+ **RSA1024**)
+ :param bool discard_key: avoid providing the key back in our response
+ :param bool detached: continue this hidden service even after this control
+ connection is closed if **True**
+ :param bool await_publication: blocks until our descriptor is successfully
+ published if **True**
+
+ :returns: :class:`~stem.response.add_onion.AddOnionResponse` with the response
+
+ :raises: :class:`stem.ControllerError` if the call fails
+ """
+
+ if self.get_version() < stem.version.Requirement.ADD_ONION:
+ raise stem.UnsatisfiableRequest(message = 'Ephemeral hidden services were added in tor version %s' % stem.version.Requirement.ADD_ONION)
+
+ hs_desc_queue, hs_desc_listener = queue.Queue(), None
+
+ if await_publication:
+ def hs_desc_listener(event):
+ hs_desc_queue.put(event)
+
+ self.add_event_listener(hs_desc_listener, EventType.HS_DESC)
+
+ request = 'ADD_ONION %s:%s' % (key_type, key_content)
+
+ flags = []
+
+ if discard_key:
+ flags.append('DiscardPK')
+
+ if detached:
+ flags.append('Detach')
+
+ if flags:
+ request += ' Flags=%s' % ','.join(flags)
+
+ if isinstance(ports, int):
+ request += ' Port=%s' % ports
+ elif isinstance(ports, list):
+ for port in ports:
+ request += ' Port=%s' % port
+ elif isinstance(ports, dict):
+ for port, target in ports.items():
+ request += ' Port=%s,%s' % (port, target)
+ else:
+ raise ValueError("The 'ports' argument of create_ephemeral_hidden_service() needs to be an int, list, or dict")
+
+ response = self.msg(request)
+ stem.response.convert('ADD_ONION', response)
+
+ if await_publication:
+ # We should receive five UPLOAD events, followed by up to another five
+ # UPLOADED to indicate they've finished. Presently tor seems to have an
+ # issue where the address is provided for UPLOAD but not UPLOADED so need
+ # to just guess that if it's for the same hidden service authority then
+ # it's what we're looking for.
+
+ directories_uploaded_to, failures = [], []
+
+ try:
+ while True:
+ event = hs_desc_queue.get()
+
+ if event.action == stem.HSDescAction.UPLOAD and event.address == response.service_id:
+ directories_uploaded_to.append(event.directory_fingerprint)
+ elif event.action == stem.HSDescAction.UPLOADED and event.directory_fingerprint in directories_uploaded_to:
+ break # successfully uploaded to a HS authority... maybe
+ elif event.action == stem.HSDescAction.FAILED and event.directory_fingerprint in directories_uploaded_to:
+ failures.append('%s (%s)' % (event.directory_fingerprint, event.reason))
+
+ if len(directories_uploaded_to) == len(failures):
+ raise stem.OperationFailed(message = 'Failed to upload our hidden service descriptor to %s' % ', '.join(failures))
+ finally:
+ self.remove_event_listener(hs_desc_listener)
+
+ return response
+
+ def remove_ephemeral_hidden_service(self, service_id):
+ """
+ Discontinues a given hidden service that was created with
+ :func:`~stem.control.Controller.create_ephemeral_hidden_service`.
+
+ .. versionadded:: 1.4.0
+
+ :param str service_id: hidden service address without the '.onion' suffix
+
+ :returns: **True** if the hidden service is discontinued, **False** if it
+ wasn't running in the first place
+
+ :raises: :class:`stem.ControllerError` if the call fails
+ """
+
+ if self.get_version() < stem.version.Requirement.ADD_ONION:
+ raise stem.UnsatisfiableRequest(message = 'Ephemeral hidden services were added in tor version %s' % stem.version.Requirement.ADD_ONION)
+
+ response = self.msg('DEL_ONION %s' % service_id)
+ stem.response.convert('SINGLELINE', response)
+
+ if response.is_ok():
+ return True
+ elif response.code == '552':
+ return False # no hidden service to discontinue
+ else:
+ raise stem.ProtocolError('DEL_ONION returned unexpected response code: %s' % response.code)
+
+ def add_event_listener(self, listener, *events):
+ """
+ Directs further tor controller events to a given function. The function is
+ expected to take a single argument, which is a
+ :class:`~stem.response.events.Event` subclass. For instance the following
+ would print the bytes sent and received by tor over five seconds...
+
+ ::
+
+ import time
+ from stem.control import Controller, EventType
+
+ def print_bw(event):
+ print('sent: %i, received: %i' % (event.written, event.read))
+
+ with Controller.from_port(port = 9051) as controller:
+ controller.authenticate()
+ controller.add_event_listener(print_bw, EventType.BW)
+ time.sleep(5)
+
+ If a new control connection is initialized then this listener will be
+ reattached.
+
+ :param functor listener: function to be called when an event is received
+ :param stem.control.EventType events: event types to be listened for
+
+ :raises: :class:`stem.ProtocolError` if unable to set the events
+ """
+
+ # first checking that tor supports these event types
+
+ with self._event_listeners_lock:
+ if self.is_authenticated():
+ for event_type in events:
+ event_type = stem.response.events.EVENT_TYPE_TO_CLASS.get(event_type)
+
+ if event_type and (self.get_version() < event_type._VERSION_ADDED):
+ raise stem.InvalidRequest(552, '%s event requires Tor version %s or later' % (event_type, event_type._VERSION_ADDED))
+
+ for event_type in events:
+ self._event_listeners.setdefault(event_type, []).append(listener)
+
+ failed_events = self._attach_listeners()[1]
+
+ # restricted the failures to just things we requested
+
+ failed_events = set(failed_events).intersection(set(events))
+
+ if failed_events:
+ raise stem.ProtocolError('SETEVENTS rejected %s' % ', '.join(failed_events))
+
+ def remove_event_listener(self, listener):
+ """
+ Stops a listener from being notified of further tor events.
+
+ :param stem.control.EventListener listener: listener to be removed
+
+ :raises: :class:`stem.ProtocolError` if unable to set the events
+ """
+
+ with self._event_listeners_lock:
+ event_types_changed = False
+
+ for event_type, event_listeners in list(self._event_listeners.items()):
+ if listener in event_listeners:
+ event_listeners.remove(listener)
+
+ if len(event_listeners) == 0:
+ event_types_changed = True
+ del self._event_listeners[event_type]
+
+ if event_types_changed:
+ response = self.msg('SETEVENTS %s' % ' '.join(self._event_listeners.keys()))
+
+ if not response.is_ok():
+ raise stem.ProtocolError('SETEVENTS received unexpected response\n%s' % response)
+
+ def _get_cache(self, param, namespace = None):
+ """
+ Queries our request cache for the given key.
+
+ :param str param: key to be queried
+ :param str namespace: namespace in which to check for the key
+
+ :returns: cached value corresponding to key or **None** if the key wasn't found
+ """
+
+ return self._get_cache_map([param], namespace).get(param, None)
+
+ def _get_cache_map(self, params, namespace = None):
+ """
+ Queries our request cache for multiple entries.
+
+ :param list params: keys to be queried
+ :param str namespace: namespace in which to check for the keys
+
+ :returns: **dict** of 'param => cached value' pairs of keys present in cache
+ """
+
+ with self._cache_lock:
+ cached_values = {}
+
+ if self.is_caching_enabled():
+ for param in params:
+ if namespace:
+ cache_key = '%s.%s' % (namespace, param)
+ else:
+ cache_key = param
+
+ if cache_key in self._request_cache:
+ cached_values[param] = self._request_cache[cache_key]
+
+ return cached_values
+
+ def _set_cache(self, params, namespace = None):
+ """
+ Sets the given request cache entries. If the new cache value is **None**
+ then it is removed from our cache.
+
+ :param dict params: **dict** of 'cache_key => value' pairs to be cached
+ :param str namespace: namespace for the keys
+ """
+
+ with self._cache_lock:
+ if not self.is_caching_enabled():
+ return
+
+ for key, value in list(params.items()):
+ if namespace:
+ cache_key = '%s.%s' % (namespace, key)
+ else:
+ cache_key = key
+
+ if value is None:
+ if cache_key in self._request_cache:
+ del self._request_cache[cache_key]
+ else:
+ self._request_cache[cache_key] = value
+
+ def is_caching_enabled(self):
+ """
+ **True** if caching has been enabled, **False** otherwise.
+
+ :returns: bool to indicate if caching is enabled
+ """
+
+ return self._is_caching_enabled
+
+ def set_caching(self, enabled):
+ """
+ Enables or disables caching of information retrieved from tor.
+
+ :param bool enabled: **True** to enable caching, **False** to disable it
+ """
+
+ self._is_caching_enabled = enabled
+
+ if not self._is_caching_enabled:
+ self.clear_cache()
+
+ def clear_cache(self):
+ """
+ Drops any cached results.
+ """
+
+ with self._cache_lock:
+ self._request_cache = {}
+ self._last_newnym = 0.0
+ self._geoip_failure_count = 0
+
+ def load_conf(self, configtext):
+ """
+ Sends the configuration text to Tor and loads it as if it has been read from
+ the torrc.
+
+ :param str configtext: the configuration text
+
+ :raises: :class:`stem.ControllerError` if the call fails
+ """
+
+ response = self.msg('LOADCONF\n%s' % configtext)
+ stem.response.convert('SINGLELINE', response)
+
+ if response.code in ('552', '553'):
+ if response.code == '552' and response.message.startswith('Invalid config file: Failed to parse/validate config: Unknown option'):
+ raise stem.InvalidArguments(response.code, response.message, [response.message[70:response.message.find('.', 70) - 1]])
+ raise stem.InvalidRequest(response.code, response.message)
+ elif not response.is_ok():
+ raise stem.ProtocolError('+LOADCONF Received unexpected response\n%s' % str(response))
+
+ def save_conf(self):
+ """
+ Saves the current configuration options into the active torrc file.
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails
+ * :class:`stem.OperationFailed` if the client is unable to save
+ the configuration file
+ """
+
+ response = self.msg('SAVECONF')
+ stem.response.convert('SINGLELINE', response)
+
+ if response.is_ok():
+ return True
+ elif response.code == '551':
+ raise stem.OperationFailed(response.code, response.message)
+ else:
+ raise stem.ProtocolError('SAVECONF returned unexpected response code')
+
+ def is_feature_enabled(self, feature):
+ """
+ Checks if a control connection feature is enabled. These features can be
+ enabled using :func:`~stem.control.Controller.enable_feature`.
+
+ :param str feature: feature to be checked
+
+ :returns: **True** if feature is enabled, **False** otherwise
+ """
+
+ feature = feature.upper()
+
+ if feature in self._enabled_features:
+ return True
+ else:
+ # check if this feature is on by default
+ defaulted_version = None
+
+ if feature == 'EXTENDED_EVENTS':
+ defaulted_version = stem.version.Requirement.FEATURE_EXTENDED_EVENTS
+ elif feature == 'VERBOSE_NAMES':
+ defaulted_version = stem.version.Requirement.FEATURE_VERBOSE_NAMES
+
+ if defaulted_version:
+ our_version = self.get_version(None)
+
+ if our_version and our_version >= defaulted_version:
+ self._enabled_features.append(feature)
+
+ return feature in self._enabled_features
+
+ def enable_feature(self, features):
+ """
+ Enables features that are disabled by default to maintain backward
+ compatibility. Once enabled, a feature cannot be disabled and a new
+ control connection must be opened to get a connection with the feature
+ disabled. Feature names are case-insensitive.
+
+ The following features are currently accepted:
+
+ * EXTENDED_EVENTS - Requests the extended event syntax
+ * VERBOSE_NAMES - Replaces ServerID with LongName in events and GETINFO results
+
+ :param str,list features: a single feature or a list of features to be enabled
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails
+ * :class:`stem.InvalidArguments` if features passed were invalid
+ """
+
+ if isinstance(features, (bytes, str_type)):
+ features = [features]
+
+ response = self.msg('USEFEATURE %s' % ' '.join(features))
+ stem.response.convert('SINGLELINE', response)
+
+ if not response.is_ok():
+ if response.code == '552':
+ invalid_feature = []
+
+ if response.message.startswith('Unrecognized feature "'):
+ invalid_feature = [response.message[22:response.message.find('"', 22)]]
+
+ raise stem.InvalidArguments(response.code, response.message, invalid_feature)
+
+ raise stem.ProtocolError('USEFEATURE provided an invalid response code: %s' % response.code)
+
+ self._enabled_features += [entry.upper() for entry in features]
+
+ @with_default()
+ def get_circuit(self, circuit_id, default = UNDEFINED):
+ """
+ get_circuit(circuit_id, default = UNDEFINED)
+
+ Provides a circuit currently available from tor.
+
+ :param int circuit_id: circuit to be fetched
+ :param object default: response if the query fails
+
+ :returns: :class:`stem.response.events.CircuitEvent` for the given circuit
+
+ :raises:
+ * :class:`stem.ControllerError` if the call fails
+ * **ValueError** if the circuit doesn't exist
+
+ An exception is only raised if we weren't provided a default response.
+ """
+
+ for circ in self.get_circuits():
+ if circ.id == circuit_id:
+ return circ
+
+ raise ValueError("Tor currently does not have a circuit with the id of '%s'" % circuit_id)
+
+ @with_default()
+ def get_circuits(self, default = UNDEFINED):
+ """
+ get_circuits(default = UNDEFINED)
+
+ Provides tor's currently available circuits.
+
+ :param object default: response if the query fails
+
+ :returns: **list** of :class:`stem.response.events.CircuitEvent` for our circuits
+
+ :raises: :class:`stem.ControllerError` if the call fails and no default was provided
+ """
+
+ circuits = []
+ response = self.get_info('circuit-status')
+
+ for circ in response.splitlines():
+ circ_message = stem.socket.recv_message(StringIO('650 CIRC ' + circ + '\r\n'))
+ stem.response.convert('EVENT', circ_message, arrived_at = 0)
+ circuits.append(circ_message)
+
+ return circuits
+
+ def new_circuit(self, path = None, purpose = 'general', await_build = False):
+ """
+ Requests a new circuit. If the path isn't provided, one is automatically
+ selected.
+
+ :param list,str path: one or more relays to make a circuit through
+ :param str purpose: 'general' or 'controller'
+ :param bool await_build: blocks until the circuit is built if **True**
+
+ :returns: str of the circuit id of the newly created circuit
+
+ :raises: :class:`stem.ControllerError` if the call fails
+ """
+
+ return self.extend_circuit('0', path, purpose, await_build)
+
+ def extend_circuit(self, circuit_id = '0', path = None, purpose = 'general', await_build = False):
+ """
+ Either requests the creation of a new circuit or extends an existing one.
+
+ When called with a circuit value of zero (the default) a new circuit is
+ created, and when non-zero the circuit with that id is extended. If the
+ path isn't provided, one is automatically selected.
+
+ A python interpreter session used to create circuits could look like this...
+
+ ::
+
+ >>> controller.extend_circuit('0', ['718BCEA286B531757ACAFF93AE04910EA73DE617', '30BAB8EE7606CBD12F3CC269AE976E0153E7A58D', '2765D8A8C4BBA3F89585A9FFE0E8575615880BEB'])
+ 19
+ >>> controller.extend_circuit('0')
+ 20
+ >>> print(controller.get_info('circuit-status'))
+ 20 EXTENDED $718BCEA286B531757ACAFF93AE04910EA73DE617=KsmoinOK,$649F2D0ACF418F7CFC6539AB2257EB2D5297BAFA=Eskimo BUILD_FLAGS=NEED_CAPACITY PURPOSE=GENERAL TIME_CREATED=2012-12-06T13:51:11.433755
+ 19 BUILT $718BCEA286B531757ACAFF93AE04910EA73DE617=KsmoinOK,$30BAB8EE7606CBD12F3CC269AE976E0153E7A58D=Pascal1,$2765D8A8C4BBA3F89585A9FFE0E8575615880BEB=Anthracite PURPOSE=GENERAL TIME_CREATED=2012-12-06T13:50:56.969938
+
+ :param str circuit_id: id of a circuit to be extended
+ :param list,str path: one or more relays to make a circuit through, this is
+ required if the circuit id is non-zero
+ :param str purpose: 'general' or 'controller'
+ :param bool await_build: blocks until the circuit is built if **True**
+
+ :returns: str of the circuit id of the created or extended circuit
+
+ :raises:
+ * :class:`stem.InvalidRequest` if one of the parameters were invalid
+ * :class:`stem.CircuitExtensionFailed` if we were waiting for the circuit
+ to build but it failed
+ * :class:`stem.ControllerError` if the call fails
+ """
+
+ # Attaches a temporary listener for CIRC events if we'll be waiting for it
+ # to build. This is icky, but we can't reliably do this via polling since
+ # we then can't get the failure if it can't be created.
+
+ circ_queue, circ_listener = queue.Queue(), None
+
+ if await_build:
+ def circ_listener(event):
+ circ_queue.put(event)
+
+ self.add_event_listener(circ_listener, EventType.CIRC)
+
+ try:
+ # we might accidently get integer circuit ids
+ circuit_id = str(circuit_id)
+
+ if path is None and circuit_id == '0':
+ path_opt_version = stem.version.Requirement.EXTENDCIRCUIT_PATH_OPTIONAL
+
+ if not self.get_version() >= path_opt_version:
+ raise stem.InvalidRequest(512, 'EXTENDCIRCUIT requires the path prior to version %s' % path_opt_version)
+
+ args = [circuit_id]
+
+ if isinstance(path, (bytes, str_type)):
+ path = [path]
+
+ if path:
+ args.append(','.join(path))
+
+ if purpose:
+ args.append('purpose=%s' % purpose)
+
+ response = self.msg('EXTENDCIRCUIT %s' % ' '.join(args))
+ stem.response.convert('SINGLELINE', response)
+
+ if response.code in ('512', '552'):
+ raise stem.InvalidRequest(response.code, response.message)
+ elif not response.is_ok():
+ raise stem.ProtocolError('EXTENDCIRCUIT returned unexpected response code: %s' % response.code)
+
+ if not response.message.startswith('EXTENDED '):
+ raise stem.ProtocolError('EXTENDCIRCUIT response invalid:\n%s', response)
+
+ new_circuit = response.message.split(' ', 1)[1]
+
+ if await_build:
+ while True:
+ circ = circ_queue.get()
+
+ if circ.id == new_circuit:
+ if circ.status == CircStatus.BUILT:
+ break
+ elif circ.status == CircStatus.FAILED:
+ raise stem.CircuitExtensionFailed('Circuit failed to be created: %s' % circ.reason, circ)
+ elif circ.status == CircStatus.CLOSED:
+ raise stem.CircuitExtensionFailed('Circuit was closed prior to build', circ)
+
+ return new_circuit
+ finally:
+ if circ_listener:
+ self.remove_event_listener(circ_listener)
+
+ def repurpose_circuit(self, circuit_id, purpose):
+ """
+ Changes a circuit's purpose. Currently, two purposes are recognized...
+ * general
+ * controller
+
+ :param str circuit_id: id of the circuit whose purpose is to be changed
+ :param str purpose: purpose (either 'general' or 'controller')
+
+ :raises: :class:`stem.InvalidArguments` if the circuit doesn't exist or if the purpose was invalid
+ """
+
+ response = self.msg('SETCIRCUITPURPOSE %s purpose=%s' % (circuit_id, purpose))
+ stem.response.convert('SINGLELINE', response)
+
+ if not response.is_ok():
+ if response.code == '552':
+ raise stem.InvalidRequest(response.code, response.message)
+ else:
+ raise stem.ProtocolError('SETCIRCUITPURPOSE returned unexpected response code: %s' % response.code)
+
+ def close_circuit(self, circuit_id, flag = ''):
+ """
+ Closes the specified circuit.
+
+ :param str circuit_id: id of the circuit to be closed
+ :param str flag: optional value to modify closing, the only flag available
+ is 'IfUnused' which will not close the circuit unless it is unused
+
+ :raises: :class:`stem.InvalidArguments` if the circuit is unknown
+ :raises: :class:`stem.InvalidRequest` if not enough information is provided
+ """
+
+ response = self.msg('CLOSECIRCUIT %s %s' % (circuit_id, flag))
+ stem.response.convert('SINGLELINE', response)
+
+ if not response.is_ok():
+ if response.code in ('512', '552'):
+ if response.message.startswith('Unknown circuit '):
+ raise stem.InvalidArguments(response.code, response.message, [circuit_id])
+ raise stem.InvalidRequest(response.code, response.message)
+ else:
+ raise stem.ProtocolError('CLOSECIRCUIT returned unexpected response code: %s' % response.code)
+
+ @with_default()
+ def get_streams(self, default = UNDEFINED):
+ """
+ get_streams(default = UNDEFINED)
+
+ Provides the list of streams tor is currently handling.
+
+ :param object default: response if the query fails
+
+ :returns: list of :class:`stem.response.events.StreamEvent` objects
+
+ :raises: :class:`stem.ControllerError` if the call fails and no default was
+ provided
+ """
+
+ streams = []
+ response = self.get_info('stream-status')
+
+ for stream in response.splitlines():
+ message = stem.socket.recv_message(StringIO('650 STREAM ' + stream + '\r\n'))
+ stem.response.convert('EVENT', message, arrived_at = 0)
+ streams.append(message)
+
+ return streams
+
+ def attach_stream(self, stream_id, circuit_id, exiting_hop = None):
+ """
+ Attaches a stream to a circuit.
+
+ Note: Tor attaches streams to circuits automatically unless the
+ __LeaveStreamsUnattached configuration variable is set to '1'
+
+ :param str stream_id: id of the stream that must be attached
+ :param str circuit_id: id of the circuit to which it must be attached
+ :param int exiting_hop: hop in the circuit where traffic should exit
+
+ :raises:
+ * :class:`stem.InvalidRequest` if the stream or circuit id were unrecognized
+ * :class:`stem.UnsatisfiableRequest` if the stream isn't in a state where it can be attached
+ * :class:`stem.OperationFailed` if the stream couldn't be attached for any other reason
+ """
+
+ query = 'ATTACHSTREAM %s %s' % (stream_id, circuit_id)
+
+ if exiting_hop:
+ query += ' HOP=%s' % exiting_hop
+
+ response = self.msg(query)
+ stem.response.convert('SINGLELINE', response)
+
+ if not response.is_ok():
+ if response.code == '552':
+ raise stem.InvalidRequest(response.code, response.message)
+ elif response.code == '551':
+ raise stem.OperationFailed(response.code, response.message)
+ elif response.code == '555':
+ raise stem.UnsatisfiableRequest(response.code, response.message)
+ else:
+ raise stem.ProtocolError('ATTACHSTREAM returned unexpected response code: %s' % response.code)
+
+ def close_stream(self, stream_id, reason = stem.RelayEndReason.MISC, flag = ''):
+ """
+ Closes the specified stream.
+
+ :param str stream_id: id of the stream to be closed
+ :param stem.RelayEndReason reason: reason the stream is closing
+ :param str flag: not currently used
+
+ :raises:
+ * :class:`stem.InvalidArguments` if the stream or reason are not recognized
+ * :class:`stem.InvalidRequest` if the stream and/or reason are missing
+ """
+
+ # there's a single value offset between RelayEndReason.index_of() and the
+ # value that tor expects since tor's value starts with the index of one
+
+ response = self.msg('CLOSESTREAM %s %s %s' % (stream_id, stem.RelayEndReason.index_of(reason) + 1, flag))
+ stem.response.convert('SINGLELINE', response)
+
+ if not response.is_ok():
+ if response.code in ('512', '552'):
+ if response.message.startswith('Unknown stream '):
+ raise stem.InvalidArguments(response.code, response.message, [stream_id])
+ elif response.message.startswith('Unrecognized reason '):
+ raise stem.InvalidArguments(response.code, response.message, [reason])
+ raise stem.InvalidRequest(response.code, response.message)
+ else:
+ raise stem.ProtocolError('CLOSESTREAM returned unexpected response code: %s' % response.code)
+
+ def signal(self, signal):
+ """
+ Sends a signal to the Tor client.
+
+ :param stem.Signal signal: type of signal to be sent
+
+ :raises: :class:`stem.InvalidArguments` if signal provided wasn't recognized
+ """
+
+ response = self.msg('SIGNAL %s' % signal)
+ stem.response.convert('SINGLELINE', response)
+
+ if response.is_ok():
+ if signal == stem.Signal.NEWNYM:
+ self._last_newnym = time.time()
+ else:
+ if response.code == '552':
+ raise stem.InvalidArguments(response.code, response.message, [signal])
+
+ raise stem.ProtocolError('SIGNAL response contained unrecognized status code: %s' % response.code)
+
+ def is_newnym_available(self):
+ """
+ Indicates if tor would currently accept a NEWNYM signal. This can only
+ account for signals sent via this controller.
+
+ .. versionadded:: 1.2.0
+
+ :returns: **True** if tor would currently accept a NEWNYM signal, **False**
+ otherwise
+ """
+
+ if self.is_alive():
+ return self.get_newnym_wait() == 0.0
+ else:
+ return False
+
+ def get_newnym_wait(self):
+ """
+ Provides the number of seconds until a NEWNYM signal would be respected.
+ This can only account for signals sent via this controller.
+
+ .. versionadded:: 1.2.0
+
+ :returns: **float** for the number of seconds until tor would respect
+ another NEWNYM signal
+ """
+
+ return max(0.0, self._last_newnym + 10 - time.time())
+
+ @with_default()
+ def get_effective_rate(self, default = UNDEFINED, burst = False):
+ """
+ get_effective_rate(default = UNDEFINED, burst = False)
+
+ Provides the maximum rate this relay is configured to relay in bytes per
+ second. This is based on multiple torrc parameters if they're set...
+
+ * Effective Rate = min(BandwidthRate, RelayBandwidthRate, MaxAdvertisedBandwidth)
+ * Effective Burst = min(BandwidthBurst, RelayBandwidthBurst)
+
+ .. versionadded:: 1.3.0
+
+ :param object default: response if the query fails
+ :param bool burst: provides the burst bandwidth, otherwise this provides
+ the standard rate
+
+ :returns: **int** with the effective bandwidth rate in bytes per second
+
+ :raises: :class:`stem.ControllerError` if the call fails and no default was
+ provided
+ """
+
+ if not burst:
+ attributes = ('BandwidthRate', 'RelayBandwidthRate', 'MaxAdvertisedBandwidth')
+ else:
+ attributes = ('BandwidthBurst', 'RelayBandwidthBurst')
+
+ value = None
+
+ for attr in attributes:
+ attr_value = int(self.get_conf(attr))
+
+ if attr_value == 0 and attr.startswith('Relay'):
+ continue # RelayBandwidthRate and RelayBandwidthBurst default to zero
+
+ value = min(value, attr_value) if value else attr_value
+
+ return value
+
+ def is_geoip_unavailable(self):
+ """
+ Provides **True** if we've concluded hat our geoip database is unavailable,
+ **False** otherwise. This is determined by having our 'GETINFO
+ ip-to-country/\*' lookups fail so this will default to **False** if we
+ aren't making those queries.
+
+ Geoip failures will be untracked if caching is disabled.
+
+ :returns: **bool** to indicate if we've concluded our geoip database to be
+ unavailable or not
+ """
+
+ return self._geoip_failure_count >= GEOIP_FAILURE_THRESHOLD
+
+ def map_address(self, mapping):
+ """
+ Map addresses to replacement addresses. Tor replaces subseqent connections
+ to the original addresses with the replacement addresses.
+
+ If the original address is a null address, i.e., one of '0.0.0.0', '::0', or
+ '.' Tor picks an original address itself and returns it in the reply. If the
+ original address is already mapped to a different address the mapping is
+ removed.
+
+ :param dict mapping: mapping of original addresses to replacement addresses
+
+ :raises:
+ * :class:`stem.InvalidRequest` if the addresses are malformed
+ * :class:`stem.OperationFailed` if Tor couldn't fulfill the request
+
+ :returns: **dict** with 'original -> replacement' address mappings
+ """
+
+ mapaddress_arg = ' '.join(['%s=%s' % (k, v) for (k, v) in list(mapping.items())])
+ response = self.msg('MAPADDRESS %s' % mapaddress_arg)
+ stem.response.convert('MAPADDRESS', response)
+
+ return response.entries
+
+ def drop_guards(self):
+ """
+ Drops our present guard nodes and picks a new set.
+
+ .. versionadded:: 1.2.0
+
+ :raises: :class:`stem.ControllerError` if Tor couldn't fulfill the request
+ """
+
+ if self.get_version() < stem.version.Requirement.DROPGUARDS:
+ raise stem.UnsatisfiableRequest(message = 'DROPGUARDS was added in tor version %s' % stem.version.Requirement.DROPGUARDS)
+
+ self.msg('DROPGUARDS')
+
+ def _post_authentication(self):
+ super(Controller, self)._post_authentication()
+
+ # try to re-attach event listeners to the new instance
+
+ with self._event_listeners_lock:
+ try:
+ failed_events = self._attach_listeners()[1]
+
+ if failed_events:
+ # remove our listeners for these so we don't keep failing
+ for event_type in failed_events:
+ del self._event_listeners[event_type]
+
+ logging_id = 'stem.controller.event_reattach-%s' % '-'.join(failed_events)
+ log.log_once(logging_id, log.WARN, 'We were unable to re-attach our event listeners to the new tor instance for: %s' % ', '.join(failed_events))
+ except stem.ProtocolError as exc:
+ log.warn('Unable to issue the SETEVENTS request to re-attach our listeners (%s)' % exc)
+
+ # issue TAKEOWNERSHIP if we're the owning process for this tor instance
+
+ owning_pid = self.get_conf('__OwningControllerProcess', None)
+
+ if owning_pid == str(os.getpid()) and self.is_localhost():
+ response = self.msg('TAKEOWNERSHIP')
+ stem.response.convert('SINGLELINE', response)
+
+ if response.is_ok():
+ # Now that tor is tracking our ownership of the process via the control
+ # connection, we can stop having it check for us via our pid.
+
+ try:
+ self.reset_conf('__OwningControllerProcess')
+ except stem.ControllerError as exc:
+ log.warn("We were unable to reset tor's __OwningControllerProcess configuration. It will continue to periodically check if our pid exists. (%s)" % exc)
+ else:
+ log.warn('We were unable assert ownership of tor through TAKEOWNERSHIP, despite being configured to be the owning process through __OwningControllerProcess. (%s)' % response)
+
+ def _handle_event(self, event_message):
+ stem.response.convert('EVENT', event_message, arrived_at = time.time())
+
+ with self._event_listeners_lock:
+ for event_type, event_listeners in list(self._event_listeners.items()):
+ if event_type == event_message.type:
+ for listener in event_listeners:
+ listener(event_message)
+
+ def _attach_listeners(self):
+ """
+ Attempts to subscribe to the self._event_listeners events from tor. This is
+ a no-op if we're not currently authenticated.
+
+ :returns: tuple of the form (set_events, failed_events)
+
+ :raises: :class:`stem.ControllerError` if unable to make our request to tor
+ """
+
+ set_events, failed_events = [], []
+
+ with self._event_listeners_lock:
+ if self.is_authenticated():
+ # try to set them all
+ response = self.msg('SETEVENTS %s' % ' '.join(self._event_listeners.keys()))
+
+ if response.is_ok():
+ set_events = list(self._event_listeners.keys())
+ else:
+ # One of the following likely happened...
+ #
+ # * Our user attached listeners before having an authenticated
+ # connection, so we couldn't check if we met the version
+ # requirement.
+ #
+ # * User attached listeners to one tor instance, then connected us to
+ # an older tor instancce.
+ #
+ # * Some other controller hiccup (far less likely).
+ #
+ # See if we can set some subset of our events.
+
+ for event in list(self._event_listeners.keys()):
+ response = self.msg('SETEVENTS %s' % ' '.join(set_events + [event]))
+
+ if response.is_ok():
+ set_events.append(event)
+ else:
+ failed_events.append(event)
+
+ return (set_events, failed_events)
+
+
+def _parse_circ_path(path):
+ """
+ Parses a circuit path as a list of **(fingerprint, nickname)** tuples. Tor
+ circuit paths are defined as being of the form...
+
+ ::
+
+ Path = LongName *("," LongName)
+ LongName = Fingerprint [ ( "=" / "~" ) Nickname ]
+
+ example:
+ $999A226EBED397F331B612FE1E4CFAE5C1F201BA=piyaz
+
+ ... *unless* this is prior to tor version 0.2.2.1 with the VERBOSE_NAMES
+ feature turned off (or before version 0.1.2.2 where the feature was
+ introduced). In that case either the fingerprint or nickname in the tuple
+ will be **None**, depending on which is missing.
+
+ ::
+
+ Path = ServerID *("," ServerID)
+ ServerID = Nickname / Fingerprint
+
+ example:
+ $E57A476CD4DFBD99B4EE52A100A58610AD6E80B9,hamburgerphone,PrivacyRepublic14
+
+ :param str path: circuit path to be parsed
+
+ :returns: list of **(fingerprint, nickname)** tuples, fingerprints do not have a proceeding '$'
+
+ :raises: :class:`stem.ProtocolError` if the path is malformed
+ """
+
+ if path:
+ try:
+ return [_parse_circ_entry(entry) for entry in path.split(',')]
+ except stem.ProtocolError as exc:
+ # include the path with the exception
+ raise stem.ProtocolError('%s: %s' % (exc, path))
+ else:
+ return []
+
+
+def _parse_circ_entry(entry):
+ """
+ Parses a single relay's 'LongName' or 'ServerID'. See the
+ :func:`~stem.control._parse_circ_path` function for more information.
+
+ :param str entry: relay information to be parsed
+
+ :returns: **(fingerprint, nickname)** tuple
+
+ :raises: :class:`stem.ProtocolError` if the entry is malformed
+ """
+
+ if '=' in entry:
+ # common case
+ fingerprint, nickname = entry.split('=')
+ elif '~' in entry:
+ # this is allowed for by the spec, but I've never seen it used
+ fingerprint, nickname = entry.split('~')
+ elif entry[0] == '$':
+ # old style, fingerprint only
+ fingerprint, nickname = entry, None
+ else:
+ # old style, nickname only
+ fingerprint, nickname = None, entry
+
+ if fingerprint is not None:
+ if not stem.util.tor_tools.is_valid_fingerprint(fingerprint, True):
+ raise stem.ProtocolError('Fingerprint in the circuit path is malformed (%s)' % fingerprint)
+
+ fingerprint = fingerprint[1:] # strip off the leading '$'
+
+ if nickname is not None and not stem.util.tor_tools.is_valid_nickname(nickname):
+ raise stem.ProtocolError('Nickname in the circuit path is malformed (%s)' % nickname)
+
+ return (fingerprint, nickname)
+
+
+@with_default()
+def _case_insensitive_lookup(entries, key, default = UNDEFINED):
+ """
+ Makes a case insensitive lookup within a list or dictionary, providing the
+ first matching entry that we come across.
+
+ :param list,dict entries: list or dictionary to be searched
+ :param str key: entry or key value to look up
+ :param object default: value to be returned if the key doesn't exist
+
+ :returns: case insensitive match or default if one was provided and key wasn't found
+
+ :raises: **ValueError** if no such value exists
+ """
+
+ if entries is not None:
+ if isinstance(entries, dict):
+ for k, v in list(entries.items()):
+ if k.lower() == key.lower():
+ return v
+ else:
+ for entry in entries:
+ if entry.lower() == key.lower():
+ return entry
+
+ raise ValueError("key '%s' doesn't exist in dict: %s" % (key, entries))
diff --git a/ext/stem/descriptor/__init__.py b/ext/stem/descriptor/__init__.py
new file mode 100644
index 00000000..e8b581ed
--- /dev/null
+++ b/ext/stem/descriptor/__init__.py
@@ -0,0 +1,847 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Package for parsing and processing descriptor data.
+
+**Module Overview:**
+
+::
+
+ parse_file - Parses the descriptors in a file.
+
+ Descriptor - Common parent for all descriptor file types.
+ |- get_path - location of the descriptor on disk if it came from a file
+ |- get_archive_path - location of the descriptor within the archive it came from
+ |- get_bytes - similar to str(), but provides our original bytes content
+ |- get_unrecognized_lines - unparsed descriptor content
+ +- __str__ - string that the descriptor was made from
+
+.. data:: DocumentHandler (enum)
+
+ Ways in which we can parse a
+ :class:`~stem.descriptor.networkstatus.NetworkStatusDocument`.
+
+ Both **ENTRIES** and **BARE_DOCUMENT** have a 'thin' document, which doesn't
+ have a populated **routers** attribute. This allows for lower memory usage
+ and upfront runtime. However, if read time and memory aren't a concern then
+ **DOCUMENT** can provide you with a fully populated document.
+
+ =================== ===========
+ DocumentHandler Description
+ =================== ===========
+ **ENTRIES** Iterates over the contained :class:`~stem.descriptor.router_status_entry.RouterStatusEntry`. Each has a reference to the bare document it came from (through its **document** attribute).
+ **DOCUMENT** :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` with the :class:`~stem.descriptor.router_status_entry.RouterStatusEntry` it contains (through its **routers** attribute).
+ **BARE_DOCUMENT** :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` **without** a reference to its contents (the :class:`~stem.descriptor.router_status_entry.RouterStatusEntry` are unread).
+ =================== ===========
+"""
+
+__all__ = [
+ 'export',
+ 'reader',
+ 'remote',
+ 'extrainfo_descriptor',
+ 'server_descriptor',
+ 'microdescriptor',
+ 'networkstatus',
+ 'router_status_entry',
+ 'tordnsel',
+ 'parse_file',
+ 'Descriptor',
+]
+
+import base64
+import codecs
+import copy
+import hashlib
+import os
+import re
+import tarfile
+
+import stem.prereq
+import stem.util.enum
+import stem.util.str_tools
+import stem.util.system
+
+from stem import str_type
+
+try:
+ # added in python 2.7
+ from collections import OrderedDict
+except ImportError:
+ from stem.util.ordereddict import OrderedDict
+
+KEYWORD_CHAR = 'a-zA-Z0-9-'
+WHITESPACE = ' \t'
+KEYWORD_LINE = re.compile('^([%s]+)(?:[%s]+(.*))?$' % (KEYWORD_CHAR, WHITESPACE))
+SPECIFIC_KEYWORD_LINE = '^(%%s)(?:[%s]+(.*))?$' % WHITESPACE
+PGP_BLOCK_START = re.compile('^-----BEGIN ([%s%s]+)-----$' % (KEYWORD_CHAR, WHITESPACE))
+PGP_BLOCK_END = '-----END %s-----'
+
+DocumentHandler = stem.util.enum.UppercaseEnum(
+ 'ENTRIES',
+ 'DOCUMENT',
+ 'BARE_DOCUMENT',
+)
+
+
+def parse_file(descriptor_file, descriptor_type = None, validate = False, document_handler = DocumentHandler.ENTRIES, **kwargs):
+ """
+ Simple function to read the descriptor contents from a file, providing an
+ iterator for its :class:`~stem.descriptor.__init__.Descriptor` contents.
+
+ If you don't provide a **descriptor_type** argument then this automatically
+ tries to determine the descriptor type based on the following...
+
+ * The @type annotation on the first line. These are generally only found in
+ the `CollecTor archives <https://collector.torproject.org/formats.html#relay-descriptors>`_.
+
+ * The filename if it matches something from tor's data directory. For
+ instance, tor's 'cached-descriptors' contains server descriptors.
+
+ This is a handy function for simple usage, but if you're reading multiple
+ descriptor files you might want to consider the
+ :class:`~stem.descriptor.reader.DescriptorReader`.
+
+ Descriptor types include the following, including further minor versions (ie.
+ if we support 1.1 then we also support everything from 1.0 and most things
+ from 1.2, but not 2.0)...
+
+ ========================================= =====
+ Descriptor Type Class
+ ========================================= =====
+ server-descriptor 1.0 :class:`~stem.descriptor.server_descriptor.RelayDescriptor`
+ extra-info 1.0 :class:`~stem.descriptor.extrainfo_descriptor.RelayExtraInfoDescriptor`
+ microdescriptor 1.0 :class:`~stem.descriptor.microdescriptor.Microdescriptor`
+ directory 1.0 **unsupported**
+ network-status-2 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV2` (with a :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV2`)
+ dir-key-certificate-3 1.0 :class:`~stem.descriptor.networkstatus.KeyCertificate`
+ network-status-consensus-3 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` (with a :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV3`)
+ network-status-vote-3 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` (with a :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV3`)
+ network-status-microdesc-consensus-3 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryMicroV3` (with a :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV3`)
+ bridge-network-status 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` (with a :class:`~stem.descriptor.networkstatus.BridgeNetworkStatusDocument`)
+ bridge-server-descriptor 1.0 :class:`~stem.descriptor.server_descriptor.BridgeDescriptor`
+ bridge-extra-info 1.1 or 1.2 :class:`~stem.descriptor.extrainfo_descriptor.BridgeExtraInfoDescriptor`
+ torperf 1.0 **unsupported**
+ bridge-pool-assignment 1.0 **unsupported**
+ tordnsel 1.0 :class:`~stem.descriptor.tordnsel.TorDNSEL`
+ hidden-service-descriptor 1.0 :class:`~stem.descriptor.hidden_service_descriptor.HiddenServiceDescriptor`
+ ========================================= =====
+
+ If you're using **python 3** then beware that the open() function defaults to
+ using text mode. **Binary mode** is strongly suggested because it's both
+ faster (by my testing by about 33x) and doesn't do universal newline
+ translation which can make us misparse the document.
+
+ ::
+
+ my_descriptor_file = open(descriptor_path, 'rb')
+
+ :param str,file,tarfile descriptor_file: path or opened file with the descriptor contents
+ :param str descriptor_type: `descriptor type <https://collector.torproject.org/formats.html>`_, this is guessed if not provided
+ :param bool validate: checks the validity of the descriptor's content if
+ **True**, skips these checks otherwise
+ :param stem.descriptor.__init__.DocumentHandler document_handler: method in
+ which to parse the :class:`~stem.descriptor.networkstatus.NetworkStatusDocument`
+ :param dict kwargs: additional arguments for the descriptor constructor
+
+ :returns: iterator for :class:`~stem.descriptor.__init__.Descriptor` instances in the file
+
+ :raises:
+ * **ValueError** if the contents is malformed and validate is True
+ * **TypeError** if we can't match the contents of the file to a descriptor type
+ * **IOError** if unable to read from the descriptor_file
+ """
+
+ # Delegate to a helper if this is a path or tarfile.
+
+ handler = None
+
+ if isinstance(descriptor_file, (bytes, str_type)):
+ if stem.util.system.is_tarfile(descriptor_file):
+ handler = _parse_file_for_tar_path
+ else:
+ handler = _parse_file_for_path
+ elif isinstance(descriptor_file, tarfile.TarFile):
+ handler = _parse_file_for_tarfile
+
+ if handler:
+ for desc in handler(descriptor_file, descriptor_type, validate, document_handler, **kwargs):
+ yield desc
+
+ return
+
+ # The tor descriptor specifications do not provide a reliable method for
+ # identifying a descriptor file's type and version so we need to guess
+ # based on its filename. Metrics descriptors, however, can be identified
+ # by an annotation on their first line...
+ # https://trac.torproject.org/5651
+
+ initial_position = descriptor_file.tell()
+ first_line = stem.util.str_tools._to_unicode(descriptor_file.readline().strip())
+ metrics_header_match = re.match('^@type (\S+) (\d+).(\d+)$', first_line)
+
+ if not metrics_header_match:
+ descriptor_file.seek(initial_position)
+
+ descriptor_path = getattr(descriptor_file, 'name', None)
+ filename = '<undefined>' if descriptor_path is None else os.path.basename(descriptor_file.name)
+ file_parser = None
+
+ if descriptor_type is not None:
+ descriptor_type_match = re.match('^(\S+) (\d+).(\d+)$', descriptor_type)
+
+ if descriptor_type_match:
+ desc_type, major_version, minor_version = descriptor_type_match.groups()
+ file_parser = lambda f: _parse_metrics_file(desc_type, int(major_version), int(minor_version), f, validate, document_handler, **kwargs)
+ else:
+ raise ValueError("The descriptor_type must be of the form '<type> <major_version>.<minor_version>'")
+ elif metrics_header_match:
+ # Metrics descriptor handling
+
+ desc_type, major_version, minor_version = metrics_header_match.groups()
+ file_parser = lambda f: _parse_metrics_file(desc_type, int(major_version), int(minor_version), f, validate, document_handler, **kwargs)
+ else:
+ # Cached descriptor handling. These contain multiple descriptors per file.
+
+ if filename == 'cached-descriptors' or filename == 'cached-descriptors.new':
+ file_parser = lambda f: stem.descriptor.server_descriptor._parse_file(f, validate = validate, **kwargs)
+ elif filename == 'cached-extrainfo' or filename == 'cached-extrainfo.new':
+ file_parser = lambda f: stem.descriptor.extrainfo_descriptor._parse_file(f, validate = validate, **kwargs)
+ elif filename == 'cached-microdescs' or filename == 'cached-microdescs.new':
+ file_parser = lambda f: stem.descriptor.microdescriptor._parse_file(f, validate = validate, **kwargs)
+ elif filename == 'cached-consensus':
+ file_parser = lambda f: stem.descriptor.networkstatus._parse_file(f, validate = validate, document_handler = document_handler, **kwargs)
+ elif filename == 'cached-microdesc-consensus':
+ file_parser = lambda f: stem.descriptor.networkstatus._parse_file(f, is_microdescriptor = True, validate = validate, document_handler = document_handler, **kwargs)
+
+ if file_parser:
+ for desc in file_parser(descriptor_file):
+ if descriptor_path is not None:
+ desc._set_path(os.path.abspath(descriptor_path))
+
+ yield desc
+
+ return
+
+ # Not recognized as a descriptor file.
+
+ raise TypeError("Unable to determine the descriptor's type. filename: '%s', first line: '%s'" % (filename, first_line))
+
+
+def _parse_file_for_path(descriptor_file, *args, **kwargs):
+ with open(descriptor_file, 'rb') as desc_file:
+ for desc in parse_file(desc_file, *args, **kwargs):
+ yield desc
+
+
+def _parse_file_for_tar_path(descriptor_file, *args, **kwargs):
+ # TODO: use 'with' for tarfile after dropping python 2.6 support
+ tar_file = tarfile.open(descriptor_file)
+
+ try:
+ for desc in parse_file(tar_file, *args, **kwargs):
+ desc._set_path(os.path.abspath(descriptor_file))
+ yield desc
+ finally:
+ if tar_file:
+ tar_file.close()
+
+
+def _parse_file_for_tarfile(descriptor_file, *args, **kwargs):
+ for tar_entry in descriptor_file:
+ if tar_entry.isfile():
+ entry = descriptor_file.extractfile(tar_entry)
+
+ try:
+ for desc in parse_file(entry, *args, **kwargs):
+ desc._set_archive_path(entry.name)
+ yield desc
+ finally:
+ entry.close()
+
+
+def _parse_metrics_file(descriptor_type, major_version, minor_version, descriptor_file, validate, document_handler, **kwargs):
+ # Parses descriptor files from metrics, yielding individual descriptors. This
+ # throws a TypeError if the descriptor_type or version isn't recognized.
+
+ if descriptor_type == 'server-descriptor' and major_version == 1:
+ for desc in stem.descriptor.server_descriptor._parse_file(descriptor_file, is_bridge = False, validate = validate, **kwargs):
+ yield desc
+ elif descriptor_type == 'bridge-server-descriptor' and major_version == 1:
+ for desc in stem.descriptor.server_descriptor._parse_file(descriptor_file, is_bridge = True, validate = validate, **kwargs):
+ yield desc
+ elif descriptor_type == 'extra-info' and major_version == 1:
+ for desc in stem.descriptor.extrainfo_descriptor._parse_file(descriptor_file, is_bridge = False, validate = validate, **kwargs):
+ yield desc
+ elif descriptor_type == 'microdescriptor' and major_version == 1:
+ for desc in stem.descriptor.microdescriptor._parse_file(descriptor_file, validate = validate, **kwargs):
+ yield desc
+ elif descriptor_type == 'bridge-extra-info' and major_version == 1:
+ # version 1.1 introduced a 'transport' field...
+ # https://trac.torproject.org/6257
+
+ for desc in stem.descriptor.extrainfo_descriptor._parse_file(descriptor_file, is_bridge = True, validate = validate, **kwargs):
+ yield desc
+ elif descriptor_type == 'network-status-2' and major_version == 1:
+ document_type = stem.descriptor.networkstatus.NetworkStatusDocumentV2
+
+ for desc in stem.descriptor.networkstatus._parse_file(descriptor_file, document_type, validate = validate, document_handler = document_handler, **kwargs):
+ yield desc
+ elif descriptor_type == 'dir-key-certificate-3' and major_version == 1:
+ for desc in stem.descriptor.networkstatus._parse_file_key_certs(descriptor_file, validate = validate, **kwargs):
+ yield desc
+ elif descriptor_type in ('network-status-consensus-3', 'network-status-vote-3') and major_version == 1:
+ document_type = stem.descriptor.networkstatus.NetworkStatusDocumentV3
+
+ for desc in stem.descriptor.networkstatus._parse_file(descriptor_file, document_type, validate = validate, document_handler = document_handler, **kwargs):
+ yield desc
+ elif descriptor_type == 'network-status-microdesc-consensus-3' and major_version == 1:
+ document_type = stem.descriptor.networkstatus.NetworkStatusDocumentV3
+
+ for desc in stem.descriptor.networkstatus._parse_file(descriptor_file, document_type, is_microdescriptor = True, validate = validate, document_handler = document_handler, **kwargs):
+ yield desc
+ elif descriptor_type == 'bridge-network-status' and major_version == 1:
+ document_type = stem.descriptor.networkstatus.BridgeNetworkStatusDocument
+
+ for desc in stem.descriptor.networkstatus._parse_file(descriptor_file, document_type, validate = validate, document_handler = document_handler, **kwargs):
+ yield desc
+ elif descriptor_type == 'tordnsel' and major_version == 1:
+ document_type = stem.descriptor.tordnsel.TorDNSEL
+
+ for desc in stem.descriptor.tordnsel._parse_file(descriptor_file, validate = validate, **kwargs):
+ yield desc
+ elif descriptor_type == 'hidden-service-descriptor' and major_version == 1:
+ document_type = stem.descriptor.hidden_service_descriptor.HiddenServiceDescriptor
+
+ for desc in stem.descriptor.hidden_service_descriptor._parse_file(descriptor_file, validate = validate, **kwargs):
+ yield desc
+ else:
+ raise TypeError("Unrecognized metrics descriptor format. type: '%s', version: '%i.%i'" % (descriptor_type, major_version, minor_version))
+
+
+def _value(line, entries):
+ return entries[line][0][0]
+
+
+def _values(line, entries):
+ return [entry[0] for entry in entries[line]]
+
+
+def _parse_simple_line(keyword, attribute):
+ def _parse(descriptor, entries):
+ setattr(descriptor, attribute, _value(keyword, entries))
+
+ return _parse
+
+
+def _parse_bytes_line(keyword, attribute):
+ def _parse(descriptor, entries):
+ line_match = re.search(stem.util.str_tools._to_bytes('^(opt )?%s(?:[%s]+(.*))?$' % (keyword, WHITESPACE)), descriptor.get_bytes(), re.MULTILINE)
+ result = None
+
+ if line_match:
+ value = line_match.groups()[1]
+ result = b'' if value is None else value
+
+ setattr(descriptor, attribute, result)
+
+ return _parse
+
+
+def _parse_timestamp_line(keyword, attribute):
+ # "<keyword>" YYYY-MM-DD HH:MM:SS
+
+ def _parse(descriptor, entries):
+ value = _value(keyword, entries)
+
+ try:
+ setattr(descriptor, attribute, stem.util.str_tools._parse_timestamp(value))
+ except ValueError:
+ raise ValueError("Timestamp on %s line wasn't parsable: %s %s" % (keyword, keyword, value))
+
+ return _parse
+
+
+def _parse_forty_character_hex(keyword, attribute):
+ # format of fingerprints, sha1 digests, etc
+
+ def _parse(descriptor, entries):
+ value = _value(keyword, entries)
+
+ if not stem.util.tor_tools.is_hex_digits(value, 40):
+ raise ValueError('%s line had an invalid value (should be 40 hex characters): %s %s' % (keyword, keyword, value))
+
+ setattr(descriptor, attribute, value)
+
+ return _parse
+
+
+def _parse_key_block(keyword, attribute, expected_block_type, value_attribute = None):
+ def _parse(descriptor, entries):
+ value, block_type, block_contents = entries[keyword][0]
+
+ if not block_contents or block_type != expected_block_type:
+ raise ValueError("'%s' should be followed by a %s block, but was a %s" % (keyword, expected_block_type, block_type))
+
+ setattr(descriptor, attribute, block_contents)
+
+ if value_attribute:
+ setattr(descriptor, value_attribute, value)
+
+ return _parse
+
+
+class Descriptor(object):
+ """
+ Common parent for all types of descriptors.
+ """
+
+ ATTRIBUTES = {} # mapping of 'attribute' => (default_value, parsing_function)
+ PARSER_FOR_LINE = {} # line keyword to its associated parsing function
+
+ def __init__(self, contents, lazy_load = False):
+ self._path = None
+ self._archive_path = None
+ self._raw_contents = contents
+ self._lazy_loading = lazy_load
+ self._entries = {}
+ self._unrecognized_lines = []
+
+ def get_path(self):
+ """
+ Provides the absolute path that we loaded this descriptor from.
+
+ :returns: **str** with the absolute path of the descriptor source
+ """
+
+ return self._path
+
+ def get_archive_path(self):
+ """
+ If this descriptor came from an archive then provides its path within the
+ archive. This is only set if the descriptor came from a
+ :class:`~stem.descriptor.reader.DescriptorReader`, and is **None** if this
+ descriptor didn't come from an archive.
+
+ :returns: **str** with the descriptor's path within the archive
+ """
+
+ return self._archive_path
+
+ def get_bytes(self):
+ """
+ Provides the ASCII **bytes** of the descriptor. This only differs from
+ **str()** if you're running python 3.x, in which case **str()** provides a
+ **unicode** string.
+
+ :returns: **bytes** for the descriptor's contents
+ """
+
+ return self._raw_contents
+
+ def get_unrecognized_lines(self):
+ """
+ Provides a list of lines that were either ignored or had data that we did
+ not know how to process. This is most common due to new descriptor fields
+ that this library does not yet know how to process. Patches welcome!
+
+ :returns: **list** of lines of unrecognized content
+ """
+
+ if self._lazy_loading:
+ # we need to go ahead and parse the whole document to figure this out
+ self._parse(self._entries, False)
+ self._lazy_loading = False
+
+ return list(self._unrecognized_lines)
+
+ def _parse(self, entries, validate, parser_for_line = None):
+ """
+ Parses a series of 'keyword => (value, pgp block)' mappings and applies
+ them as attributes.
+
+ :param dict entries: descriptor contents to be applied
+ :param bool validate: checks the validity of descriptor content if True
+ :param dict parsers: mapping of lines to the function for parsing it
+
+ :raises: **ValueError** if an error occurs in validation
+ """
+
+ if parser_for_line is None:
+ parser_for_line = self.PARSER_FOR_LINE
+
+ # set defaults
+
+ for attr in self.ATTRIBUTES:
+ if not hasattr(self, attr):
+ setattr(self, attr, copy.copy(self.ATTRIBUTES[attr][0]))
+
+ for keyword, values in list(entries.items()):
+ try:
+ if keyword in parser_for_line:
+ parser_for_line[keyword](self, entries)
+ else:
+ for value, block_type, block_contents in values:
+ line = '%s %s' % (keyword, value)
+
+ if block_contents:
+ line += '\n%s' % block_contents
+
+ self._unrecognized_lines.append(line)
+ except ValueError as exc:
+ if validate:
+ raise exc
+
+ def _set_path(self, path):
+ self._path = path
+
+ def _set_archive_path(self, path):
+ self._archive_path = path
+
+ def _name(self, is_plural = False):
+ return str(type(self))
+
+ def _digest_for_signature(self, signing_key, signature):
+ """
+ Provides the signed digest we should have given this key and signature.
+
+ :param str signing_key: key block used to make this signature
+ :param str signature: signed digest for this descriptor content
+
+ :returns: the digest string encoded in uppercase hex
+
+ :raises: ValueError if unable to provide a validly signed digest
+ """
+
+ if not stem.prereq.is_crypto_available():
+ raise ValueError('Generating the signed digest requires pycrypto')
+
+ from Crypto.Util import asn1
+ from Crypto.Util.number import bytes_to_long, long_to_bytes
+
+ # get the ASN.1 sequence
+
+ seq = asn1.DerSequence()
+ seq.decode(_bytes_for_block(signing_key))
+ modulus, public_exponent = seq[0], seq[1]
+
+ sig_as_bytes = _bytes_for_block(signature)
+ sig_as_long = bytes_to_long(sig_as_bytes) # convert signature to an int
+ blocksize = 128 # block size will always be 128 for a 1024 bit key
+
+ # use the public exponent[e] & the modulus[n] to decrypt the int
+
+ decrypted_int = pow(sig_as_long, public_exponent, modulus)
+
+ # convert the int to a byte array
+
+ decrypted_bytes = long_to_bytes(decrypted_int, blocksize)
+
+ ############################################################################
+ # The decrypted bytes should have a structure exactly along these lines.
+ # 1 byte - [null '\x00']
+ # 1 byte - [block type identifier '\x01'] - Should always be 1
+ # N bytes - [padding '\xFF' ]
+ # 1 byte - [separator '\x00' ]
+ # M bytes - [message]
+ # Total - 128 bytes
+ # More info here http://www.ietf.org/rfc/rfc2313.txt
+ # esp the Notes in section 8.1
+ ############################################################################
+
+ try:
+ if decrypted_bytes.index(b'\x00\x01') != 0:
+ raise ValueError('Verification failed, identifier missing')
+ except ValueError:
+ raise ValueError('Verification failed, malformed data')
+
+ try:
+ identifier_offset = 2
+
+ # find the separator
+ seperator_index = decrypted_bytes.index(b'\x00', identifier_offset)
+ except ValueError:
+ raise ValueError('Verification failed, seperator not found')
+
+ digest_hex = codecs.encode(decrypted_bytes[seperator_index + 1:], 'hex_codec')
+ return stem.util.str_tools._to_unicode(digest_hex.upper())
+
+ def _digest_for_content(self, start, end):
+ """
+ Provides the digest of our descriptor's content in a given range.
+
+ :param bytes start: start of the range to generate a digest for
+ :param bytes end: end of the range to generate a digest for
+
+ :returns: the digest string encoded in uppercase hex
+
+ :raises: ValueError if the digest canot be calculated
+ """
+
+ raw_descriptor = self.get_bytes()
+
+ start_index = raw_descriptor.find(start)
+ end_index = raw_descriptor.find(end, start_index)
+
+ if start_index == -1:
+ raise ValueError("Digest is for the range starting with '%s' but that isn't in our descriptor" % start)
+ elif end_index == -1:
+ raise ValueError("Digest is for the range ending with '%s' but that isn't in our descriptor" % end)
+
+ digest_content = raw_descriptor[start_index:end_index + len(end)]
+ digest_hash = hashlib.sha1(stem.util.str_tools._to_bytes(digest_content))
+ return stem.util.str_tools._to_unicode(digest_hash.hexdigest().upper())
+
+ def __getattr__(self, name):
+ # Our constructor sets these, but when unpickling we might lack them. This
+ # check is needed to avoid an infinite loop in that case.
+
+ if name in ('_lazy_loading', 'ATTRIBUTES'):
+ return super(Descriptor, self).__getattribute__(name)
+
+ # If attribute isn't already present we might be lazy loading it...
+
+ if self._lazy_loading and name in self.ATTRIBUTES:
+ default, parsing_function = self.ATTRIBUTES[name]
+
+ try:
+ parsing_function(self, self._entries)
+ except (ValueError, KeyError):
+ try:
+ # despite having a validation failure check to see if we set something
+ return super(Descriptor, self).__getattribute__(name)
+ except AttributeError:
+ setattr(self, name, copy.copy(default))
+
+ return super(Descriptor, self).__getattribute__(name)
+
+ def __str__(self):
+ if stem.prereq.is_python_3():
+ return stem.util.str_tools._to_unicode(self._raw_contents)
+ else:
+ return self._raw_contents
+
+
+def _read_until_keywords(keywords, descriptor_file, inclusive = False, ignore_first = False, skip = False, end_position = None, include_ending_keyword = False):
+ """
+ Reads from the descriptor file until we get to one of the given keywords or reach the
+ end of the file.
+
+ :param str,list keywords: keyword(s) we want to read until
+ :param file descriptor_file: file with the descriptor content
+ :param bool inclusive: includes the line with the keyword if True
+ :param bool ignore_first: doesn't check if the first line read has one of the
+ given keywords
+ :param bool skip: skips buffering content, returning None
+ :param int end_position: end if we reach this point in the file
+ :param bool include_ending_keyword: provides the keyword we broke on if **True**
+
+ :returns: **list** with the lines until we find one of the keywords, this is
+ a two value tuple with the ending keyword if include_ending_keyword is
+ **True**
+ """
+
+ if skip:
+ content = None
+ content_append = lambda x: None
+ else:
+ content = []
+ content_append = content.append
+
+ ending_keyword = None
+
+ if isinstance(keywords, (bytes, str_type)):
+ keywords = (keywords,)
+
+ if ignore_first:
+ first_line = descriptor_file.readline()
+
+ if first_line:
+ content_append(first_line)
+
+ keyword_match = re.compile(SPECIFIC_KEYWORD_LINE % '|'.join(keywords))
+
+ while True:
+ last_position = descriptor_file.tell()
+
+ if end_position and last_position >= end_position:
+ break
+
+ line = descriptor_file.readline()
+
+ if not line:
+ break # EOF
+
+ line_match = keyword_match.match(stem.util.str_tools._to_unicode(line))
+
+ if line_match:
+ ending_keyword = line_match.groups()[0]
+
+ if not inclusive:
+ descriptor_file.seek(last_position)
+ else:
+ content_append(line)
+
+ break
+ else:
+ content_append(line)
+
+ if include_ending_keyword:
+ return (content, ending_keyword)
+ else:
+ return content
+
+
+def _bytes_for_block(content):
+ """
+ Provides the base64 decoded content of a pgp-style block.
+
+ :param str content: block to be decoded
+
+ :returns: decoded block content
+
+ :raises: **TypeError** if this isn't base64 encoded content
+ """
+
+ # strip the '-----BEGIN RSA PUBLIC KEY-----' header and footer
+
+ content = ''.join(content.split('\n')[1:-1])
+
+ return base64.b64decode(stem.util.str_tools._to_bytes(content))
+
+
+def _get_pseudo_pgp_block(remaining_contents):
+ """
+ Checks if given contents begins with a pseudo-Open-PGP-style block and, if
+ so, pops it off and provides it back to the caller.
+
+ :param list remaining_contents: lines to be checked for a public key block
+
+ :returns: **tuple** of the (block_type, content) or None if it doesn't exist
+
+ :raises: **ValueError** if the contents starts with a key block but it's
+ malformed (for instance, if it lacks an ending line)
+ """
+
+ if not remaining_contents:
+ return None # nothing left
+
+ block_match = PGP_BLOCK_START.match(remaining_contents[0])
+
+ if block_match:
+ block_type = block_match.groups()[0]
+ block_lines = []
+ end_line = PGP_BLOCK_END % block_type
+
+ while True:
+ if not remaining_contents:
+ raise ValueError("Unterminated pgp style block (looking for '%s'):\n%s" % (end_line, '\n'.join(block_lines)))
+
+ line = remaining_contents.pop(0)
+ block_lines.append(line)
+
+ if line == end_line:
+ return (block_type, '\n'.join(block_lines))
+ else:
+ return None
+
+
+def _get_descriptor_components(raw_contents, validate, extra_keywords = ()):
+ """
+ Initial breakup of the server descriptor contents to make parsing easier.
+
+ A descriptor contains a series of 'keyword lines' which are simply a keyword
+ followed by an optional value. Lines can also be followed by a signature
+ block.
+
+ To get a sub-listing with just certain keywords use extra_keywords. This can
+ be useful if we care about their relative ordering with respect to each
+ other. For instance, we care about the ordering of 'accept' and 'reject'
+ entries because this influences the resulting exit policy, but for everything
+ else in server descriptors the order does not matter.
+
+ :param str raw_contents: descriptor content provided by the relay
+ :param bool validate: checks the validity of the descriptor's content if
+ True, skips these checks otherwise
+ :param list extra_keywords: entity keywords to put into a separate listing
+ with ordering intact
+
+ :returns:
+ **collections.OrderedDict** with the 'keyword => (value, pgp key) entries'
+ mappings. If a extra_keywords was provided then this instead provides a two
+ value tuple, the second being a list of those entries.
+ """
+
+ if isinstance(raw_contents, bytes):
+ raw_contents = stem.util.str_tools._to_unicode(raw_contents)
+
+ entries = OrderedDict()
+ extra_entries = [] # entries with a keyword in extra_keywords
+ remaining_lines = raw_contents.split('\n')
+
+ while remaining_lines:
+ line = remaining_lines.pop(0)
+
+ # V2 network status documents explicitly can contain blank lines...
+ #
+ # "Implementations MAY insert blank lines for clarity between sections;
+ # these blank lines are ignored."
+ #
+ # ... and server descriptors end with an extra newline. But other documents
+ # don't say how blank lines should be handled so globally ignoring them.
+
+ if not line:
+ continue
+
+ # Some lines have an 'opt ' for backward compatibility. They should be
+ # ignored. This prefix is being removed in...
+ # https://trac.torproject.org/projects/tor/ticket/5124
+
+ if line.startswith('opt '):
+ line = line[4:]
+
+ line_match = KEYWORD_LINE.match(line)
+
+ if not line_match:
+ if not validate:
+ continue
+
+ raise ValueError('Line contains invalid characters: %s' % line)
+
+ keyword, value = line_match.groups()
+
+ if value is None:
+ value = ''
+
+ try:
+ block_attr = _get_pseudo_pgp_block(remaining_lines)
+
+ if block_attr:
+ block_type, block_contents = block_attr
+ else:
+ block_type, block_contents = None, None
+ except ValueError as exc:
+ if not validate:
+ continue
+
+ raise exc
+
+ if keyword in extra_keywords:
+ extra_entries.append('%s %s' % (keyword, value))
+ else:
+ entries.setdefault(keyword, []).append((value, block_type, block_contents))
+
+ if extra_keywords:
+ return entries, extra_entries
+ else:
+ return entries
+
+# importing at the end to avoid circular dependencies on our Descriptor class
+
+import stem.descriptor.server_descriptor
+import stem.descriptor.extrainfo_descriptor
+import stem.descriptor.networkstatus
+import stem.descriptor.microdescriptor
+import stem.descriptor.tordnsel
+import stem.descriptor.hidden_service_descriptor
diff --git a/ext/stem/descriptor/export.py b/ext/stem/descriptor/export.py
new file mode 100644
index 00000000..f90a607c
--- /dev/null
+++ b/ext/stem/descriptor/export.py
@@ -0,0 +1,110 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Toolkit for exporting descriptors to other formats.
+
+**Module Overview:**
+
+::
+
+ export_csv - Exports descriptors to a CSV
+ export_csv_file - Writes exported CSV output to a file
+"""
+
+import csv
+
+try:
+ from cStringIO import StringIO
+except ImportError:
+ from io import StringIO
+
+import stem.descriptor
+import stem.prereq
+
+
+class _ExportDialect(csv.excel):
+ lineterminator = '\n'
+
+
+def export_csv(descriptors, included_fields = (), excluded_fields = (), header = True):
+ """
+ Provides a newline separated CSV for one or more descriptors. If simply
+ provided with descriptors then the CSV contains all of its attributes,
+ labeled with a header row. Either 'included_fields' or 'excluded_fields' can
+ be used for more granular control over its attributes and the order.
+
+ :param Descriptor,list descriptors: either a
+ :class:`~stem.descriptor.Descriptor` or list of descriptors to be exported
+ :param list included_fields: attributes to include in the csv
+ :param list excluded_fields: attributes to exclude from the csv
+ :param bool header: if **True** then the first line will be a comma separated
+ list of the attribute names (**only supported in python 2.7 and higher**)
+
+ :returns: **str** of the CSV for the descriptors, one per line
+ :raises: **ValueError** if descriptors contain more than one descriptor type
+ """
+
+ output_buffer = StringIO()
+ export_csv_file(output_buffer, descriptors, included_fields, excluded_fields, header)
+ return output_buffer.getvalue()
+
+
+def export_csv_file(output_file, descriptors, included_fields = (), excluded_fields = (), header = True):
+ """
+ Similar to :func:`stem.descriptor.export.export_csv`, except that the CSV is
+ written directly to a file.
+
+ :param file output_file: file to be written to
+ :param Descriptor,list descriptors: either a
+ :class:`~stem.descriptor.Descriptor` or list of descriptors to be exported
+ :param list included_fields: attributes to include in the csv
+ :param list excluded_fields: attributes to exclude from the csv
+ :param bool header: if **True** then the first line will be a comma separated
+ list of the attribute names (**only supported in python 2.7 and higher**)
+
+ :returns: **str** of the CSV for the descriptors, one per line
+ :raises: **ValueError** if descriptors contain more than one descriptor type
+ """
+
+ if isinstance(descriptors, stem.descriptor.Descriptor):
+ descriptors = (descriptors,)
+
+ if not descriptors:
+ return
+
+ descriptor_type = type(descriptors[0])
+ descriptor_type_label = descriptor_type.__name__
+ included_fields = list(included_fields)
+
+ # If the user didn't specify the fields to include then export everything,
+ # ordered alphabetically. If they did specify fields then make sure that
+ # they exist.
+
+ desc_attr = sorted(vars(descriptors[0]).keys())
+
+ if included_fields:
+ for field in included_fields:
+ if field not in desc_attr:
+ raise ValueError("%s does not have a '%s' attribute, valid fields are: %s" % (descriptor_type_label, field, ', '.join(desc_attr)))
+ else:
+ included_fields = [attr for attr in desc_attr if not attr.startswith('_')]
+
+ for field in excluded_fields:
+ try:
+ included_fields.remove(field)
+ except ValueError:
+ pass
+
+ writer = csv.DictWriter(output_file, included_fields, dialect = _ExportDialect(), extrasaction='ignore')
+
+ if header and stem.prereq.is_python_27():
+ writer.writeheader()
+
+ for desc in descriptors:
+ if not isinstance(desc, stem.descriptor.Descriptor):
+ raise ValueError('Unable to export a descriptor CSV since %s is not a descriptor.' % type(desc).__name__)
+ elif descriptor_type != type(desc):
+ raise ValueError('To export a descriptor CSV all of the descriptors must be of the same type. First descriptor was a %s but we later got a %s.' % (descriptor_type_label, type(desc)))
+
+ writer.writerow(vars(desc))
diff --git a/ext/stem/descriptor/extrainfo_descriptor.py b/ext/stem/descriptor/extrainfo_descriptor.py
new file mode 100644
index 00000000..607bbbec
--- /dev/null
+++ b/ext/stem/descriptor/extrainfo_descriptor.py
@@ -0,0 +1,939 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Parsing for Tor extra-info descriptors. These are published by relays whenever
+their server descriptor is published and have a similar format. However, unlike
+server descriptors these don't contain information that Tor clients require to
+function and as such aren't fetched by default.
+
+Defined in section 2.2 of the `dir-spec
+<https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt>`_,
+extra-info descriptors contain interesting but non-vital information such as
+usage statistics. Tor clients cannot request these documents for bridges.
+
+Extra-info descriptors are available from a few sources...
+
+* If you have 'DownloadExtraInfo 1' in your torrc...
+
+ * control port via 'GETINFO extra-info/digest/\*' queries
+ * the 'cached-extrainfo' file in tor's data directory
+
+* Archived descriptors provided by CollecTor
+ (https://collector.torproject.org/).
+
+* Directory authorities and mirrors via their DirPort.
+
+**Module Overview:**
+
+::
+
+ ExtraInfoDescriptor - Tor extra-info descriptor.
+ |- RelayExtraInfoDescriptor - Extra-info descriptor for a relay.
+ |- BridgeExtraInfoDescriptor - Extra-info descriptor for a bridge.
+ |
+ +- digest - calculates the upper-case hex digest value for our content
+
+.. data:: DirResponse (enum)
+
+ Enumeration for known statuses for ExtraInfoDescriptor's dir_*_responses.
+
+ =================== ===========
+ DirResponse Description
+ =================== ===========
+ **OK** network status requests that were answered
+ **NOT_ENOUGH_SIGS** network status wasn't signed by enough authorities
+ **UNAVAILABLE** requested network status was unavailable
+ **NOT_FOUND** requested network status was not found
+ **NOT_MODIFIED** network status unmodified since If-Modified-Since time
+ **BUSY** directory was busy
+ =================== ===========
+
+.. data:: DirStat (enum)
+
+ Enumeration for known stats for ExtraInfoDescriptor's dir_*_direct_dl and
+ dir_*_tunneled_dl.
+
+ ===================== ===========
+ DirStat Description
+ ===================== ===========
+ **COMPLETE** requests that completed successfully
+ **TIMEOUT** requests that didn't complete within a ten minute timeout
+ **RUNNING** requests still in process when measurement's taken
+ **MIN** smallest rate at which a descriptor was downloaded in B/s
+ **MAX** largest rate at which a descriptor was downloaded in B/s
+ **D1-4** and **D6-9** rate of the slowest x/10 download rates in B/s
+ **Q1** and **Q3** rate of the slowest and fastest quarter download rates in B/s
+ **MD** median download rate in B/s
+ ===================== ===========
+"""
+
+import functools
+import hashlib
+import re
+
+import stem.util.connection
+import stem.util.enum
+import stem.util.str_tools
+
+from stem.descriptor import (
+ PGP_BLOCK_END,
+ Descriptor,
+ _read_until_keywords,
+ _get_descriptor_components,
+ _value,
+ _values,
+ _parse_timestamp_line,
+ _parse_forty_character_hex,
+ _parse_key_block,
+)
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+# known statuses for dirreq-v2-resp and dirreq-v3-resp...
+DirResponse = stem.util.enum.Enum(
+ ('OK', 'ok'),
+ ('NOT_ENOUGH_SIGS', 'not-enough-sigs'),
+ ('UNAVAILABLE', 'unavailable'),
+ ('NOT_FOUND', 'not-found'),
+ ('NOT_MODIFIED', 'not-modified'),
+ ('BUSY', 'busy'),
+)
+
+# known stats for dirreq-v2/3-direct-dl and dirreq-v2/3-tunneled-dl...
+dir_stats = ['complete', 'timeout', 'running', 'min', 'max', 'q1', 'q3', 'md']
+dir_stats += ['d%i' % i for i in range(1, 5)]
+dir_stats += ['d%i' % i for i in range(6, 10)]
+DirStat = stem.util.enum.Enum(*[(stat.upper(), stat) for stat in dir_stats])
+
+# relay descriptors must have exactly one of the following
+REQUIRED_FIELDS = (
+ 'extra-info',
+ 'published',
+ 'router-signature',
+)
+
+# optional entries that can appear at most once
+SINGLE_FIELDS = (
+ 'read-history',
+ 'write-history',
+ 'geoip-db-digest',
+ 'geoip6-db-digest',
+ 'bridge-stats-end',
+ 'bridge-ips',
+ 'dirreq-stats-end',
+ 'dirreq-v2-ips',
+ 'dirreq-v3-ips',
+ 'dirreq-v2-reqs',
+ 'dirreq-v3-reqs',
+ 'dirreq-v2-share',
+ 'dirreq-v3-share',
+ 'dirreq-v2-resp',
+ 'dirreq-v3-resp',
+ 'dirreq-v2-direct-dl',
+ 'dirreq-v3-direct-dl',
+ 'dirreq-v2-tunneled-dl',
+ 'dirreq-v3-tunneled-dl',
+ 'dirreq-read-history',
+ 'dirreq-write-history',
+ 'entry-stats-end',
+ 'entry-ips',
+ 'cell-stats-end',
+ 'cell-processed-cells',
+ 'cell-queued-cells',
+ 'cell-time-in-queue',
+ 'cell-circuits-per-decile',
+ 'conn-bi-direct',
+ 'exit-stats-end',
+ 'exit-kibibytes-written',
+ 'exit-kibibytes-read',
+ 'exit-streams-opened',
+)
+
+
+_timestamp_re = re.compile('^(.*) \(([0-9]+) s\)( .*)?$')
+_locale_re = re.compile('^[a-zA-Z0-9\?]{2}$')
+
+
+def _parse_file(descriptor_file, is_bridge = False, validate = False, **kwargs):
+ """
+ Iterates over the extra-info descriptors in a file.
+
+ :param file descriptor_file: file with descriptor content
+ :param bool is_bridge: parses the file as being a bridge descriptor
+ :param bool validate: checks the validity of the descriptor's content if
+ **True**, skips these checks otherwise
+ :param dict kwargs: additional arguments for the descriptor constructor
+
+ :returns: iterator for :class:`~stem.descriptor.extrainfo_descriptor.ExtraInfoDescriptor`
+ instances in the file
+
+ :raises:
+ * **ValueError** if the contents is malformed and validate is **True**
+ * **IOError** if the file can't be read
+ """
+
+ while True:
+ if not is_bridge:
+ extrainfo_content = _read_until_keywords('router-signature', descriptor_file)
+
+ # we've reached the 'router-signature', now include the pgp style block
+
+ block_end_prefix = PGP_BLOCK_END.split(' ', 1)[0]
+ extrainfo_content += _read_until_keywords(block_end_prefix, descriptor_file, True)
+ else:
+ extrainfo_content = _read_until_keywords('router-digest', descriptor_file, True)
+
+ if extrainfo_content:
+ if extrainfo_content[0].startswith(b'@type'):
+ extrainfo_content = extrainfo_content[1:]
+
+ if is_bridge:
+ yield BridgeExtraInfoDescriptor(bytes.join(b'', extrainfo_content), validate, **kwargs)
+ else:
+ yield RelayExtraInfoDescriptor(bytes.join(b'', extrainfo_content), validate, **kwargs)
+ else:
+ break # done parsing file
+
+
+def _parse_timestamp_and_interval(keyword, content):
+ """
+ Parses a 'YYYY-MM-DD HH:MM:SS (NSEC s) *' entry.
+
+ :param str keyword: line's keyword
+ :param str content: line content to be parsed
+
+ :returns: **tuple** of the form (timestamp (**datetime**), interval
+ (**int**), remaining content (**str**))
+
+ :raises: **ValueError** if the content is malformed
+ """
+
+ line = '%s %s' % (keyword, content)
+ content_match = _timestamp_re.match(content)
+
+ if not content_match:
+ raise ValueError('Malformed %s line: %s' % (keyword, line))
+
+ timestamp_str, interval, remainder = content_match.groups()
+
+ if remainder:
+ remainder = remainder[1:] # remove leading space
+
+ if not interval.isdigit():
+ raise ValueError("%s line's interval wasn't a number: %s" % (keyword, line))
+
+ try:
+ timestamp = stem.util.str_tools._parse_timestamp(timestamp_str)
+ return timestamp, int(interval), remainder
+ except ValueError:
+ raise ValueError("%s line's timestamp wasn't parsable: %s" % (keyword, line))
+
+
+def _parse_extra_info_line(descriptor, entries):
+ # "extra-info" Nickname Fingerprint
+
+ value = _value('extra-info', entries)
+ extra_info_comp = value.split()
+
+ if len(extra_info_comp) < 2:
+ raise ValueError('Extra-info line must have two values: extra-info %s' % value)
+ elif not stem.util.tor_tools.is_valid_nickname(extra_info_comp[0]):
+ raise ValueError("Extra-info line entry isn't a valid nickname: %s" % extra_info_comp[0])
+ elif not stem.util.tor_tools.is_valid_fingerprint(extra_info_comp[1]):
+ raise ValueError('Tor relay fingerprints consist of forty hex digits: %s' % extra_info_comp[1])
+
+ descriptor.nickname = extra_info_comp[0]
+ descriptor.fingerprint = extra_info_comp[1]
+
+
+def _parse_transport_line(descriptor, entries):
+ # "transport" transportname address:port [arglist]
+ # Everything after the transportname is scrubbed in published bridge
+ # descriptors, so we'll never see it in practice.
+ #
+ # These entries really only make sense for bridges, but have been seen
+ # on non-bridges in the wild when the relay operator configured it this
+ # way.
+
+ transports = {}
+
+ for value in _values('transport', entries):
+ name, address, port, args = None, None, None, None
+
+ if ' ' not in value:
+ # scrubbed
+ name = value
+ else:
+ # not scrubbed
+ value_comp = value.split()
+
+ if len(value_comp) < 1:
+ raise ValueError('Transport line is missing its transport name: transport %s' % value)
+ elif len(value_comp) < 2:
+ raise ValueError('Transport line is missing its address:port value: transport %s' % value)
+ elif ':' not in value_comp[1]:
+ raise ValueError("Transport line's address:port entry is missing a colon: transport %s" % value)
+
+ name = value_comp[0]
+ address, port_str = value_comp[1].split(':', 1)
+
+ if not stem.util.connection.is_valid_ipv4_address(address) or \
+ stem.util.connection.is_valid_ipv6_address(address):
+ raise ValueError('Transport line has a malformed address: transport %s' % value)
+ elif not stem.util.connection.is_valid_port(port_str):
+ raise ValueError('Transport line has a malformed port: transport %s' % value)
+
+ port = int(port_str)
+ args = value_comp[2:] if len(value_comp) >= 3 else []
+
+ transports[name] = (address, port, args)
+
+ descriptor.transport = transports
+
+
+def _parse_cell_circuits_per_decline_line(descriptor, entries):
+ # "cell-circuits-per-decile" num
+
+ value = _value('cell-circuits-per-decile', entries)
+
+ if not value.isdigit():
+ raise ValueError('Non-numeric cell-circuits-per-decile value: %s' % value)
+ elif int(value) < 0:
+ raise ValueError('Negative cell-circuits-per-decile value: %s' % value)
+
+ descriptor.cell_circuits_per_decile = int(value)
+
+
+def _parse_dirreq_line(keyword, recognized_counts_attr, unrecognized_counts_attr, descriptor, entries):
+ value = _value(keyword, entries)
+
+ recognized_counts = {}
+ unrecognized_counts = {}
+
+ is_response_stats = keyword in ('dirreq-v2-resp', 'dirreq-v3-resp')
+ key_set = DirResponse if is_response_stats else DirStat
+
+ key_type = 'STATUS' if is_response_stats else 'STAT'
+ error_msg = '%s lines should contain %s=COUNT mappings: %s %s' % (keyword, key_type, keyword, value)
+
+ if value:
+ for entry in value.split(','):
+ if '=' not in entry:
+ raise ValueError(error_msg)
+
+ status, count = entry.split('=', 1)
+
+ if count.isdigit():
+ if status in key_set:
+ recognized_counts[status] = int(count)
+ else:
+ unrecognized_counts[status] = int(count)
+ else:
+ raise ValueError(error_msg)
+
+ setattr(descriptor, recognized_counts_attr, recognized_counts)
+ setattr(descriptor, unrecognized_counts_attr, unrecognized_counts)
+
+
+def _parse_dirreq_share_line(keyword, attribute, descriptor, entries):
+ value = _value(keyword, entries)
+
+ if not value.endswith('%'):
+ raise ValueError('%s lines should be a percentage: %s %s' % (keyword, keyword, value))
+ elif float(value[:-1]) < 0:
+ raise ValueError('Negative percentage value: %s %s' % (keyword, value))
+
+ # bug means it might be above 100%: https://lists.torproject.org/pipermail/tor-dev/2012-June/003679.html
+
+ setattr(descriptor, attribute, float(value[:-1]) / 100)
+
+
+def _parse_cell_line(keyword, attribute, descriptor, entries):
+ # "<keyword>" num,...,num
+
+ value = _value(keyword, entries)
+ entries, exc = [], None
+
+ if value:
+ for entry in value.split(','):
+ try:
+ # Values should be positive but as discussed in ticket #5849
+ # there was a bug around this. It was fixed in tor 0.2.2.1.
+
+ entries.append(float(entry))
+ except ValueError:
+ exc = ValueError('Non-numeric entry in %s listing: %s %s' % (keyword, keyword, value))
+
+ setattr(descriptor, attribute, entries)
+
+ if exc:
+ raise exc
+
+
+def _parse_timestamp_and_interval_line(keyword, end_attribute, interval_attribute, descriptor, entries):
+ # "<keyword>" YYYY-MM-DD HH:MM:SS (NSEC s)
+
+ timestamp, interval, _ = _parse_timestamp_and_interval(keyword, _value(keyword, entries))
+ setattr(descriptor, end_attribute, timestamp)
+ setattr(descriptor, interval_attribute, interval)
+
+
+def _parse_conn_bi_direct_line(descriptor, entries):
+ # "conn-bi-direct" YYYY-MM-DD HH:MM:SS (NSEC s) BELOW,READ,WRITE,BOTH
+
+ value = _value('conn-bi-direct', entries)
+ timestamp, interval, remainder = _parse_timestamp_and_interval('conn-bi-direct', value)
+ stats = remainder.split(',')
+
+ if len(stats) != 4 or not (stats[0].isdigit() and stats[1].isdigit() and stats[2].isdigit() and stats[3].isdigit()):
+ raise ValueError('conn-bi-direct line should end with four numeric values: conn-bi-direct %s' % value)
+
+ descriptor.conn_bi_direct_end = timestamp
+ descriptor.conn_bi_direct_interval = interval
+ descriptor.conn_bi_direct_below = int(stats[0])
+ descriptor.conn_bi_direct_read = int(stats[1])
+ descriptor.conn_bi_direct_write = int(stats[2])
+ descriptor.conn_bi_direct_both = int(stats[3])
+
+
+def _parse_history_line(keyword, end_attribute, interval_attribute, values_attribute, descriptor, entries):
+ # "<keyword>" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM...
+
+ value = _value(keyword, entries)
+ timestamp, interval, remainder = _parse_timestamp_and_interval(keyword, value)
+ history_values = []
+
+ if remainder:
+ try:
+ history_values = [int(entry) for entry in remainder.split(',')]
+ except ValueError:
+ raise ValueError('%s line has non-numeric values: %s %s' % (keyword, keyword, value))
+
+ setattr(descriptor, end_attribute, timestamp)
+ setattr(descriptor, interval_attribute, interval)
+ setattr(descriptor, values_attribute, history_values)
+
+
+def _parse_port_count_line(keyword, attribute, descriptor, entries):
+ # "<keyword>" port=N,port=N,...
+
+ value, port_mappings = _value(keyword, entries), {}
+ error_msg = 'Entries in %s line should only be PORT=N entries: %s %s' % (keyword, keyword, value)
+
+ if value:
+ for entry in value.split(','):
+ if '=' not in entry:
+ raise ValueError(error_msg)
+
+ port, stat = entry.split('=', 1)
+
+ if (port == 'other' or stem.util.connection.is_valid_port(port)) and stat.isdigit():
+ if port != 'other':
+ port = int(port)
+
+ port_mappings[port] = int(stat)
+ else:
+ raise ValueError(error_msg)
+
+ setattr(descriptor, attribute, port_mappings)
+
+
+def _parse_geoip_to_count_line(keyword, attribute, descriptor, entries):
+ # "<keyword>" CC=N,CC=N,...
+ #
+ # The maxmind geoip (https://www.maxmind.com/app/iso3166) has numeric
+ # locale codes for some special values, for instance...
+ # A1,"Anonymous Proxy"
+ # A2,"Satellite Provider"
+ # ??,"Unknown"
+
+ value, locale_usage = _value(keyword, entries), {}
+ error_msg = 'Entries in %s line should only be CC=N entries: %s %s' % (keyword, keyword, value)
+
+ if value:
+ for entry in value.split(','):
+ if '=' not in entry:
+ raise ValueError(error_msg)
+
+ locale, count = entry.split('=', 1)
+
+ if _locale_re.match(locale) and count.isdigit():
+ locale_usage[locale] = int(count)
+ else:
+ raise ValueError(error_msg)
+
+ setattr(descriptor, attribute, locale_usage)
+
+
+def _parse_bridge_ip_versions_line(descriptor, entries):
+ value, ip_versions = _value('bridge-ip-versions', entries), {}
+
+ if value:
+ for entry in value.split(','):
+ if '=' not in entry:
+ raise stem.ProtocolError("The bridge-ip-versions should be a comma separated listing of '<protocol>=<count>' mappings: bridge-ip-versions %s" % value)
+
+ protocol, count = entry.split('=', 1)
+
+ if not count.isdigit():
+ raise stem.ProtocolError('IP protocol count was non-numeric (%s): bridge-ip-versions %s' % (count, value))
+
+ ip_versions[protocol] = int(count)
+
+ descriptor.ip_versions = ip_versions
+
+
+def _parse_bridge_ip_transports_line(descriptor, entries):
+ value, ip_transports = _value('bridge-ip-transports', entries), {}
+
+ if value:
+ for entry in value.split(','):
+ if '=' not in entry:
+ raise stem.ProtocolError("The bridge-ip-transports should be a comma separated listing of '<protocol>=<count>' mappings: bridge-ip-transports %s" % value)
+
+ protocol, count = entry.split('=', 1)
+
+ if not count.isdigit():
+ raise stem.ProtocolError('Transport count was non-numeric (%s): bridge-ip-transports %s' % (count, value))
+
+ ip_transports[protocol] = int(count)
+
+ descriptor.ip_transports = ip_transports
+
+
+def _parse_hs_stats(keyword, stat_attribute, extra_attribute, descriptor, entries):
+ # "<keyword>" num key=val key=val...
+
+ value, stat, extra = _value(keyword, entries), None, {}
+
+ if value is not None:
+ value_comp = value.split()
+
+ if not value_comp:
+ raise ValueError("'%s' line was blank" % keyword)
+
+ try:
+ stat = int(value_comp[0])
+ except ValueError:
+ raise ValueError("'%s' stat was non-numeric (%s): %s %s" % (keyword, value_comp[0], keyword, value))
+
+ for entry in value_comp[1:]:
+ if '=' not in entry:
+ raise ValueError('Entries after the stat in %s lines should only be key=val entries: %s %s' % (keyword, keyword, value))
+
+ key, val = entry.split('=', 1)
+ extra[key] = val
+
+ setattr(descriptor, stat_attribute, stat)
+ setattr(descriptor, extra_attribute, extra)
+
+
+_parse_geoip_db_digest_line = _parse_forty_character_hex('geoip-db-digest', 'geoip_db_digest')
+_parse_geoip6_db_digest_line = _parse_forty_character_hex('geoip6-db-digest', 'geoip6_db_digest')
+_parse_dirreq_v2_resp_line = functools.partial(_parse_dirreq_line, 'dirreq-v2-resp', 'dir_v2_responses', 'dir_v2_responses_unknown')
+_parse_dirreq_v3_resp_line = functools.partial(_parse_dirreq_line, 'dirreq-v3-resp', 'dir_v3_responses', 'dir_v3_responses_unknown')
+_parse_dirreq_v2_direct_dl_line = functools.partial(_parse_dirreq_line, 'dirreq-v2-direct-dl', 'dir_v2_direct_dl', 'dir_v2_direct_dl_unknown')
+_parse_dirreq_v3_direct_dl_line = functools.partial(_parse_dirreq_line, 'dirreq-v3-direct-dl', 'dir_v3_direct_dl', 'dir_v3_direct_dl_unknown')
+_parse_dirreq_v2_tunneled_dl_line = functools.partial(_parse_dirreq_line, 'dirreq-v2-tunneled-dl', 'dir_v2_tunneled_dl', 'dir_v2_tunneled_dl_unknown')
+_parse_dirreq_v3_tunneled_dl_line = functools.partial(_parse_dirreq_line, 'dirreq-v3-tunneled-dl', 'dir_v3_tunneled_dl', 'dir_v3_tunneled_dl_unknown')
+_parse_dirreq_v2_share_line = functools.partial(_parse_dirreq_share_line, 'dirreq-v2-share', 'dir_v2_share')
+_parse_dirreq_v3_share_line = functools.partial(_parse_dirreq_share_line, 'dirreq-v3-share', 'dir_v3_share')
+_parse_cell_processed_cells_line = functools.partial(_parse_cell_line, 'cell-processed-cells', 'cell_processed_cells')
+_parse_cell_queued_cells_line = functools.partial(_parse_cell_line, 'cell-queued-cells', 'cell_queued_cells')
+_parse_cell_time_in_queue_line = functools.partial(_parse_cell_line, 'cell-time-in-queue', 'cell_time_in_queue')
+_parse_published_line = _parse_timestamp_line('published', 'published')
+_parse_geoip_start_time_line = _parse_timestamp_line('geoip-start-time', 'geoip_start_time')
+_parse_cell_stats_end_line = functools.partial(_parse_timestamp_and_interval_line, 'cell-stats-end', 'cell_stats_end', 'cell_stats_interval')
+_parse_entry_stats_end_line = functools.partial(_parse_timestamp_and_interval_line, 'entry-stats-end', 'entry_stats_end', 'entry_stats_interval')
+_parse_exit_stats_end_line = functools.partial(_parse_timestamp_and_interval_line, 'exit-stats-end', 'exit_stats_end', 'exit_stats_interval')
+_parse_bridge_stats_end_line = functools.partial(_parse_timestamp_and_interval_line, 'bridge-stats-end', 'bridge_stats_end', 'bridge_stats_interval')
+_parse_dirreq_stats_end_line = functools.partial(_parse_timestamp_and_interval_line, 'dirreq-stats-end', 'dir_stats_end', 'dir_stats_interval')
+_parse_read_history_line = functools.partial(_parse_history_line, 'read-history', 'read_history_end', 'read_history_interval', 'read_history_values')
+_parse_write_history_line = functools.partial(_parse_history_line, 'write-history', 'write_history_end', 'write_history_interval', 'write_history_values')
+_parse_dirreq_read_history_line = functools.partial(_parse_history_line, 'dirreq-read-history', 'dir_read_history_end', 'dir_read_history_interval', 'dir_read_history_values')
+_parse_dirreq_write_history_line = functools.partial(_parse_history_line, 'dirreq-write-history', 'dir_write_history_end', 'dir_write_history_interval', 'dir_write_history_values')
+_parse_exit_kibibytes_written_line = functools.partial(_parse_port_count_line, 'exit-kibibytes-written', 'exit_kibibytes_written')
+_parse_exit_kibibytes_read_line = functools.partial(_parse_port_count_line, 'exit-kibibytes-read', 'exit_kibibytes_read')
+_parse_exit_streams_opened_line = functools.partial(_parse_port_count_line, 'exit-streams-opened', 'exit_streams_opened')
+_parse_hidden_service_stats_end_line = _parse_timestamp_line('hidserv-stats-end', 'hs_stats_end')
+_parse_hidden_service_rend_relayed_cells_line = functools.partial(_parse_hs_stats, 'hidserv-rend-relayed-cells', 'hs_rend_cells', 'hs_rend_cells_attr')
+_parse_hidden_service_dir_onions_seen_line = functools.partial(_parse_hs_stats, 'hidserv-dir-onions-seen', 'hs_dir_onions_seen', 'hs_dir_onions_seen_attr')
+_parse_dirreq_v2_ips_line = functools.partial(_parse_geoip_to_count_line, 'dirreq-v2-ips', 'dir_v2_ips')
+_parse_dirreq_v3_ips_line = functools.partial(_parse_geoip_to_count_line, 'dirreq-v3-ips', 'dir_v3_ips')
+_parse_dirreq_v2_reqs_line = functools.partial(_parse_geoip_to_count_line, 'dirreq-v2-reqs', 'dir_v2_requests')
+_parse_dirreq_v3_reqs_line = functools.partial(_parse_geoip_to_count_line, 'dirreq-v3-reqs', 'dir_v3_requests')
+_parse_geoip_client_origins_line = functools.partial(_parse_geoip_to_count_line, 'geoip-client-origins', 'geoip_client_origins')
+_parse_entry_ips_line = functools.partial(_parse_geoip_to_count_line, 'entry-ips', 'entry_ips')
+_parse_bridge_ips_line = functools.partial(_parse_geoip_to_count_line, 'bridge-ips', 'bridge_ips')
+_parse_router_digest_line = _parse_forty_character_hex('router-digest', '_digest')
+_parse_router_signature_line = _parse_key_block('router-signature', 'signature', 'SIGNATURE')
+
+
+class ExtraInfoDescriptor(Descriptor):
+ """
+ Extra-info descriptor document.
+
+ :var str nickname: **\*** relay's nickname
+ :var str fingerprint: **\*** identity key fingerprint
+ :var datetime published: **\*** time in UTC when this descriptor was made
+ :var str geoip_db_digest: sha1 of the geoIP database file for IPv4 addresses
+ :var str geoip6_db_digest: sha1 of the geoIP database file for IPv6 addresses
+ :var dict transport: **\*** mapping of transport methods to their (address,
+ port, args) tuple, these usually appear on bridges in which case all of
+ those are **None**
+
+ **Bi-directional connection usage:**
+
+ :var datetime conn_bi_direct_end: end of the sampling interval
+ :var int conn_bi_direct_interval: seconds per interval
+ :var int conn_bi_direct_below: connections that read/wrote less than 20 KiB
+ :var int conn_bi_direct_read: connections that read at least 10x more than wrote
+ :var int conn_bi_direct_write: connections that wrote at least 10x more than read
+ :var int conn_bi_direct_both: remaining connections
+
+ **Bytes read/written for relayed traffic:**
+
+ :var datetime read_history_end: end of the sampling interval
+ :var int read_history_interval: seconds per interval
+ :var list read_history_values: bytes read during each interval
+
+ :var datetime write_history_end: end of the sampling interval
+ :var int write_history_interval: seconds per interval
+ :var list write_history_values: bytes written during each interval
+
+ **Cell relaying statistics:**
+
+ :var datetime cell_stats_end: end of the period when stats were gathered
+ :var int cell_stats_interval: length in seconds of the interval
+ :var list cell_processed_cells: measurement of processed cells per circuit
+ :var list cell_queued_cells: measurement of queued cells per circuit
+ :var list cell_time_in_queue: mean enqueued time in milliseconds for cells
+ :var int cell_circuits_per_decile: mean number of circuits in a decile
+
+ **Directory Mirror Attributes:**
+
+ :var datetime dir_stats_end: end of the period when stats were gathered
+ :var int dir_stats_interval: length in seconds of the interval
+ :var dict dir_v2_ips: mapping of locales to rounded count of requester ips
+ :var dict dir_v3_ips: mapping of locales to rounded count of requester ips
+ :var float dir_v2_share: percent of total directory traffic it expects to serve
+ :var float dir_v3_share: percent of total directory traffic it expects to serve
+ :var dict dir_v2_requests: mapping of locales to rounded count of requests
+ :var dict dir_v3_requests: mapping of locales to rounded count of requests
+
+ :var dict dir_v2_responses: mapping of :data:`~stem.descriptor.extrainfo_descriptor.DirResponse` to their rounded count
+ :var dict dir_v3_responses: mapping of :data:`~stem.descriptor.extrainfo_descriptor.DirResponse` to their rounded count
+ :var dict dir_v2_responses_unknown: mapping of unrecognized statuses to their count
+ :var dict dir_v3_responses_unknown: mapping of unrecognized statuses to their count
+
+ :var dict dir_v2_direct_dl: mapping of :data:`~stem.descriptor.extrainfo_descriptor.DirStat` to measurement over DirPort
+ :var dict dir_v3_direct_dl: mapping of :data:`~stem.descriptor.extrainfo_descriptor.DirStat` to measurement over DirPort
+ :var dict dir_v2_direct_dl_unknown: mapping of unrecognized stats to their measurement
+ :var dict dir_v3_direct_dl_unknown: mapping of unrecognized stats to their measurement
+
+ :var dict dir_v2_tunneled_dl: mapping of :data:`~stem.descriptor.extrainfo_descriptor.DirStat` to measurement over ORPort
+ :var dict dir_v3_tunneled_dl: mapping of :data:`~stem.descriptor.extrainfo_descriptor.DirStat` to measurement over ORPort
+ :var dict dir_v2_tunneled_dl_unknown: mapping of unrecognized stats to their measurement
+ :var dict dir_v3_tunneled_dl_unknown: mapping of unrecognized stats to their measurement
+
+ **Bytes read/written for directory mirroring:**
+
+ :var datetime dir_read_history_end: end of the sampling interval
+ :var int dir_read_history_interval: seconds per interval
+ :var list dir_read_history_values: bytes read during each interval
+
+ :var datetime dir_write_history_end: end of the sampling interval
+ :var int dir_write_history_interval: seconds per interval
+ :var list dir_write_history_values: bytes read during each interval
+
+ **Guard Attributes:**
+
+ :var datetime entry_stats_end: end of the period when stats were gathered
+ :var int entry_stats_interval: length in seconds of the interval
+ :var dict entry_ips: mapping of locales to rounded count of unique user ips
+
+ **Exit Attributes:**
+
+ :var datetime exit_stats_end: end of the period when stats were gathered
+ :var int exit_stats_interval: length in seconds of the interval
+ :var dict exit_kibibytes_written: traffic per port (keys are ints or 'other')
+ :var dict exit_kibibytes_read: traffic per port (keys are ints or 'other')
+ :var dict exit_streams_opened: streams per port (keys are ints or 'other')
+
+ **Hidden Service Attributes:**
+
+ :var datetime hs_stats_end: end of the sampling interval
+ :var int hs_rend_cells: rounded count of the RENDEZVOUS1 cells seen
+ :var int hs_rend_cells_attr: **\*** attributes provided for the hs_rend_cells
+ :var int hs_dir_onions_seen: rounded count of the identities seen
+ :var int hs_dir_onions_seen_attr: **\*** attributes provided for the hs_dir_onions_seen
+
+ **Bridge Attributes:**
+
+ :var datetime bridge_stats_end: end of the period when stats were gathered
+ :var int bridge_stats_interval: length in seconds of the interval
+ :var dict bridge_ips: mapping of locales to rounded count of unique user ips
+ :var datetime geoip_start_time: replaced by bridge_stats_end (deprecated)
+ :var dict geoip_client_origins: replaced by bridge_ips (deprecated)
+ :var dict ip_versions: mapping of ip protocols to a rounded count for the number of users
+ :var dict ip_versions: mapping of ip transports to a count for the number of users
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as **None** if undefined
+
+ .. versionchanged:: 1.4.0
+ Added the hs_stats_end, hs_rend_cells, hs_rend_cells_attr,
+ hs_dir_onions_seen, and hs_dir_onions_seen_attr attributes.
+ """
+
+ ATTRIBUTES = {
+ 'nickname': (None, _parse_extra_info_line),
+ 'fingerprint': (None, _parse_extra_info_line),
+ 'published': (None, _parse_published_line),
+ 'geoip_db_digest': (None, _parse_geoip_db_digest_line),
+ 'geoip6_db_digest': (None, _parse_geoip6_db_digest_line),
+ 'transport': ({}, _parse_transport_line),
+
+ 'conn_bi_direct_end': (None, _parse_conn_bi_direct_line),
+ 'conn_bi_direct_interval': (None, _parse_conn_bi_direct_line),
+ 'conn_bi_direct_below': (None, _parse_conn_bi_direct_line),
+ 'conn_bi_direct_read': (None, _parse_conn_bi_direct_line),
+ 'conn_bi_direct_write': (None, _parse_conn_bi_direct_line),
+ 'conn_bi_direct_both': (None, _parse_conn_bi_direct_line),
+
+ 'read_history_end': (None, _parse_read_history_line),
+ 'read_history_interval': (None, _parse_read_history_line),
+ 'read_history_values': (None, _parse_read_history_line),
+
+ 'write_history_end': (None, _parse_write_history_line),
+ 'write_history_interval': (None, _parse_write_history_line),
+ 'write_history_values': (None, _parse_write_history_line),
+
+ 'cell_stats_end': (None, _parse_cell_stats_end_line),
+ 'cell_stats_interval': (None, _parse_cell_stats_end_line),
+ 'cell_processed_cells': (None, _parse_cell_processed_cells_line),
+ 'cell_queued_cells': (None, _parse_cell_queued_cells_line),
+ 'cell_time_in_queue': (None, _parse_cell_time_in_queue_line),
+ 'cell_circuits_per_decile': (None, _parse_cell_circuits_per_decline_line),
+
+ 'dir_stats_end': (None, _parse_dirreq_stats_end_line),
+ 'dir_stats_interval': (None, _parse_dirreq_stats_end_line),
+ 'dir_v2_ips': (None, _parse_dirreq_v2_ips_line),
+ 'dir_v3_ips': (None, _parse_dirreq_v3_ips_line),
+ 'dir_v2_share': (None, _parse_dirreq_v2_share_line),
+ 'dir_v3_share': (None, _parse_dirreq_v3_share_line),
+ 'dir_v2_requests': (None, _parse_dirreq_v2_reqs_line),
+ 'dir_v3_requests': (None, _parse_dirreq_v3_reqs_line),
+ 'dir_v2_responses': (None, _parse_dirreq_v2_resp_line),
+ 'dir_v3_responses': (None, _parse_dirreq_v3_resp_line),
+ 'dir_v2_responses_unknown': (None, _parse_dirreq_v2_resp_line),
+ 'dir_v3_responses_unknown': (None, _parse_dirreq_v3_resp_line),
+ 'dir_v2_direct_dl': (None, _parse_dirreq_v2_direct_dl_line),
+ 'dir_v3_direct_dl': (None, _parse_dirreq_v3_direct_dl_line),
+ 'dir_v2_direct_dl_unknown': (None, _parse_dirreq_v2_direct_dl_line),
+ 'dir_v3_direct_dl_unknown': (None, _parse_dirreq_v3_direct_dl_line),
+ 'dir_v2_tunneled_dl': (None, _parse_dirreq_v2_tunneled_dl_line),
+ 'dir_v3_tunneled_dl': (None, _parse_dirreq_v3_tunneled_dl_line),
+ 'dir_v2_tunneled_dl_unknown': (None, _parse_dirreq_v2_tunneled_dl_line),
+ 'dir_v3_tunneled_dl_unknown': (None, _parse_dirreq_v3_tunneled_dl_line),
+
+ 'dir_read_history_end': (None, _parse_dirreq_read_history_line),
+ 'dir_read_history_interval': (None, _parse_dirreq_read_history_line),
+ 'dir_read_history_values': (None, _parse_dirreq_read_history_line),
+
+ 'dir_write_history_end': (None, _parse_dirreq_write_history_line),
+ 'dir_write_history_interval': (None, _parse_dirreq_write_history_line),
+ 'dir_write_history_values': (None, _parse_dirreq_write_history_line),
+
+ 'entry_stats_end': (None, _parse_entry_stats_end_line),
+ 'entry_stats_interval': (None, _parse_entry_stats_end_line),
+ 'entry_ips': (None, _parse_entry_ips_line),
+
+ 'exit_stats_end': (None, _parse_exit_stats_end_line),
+ 'exit_stats_interval': (None, _parse_exit_stats_end_line),
+ 'exit_kibibytes_written': (None, _parse_exit_kibibytes_written_line),
+ 'exit_kibibytes_read': (None, _parse_exit_kibibytes_read_line),
+ 'exit_streams_opened': (None, _parse_exit_streams_opened_line),
+
+ 'hs_stats_end': (None, _parse_hidden_service_stats_end_line),
+ 'hs_rend_cells': (None, _parse_hidden_service_rend_relayed_cells_line),
+ 'hs_rend_cells_attr': ({}, _parse_hidden_service_rend_relayed_cells_line),
+ 'hs_dir_onions_seen': (None, _parse_hidden_service_dir_onions_seen_line),
+ 'hs_dir_onions_seen_attr': ({}, _parse_hidden_service_dir_onions_seen_line),
+
+ 'bridge_stats_end': (None, _parse_bridge_stats_end_line),
+ 'bridge_stats_interval': (None, _parse_bridge_stats_end_line),
+ 'bridge_ips': (None, _parse_bridge_ips_line),
+ 'geoip_start_time': (None, _parse_geoip_start_time_line),
+ 'geoip_client_origins': (None, _parse_geoip_client_origins_line),
+
+ 'ip_versions': (None, _parse_bridge_ip_versions_line),
+ 'ip_transports': (None, _parse_bridge_ip_transports_line),
+ }
+
+ PARSER_FOR_LINE = {
+ 'extra-info': _parse_extra_info_line,
+ 'geoip-db-digest': _parse_geoip_db_digest_line,
+ 'geoip6-db-digest': _parse_geoip6_db_digest_line,
+ 'transport': _parse_transport_line,
+ 'cell-circuits-per-decile': _parse_cell_circuits_per_decline_line,
+ 'dirreq-v2-resp': _parse_dirreq_v2_resp_line,
+ 'dirreq-v3-resp': _parse_dirreq_v3_resp_line,
+ 'dirreq-v2-direct-dl': _parse_dirreq_v2_direct_dl_line,
+ 'dirreq-v3-direct-dl': _parse_dirreq_v3_direct_dl_line,
+ 'dirreq-v2-tunneled-dl': _parse_dirreq_v2_tunneled_dl_line,
+ 'dirreq-v3-tunneled-dl': _parse_dirreq_v3_tunneled_dl_line,
+ 'dirreq-v2-share': _parse_dirreq_v2_share_line,
+ 'dirreq-v3-share': _parse_dirreq_v3_share_line,
+ 'cell-processed-cells': _parse_cell_processed_cells_line,
+ 'cell-queued-cells': _parse_cell_queued_cells_line,
+ 'cell-time-in-queue': _parse_cell_time_in_queue_line,
+ 'published': _parse_published_line,
+ 'geoip-start-time': _parse_geoip_start_time_line,
+ 'cell-stats-end': _parse_cell_stats_end_line,
+ 'entry-stats-end': _parse_entry_stats_end_line,
+ 'exit-stats-end': _parse_exit_stats_end_line,
+ 'bridge-stats-end': _parse_bridge_stats_end_line,
+ 'dirreq-stats-end': _parse_dirreq_stats_end_line,
+ 'conn-bi-direct': _parse_conn_bi_direct_line,
+ 'read-history': _parse_read_history_line,
+ 'write-history': _parse_write_history_line,
+ 'dirreq-read-history': _parse_dirreq_read_history_line,
+ 'dirreq-write-history': _parse_dirreq_write_history_line,
+ 'exit-kibibytes-written': _parse_exit_kibibytes_written_line,
+ 'exit-kibibytes-read': _parse_exit_kibibytes_read_line,
+ 'exit-streams-opened': _parse_exit_streams_opened_line,
+ 'hidserv-stats-end': _parse_hidden_service_stats_end_line,
+ 'hidserv-rend-relayed-cells': _parse_hidden_service_rend_relayed_cells_line,
+ 'hidserv-dir-onions-seen': _parse_hidden_service_dir_onions_seen_line,
+ 'dirreq-v2-ips': _parse_dirreq_v2_ips_line,
+ 'dirreq-v3-ips': _parse_dirreq_v3_ips_line,
+ 'dirreq-v2-reqs': _parse_dirreq_v2_reqs_line,
+ 'dirreq-v3-reqs': _parse_dirreq_v3_reqs_line,
+ 'geoip-client-origins': _parse_geoip_client_origins_line,
+ 'entry-ips': _parse_entry_ips_line,
+ 'bridge-ips': _parse_bridge_ips_line,
+ 'bridge-ip-versions': _parse_bridge_ip_versions_line,
+ 'bridge-ip-transports': _parse_bridge_ip_transports_line,
+ }
+
+ def __init__(self, raw_contents, validate = False):
+ """
+ Extra-info descriptor constructor. By default this validates the
+ descriptor's content as it's parsed. This validation can be disabled to
+ either improve performance or be accepting of malformed data.
+
+ :param str raw_contents: extra-info content provided by the relay
+ :param bool validate: checks the validity of the extra-info descriptor if
+ **True**, skips these checks otherwise
+
+ :raises: **ValueError** if the contents is malformed and validate is True
+ """
+
+ super(ExtraInfoDescriptor, self).__init__(raw_contents, lazy_load = not validate)
+ entries = _get_descriptor_components(raw_contents, validate)
+
+ if validate:
+ for keyword in self._required_fields():
+ if keyword not in entries:
+ raise ValueError("Extra-info descriptor must have a '%s' entry" % keyword)
+
+ for keyword in self._required_fields() + SINGLE_FIELDS:
+ if keyword in entries and len(entries[keyword]) > 1:
+ raise ValueError("The '%s' entry can only appear once in an extra-info descriptor" % keyword)
+
+ expected_first_keyword = self._first_keyword()
+ if expected_first_keyword and expected_first_keyword != list(entries.keys())[0]:
+ raise ValueError("Extra-info descriptor must start with a '%s' entry" % expected_first_keyword)
+
+ expected_last_keyword = self._last_keyword()
+ if expected_last_keyword and expected_last_keyword != list(entries.keys())[-1]:
+ raise ValueError("Descriptor must end with a '%s' entry" % expected_last_keyword)
+
+ self._parse(entries, validate)
+ else:
+ self._entries = entries
+
+ def digest(self):
+ """
+ Provides the upper-case hex encoded sha1 of our content. This value is part
+ of the server descriptor entry for this relay.
+
+ :returns: **str** with the upper-case hex digest value for this server
+ descriptor
+ """
+
+ raise NotImplementedError('Unsupported Operation: this should be implemented by the ExtraInfoDescriptor subclass')
+
+ def _required_fields(self):
+ return REQUIRED_FIELDS
+
+ def _first_keyword(self):
+ return 'extra-info'
+
+ def _last_keyword(self):
+ return 'router-signature'
+
+
+class RelayExtraInfoDescriptor(ExtraInfoDescriptor):
+ """
+ Relay extra-info descriptor, constructed from data such as that provided by
+ 'GETINFO extra-info/digest/\*', cached descriptors, and metrics
+ (`specification <https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt>`_).
+
+ :var str signature: **\*** signature for this extrainfo descriptor
+
+ **\*** attribute is required when we're parsed with validation
+ """
+
+ ATTRIBUTES = dict(ExtraInfoDescriptor.ATTRIBUTES, **{
+ 'signature': (None, _parse_router_signature_line),
+ })
+
+ PARSER_FOR_LINE = dict(ExtraInfoDescriptor.PARSER_FOR_LINE, **{
+ 'router-signature': _parse_router_signature_line,
+ })
+
+ @lru_cache()
+ def digest(self):
+ # our digest is calculated from everything except our signature
+ raw_content, ending = str(self), '\nrouter-signature\n'
+ raw_content = raw_content[:raw_content.find(ending) + len(ending)]
+ return hashlib.sha1(stem.util.str_tools._to_bytes(raw_content)).hexdigest().upper()
+
+
+class BridgeExtraInfoDescriptor(ExtraInfoDescriptor):
+ """
+ Bridge extra-info descriptor (`bridge descriptor specification
+ <https://collector.torproject.org/formats.html#bridge-descriptors>`_)
+ """
+
+ ATTRIBUTES = dict(ExtraInfoDescriptor.ATTRIBUTES, **{
+ '_digest': (None, _parse_router_digest_line),
+ })
+
+ PARSER_FOR_LINE = dict(ExtraInfoDescriptor.PARSER_FOR_LINE, **{
+ 'router-digest': _parse_router_digest_line,
+ })
+
+ def digest(self):
+ return self._digest
+
+ def _required_fields(self):
+ excluded_fields = [
+ 'router-signature',
+ ]
+
+ included_fields = [
+ 'router-digest',
+ ]
+
+ return tuple(included_fields + [f for f in REQUIRED_FIELDS if f not in excluded_fields])
+
+ def _last_keyword(self):
+ return None
diff --git a/ext/stem/descriptor/hidden_service_descriptor.py b/ext/stem/descriptor/hidden_service_descriptor.py
new file mode 100644
index 00000000..31a99cc4
--- /dev/null
+++ b/ext/stem/descriptor/hidden_service_descriptor.py
@@ -0,0 +1,422 @@
+# Copyright 2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Parsing for Tor hidden service descriptors as described in Tor's `rend-spec
+<https://gitweb.torproject.org/torspec.git/tree/rend-spec.txt>`_.
+
+Unlike other descriptor types these describe a hidden service rather than a
+relay. They're created by the service, and can only be fetched via relays with
+the HSDir flag.
+
+**Module Overview:**
+
+::
+
+ HiddenServiceDescriptor - Tor hidden service descriptor.
+
+.. versionadded:: 1.4.0
+"""
+
+# TODO: Add a description for how to retrieve them when tor supports that
+# (#14847) and then update #15009.
+
+import base64
+import binascii
+import collections
+import hashlib
+import io
+
+import stem.util.connection
+import stem.util.str_tools
+
+from stem.descriptor import (
+ PGP_BLOCK_END,
+ Descriptor,
+ _get_descriptor_components,
+ _read_until_keywords,
+ _bytes_for_block,
+ _value,
+ _parse_simple_line,
+ _parse_timestamp_line,
+ _parse_key_block,
+)
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+REQUIRED_FIELDS = (
+ 'rendezvous-service-descriptor',
+ 'version',
+ 'permanent-key',
+ 'secret-id-part',
+ 'publication-time',
+ 'protocol-versions',
+ 'signature',
+)
+
+INTRODUCTION_POINTS_ATTR = {
+ 'identifier': None,
+ 'address': None,
+ 'port': None,
+ 'onion_key': None,
+ 'service_key': None,
+ 'intro_authentication': [],
+}
+
+# introduction-point fields that can only appear once
+
+SINGLE_INTRODUCTION_POINT_FIELDS = [
+ 'introduction-point',
+ 'ip-address',
+ 'onion-port',
+ 'onion-key',
+ 'service-key',
+]
+
+BASIC_AUTH = 1
+STEALTH_AUTH = 2
+
+IntroductionPoint = collections.namedtuple('IntroductionPoints', INTRODUCTION_POINTS_ATTR.keys())
+
+
+class DecryptionFailure(Exception):
+ """
+ Failure to decrypt the hidden service descriptor's introduction-points.
+ """
+
+
+def _parse_file(descriptor_file, validate = False, **kwargs):
+ """
+ Iterates over the hidden service descriptors in a file.
+
+ :param file descriptor_file: file with descriptor content
+ :param bool validate: checks the validity of the descriptor's content if
+ **True**, skips these checks otherwise
+ :param dict kwargs: additional arguments for the descriptor constructor
+
+ :returns: iterator for :class:`~stem.descriptor.hidden_service_descriptor.HiddenServiceDescriptor`
+ instances in the file
+
+ :raises:
+ * **ValueError** if the contents is malformed and validate is **True**
+ * **IOError** if the file can't be read
+ """
+
+ while True:
+ descriptor_content = _read_until_keywords('signature', descriptor_file)
+
+ # we've reached the 'signature', now include the pgp style block
+ block_end_prefix = PGP_BLOCK_END.split(' ', 1)[0]
+ descriptor_content += _read_until_keywords(block_end_prefix, descriptor_file, True)
+
+ if descriptor_content:
+ if descriptor_content[0].startswith(b'@type'):
+ descriptor_content = descriptor_content[1:]
+
+ yield HiddenServiceDescriptor(bytes.join(b'', descriptor_content), validate, **kwargs)
+ else:
+ break # done parsing file
+
+
+def _parse_version_line(descriptor, entries):
+ value = _value('version', entries)
+
+ if value.isdigit():
+ descriptor.version = int(value)
+ else:
+ raise ValueError('version line must have a positive integer value: %s' % value)
+
+
+def _parse_protocol_versions_line(descriptor, entries):
+ value = _value('protocol-versions', entries)
+
+ try:
+ versions = [int(entry) for entry in value.split(',')]
+ except ValueError:
+ raise ValueError('protocol-versions line has non-numeric versoins: protocol-versions %s' % value)
+
+ for v in versions:
+ if v <= 0:
+ raise ValueError('protocol-versions must be positive integers: %s' % value)
+
+ descriptor.protocol_versions = versions
+
+
+def _parse_introduction_points_line(descriptor, entries):
+ _, block_type, block_contents = entries['introduction-points'][0]
+
+ if not block_contents or block_type != 'MESSAGE':
+ raise ValueError("'introduction-points' should be followed by a MESSAGE block, but was a %s" % block_type)
+
+ descriptor.introduction_points_encoded = block_contents
+
+ try:
+ decoded_field = _bytes_for_block(block_contents)
+ except TypeError:
+ raise ValueError("'introduction-points' isn't base64 encoded content:\n%s" % block_contents)
+
+ auth_types = []
+
+ while decoded_field.startswith(b'service-authentication ') and b'\n' in decoded_field:
+ auth_line, decoded_field = decoded_field.split(b'\n', 1)
+ auth_line_comp = auth_line.split(b' ')
+
+ if len(auth_line_comp) < 3:
+ raise ValueError("Within introduction-points we expected 'service-authentication [auth_type] [auth_data]', but had '%s'" % auth_line)
+
+ auth_types.append((auth_line_comp[1], auth_line_comp[2]))
+
+ descriptor.introduction_points_auth = auth_types
+ descriptor.introduction_points_content = decoded_field
+
+_parse_rendezvous_service_descriptor_line = _parse_simple_line('rendezvous-service-descriptor', 'descriptor_id')
+_parse_permanent_key_line = _parse_key_block('permanent-key', 'permanent_key', 'RSA PUBLIC KEY')
+_parse_secret_id_part_line = _parse_simple_line('secret-id-part', 'secret_id_part')
+_parse_publication_time_line = _parse_timestamp_line('publication-time', 'published')
+_parse_signature_line = _parse_key_block('signature', 'signature', 'SIGNATURE')
+
+
+class HiddenServiceDescriptor(Descriptor):
+ """
+ Hidden service descriptor.
+
+ :var str descriptor_id: **\*** identifier for this descriptor, this is a base32 hash of several fields
+ :var int version: **\*** hidden service descriptor version
+ :var str permanent_key: **\*** long term key of the hidden service
+ :var str secret_id_part: **\*** hash of the time period, cookie, and replica
+ values so our descriptor_id can be validated
+ :var datetime published: **\*** time in UTC when this descriptor was made
+ :var list protocol_versions: **\*** list of **int** versions that are supported when establishing a connection
+ :var str introduction_points_encoded: raw introduction points blob
+ :var list introduction_points_auth: **\*** tuples of the form
+ (auth_method, auth_data) for our introduction_points_content
+ :var bytes introduction_points_content: decoded introduction-points content
+ without authentication data, if using cookie authentication this is
+ encrypted
+ :var str signature: signature of the descriptor content
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as **None** if undefined
+ """
+
+ ATTRIBUTES = {
+ 'descriptor_id': (None, _parse_rendezvous_service_descriptor_line),
+ 'version': (None, _parse_version_line),
+ 'permanent_key': (None, _parse_permanent_key_line),
+ 'secret_id_part': (None, _parse_secret_id_part_line),
+ 'published': (None, _parse_publication_time_line),
+ 'protocol_versions': ([], _parse_protocol_versions_line),
+ 'introduction_points_encoded': (None, _parse_introduction_points_line),
+ 'introduction_points_auth': ([], _parse_introduction_points_line),
+ 'introduction_points_content': (None, _parse_introduction_points_line),
+ 'signature': (None, _parse_signature_line),
+ }
+
+ PARSER_FOR_LINE = {
+ 'rendezvous-service-descriptor': _parse_rendezvous_service_descriptor_line,
+ 'version': _parse_version_line,
+ 'permanent-key': _parse_permanent_key_line,
+ 'secret-id-part': _parse_secret_id_part_line,
+ 'publication-time': _parse_publication_time_line,
+ 'protocol-versions': _parse_protocol_versions_line,
+ 'introduction-points': _parse_introduction_points_line,
+ 'signature': _parse_signature_line,
+ }
+
+ def __init__(self, raw_contents, validate = False):
+ super(HiddenServiceDescriptor, self).__init__(raw_contents, lazy_load = not validate)
+ entries = _get_descriptor_components(raw_contents, validate)
+
+ if validate:
+ for keyword in REQUIRED_FIELDS:
+ if keyword not in entries:
+ raise ValueError("Hidden service descriptor must have a '%s' entry" % keyword)
+ elif keyword in entries and len(entries[keyword]) > 1:
+ raise ValueError("The '%s' entry can only appear once in a hidden service descriptor" % keyword)
+
+ if 'rendezvous-service-descriptor' != list(entries.keys())[0]:
+ raise ValueError("Hidden service descriptor must start with a 'rendezvous-service-descriptor' entry")
+ elif 'signature' != list(entries.keys())[-1]:
+ raise ValueError("Hidden service descriptor must end with a 'signature' entry")
+
+ self._parse(entries, validate)
+
+ if stem.prereq.is_crypto_available():
+ signed_digest = self._digest_for_signature(self.permanent_key, self.signature)
+ content_digest = self._digest_for_content(b'rendezvous-service-descriptor ', b'\nsignature\n')
+
+ if signed_digest != content_digest:
+ raise ValueError('Decrypted digest does not match local digest (calculated: %s, local: %s)' % (signed_digest, content_digest))
+ else:
+ self._entries = entries
+
+ @lru_cache()
+ def introduction_points(self, authentication_cookie = None):
+ """
+ Provided this service's introduction points. This provides a list of
+ IntroductionPoint instances, which have the following attributes...
+
+ * **identifier** (str): hash of this introduction point's identity key
+ * **address** (str): address of this introduction point
+ * **port** (int): port where this introduction point is listening
+ * **onion_key** (str): public key for communicating with this introduction point
+ * **service_key** (str): public key for communicating with this hidden service
+ * **intro_authentication** (list): tuples of the form (auth_type, auth_data)
+ for establishing a connection
+
+ :param str authentication_cookie: cookie to decrypt the introduction-points
+ if it's encrypted
+
+ :returns: **list** of IntroductionPoints instances
+
+ :raises:
+ * **ValueError** if the our introduction-points is malformed
+ * **DecryptionFailure** if unable to decrypt this field
+ """
+
+ content = self.introduction_points_content
+
+ if not content:
+ return []
+ elif authentication_cookie:
+ if not stem.prereq.is_crypto_available():
+ raise DecryptionFailure('Decrypting introduction-points requires pycrypto')
+
+ try:
+ missing_padding = len(authentication_cookie) % 4
+ authentication_cookie = base64.b64decode(stem.util.str_tools._to_bytes(authentication_cookie) + b'=' * missing_padding)
+ except TypeError as exc:
+ raise DecryptionFailure('authentication_cookie must be a base64 encoded string (%s)' % exc)
+
+ authentication_type = int(binascii.hexlify(content[0:1]), 16)
+
+ if authentication_type == BASIC_AUTH:
+ content = HiddenServiceDescriptor._decrypt_basic_auth(content, authentication_cookie)
+ elif authentication_type == STEALTH_AUTH:
+ content = HiddenServiceDescriptor._decrypt_stealth_auth(content, authentication_cookie)
+ else:
+ raise DecryptionFailure("Unrecognized authentication type '%s', currently we only support basic auth (%s) and stealth auth (%s)" % (authentication_type, BASIC_AUTH, STEALTH_AUTH))
+
+ if not content.startswith(b'introduction-point '):
+ raise DecryptionFailure('Unable to decrypt the introduction-points, maybe this is the wrong key?')
+ elif not content.startswith(b'introduction-point '):
+ raise DecryptionFailure('introduction-points content is encrypted, you need to provide its authentication_cookie')
+
+ return HiddenServiceDescriptor._parse_introduction_points(content)
+
+ @staticmethod
+ def _decrypt_basic_auth(content, authentication_cookie):
+ from Crypto.Cipher import AES
+ from Crypto.Util import Counter
+ from Crypto.Util.number import bytes_to_long
+
+ try:
+ client_blocks = int(binascii.hexlify(content[1:2]), 16)
+ except ValueError:
+ raise DecryptionFailure("When using basic auth the content should start with a number of blocks but wasn't a hex digit: %s" % binascii.hexlify(content[1:2]))
+
+ # parse the client id and encrypted session keys
+
+ client_entries_length = client_blocks * 16 * 20
+ client_entries = content[2:2 + client_entries_length]
+ client_keys = [(client_entries[i:i + 4], client_entries[i + 4:i + 20]) for i in range(0, client_entries_length, 4 + 16)]
+
+ iv = content[2 + client_entries_length:2 + client_entries_length + 16]
+ encrypted = content[2 + client_entries_length + 16:]
+
+ client_id = hashlib.sha1(authentication_cookie + iv).digest()[:4]
+
+ for entry_id, encrypted_session_key in client_keys:
+ if entry_id != client_id:
+ continue # not the session key for this client
+
+ # try decrypting the session key
+
+ counter = Counter.new(128, initial_value = 0)
+ cipher = AES.new(authentication_cookie, AES.MODE_CTR, counter = counter)
+ session_key = cipher.decrypt(encrypted_session_key)
+
+ # attempt to decrypt the intro points with the session key
+
+ counter = Counter.new(128, initial_value = bytes_to_long(iv))
+ cipher = AES.new(session_key, AES.MODE_CTR, counter = counter)
+ decrypted = cipher.decrypt(encrypted)
+
+ # check if the decryption looks correct
+
+ if decrypted.startswith(b'introduction-point '):
+ return decrypted
+
+ return content # nope, unable to decrypt the content
+
+ @staticmethod
+ def _decrypt_stealth_auth(content, authentication_cookie):
+ from Crypto.Cipher import AES
+ from Crypto.Util import Counter
+ from Crypto.Util.number import bytes_to_long
+
+ # byte 1 = authentication type, 2-17 = input vector, 18 on = encrypted content
+
+ iv, encrypted = content[1:17], content[17:]
+ counter = Counter.new(128, initial_value = bytes_to_long(iv))
+ cipher = AES.new(authentication_cookie, AES.MODE_CTR, counter = counter)
+
+ return cipher.decrypt(encrypted)
+
+ @staticmethod
+ def _parse_introduction_points(content):
+ """
+ Provides the parsed list of IntroductionPoint for the unencrypted content.
+ """
+
+ introduction_points = []
+ content_io = io.BytesIO(content)
+
+ while True:
+ content = b''.join(_read_until_keywords('introduction-point', content_io, ignore_first = True))
+
+ if not content:
+ break # reached the end
+
+ attr = dict(INTRODUCTION_POINTS_ATTR)
+ entries = _get_descriptor_components(content, False)
+
+ for keyword, values in list(entries.items()):
+ value, block_type, block_contents = values[0]
+
+ if keyword in SINGLE_INTRODUCTION_POINT_FIELDS and len(values) > 1:
+ raise ValueError("'%s' can only appear once in an introduction-point block, but appeared %i times" % (keyword, len(values)))
+
+ if keyword == 'introduction-point':
+ attr['identifier'] = value
+ elif keyword == 'ip-address':
+ if not stem.util.connection.is_valid_ipv4_address(value):
+ raise ValueError("'%s' is an invalid IPv4 address" % value)
+
+ attr['address'] = value
+ elif keyword == 'onion-port':
+ if not stem.util.connection.is_valid_port(value):
+ raise ValueError("'%s' is an invalid port" % value)
+
+ attr['port'] = int(value)
+ elif keyword == 'onion-key':
+ attr['onion_key'] = block_contents
+ elif keyword == 'service-key':
+ attr['service_key'] = block_contents
+ elif keyword == 'intro-authentication':
+ auth_entries = []
+
+ for auth_value, _, _ in values:
+ if ' ' not in auth_value:
+ raise ValueError("We expected 'intro-authentication [auth_type] [auth_data]', but had '%s'" % auth_value)
+
+ auth_type, auth_data = auth_value.split(' ')[:2]
+ auth_entries.append((auth_type, auth_data))
+
+ introduction_points.append(IntroductionPoint(**attr))
+
+ return introduction_points
diff --git a/ext/stem/descriptor/microdescriptor.py b/ext/stem/descriptor/microdescriptor.py
new file mode 100644
index 00000000..ffbec438
--- /dev/null
+++ b/ext/stem/descriptor/microdescriptor.py
@@ -0,0 +1,314 @@
+# Copyright 2013-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Parsing for Tor microdescriptors, which contain a distilled version of a
+relay's server descriptor. As of Tor version 0.2.3.3-alpha Tor no longer
+downloads server descriptors by default, opting for microdescriptors instead.
+
+Unlike most descriptor documents these aren't available on the metrics site
+(since they don't contain any information that the server descriptors don't).
+
+The limited information in microdescriptors make them rather clunky to use
+compared with server descriptors. For instance microdescriptors lack the
+relay's fingerprint, making it difficut to use them to look up the relay's
+other descriptors.
+
+To do so you need to match the microdescriptor's digest against its
+corresponding router status entry. For added fun as of this writing the
+controller doesn't even surface those router status entries
+(:trac:`7953`).
+
+For instance, here's an example that prints the nickname and fignerprints of
+the exit relays.
+
+::
+
+ import os
+
+ from stem.control import Controller
+ from stem.descriptor import parse_file
+
+ with Controller.from_port(port = 9051) as controller:
+ controller.authenticate()
+
+ exit_digests = set()
+ data_dir = controller.get_conf('DataDirectory')
+
+ for desc in controller.get_microdescriptors():
+ if desc.exit_policy.is_exiting_allowed():
+ exit_digests.add(desc.digest)
+
+ print 'Exit Relays:'
+
+ for desc in parse_file(os.path.join(data_dir, 'cached-microdesc-consensus')):
+ if desc.digest in exit_digests:
+ print ' %s (%s)' % (desc.nickname, desc.fingerprint)
+
+Doing the same is trivial with server descriptors...
+
+::
+
+ from stem.descriptor import parse_file
+
+ print 'Exit Relays:'
+
+ for desc in parse_file('/home/atagar/.tor/cached-descriptors'):
+ if desc.exit_policy.is_exiting_allowed():
+ print ' %s (%s)' % (desc.nickname, desc.fingerprint)
+
+**Module Overview:**
+
+::
+
+ Microdescriptor - Tor microdescriptor.
+"""
+
+import hashlib
+
+import stem.exit_policy
+
+from stem.descriptor import (
+ Descriptor,
+ _get_descriptor_components,
+ _read_until_keywords,
+ _value,
+ _parse_simple_line,
+ _parse_key_block,
+)
+
+from stem.descriptor.router_status_entry import (
+ _parse_a_line,
+ _parse_p_line,
+)
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+REQUIRED_FIELDS = (
+ 'onion-key',
+)
+
+SINGLE_FIELDS = (
+ 'onion-key',
+ 'ntor-onion-key',
+ 'family',
+ 'p',
+ 'p6',
+)
+
+
+def _parse_file(descriptor_file, validate = False, **kwargs):
+ """
+ Iterates over the microdescriptors in a file.
+
+ :param file descriptor_file: file with descriptor content
+ :param bool validate: checks the validity of the descriptor's content if
+ **True**, skips these checks otherwise
+ :param dict kwargs: additional arguments for the descriptor constructor
+
+ :returns: iterator for Microdescriptor instances in the file
+
+ :raises:
+ * **ValueError** if the contents is malformed and validate is True
+ * **IOError** if the file can't be read
+ """
+
+ while True:
+ annotations = _read_until_keywords('onion-key', descriptor_file)
+
+ # read until we reach an annotation or onion-key line
+ descriptor_lines = []
+
+ # read the onion-key line, done if we're at the end of the document
+
+ onion_key_line = descriptor_file.readline()
+
+ if onion_key_line:
+ descriptor_lines.append(onion_key_line)
+ else:
+ break
+
+ while True:
+ last_position = descriptor_file.tell()
+ line = descriptor_file.readline()
+
+ if not line:
+ break # EOF
+ elif line.startswith(b'@') or line.startswith(b'onion-key'):
+ descriptor_file.seek(last_position)
+ break
+ else:
+ descriptor_lines.append(line)
+
+ if descriptor_lines:
+ if descriptor_lines[0].startswith(b'@type'):
+ descriptor_lines = descriptor_lines[1:]
+
+ # strip newlines from annotations
+ annotations = list(map(bytes.strip, annotations))
+
+ descriptor_text = bytes.join(b'', descriptor_lines)
+
+ yield Microdescriptor(descriptor_text, validate, annotations, **kwargs)
+ else:
+ break # done parsing descriptors
+
+
+def _parse_id_line(descriptor, entries):
+ value = _value('id', entries)
+ value_comp = value.split()
+
+ if len(value_comp) >= 2:
+ descriptor.identifier_type = value_comp[0]
+ descriptor.identifier = value_comp[1]
+ else:
+ raise ValueError("'id' lines should contain both the key type and digest: id %s" % value)
+
+
+_parse_digest = lambda descriptor, entries: setattr(descriptor, 'digest', hashlib.sha256(descriptor.get_bytes()).hexdigest().upper())
+_parse_onion_key_line = _parse_key_block('onion-key', 'onion_key', 'RSA PUBLIC KEY')
+_parse_ntor_onion_key_line = _parse_simple_line('ntor-onion-key', 'ntor_onion_key')
+_parse_family_line = lambda descriptor, entries: setattr(descriptor, 'family', _value('family', entries).split(' '))
+_parse_p6_line = lambda descriptor, entries: setattr(descriptor, 'exit_policy_v6', stem.exit_policy.MicroExitPolicy(_value('p6', entries)))
+
+
+class Microdescriptor(Descriptor):
+ """
+ Microdescriptor (`descriptor specification
+ <https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt>`_)
+
+ :var str digest: **\*** hex digest for this microdescriptor, this can be used
+ to match against the corresponding digest attribute of a
+ :class:`~stem.descriptor.router_status_entry.RouterStatusEntryMicroV3`
+ :var str onion_key: **\*** key used to encrypt EXTEND cells
+ :var str ntor_onion_key: base64 key used to encrypt EXTEND in the ntor protocol
+ :var list or_addresses: **\*** alternative for our address/or_port attributes, each
+ entry is a tuple of the form (address (**str**), port (**int**), is_ipv6
+ (**bool**))
+ :var list family: **\*** nicknames or fingerprints of declared family
+ :var stem.exit_policy.MicroExitPolicy exit_policy: **\*** relay's exit policy
+ :var stem.exit_policy.MicroExitPolicy exit_policy_v6: **\*** exit policy for IPv6
+ :var str identifier_type: identity digest key type
+ :var str identifier: base64 encoded identity digest, this is only used for collision prevention (:trac:`11743`)
+
+ **\*** attribute is required when we're parsed with validation
+
+ .. versionchanged:: 1.1.0
+ Added the identifier and identifier_type attributes.
+ """
+
+ ATTRIBUTES = {
+ 'onion_key': (None, _parse_onion_key_line),
+ 'ntor_onion_key': (None, _parse_ntor_onion_key_line),
+ 'or_addresses': ([], _parse_a_line),
+ 'family': ([], _parse_family_line),
+ 'exit_policy': (stem.exit_policy.MicroExitPolicy('reject 1-65535'), _parse_p_line),
+ 'exit_policy_v6': (None, _parse_p6_line),
+ 'identifier_type': (None, _parse_id_line),
+ 'identifier': (None, _parse_id_line),
+ 'digest': (None, _parse_digest),
+ }
+
+ PARSER_FOR_LINE = {
+ 'onion-key': _parse_onion_key_line,
+ 'ntor-onion-key': _parse_ntor_onion_key_line,
+ 'a': _parse_a_line,
+ 'family': _parse_family_line,
+ 'p': _parse_p_line,
+ 'p6': _parse_p6_line,
+ 'id': _parse_id_line,
+ }
+
+ def __init__(self, raw_contents, validate = False, annotations = None):
+ super(Microdescriptor, self).__init__(raw_contents, lazy_load = not validate)
+ self._annotation_lines = annotations if annotations else []
+ entries = _get_descriptor_components(raw_contents, validate)
+
+ if validate:
+ self.digest = hashlib.sha256(self.get_bytes()).hexdigest().upper()
+ self._parse(entries, validate)
+ self._check_constraints(entries)
+ else:
+ self._entries = entries
+
+ @lru_cache()
+ def get_annotations(self):
+ """
+ Provides content that appeared prior to the descriptor. If this comes from
+ the cached-microdescs then this commonly contains content like...
+
+ ::
+
+ @last-listed 2013-02-24 00:18:30
+
+ :returns: **dict** with the key/value pairs in our annotations
+ """
+
+ annotation_dict = {}
+
+ for line in self._annotation_lines:
+ if b' ' in line:
+ key, value = line.split(b' ', 1)
+ annotation_dict[key] = value
+ else:
+ annotation_dict[line] = None
+
+ return annotation_dict
+
+ def get_annotation_lines(self):
+ """
+ Provides the lines of content that appeared prior to the descriptor. This
+ is the same as the
+ :func:`~stem.descriptor.microdescriptor.Microdescriptor.get_annotations`
+ results, but with the unparsed lines and ordering retained.
+
+ :returns: **list** with the lines of annotation that came before this descriptor
+ """
+
+ return self._annotation_lines
+
+ def _check_constraints(self, entries):
+ """
+ Does a basic check that the entries conform to this descriptor type's
+ constraints.
+
+ :param dict entries: keyword => (value, pgp key) entries
+
+ :raises: **ValueError** if an issue arises in validation
+ """
+
+ for keyword in REQUIRED_FIELDS:
+ if keyword not in entries:
+ raise ValueError("Microdescriptor must have a '%s' entry" % keyword)
+
+ for keyword in SINGLE_FIELDS:
+ if keyword in entries and len(entries[keyword]) > 1:
+ raise ValueError("The '%s' entry can only appear once in a microdescriptor" % keyword)
+
+ if 'onion-key' != list(entries.keys())[0]:
+ raise ValueError("Microdescriptor must start with a 'onion-key' entry")
+
+ def _name(self, is_plural = False):
+ return 'microdescriptors' if is_plural else 'microdescriptor'
+
+ def _compare(self, other, method):
+ if not isinstance(other, Microdescriptor):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __hash__(self):
+ return hash(str(self).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
diff --git a/ext/stem/descriptor/networkstatus.py b/ext/stem/descriptor/networkstatus.py
new file mode 100644
index 00000000..a162e2ec
--- /dev/null
+++ b/ext/stem/descriptor/networkstatus.py
@@ -0,0 +1,1444 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Parsing for Tor network status documents. This supports both the v2 and v3
+dir-spec. Documents can be obtained from a few sources...
+
+* The 'cached-consensus' file in Tor's data directory.
+
+* Archived descriptors provided by CollecTor
+ (https://collector.torproject.org/).
+
+* Directory authorities and mirrors via their DirPort.
+
+... and contain the following sections...
+
+* document header
+* list of :class:`stem.descriptor.networkstatus.DirectoryAuthority`
+* list of :class:`stem.descriptor.router_status_entry.RouterStatusEntry`
+* document footer
+
+Of these, the router status entry section can be quite large (on the order of
+hundreds of kilobytes). As such we provide a couple of methods for reading
+network status documents through :func:`~stem.descriptor.__init__.parse_file`.
+For more information see :func:`~stem.descriptor.__init__.DocumentHandler`...
+
+::
+
+ from stem.descriptor import parse_file, DocumentHandler
+
+ with open('.tor/cached-consensus', 'rb') as consensus_file:
+ # Processes the routers as we read them in. The routers refer to a document
+ # with an unset 'routers' attribute.
+
+ for router in parse_file(consensus_file, 'network-status-consensus-3 1.0', document_handler = DocumentHandler.ENTRIES):
+ print router.nickname
+
+**Module Overview:**
+
+::
+
+ NetworkStatusDocument - Network status document
+ |- NetworkStatusDocumentV2 - Version 2 network status document
+ |- NetworkStatusDocumentV3 - Version 3 network status document
+ +- BridgeNetworkStatusDocument - Version 3 network status document for bridges
+
+ KeyCertificate - Certificate used to authenticate an authority
+ DocumentSignature - Signature of a document by a directory authority
+ DirectoryAuthority - Directory authority as defined in a v3 network status document
+
+
+.. data:: PackageVersion
+
+ Latest recommended version of a package that's available.
+
+ :var str name: name of the package
+ :var str version: latest recommended version
+ :var str url: package's url
+ :var dict digests: mapping of digest types to their value
+"""
+
+import collections
+import io
+
+import stem.descriptor.router_status_entry
+import stem.util.str_tools
+import stem.util.tor_tools
+import stem.version
+
+from stem.descriptor import (
+ PGP_BLOCK_END,
+ Descriptor,
+ DocumentHandler,
+ _get_descriptor_components,
+ _read_until_keywords,
+ _value,
+ _parse_simple_line,
+ _parse_timestamp_line,
+ _parse_forty_character_hex,
+ _parse_key_block,
+)
+
+from stem.descriptor.router_status_entry import (
+ RouterStatusEntryV2,
+ RouterStatusEntryV3,
+ RouterStatusEntryMicroV3,
+)
+
+PackageVersion = collections.namedtuple('PackageVersion', [
+ 'name',
+ 'version',
+ 'url',
+ 'digests',
+])
+
+# Version 2 network status document fields, tuples of the form...
+# (keyword, is_mandatory)
+
+NETWORK_STATUS_V2_FIELDS = (
+ ('network-status-version', True),
+ ('dir-source', True),
+ ('fingerprint', True),
+ ('contact', True),
+ ('dir-signing-key', True),
+ ('client-versions', False),
+ ('server-versions', False),
+ ('published', True),
+ ('dir-options', False),
+ ('directory-signature', True),
+)
+
+# Network status document are either a 'vote' or 'consensus', with different
+# mandatory fields for each. Both though require that their fields appear in a
+# specific order. This is an ordered listing of the following...
+#
+# (field, in_votes, in_consensus, is_mandatory)
+
+HEADER_STATUS_DOCUMENT_FIELDS = (
+ ('network-status-version', True, True, True),
+ ('vote-status', True, True, True),
+ ('consensus-methods', True, False, False),
+ ('consensus-method', False, True, False),
+ ('published', True, False, True),
+ ('valid-after', True, True, True),
+ ('fresh-until', True, True, True),
+ ('valid-until', True, True, True),
+ ('voting-delay', True, True, True),
+ ('client-versions', True, True, False),
+ ('server-versions', True, True, False),
+ ('package', True, True, False),
+ ('known-flags', True, True, True),
+ ('flag-thresholds', True, False, False),
+ ('params', True, True, False),
+)
+
+FOOTER_STATUS_DOCUMENT_FIELDS = (
+ ('directory-footer', True, True, False),
+ ('bandwidth-weights', False, True, False),
+ ('directory-signature', True, True, True),
+)
+
+HEADER_FIELDS = [attr[0] for attr in HEADER_STATUS_DOCUMENT_FIELDS]
+FOOTER_FIELDS = [attr[0] for attr in FOOTER_STATUS_DOCUMENT_FIELDS]
+
+AUTH_START = 'dir-source'
+ROUTERS_START = 'r'
+FOOTER_START = 'directory-footer'
+V2_FOOTER_START = 'directory-signature'
+
+DEFAULT_PARAMS = {
+ 'bwweightscale': 10000,
+ 'cbtdisabled': 0,
+ 'cbtnummodes': 3,
+ 'cbtrecentcount': 20,
+ 'cbtmaxtimeouts': 18,
+ 'cbtmincircs': 100,
+ 'cbtquantile': 80,
+ 'cbtclosequantile': 95,
+ 'cbttestfreq': 60,
+ 'cbtmintimeout': 2000,
+ 'cbtinitialtimeout': 60000,
+ 'Support022HiddenServices': 1,
+ 'usecreatefast': 1,
+}
+
+# KeyCertificate fields, tuple is of the form...
+# (keyword, is_mandatory)
+
+KEY_CERTIFICATE_PARAMS = (
+ ('dir-key-certificate-version', True),
+ ('dir-address', False),
+ ('fingerprint', True),
+ ('dir-identity-key', True),
+ ('dir-key-published', True),
+ ('dir-key-expires', True),
+ ('dir-signing-key', True),
+ ('dir-key-crosscert', False),
+ ('dir-key-certification', True),
+)
+
+# all parameters are constrained to int32 range
+MIN_PARAM, MAX_PARAM = -2147483648, 2147483647
+
+PARAM_RANGE = {
+ 'circwindow': (100, 1000),
+ 'CircuitPriorityHalflifeMsec': (-1, MAX_PARAM),
+ 'perconnbwrate': (-1, MAX_PARAM),
+ 'perconnbwburst': (-1, MAX_PARAM),
+ 'refuseunknownexits': (0, 1),
+ 'bwweightscale': (1, MAX_PARAM),
+ 'cbtdisabled': (0, 1),
+ 'cbtnummodes': (1, 20),
+ 'cbtrecentcount': (3, 1000),
+ 'cbtmaxtimeouts': (3, 10000),
+ 'cbtmincircs': (1, 10000),
+ 'cbtquantile': (10, 99),
+ 'cbtclosequantile': (MIN_PARAM, 99),
+ 'cbttestfreq': (1, MAX_PARAM),
+ 'cbtmintimeout': (500, MAX_PARAM),
+ 'UseOptimisticData': (0, 1),
+ 'Support022HiddenServices': (0, 1),
+ 'usecreatefast': (0, 1),
+ 'UseNTorHandshake': (0, 1),
+ 'FastFlagMinThreshold': (4, MAX_PARAM),
+ 'NumDirectoryGuards': (0, 10),
+ 'NumEntryGuards': (1, 10),
+ 'GuardLifetime': (2592000, 157766400), # min: 30 days, max: 1826 days
+ 'NumNTorsPerTAP': (1, 100000),
+ 'AllowNonearlyExtend': (0, 1),
+}
+
+
+def _parse_file(document_file, document_type = None, validate = False, is_microdescriptor = False, document_handler = DocumentHandler.ENTRIES, **kwargs):
+ """
+ Parses a network status and iterates over the RouterStatusEntry in it. The
+ document that these instances reference have an empty 'routers' attribute to
+ allow for limited memory usage.
+
+ :param file document_file: file with network status document content
+ :param class document_type: NetworkStatusDocument subclass
+ :param bool validate: checks the validity of the document's contents if
+ **True**, skips these checks otherwise
+ :param bool is_microdescriptor: **True** if this is for a microdescriptor
+ consensus, **False** otherwise
+ :param stem.descriptor.__init__.DocumentHandler document_handler: method in
+ which to parse :class:`~stem.descriptor.networkstatus.NetworkStatusDocument`
+ :param dict kwargs: additional arguments for the descriptor constructor
+
+ :returns: :class:`stem.descriptor.networkstatus.NetworkStatusDocument` object
+
+ :raises:
+ * **ValueError** if the document_version is unrecognized or the contents is
+ malformed and validate is **True**
+ * **IOError** if the file can't be read
+ """
+
+ # we can't properly default this since NetworkStatusDocumentV3 isn't defined yet
+
+ if document_type is None:
+ document_type = NetworkStatusDocumentV3
+
+ if document_type == NetworkStatusDocumentV2:
+ document_type, router_type = NetworkStatusDocumentV2, RouterStatusEntryV2
+ elif document_type == NetworkStatusDocumentV3:
+ router_type = RouterStatusEntryMicroV3 if is_microdescriptor else RouterStatusEntryV3
+ elif document_type == BridgeNetworkStatusDocument:
+ document_type, router_type = BridgeNetworkStatusDocument, RouterStatusEntryV2
+ else:
+ raise ValueError("Document type %i isn't recognized (only able to parse v2, v3, and bridge)" % document_type)
+
+ if document_handler == DocumentHandler.DOCUMENT:
+ yield document_type(document_file.read(), validate, **kwargs)
+ return
+
+ # getting the document without the routers section
+
+ header = _read_until_keywords((ROUTERS_START, FOOTER_START, V2_FOOTER_START), document_file)
+
+ if header and header[0].startswith(b'@type'):
+ header = header[1:]
+
+ routers_start = document_file.tell()
+ _read_until_keywords((FOOTER_START, V2_FOOTER_START), document_file, skip = True)
+ routers_end = document_file.tell()
+
+ footer = document_file.readlines()
+ document_content = bytes.join(b'', header + footer)
+
+ if document_handler == DocumentHandler.BARE_DOCUMENT:
+ yield document_type(document_content, validate, **kwargs)
+ elif document_handler == DocumentHandler.ENTRIES:
+ desc_iterator = stem.descriptor.router_status_entry._parse_file(
+ document_file,
+ validate,
+ entry_class = router_type,
+ entry_keyword = ROUTERS_START,
+ start_position = routers_start,
+ end_position = routers_end,
+ extra_args = (document_type(document_content, validate),),
+ **kwargs
+ )
+
+ for desc in desc_iterator:
+ yield desc
+ else:
+ raise ValueError('Unrecognized document_handler: %s' % document_handler)
+
+
+def _parse_file_key_certs(certificate_file, validate = False):
+ """
+ Parses a file containing one or more authority key certificates.
+
+ :param file certificate_file: file with key certificates
+ :param bool validate: checks the validity of the certificate's contents if
+ **True**, skips these checks otherwise
+
+ :returns: iterator for :class:`stem.descriptor.networkstatus.KeyCertificate`
+ instance in the file
+
+ :raises:
+ * **ValueError** if the key certificate content is invalid and validate is
+ **True**
+ * **IOError** if the file can't be read
+ """
+
+ while True:
+ keycert_content = _read_until_keywords('dir-key-certification', certificate_file)
+
+ # we've reached the 'router-signature', now include the pgp style block
+ block_end_prefix = PGP_BLOCK_END.split(' ', 1)[0]
+ keycert_content += _read_until_keywords(block_end_prefix, certificate_file, True)
+
+ if keycert_content:
+ yield stem.descriptor.networkstatus.KeyCertificate(bytes.join(b'', keycert_content), validate = validate)
+ else:
+ break # done parsing file
+
+
+class NetworkStatusDocument(Descriptor):
+ """
+ Common parent for network status documents.
+ """
+
+
+def _parse_version_line(keyword, attribute, expected_version):
+ def _parse(descriptor, entries):
+ value = _value(keyword, entries)
+
+ if not value.isdigit():
+ raise ValueError('Document has a non-numeric version: %s %s' % (keyword, value))
+
+ setattr(descriptor, attribute, int(value))
+
+ if int(value) != expected_version:
+ raise ValueError("Expected a version %i document, but got version '%s' instead" % (expected_version, value))
+
+ return _parse
+
+
+def _parse_dir_source_line(descriptor, entries):
+ value = _value('dir-source', entries)
+ dir_source_comp = value.split()
+
+ if len(dir_source_comp) < 3:
+ raise ValueError("The 'dir-source' line of a v2 network status document must have three values: dir-source %s" % value)
+
+ if not dir_source_comp[0]:
+ # https://trac.torproject.org/7055
+ raise ValueError("Authority's hostname can't be blank: dir-source %s" % value)
+ elif not stem.util.connection.is_valid_ipv4_address(dir_source_comp[1]):
+ raise ValueError("Authority's address isn't a valid IPv4 address: %s" % dir_source_comp[1])
+ elif not stem.util.connection.is_valid_port(dir_source_comp[2], allow_zero = True):
+ raise ValueError("Authority's DirPort is invalid: %s" % dir_source_comp[2])
+
+ descriptor.hostname = dir_source_comp[0]
+ descriptor.address = dir_source_comp[1]
+ descriptor.dir_port = None if dir_source_comp[2] == '0' else int(dir_source_comp[2])
+
+
+_parse_network_status_version_line = _parse_version_line('network-status-version', 'version', 2)
+_parse_fingerprint_line = _parse_forty_character_hex('fingerprint', 'fingerprint')
+_parse_contact_line = _parse_simple_line('contact', 'contact')
+_parse_dir_signing_key_line = _parse_key_block('dir-signing-key', 'signing_key', 'RSA PUBLIC KEY')
+_parse_client_versions_line = lambda descriptor, entries: setattr(descriptor, 'client_versions', _value('client-versions', entries).split(','))
+_parse_server_versions_line = lambda descriptor, entries: setattr(descriptor, 'server_versions', _value('server-versions', entries).split(','))
+_parse_published_line = _parse_timestamp_line('published', 'published')
+_parse_dir_options_line = lambda descriptor, entries: setattr(descriptor, 'options', _value('dir-options', entries).split())
+_parse_directory_signature_line = _parse_key_block('directory-signature', 'signature', 'SIGNATURE', value_attribute = 'signing_authority')
+
+
+class NetworkStatusDocumentV2(NetworkStatusDocument):
+ """
+ Version 2 network status document. These have been deprecated and are no
+ longer generated by Tor.
+
+ :var dict routers: fingerprints to :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV2`
+ contained in the document
+
+ :var int version: **\*** document version
+
+ :var str hostname: **\*** hostname of the authority
+ :var str address: **\*** authority's IP address
+ :var int dir_port: **\*** authority's DirPort
+ :var str fingerprint: **\*** authority's fingerprint
+ :var str contact: **\*** authority's contact information
+ :var str signing_key: **\*** authority's public signing key
+
+ :var list client_versions: list of recommended client tor version strings
+ :var list server_versions: list of recommended server tor version strings
+ :var datetime published: **\*** time when the document was published
+ :var list options: **\*** list of things that this authority decides
+
+ :var str signing_authority: **\*** name of the authority signing the document
+ :var str signature: **\*** authority's signature for the document
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as **None** if undefined
+ """
+
+ ATTRIBUTES = {
+ 'version': (None, _parse_network_status_version_line),
+ 'hostname': (None, _parse_dir_source_line),
+ 'address': (None, _parse_dir_source_line),
+ 'dir_port': (None, _parse_dir_source_line),
+ 'fingerprint': (None, _parse_fingerprint_line),
+ 'contact': (None, _parse_contact_line),
+ 'signing_key': (None, _parse_dir_signing_key_line),
+
+ 'client_versions': ([], _parse_client_versions_line),
+ 'server_versions': ([], _parse_server_versions_line),
+ 'published': (None, _parse_published_line),
+ 'options': ([], _parse_dir_options_line),
+
+ 'signing_authority': (None, _parse_directory_signature_line),
+ 'signatures': (None, _parse_directory_signature_line),
+ }
+
+ PARSER_FOR_LINE = {
+ 'network-status-version': _parse_network_status_version_line,
+ 'dir-source': _parse_dir_source_line,
+ 'fingerprint': _parse_fingerprint_line,
+ 'contact': _parse_contact_line,
+ 'dir-signing-key': _parse_dir_signing_key_line,
+ 'client-versions': _parse_client_versions_line,
+ 'server-versions': _parse_server_versions_line,
+ 'published': _parse_published_line,
+ 'dir-options': _parse_dir_options_line,
+ 'directory-signature': _parse_directory_signature_line,
+ }
+
+ def __init__(self, raw_content, validate = False):
+ super(NetworkStatusDocumentV2, self).__init__(raw_content, lazy_load = not validate)
+
+ # Splitting the document from the routers. Unlike v3 documents we're not
+ # bending over backwards on the validation by checking the field order or
+ # that header/footer attributes aren't in the wrong section. This is a
+ # deprecated descriptor type - patches welcome if you want those checks.
+
+ document_file = io.BytesIO(raw_content)
+ document_content = bytes.join(b'', _read_until_keywords((ROUTERS_START, V2_FOOTER_START), document_file))
+
+ router_iter = stem.descriptor.router_status_entry._parse_file(
+ document_file,
+ validate,
+ entry_class = RouterStatusEntryV2,
+ entry_keyword = ROUTERS_START,
+ section_end_keywords = (V2_FOOTER_START,),
+ extra_args = (self,),
+ )
+
+ self.routers = dict((desc.fingerprint, desc) for desc in router_iter)
+
+ entries = _get_descriptor_components(document_content + b'\n' + document_file.read(), validate)
+
+ if validate:
+ self._check_constraints(entries)
+ self._parse(entries, validate)
+
+ # 'client-versions' and 'server-versions' are only required if 'Versions'
+ # is among the options
+
+ if 'Versions' in self.options and not ('client-versions' in entries and 'server-versions' in entries):
+ raise ValueError("Version 2 network status documents must have a 'client-versions' and 'server-versions' when 'Versions' is listed among its dir-options:\n%s" % str(self))
+ else:
+ self._entries = entries
+
+ def _check_constraints(self, entries):
+ required_fields = [field for (field, is_mandatory) in NETWORK_STATUS_V2_FIELDS if is_mandatory]
+ for keyword in required_fields:
+ if keyword not in entries:
+ raise ValueError("Network status document (v2) must have a '%s' line:\n%s" % (keyword, str(self)))
+
+ # all recognized fields can only appear once
+ single_fields = [field for (field, _) in NETWORK_STATUS_V2_FIELDS]
+ for keyword in single_fields:
+ if keyword in entries and len(entries[keyword]) > 1:
+ raise ValueError("Network status document (v2) can only have a single '%s' line, got %i:\n%s" % (keyword, len(entries[keyword]), str(self)))
+
+ if 'network-status-version' != list(entries.keys())[0]:
+ raise ValueError("Network status document (v2) are expected to start with a 'network-status-version' line:\n%s" % str(self))
+
+
+def _parse_header_network_status_version_line(descriptor, entries):
+ # "network-status-version" version
+
+ value = _value('network-status-version', entries)
+
+ if ' ' in value:
+ version, flavor = value.split(' ', 1)
+ else:
+ version, flavor = value, None
+
+ if not version.isdigit():
+ raise ValueError('Network status document has a non-numeric version: network-status-version %s' % value)
+
+ descriptor.version = int(version)
+ descriptor.version_flavor = flavor
+ descriptor.is_microdescriptor = flavor == 'microdesc'
+
+ if descriptor.version != 3:
+ raise ValueError("Expected a version 3 network status document, got version '%s' instead" % descriptor.version)
+
+
+def _parse_header_vote_status_line(descriptor, entries):
+ # "vote-status" type
+ #
+ # The consensus-method and consensus-methods fields are optional since
+ # they weren't included in version 1. Setting a default now that we
+ # know if we're a vote or not.
+
+ value = _value('vote-status', entries)
+
+ if value == 'consensus':
+ descriptor.is_consensus, descriptor.is_vote = True, False
+ elif value == 'vote':
+ descriptor.is_consensus, descriptor.is_vote = False, True
+ else:
+ raise ValueError("A network status document's vote-status line can only be 'consensus' or 'vote', got '%s' instead" % value)
+
+
+def _parse_header_consensus_methods_line(descriptor, entries):
+ # "consensus-methods" IntegerList
+
+ if descriptor._lazy_loading and descriptor.is_vote:
+ descriptor.consensus_methods = [1]
+
+ value, consensus_methods = _value('consensus-methods', entries), []
+
+ for entry in value.split(' '):
+ if not entry.isdigit():
+ raise ValueError("A network status document's consensus-methods must be a list of integer values, but was '%s'" % value)
+
+ consensus_methods.append(int(entry))
+
+ descriptor.consensus_methods = consensus_methods
+
+
+def _parse_header_consensus_method_line(descriptor, entries):
+ # "consensus-method" Integer
+
+ if descriptor._lazy_loading and descriptor.is_consensus:
+ descriptor.consensus_method = 1
+
+ value = _value('consensus-method', entries)
+
+ if not value.isdigit():
+ raise ValueError("A network status document's consensus-method must be an integer, but was '%s'" % value)
+
+ descriptor.consensus_method = int(value)
+
+
+def _parse_header_voting_delay_line(descriptor, entries):
+ # "voting-delay" VoteSeconds DistSeconds
+
+ value = _value('voting-delay', entries)
+ value_comp = value.split(' ')
+
+ if len(value_comp) == 2 and value_comp[0].isdigit() and value_comp[1].isdigit():
+ descriptor.vote_delay = int(value_comp[0])
+ descriptor.dist_delay = int(value_comp[1])
+ else:
+ raise ValueError("A network status document's 'voting-delay' line must be a pair of integer values, but was '%s'" % value)
+
+
+def _parse_versions_line(keyword, attribute):
+ def _parse(descriptor, entries):
+ value, entries = _value(keyword, entries), []
+
+ for entry in value.split(','):
+ try:
+ entries.append(stem.version._get_version(entry))
+ except ValueError:
+ raise ValueError("Network status document's '%s' line had '%s', which isn't a parsable tor version: %s %s" % (keyword, entry, keyword, value))
+
+ setattr(descriptor, attribute, entries)
+
+ return _parse
+
+
+def _parse_header_flag_thresholds_line(descriptor, entries):
+ # "flag-thresholds" SP THRESHOLDS
+
+ value, thresholds = _value('flag-thresholds', entries).strip(), {}
+
+ if value:
+ for entry in value.split(' '):
+ if '=' not in entry:
+ raise ValueError("Network status document's 'flag-thresholds' line is expected to be space separated key=value mappings, got: flag-thresholds %s" % value)
+
+ entry_key, entry_value = entry.split('=', 1)
+
+ try:
+ if entry_value.endswith('%'):
+ # opting for string manipulation rather than just
+ # 'float(entry_value) / 100' because floating point arithmetic
+ # will lose precision
+
+ thresholds[entry_key] = float('0.' + entry_value[:-1].replace('.', '', 1))
+ elif '.' in entry_value:
+ thresholds[entry_key] = float(entry_value)
+ else:
+ thresholds[entry_key] = int(entry_value)
+ except ValueError:
+ raise ValueError("Network status document's 'flag-thresholds' line is expected to have float values, got: flag-thresholds %s" % value)
+
+ descriptor.flag_thresholds = thresholds
+
+
+def _parse_header_parameters_line(descriptor, entries):
+ # "params" [Parameters]
+ # Parameter ::= Keyword '=' Int32
+ # Int32 ::= A decimal integer between -2147483648 and 2147483647.
+ # Parameters ::= Parameter | Parameters SP Parameter
+
+ if descriptor._lazy_loading:
+ descriptor.params = dict(DEFAULT_PARAMS) if descriptor._default_params else {}
+
+ value = _value('params', entries)
+
+ # should only appear in consensus-method 7 or later
+
+ if not descriptor.meets_consensus_method(7):
+ raise ValueError("A network status document's 'params' line should only appear in consensus-method 7 or later")
+
+ if value != '':
+ descriptor.params = _parse_int_mappings('params', value, True)
+ descriptor._check_params_constraints()
+
+
+def _parse_directory_footer_line(descriptor, entries):
+ # nothing to parse, simply checking that we don't have a value
+
+ value = _value('directory-footer', entries)
+
+ if value:
+ raise ValueError("A network status document's 'directory-footer' line shouldn't have any content, got 'directory-footer %s'" % value)
+
+
+def _parse_footer_directory_signature_line(descriptor, entries):
+ signatures = []
+
+ for sig_value, block_type, block_contents in entries['directory-signature']:
+ if sig_value.count(' ') not in (1, 2):
+ raise ValueError("Authority signatures in a network status document are expected to be of the form 'directory-signature [METHOD] FINGERPRINT KEY_DIGEST', received: %s" % sig_value)
+
+ if not block_contents or block_type != 'SIGNATURE':
+ raise ValueError("'directory-signature' should be followed by a SIGNATURE block, but was a %s" % block_type)
+
+ if sig_value.count(' ') == 1:
+ method = 'sha1' # default if none was provided
+ fingerprint, key_digest = sig_value.split(' ', 1)
+ else:
+ method, fingerprint, key_digest = sig_value.split(' ', 2)
+
+ signatures.append(DocumentSignature(method, fingerprint, key_digest, block_contents, True))
+
+ descriptor.signatures = signatures
+
+
+def _parse_package_line(descriptor, entries):
+ package_versions = []
+
+ for value, _, _ in entries['package']:
+ value_comp = value.split()
+
+ if len(value_comp) < 3:
+ raise ValueError("'package' must at least have a 'PackageName Version URL': %s" % value)
+
+ name, version, url = value_comp[:3]
+ digests = {}
+
+ for digest_entry in value_comp[3:]:
+ if '=' not in digest_entry:
+ raise ValueError("'package' digest entries should be 'key=value' pairs: %s" % value)
+
+ key, value = digest_entry.split('=', 1)
+ digests[key] = value
+
+ package_versions.append(PackageVersion(name, version, url, digests))
+
+ descriptor.packages = package_versions
+
+
+_parse_header_valid_after_line = _parse_timestamp_line('valid-after', 'valid_after')
+_parse_header_fresh_until_line = _parse_timestamp_line('fresh-until', 'fresh_until')
+_parse_header_valid_until_line = _parse_timestamp_line('valid-until', 'valid_until')
+_parse_header_client_versions_line = _parse_versions_line('client-versions', 'client_versions')
+_parse_header_server_versions_line = _parse_versions_line('server-versions', 'server_versions')
+_parse_header_known_flags_line = lambda descriptor, entries: setattr(descriptor, 'known_flags', [entry for entry in _value('known-flags', entries).split(' ') if entry])
+_parse_footer_bandwidth_weights_line = lambda descriptor, entries: setattr(descriptor, 'bandwidth_weights', _parse_int_mappings('bandwidth-weights', _value('bandwidth-weights', entries), True))
+
+
+class NetworkStatusDocumentV3(NetworkStatusDocument):
+ """
+ Version 3 network status document. This could be either a vote or consensus.
+
+ :var tuple routers: :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3`
+ contained in the document
+
+ :var int version: **\*** document version
+ :var str version_flavor: **\*** flavor associated with the document (such as 'microdesc')
+ :var bool is_consensus: **\*** **True** if the document is a consensus
+ :var bool is_vote: **\*** **True** if the document is a vote
+ :var bool is_microdescriptor: **\*** **True** if this is a microdescriptor
+ flavored document, **False** otherwise
+ :var datetime valid_after: **\*** time when the consensus became valid
+ :var datetime fresh_until: **\*** time when the next consensus should be produced
+ :var datetime valid_until: **\*** time when this consensus becomes obsolete
+ :var int vote_delay: **\*** number of seconds allowed for collecting votes
+ from all authorities
+ :var int dist_delay: **\*** number of seconds allowed for collecting
+ signatures from all authorities
+ :var list client_versions: list of recommended client tor versions
+ :var list server_versions: list of recommended server tor versions
+ :var list packages: **\*** list of :data:`~stem.descriptor.networkstatus.PackageVersion` entries
+ :var list known_flags: **\*** list of :data:`~stem.Flag` for the router's flags
+ :var dict params: **\*** dict of parameter(**str**) => value(**int**) mappings
+ :var list directory_authorities: **\*** list of :class:`~stem.descriptor.networkstatus.DirectoryAuthority`
+ objects that have generated this document
+ :var list signatures: **\*** :class:`~stem.descriptor.networkstatus.DocumentSignature`
+ of the authorities that have signed the document
+
+ **Consensus Attributes:**
+
+ :var int consensus_method: method version used to generate this consensus
+ :var dict bandwidth_weights: dict of weight(str) => value(int) mappings
+
+ **Vote Attributes:**
+
+ :var list consensus_methods: list of ints for the supported method versions
+ :var datetime published: time when the document was published
+ :var dict flag_thresholds: **\*** mapping of internal performance thresholds used while making the vote, values are **ints** or **floats**
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as None if undefined
+
+ .. versionchanged:: 1.4.0
+ Added the packages attribute.
+ """
+
+ ATTRIBUTES = {
+ 'version': (None, _parse_header_network_status_version_line),
+ 'version_flavor': (None, _parse_header_network_status_version_line),
+ 'is_consensus': (True, _parse_header_vote_status_line),
+ 'is_vote': (False, _parse_header_vote_status_line),
+ 'is_microdescriptor': (False, _parse_header_network_status_version_line),
+ 'consensus_methods': ([], _parse_header_consensus_methods_line),
+ 'published': (None, _parse_published_line),
+ 'consensus_method': (None, _parse_header_consensus_method_line),
+ 'valid_after': (None, _parse_header_valid_after_line),
+ 'fresh_until': (None, _parse_header_fresh_until_line),
+ 'valid_until': (None, _parse_header_valid_until_line),
+ 'vote_delay': (None, _parse_header_voting_delay_line),
+ 'dist_delay': (None, _parse_header_voting_delay_line),
+ 'client_versions': ([], _parse_header_client_versions_line),
+ 'server_versions': ([], _parse_header_server_versions_line),
+ 'packages': ([], _parse_package_line),
+ 'known_flags': ([], _parse_header_known_flags_line),
+ 'flag_thresholds': ({}, _parse_header_flag_thresholds_line),
+ 'params': ({}, _parse_header_parameters_line),
+
+ 'signatures': ([], _parse_footer_directory_signature_line),
+ 'bandwidth_weights': ({}, _parse_footer_bandwidth_weights_line),
+ }
+
+ HEADER_PARSER_FOR_LINE = {
+ 'network-status-version': _parse_header_network_status_version_line,
+ 'vote-status': _parse_header_vote_status_line,
+ 'consensus-methods': _parse_header_consensus_methods_line,
+ 'consensus-method': _parse_header_consensus_method_line,
+ 'published': _parse_published_line,
+ 'valid-after': _parse_header_valid_after_line,
+ 'fresh-until': _parse_header_fresh_until_line,
+ 'valid-until': _parse_header_valid_until_line,
+ 'voting-delay': _parse_header_voting_delay_line,
+ 'client-versions': _parse_header_client_versions_line,
+ 'server-versions': _parse_header_server_versions_line,
+ 'package': _parse_package_line,
+ 'known-flags': _parse_header_known_flags_line,
+ 'flag-thresholds': _parse_header_flag_thresholds_line,
+ 'params': _parse_header_parameters_line,
+ }
+
+ FOOTER_PARSER_FOR_LINE = {
+ 'directory-footer': _parse_directory_footer_line,
+ 'bandwidth-weights': _parse_footer_bandwidth_weights_line,
+ 'directory-signature': _parse_footer_directory_signature_line,
+ }
+
+ def __init__(self, raw_content, validate = False, default_params = True):
+ """
+ Parse a v3 network status document.
+
+ :param str raw_content: raw network status document data
+ :param bool validate: **True** if the document is to be validated, **False** otherwise
+ :param bool default_params: includes defaults in our params dict, otherwise
+ it just contains values from the document
+
+ :raises: **ValueError** if the document is invalid
+ """
+
+ super(NetworkStatusDocumentV3, self).__init__(raw_content, lazy_load = not validate)
+ document_file = io.BytesIO(raw_content)
+
+ self._default_params = default_params
+ self._header(document_file, validate)
+
+ self.directory_authorities = tuple(stem.descriptor.router_status_entry._parse_file(
+ document_file,
+ validate,
+ entry_class = DirectoryAuthority,
+ entry_keyword = AUTH_START,
+ section_end_keywords = (ROUTERS_START, FOOTER_START, V2_FOOTER_START),
+ extra_args = (self.is_vote,),
+ ))
+
+ if validate and self.is_vote and len(self.directory_authorities) != 1:
+ raise ValueError('Votes should only have an authority entry for the one that issued it, got %i: %s' % (len(self.directory_authorities), self.directory_authorities))
+
+ router_iter = stem.descriptor.router_status_entry._parse_file(
+ document_file,
+ validate,
+ entry_class = RouterStatusEntryMicroV3 if self.is_microdescriptor else RouterStatusEntryV3,
+ entry_keyword = ROUTERS_START,
+ section_end_keywords = (FOOTER_START, V2_FOOTER_START),
+ extra_args = (self,),
+ )
+
+ self.routers = dict((desc.fingerprint, desc) for desc in router_iter)
+ self._footer(document_file, validate)
+
+ def get_unrecognized_lines(self):
+ if self._lazy_loading:
+ self._parse(self._header_entries, False, parser_for_line = self.HEADER_PARSER_FOR_LINE)
+ self._parse(self._footer_entries, False, parser_for_line = self.FOOTER_PARSER_FOR_LINE)
+ self._lazy_loading = False
+
+ return super(NetworkStatusDocumentV3, self).get_unrecognized_lines()
+
+ def meets_consensus_method(self, method):
+ """
+ Checks if we meet the given consensus-method. This works for both votes and
+ consensuses, checking our 'consensus-method' and 'consensus-methods'
+ entries.
+
+ :param int method: consensus-method to check for
+
+ :returns: **True** if we meet the given consensus-method, and **False** otherwise
+ """
+
+ if self.consensus_method is not None:
+ return self.consensus_method >= method
+ elif self.consensus_methods is not None:
+ return bool([x for x in self.consensus_methods if x >= method])
+ else:
+ return False # malformed document
+
+ def _compare(self, other, method):
+ if not isinstance(other, NetworkStatusDocumentV3):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def _header(self, document_file, validate):
+ content = bytes.join(b'', _read_until_keywords((AUTH_START, ROUTERS_START, FOOTER_START), document_file))
+ entries = _get_descriptor_components(content, validate)
+
+ if validate:
+ # all known header fields can only appear once except
+
+ for keyword, values in list(entries.items()):
+ if len(values) > 1 and keyword in HEADER_FIELDS and keyword != 'package':
+ raise ValueError("Network status documents can only have a single '%s' line, got %i" % (keyword, len(values)))
+
+ if self._default_params:
+ self.params = dict(DEFAULT_PARAMS)
+
+ self._parse(entries, validate, parser_for_line = self.HEADER_PARSER_FOR_LINE)
+
+ _check_for_missing_and_disallowed_fields(self, entries, HEADER_STATUS_DOCUMENT_FIELDS)
+ _check_for_misordered_fields(entries, HEADER_FIELDS)
+
+ # default consensus_method and consensus_methods based on if we're a consensus or vote
+
+ if self.is_consensus and not self.consensus_method:
+ self.consensus_method = 1
+ elif self.is_vote and not self.consensus_methods:
+ self.consensus_methods = [1]
+ else:
+ self._header_entries = entries
+ self._entries.update(entries)
+
+ def _footer(self, document_file, validate):
+ entries = _get_descriptor_components(document_file.read(), validate)
+
+ if validate:
+ for keyword, values in list(entries.items()):
+ # all known footer fields can only appear once except...
+ # * 'directory-signature' in a consensus
+
+ if len(values) > 1 and keyword in FOOTER_FIELDS:
+ if not (keyword == 'directory-signature' and self.is_consensus):
+ raise ValueError("Network status documents can only have a single '%s' line, got %i" % (keyword, len(values)))
+
+ self._parse(entries, validate, parser_for_line = self.FOOTER_PARSER_FOR_LINE)
+
+ # Check that the footer has the right initial line. Prior to consensus
+ # method 9 it's a 'directory-signature' and after that footers start with
+ # 'directory-footer'.
+
+ if entries:
+ if self.meets_consensus_method(9):
+ if list(entries.keys())[0] != 'directory-footer':
+ raise ValueError("Network status document's footer should start with a 'directory-footer' line in consensus-method 9 or later")
+ else:
+ if list(entries.keys())[0] != 'directory-signature':
+ raise ValueError("Network status document's footer should start with a 'directory-signature' line prior to consensus-method 9")
+
+ _check_for_missing_and_disallowed_fields(self, entries, FOOTER_STATUS_DOCUMENT_FIELDS)
+ _check_for_misordered_fields(entries, FOOTER_FIELDS)
+ else:
+ self._footer_entries = entries
+ self._entries.update(entries)
+
+ def _check_params_constraints(self):
+ """
+ Checks that the params we know about are within their documented ranges.
+ """
+
+ for key, value in self.params.items():
+ minimum, maximum = PARAM_RANGE.get(key, (MIN_PARAM, MAX_PARAM))
+
+ # there's a few dynamic parameter ranges
+
+ if key == 'cbtclosequantile':
+ minimum = self.params.get('cbtquantile', minimum)
+ elif key == 'cbtinitialtimeout':
+ minimum = self.params.get('cbtmintimeout', minimum)
+
+ if value < minimum or value > maximum:
+ raise ValueError("'%s' value on the params line must be in the range of %i - %i, was %i" % (key, minimum, maximum, value))
+
+ def __hash__(self):
+ return hash(str(self).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
+
+
+def _check_for_missing_and_disallowed_fields(document, entries, fields):
+ """
+ Checks that we have mandatory fields for our type, and that we don't have
+ any fields exclusive to the other (ie, no vote-only fields appear in a
+ consensus or vice versa).
+
+ :param NetworkStatusDocumentV3 document: network status document
+ :param dict entries: ordered keyword/value mappings of the header or footer
+ :param list fields: expected field attributes (either
+ **HEADER_STATUS_DOCUMENT_FIELDS** or **FOOTER_STATUS_DOCUMENT_FIELDS**)
+
+ :raises: **ValueError** if we're missing mandatory fields or have fields we shouldn't
+ """
+
+ missing_fields, disallowed_fields = [], []
+
+ for field, in_votes, in_consensus, mandatory in fields:
+ if mandatory and ((document.is_consensus and in_consensus) or (document.is_vote and in_votes)):
+ # mandatory field, check that we have it
+ if field not in entries.keys():
+ missing_fields.append(field)
+ elif (document.is_consensus and not in_consensus) or (document.is_vote and not in_votes):
+ # field we shouldn't have, check that we don't
+ if field in entries.keys():
+ disallowed_fields.append(field)
+
+ if missing_fields:
+ raise ValueError('Network status document is missing mandatory field: %s' % ', '.join(missing_fields))
+
+ if disallowed_fields:
+ raise ValueError("Network status document has fields that shouldn't appear in this document type or version: %s" % ', '.join(disallowed_fields))
+
+
+def _check_for_misordered_fields(entries, expected):
+ """
+ To be valid a network status document's fiends need to appear in a specific
+ order. Checks that known fields appear in that order (unrecognized fields
+ are ignored).
+
+ :param dict entries: ordered keyword/value mappings of the header or footer
+ :param list expected: ordered list of expected fields (either
+ **HEADER_FIELDS** or **FOOTER_FIELDS**)
+
+ :raises: **ValueError** if entries aren't properly ordered
+ """
+
+ # Earlier validation has ensured that our fields either belong to our
+ # document type or are unknown. Remove the unknown fields since they
+ # reflect a spec change and can appear anywhere in the document.
+
+ actual = [field for field in entries.keys() if field in expected]
+
+ # Narrow the expected to just what we have. If the lists then match then the
+ # order's valid.
+
+ expected = [field for field in expected if field in actual]
+
+ if actual != expected:
+ actual_label = ', '.join(actual)
+ expected_label = ', '.join(expected)
+ raise ValueError("The fields in a section of the document are misordered. It should be '%s' but was '%s'" % (actual_label, expected_label))
+
+
+def _parse_int_mappings(keyword, value, validate):
+ # Parse a series of 'key=value' entries, checking the following:
+ # - values are integers
+ # - keys are sorted in lexical order
+
+ results, seen_keys = {}, []
+ for entry in value.split(' '):
+ try:
+ if '=' not in entry:
+ raise ValueError("must only have 'key=value' entries")
+
+ entry_key, entry_value = entry.split('=', 1)
+
+ try:
+ # the int() function accepts things like '+123', but we don't want to
+ if entry_value.startswith('+'):
+ raise ValueError()
+
+ entry_value = int(entry_value)
+ except ValueError:
+ raise ValueError("'%s' is a non-numeric value" % entry_value)
+
+ if validate:
+ # parameters should be in ascending order by their key
+ for prior_key in seen_keys:
+ if prior_key > entry_key:
+ raise ValueError('parameters must be sorted by their key')
+
+ results[entry_key] = entry_value
+ seen_keys.append(entry_key)
+ except ValueError as exc:
+ if not validate:
+ continue
+
+ raise ValueError("Unable to parse network status document's '%s' line (%s): %s'" % (keyword, exc, value))
+
+ return results
+
+
+def _parse_dirauth_source_line(descriptor, entries):
+ # "dir-source" nickname identity address IP dirport orport
+
+ value = _value('dir-source', entries)
+ dir_source_comp = value.split(' ')
+
+ if len(dir_source_comp) < 6:
+ raise ValueError("Authority entry's 'dir-source' line must have six values: dir-source %s" % value)
+
+ if not stem.util.tor_tools.is_valid_nickname(dir_source_comp[0].rstrip('-legacy')):
+ raise ValueError("Authority's nickname is invalid: %s" % dir_source_comp[0])
+ elif not stem.util.tor_tools.is_valid_fingerprint(dir_source_comp[1]):
+ raise ValueError("Authority's v3ident is invalid: %s" % dir_source_comp[1])
+ elif not dir_source_comp[2]:
+ # https://trac.torproject.org/7055
+ raise ValueError("Authority's hostname can't be blank: dir-source %s" % value)
+ elif not stem.util.connection.is_valid_ipv4_address(dir_source_comp[3]):
+ raise ValueError("Authority's address isn't a valid IPv4 address: %s" % dir_source_comp[3])
+ elif not stem.util.connection.is_valid_port(dir_source_comp[4], allow_zero = True):
+ raise ValueError("Authority's DirPort is invalid: %s" % dir_source_comp[4])
+ elif not stem.util.connection.is_valid_port(dir_source_comp[5]):
+ raise ValueError("Authority's ORPort is invalid: %s" % dir_source_comp[5])
+
+ descriptor.nickname = dir_source_comp[0]
+ descriptor.v3ident = dir_source_comp[1]
+ descriptor.hostname = dir_source_comp[2]
+ descriptor.address = dir_source_comp[3]
+ descriptor.dir_port = None if dir_source_comp[4] == '0' else int(dir_source_comp[4])
+ descriptor.or_port = int(dir_source_comp[5])
+ descriptor.is_legacy = descriptor.nickname.endswith('-legacy')
+
+
+_parse_legacy_dir_key_line = _parse_forty_character_hex('legacy-dir-key', 'legacy_dir_key')
+_parse_vote_digest_line = _parse_forty_character_hex('vote-digest', 'vote_digest')
+
+
+class DirectoryAuthority(Descriptor):
+ """
+ Directory authority information obtained from a v3 network status document.
+
+ Authorities can optionally use a legacy format. These are no longer found in
+ practice, but have the following differences...
+
+ * The authority's nickname ends with '-legacy'.
+ * There's no **contact** or **vote_digest** attribute.
+
+ :var str nickname: **\*** authority's nickname
+ :var str v3ident: **\*** identity key fingerprint used to sign votes and consensus
+ :var str hostname: **\*** hostname of the authority
+ :var str address: **\*** authority's IP address
+ :var int dir_port: **\*** authority's DirPort
+ :var int or_port: **\*** authority's ORPort
+ :var bool is_legacy: **\*** if the authority's using the legacy format
+ :var str contact: contact information, this is included if is_legacy is **False**
+
+ **Consensus Attributes:**
+
+ :var str vote_digest: digest of the authority that contributed to the consensus, this is included if is_legacy is **False**
+
+ **Vote Attributes:**
+
+ :var str legacy_dir_key: fingerprint of and obsolete identity key
+ :var stem.descriptor.networkstatus.KeyCertificate key_certificate: **\***
+ authority's key certificate
+
+ **\*** mandatory attribute
+
+ .. versionchanged:: 1.4.0
+ Renamed our 'fingerprint' attribute to 'v3ident' (prior attribute exists
+ for backward compatability, but is deprecated).
+ """
+
+ ATTRIBUTES = {
+ 'nickname': (None, _parse_dirauth_source_line),
+ 'v3ident': (None, _parse_dirauth_source_line),
+ 'hostname': (None, _parse_dirauth_source_line),
+ 'address': (None, _parse_dirauth_source_line),
+ 'dir_port': (None, _parse_dirauth_source_line),
+ 'or_port': (None, _parse_dirauth_source_line),
+ 'is_legacy': (False, _parse_dirauth_source_line),
+ 'contact': (None, _parse_contact_line),
+ 'vote_digest': (None, _parse_vote_digest_line),
+ 'legacy_dir_key': (None, _parse_legacy_dir_key_line),
+ }
+
+ PARSER_FOR_LINE = {
+ 'dir-source': _parse_dirauth_source_line,
+ 'contact': _parse_contact_line,
+ 'legacy-dir-key': _parse_legacy_dir_key_line,
+ 'vote-digest': _parse_vote_digest_line,
+ }
+
+ def __init__(self, raw_content, validate = False, is_vote = False):
+ """
+ Parse a directory authority entry in a v3 network status document.
+
+ :param str raw_content: raw directory authority entry information
+ :param bool validate: checks the validity of the content if True, skips
+ these checks otherwise
+ :param bool is_vote: True if this is for a vote, False if it's for a consensus
+
+ :raises: ValueError if the descriptor data is invalid
+ """
+
+ super(DirectoryAuthority, self).__init__(raw_content, lazy_load = not validate)
+ content = stem.util.str_tools._to_unicode(raw_content)
+
+ # separate the directory authority entry from its key certificate
+ key_div = content.find('\ndir-key-certificate-version')
+
+ if key_div != -1:
+ self.key_certificate = KeyCertificate(content[key_div + 1:], validate)
+ content = content[:key_div + 1]
+ else:
+ self.key_certificate = None
+
+ entries = _get_descriptor_components(content, validate)
+
+ if validate and 'dir-source' != list(entries.keys())[0]:
+ raise ValueError("Authority entries are expected to start with a 'dir-source' line:\n%s" % (content))
+
+ # check that we have mandatory fields
+
+ if validate:
+ is_legacy, dir_source_entry = False, entries.get('dir-source')
+
+ if dir_source_entry:
+ is_legacy = dir_source_entry[0][0].split()[0].endswith('-legacy')
+
+ required_fields, excluded_fields = ['dir-source'], []
+
+ if not is_legacy:
+ required_fields += ['contact']
+
+ if is_vote:
+ if not self.key_certificate:
+ raise ValueError('Authority votes must have a key certificate:\n%s' % content)
+
+ excluded_fields += ['vote-digest']
+ elif not is_vote:
+ if self.key_certificate:
+ raise ValueError("Authority consensus entries shouldn't have a key certificate:\n%s" % content)
+
+ if not is_legacy:
+ required_fields += ['vote-digest']
+
+ excluded_fields += ['legacy-dir-key']
+
+ for keyword in required_fields:
+ if keyword not in entries:
+ raise ValueError("Authority entries must have a '%s' line:\n%s" % (keyword, content))
+
+ for keyword in entries:
+ if keyword in excluded_fields:
+ type_label = 'votes' if is_vote else 'consensus entries'
+ raise ValueError("Authority %s shouldn't have a '%s' line:\n%s" % (type_label, keyword, content))
+
+ # all known attributes can only appear at most once
+ for keyword, values in list(entries.items()):
+ if len(values) > 1 and keyword in ('dir-source', 'contact', 'legacy-dir-key', 'vote-digest'):
+ raise ValueError("Authority entries can only have a single '%s' line, got %i:\n%s" % (keyword, len(values), content))
+
+ self._parse(entries, validate)
+ else:
+ self._entries = entries
+
+ # TODO: Due to a bug we had a 'fingerprint' rather than 'v3ident' attribute
+ # for a long while. Keeping this around for backward compatability, but
+ # this will be dropped in stem's 2.0 release.
+
+ self.fingerprint = self.v3ident
+
+ def _compare(self, other, method):
+ if not isinstance(other, DirectoryAuthority):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
+
+
+def _parse_dir_address_line(descriptor, entries):
+ # "dir-address" IPPort
+
+ value = _value('dir-address', entries)
+
+ if ':' not in value:
+ raise ValueError("Key certificate's 'dir-address' is expected to be of the form ADDRESS:PORT: dir-address %s" % value)
+
+ address, dirport = value.split(':', 1)
+
+ if not stem.util.connection.is_valid_ipv4_address(address):
+ raise ValueError("Key certificate's address isn't a valid IPv4 address: dir-address %s" % value)
+ elif not stem.util.connection.is_valid_port(dirport):
+ raise ValueError("Key certificate's dirport is invalid: dir-address %s" % value)
+
+ descriptor.address = address
+ descriptor.dir_port = int(dirport)
+
+
+_parse_dir_key_certificate_version_line = _parse_version_line('dir-key-certificate-version', 'version', 3)
+_parse_dir_key_published_line = _parse_timestamp_line('dir-key-published', 'published')
+_parse_dir_key_expires_line = _parse_timestamp_line('dir-key-expires', 'expires')
+_parse_identity_key_line = _parse_key_block('dir-identity-key', 'identity_key', 'RSA PUBLIC KEY')
+_parse_signing_key_line = _parse_key_block('dir-signing-key', 'signing_key', 'RSA PUBLIC KEY')
+_parse_dir_key_crosscert_line = _parse_key_block('dir-key-crosscert', 'crosscert', 'ID SIGNATURE')
+_parse_dir_key_certification_line = _parse_key_block('dir-key-certification', 'certification', 'SIGNATURE')
+
+
+class KeyCertificate(Descriptor):
+ """
+ Directory key certificate for a v3 network status document.
+
+ :var int version: **\*** version of the key certificate
+ :var str address: authority's IP address
+ :var int dir_port: authority's DirPort
+ :var str fingerprint: **\*** authority's fingerprint
+ :var str identity_key: **\*** long term authority identity key
+ :var datetime published: **\*** time when this key was generated
+ :var datetime expires: **\*** time after which this key becomes invalid
+ :var str signing_key: **\*** directory server's public signing key
+ :var str crosscert: signature made using certificate's signing key
+ :var str certification: **\*** signature of this key certificate signed with
+ the identity key
+
+ **\*** mandatory attribute
+ """
+
+ ATTRIBUTES = {
+ 'version': (None, _parse_dir_key_certificate_version_line),
+ 'address': (None, _parse_dir_address_line),
+ 'dir_port': (None, _parse_dir_address_line),
+ 'fingerprint': (None, _parse_fingerprint_line),
+ 'identity_key': (None, _parse_identity_key_line),
+ 'published': (None, _parse_dir_key_published_line),
+ 'expires': (None, _parse_dir_key_expires_line),
+ 'signing_key': (None, _parse_signing_key_line),
+ 'crosscert': (None, _parse_dir_key_crosscert_line),
+ 'certification': (None, _parse_dir_key_certification_line),
+ }
+
+ PARSER_FOR_LINE = {
+ 'dir-key-certificate-version': _parse_dir_key_certificate_version_line,
+ 'dir-address': _parse_dir_address_line,
+ 'fingerprint': _parse_fingerprint_line,
+ 'dir-key-published': _parse_dir_key_published_line,
+ 'dir-key-expires': _parse_dir_key_expires_line,
+ 'dir-identity-key': _parse_identity_key_line,
+ 'dir-signing-key': _parse_signing_key_line,
+ 'dir-key-crosscert': _parse_dir_key_crosscert_line,
+ 'dir-key-certification': _parse_dir_key_certification_line,
+ }
+
+ def __init__(self, raw_content, validate = False):
+ super(KeyCertificate, self).__init__(raw_content, lazy_load = not validate)
+ entries = _get_descriptor_components(raw_content, validate)
+
+ if validate:
+ if 'dir-key-certificate-version' != list(entries.keys())[0]:
+ raise ValueError("Key certificates must start with a 'dir-key-certificate-version' line:\n%s" % (raw_content))
+ elif 'dir-key-certification' != list(entries.keys())[-1]:
+ raise ValueError("Key certificates must end with a 'dir-key-certification' line:\n%s" % (raw_content))
+
+ # check that we have mandatory fields and that our known fields only
+ # appear once
+
+ for keyword, is_mandatory in KEY_CERTIFICATE_PARAMS:
+ if is_mandatory and keyword not in entries:
+ raise ValueError("Key certificates must have a '%s' line:\n%s" % (keyword, raw_content))
+
+ entry_count = len(entries.get(keyword, []))
+ if entry_count > 1:
+ raise ValueError("Key certificates can only have a single '%s' line, got %i:\n%s" % (keyword, entry_count, raw_content))
+
+ self._parse(entries, validate)
+ else:
+ self._entries = entries
+
+ def _compare(self, other, method):
+ if not isinstance(other, KeyCertificate):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
+
+
+class DocumentSignature(object):
+ """
+ Directory signature of a v3 network status document.
+
+ :var str method: algorithm used to make the signature
+ :var str identity: fingerprint of the authority that made the signature
+ :var str key_digest: digest of the signing key
+ :var str signature: document signature
+ :param bool validate: checks validity if **True**
+
+ :raises: **ValueError** if a validity check fails
+ """
+
+ def __init__(self, method, identity, key_digest, signature, validate = False):
+ # Checking that these attributes are valid. Technically the key
+ # digest isn't a fingerprint, but it has the same characteristics.
+
+ if validate:
+ if not stem.util.tor_tools.is_valid_fingerprint(identity):
+ raise ValueError('Malformed fingerprint (%s) in the document signature' % identity)
+
+ if not stem.util.tor_tools.is_valid_fingerprint(key_digest):
+ raise ValueError('Malformed key digest (%s) in the document signature' % key_digest)
+
+ self.method = method
+ self.identity = identity
+ self.key_digest = key_digest
+ self.signature = signature
+
+ def _compare(self, other, method):
+ if not isinstance(other, DocumentSignature):
+ return False
+
+ for attr in ('method', 'identity', 'key_digest', 'signature'):
+ if getattr(self, attr) != getattr(other, attr):
+ return method(getattr(self, attr), getattr(other, attr))
+
+ return method(True, True) # we're equal
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
+
+
+class BridgeNetworkStatusDocument(NetworkStatusDocument):
+ """
+ Network status document containing bridges. This is only available through
+ the metrics site.
+
+ :var tuple routers: :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV2`
+ contained in the document
+ :var datetime published: time when the document was published
+ """
+
+ def __init__(self, raw_content, validate = False):
+ super(BridgeNetworkStatusDocument, self).__init__(raw_content)
+
+ self.published = None
+
+ document_file = io.BytesIO(raw_content)
+ published_line = stem.util.str_tools._to_unicode(document_file.readline())
+
+ if published_line.startswith('published '):
+ published_line = published_line.split(' ', 1)[1].strip()
+
+ try:
+ self.published = stem.util.str_tools._parse_timestamp(published_line)
+ except ValueError:
+ if validate:
+ raise ValueError("Bridge network status document's 'published' time wasn't parsable: %s" % published_line)
+ elif validate:
+ raise ValueError("Bridge network status documents must start with a 'published' line:\n%s" % stem.util.str_tools._to_unicode(raw_content))
+
+ router_iter = stem.descriptor.router_status_entry._parse_file(
+ document_file,
+ validate,
+ entry_class = RouterStatusEntryV2,
+ extra_args = (self,),
+ )
+
+ self.routers = dict((desc.fingerprint, desc) for desc in router_iter)
diff --git a/ext/stem/descriptor/reader.py b/ext/stem/descriptor/reader.py
new file mode 100644
index 00000000..a96406de
--- /dev/null
+++ b/ext/stem/descriptor/reader.py
@@ -0,0 +1,574 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Utilities for reading descriptors from local directories and archives. This is
+mostly done through the :class:`~stem.descriptor.reader.DescriptorReader`
+class, which is an iterator for the descriptor data in a series of
+destinations. For example...
+
+::
+
+ my_descriptors = [
+ '/tmp/server-descriptors-2012-03.tar.bz2',
+ '/tmp/archived_descriptors/',
+ ]
+
+ # prints the contents of all the descriptor files
+ with DescriptorReader(my_descriptors) as reader:
+ for descriptor in reader:
+ print descriptor
+
+This ignores files that cannot be processed due to read errors or unparsable
+content. To be notified of skipped files you can register a listener with
+:func:`~stem.descriptor.reader.DescriptorReader.register_skip_listener`.
+
+The :class:`~stem.descriptor.reader.DescriptorReader` keeps track of the last
+modified timestamps for descriptor files that it has read so it can skip
+unchanged files if run again. This listing of processed files can also be
+persisted and applied to other
+:class:`~stem.descriptor.reader.DescriptorReader` instances. For example, the
+following prints descriptors as they're changed over the course of a minute,
+and picks up where it left off if run again...
+
+::
+
+ reader = DescriptorReader(['/tmp/descriptor_data'])
+
+ try:
+ processed_files = load_processed_files('/tmp/used_descriptors')
+ reader.set_processed_files(processed_files)
+ except: pass # could not load, maybe this is the first run
+
+ start_time = time.time()
+
+ while (time.time() - start_time) < 60:
+ # prints any descriptors that have changed since last checked
+ with reader:
+ for descriptor in reader:
+ print descriptor
+
+ time.sleep(1)
+
+ save_processed_files('/tmp/used_descriptors', reader.get_processed_files())
+
+**Module Overview:**
+
+::
+
+ load_processed_files - Loads a listing of processed files
+ save_processed_files - Saves a listing of processed files
+
+ DescriptorReader - Iterator for descriptor data on the local file system
+ |- get_processed_files - provides the listing of files that we've processed
+ |- set_processed_files - sets our tracking of the files we have processed
+ |- register_read_listener - adds a listener for when files are read
+ |- register_skip_listener - adds a listener that's notified of skipped files
+ |- start - begins reading descriptor data
+ |- stop - stops reading descriptor data
+ |- __enter__ / __exit__ - manages the descriptor reader thread in the context
+ +- __iter__ - iterates over descriptor data in unread files
+
+ FileSkipped - Base exception for a file that was skipped
+ |- AlreadyRead - We've already read a file with this last modified timestamp
+ |- ParsingFailure - Contents can't be parsed as descriptor data
+ |- UnrecognizedType - File extension indicates non-descriptor data
+ +- ReadFailed - Wraps an error that was raised while reading the file
+ +- FileMissing - File does not exist
+"""
+
+import mimetypes
+import os
+import tarfile
+import threading
+
+try:
+ import queue
+except ImportError:
+ import Queue as queue
+
+import stem.descriptor
+import stem.prereq
+import stem.util.system
+
+from stem import str_type
+
+# flag to indicate when the reader thread is out of descriptor files to read
+FINISHED = 'DONE'
+
+
+class FileSkipped(Exception):
+ "Base error when we can't provide descriptor data from a file."
+
+
+class AlreadyRead(FileSkipped):
+ """
+ Already read a file with this 'last modified' timestamp or later.
+
+ :param int last_modified: unix timestamp for when the file was last modified
+ :param int last_modified_when_read: unix timestamp for the modification time
+ when we last read this file
+ """
+
+ def __init__(self, last_modified, last_modified_when_read):
+ super(AlreadyRead, self).__init__('File has already been read since it was last modified. modification time: %s, last read: %s' % (last_modified, last_modified_when_read))
+ self.last_modified = last_modified
+ self.last_modified_when_read = last_modified_when_read
+
+
+class ParsingFailure(FileSkipped):
+ """
+ File contents could not be parsed as descriptor data.
+
+ :param ValueError exception: issue that arose when parsing
+ """
+
+ def __init__(self, parsing_exception):
+ super(ParsingFailure, self).__init__(parsing_exception)
+ self.exception = parsing_exception
+
+
+class UnrecognizedType(FileSkipped):
+ """
+ File doesn't contain descriptor data. This could either be due to its file
+ type or because it doesn't conform to a recognizable descriptor type.
+
+ :param tuple mime_type: the (type, encoding) tuple provided by mimetypes.guess_type()
+ """
+
+ def __init__(self, mime_type):
+ super(UnrecognizedType, self).__init__('Unrecognized mime type: %s (%s)' % mime_type)
+ self.mime_type = mime_type
+
+
+class ReadFailed(FileSkipped):
+ """
+ An IOError occurred while trying to read the file.
+
+ :param IOError exception: issue that arose when reading the file, **None** if
+ this arose due to the file not being present
+ """
+
+ def __init__(self, read_exception):
+ super(ReadFailed, self).__init__(read_exception)
+ self.exception = read_exception
+
+
+class FileMissing(ReadFailed):
+ 'File does not exist.'
+
+ def __init__(self):
+ super(FileMissing, self).__init__('File does not exist')
+
+
+def load_processed_files(path):
+ """
+ Loads a dictionary of 'path => last modified timestamp' mappings, as
+ persisted by :func:`~stem.descriptor.reader.save_processed_files`, from a
+ file.
+
+ :param str path: location to load the processed files dictionary from
+
+ :returns: **dict** of 'path (**str**) => last modified unix timestamp
+ (**int**)' mappings
+
+ :raises:
+ * **IOError** if unable to read the file
+ * **TypeError** if unable to parse the file's contents
+ """
+
+ processed_files = {}
+
+ with open(path) as input_file:
+ for line in input_file.readlines():
+ line = line.strip()
+
+ if not line:
+ continue # skip blank lines
+
+ if ' ' not in line:
+ raise TypeError('Malformed line: %s' % line)
+
+ path, timestamp = line.rsplit(' ', 1)
+
+ if not os.path.isabs(path):
+ raise TypeError("'%s' is not an absolute path" % path)
+ elif not timestamp.isdigit():
+ raise TypeError("'%s' is not an integer timestamp" % timestamp)
+
+ processed_files[path] = int(timestamp)
+
+ return processed_files
+
+
+def save_processed_files(path, processed_files):
+ """
+ Persists a dictionary of 'path => last modified timestamp' mappings (as
+ provided by the DescriptorReader's
+ :func:`~stem.descriptor.reader.DescriptorReader.get_processed_files` method)
+ so that they can be loaded later and applied to another
+ :class:`~stem.descriptor.reader.DescriptorReader`.
+
+ :param str path: location to save the processed files dictionary to
+ :param dict processed_files: 'path => last modified' mappings
+
+ :raises:
+ * **IOError** if unable to write to the file
+ * **TypeError** if processed_files is of the wrong type
+ """
+
+ # makes the parent directory if it doesn't already exist
+ try:
+ path_dir = os.path.dirname(path)
+
+ if not os.path.exists(path_dir):
+ os.makedirs(path_dir)
+ except OSError as exc:
+ raise IOError(exc)
+
+ with open(path, 'w') as output_file:
+ for path, timestamp in list(processed_files.items()):
+ if not os.path.isabs(path):
+ raise TypeError('Only absolute paths are acceptable: %s' % path)
+
+ output_file.write('%s %i\n' % (path, timestamp))
+
+
+class DescriptorReader(object):
+ """
+ Iterator for the descriptor data on the local file system. This can process
+ text files, tarball archives (gzip or bzip2), or recurse directories.
+
+ By default this limits the number of descriptors that we'll read ahead before
+ waiting for our caller to fetch some of them. This is included to avoid
+ unbounded memory usage.
+
+ Our persistence_path argument is a convenient method to persist the listing
+ of files we have processed between runs, however it doesn't allow for error
+ handling. If you want that then use the
+ :func:`~stem.descriptor.reader.load_processed_files` and
+ :func:`~stem.descriptor.reader.save_processed_files` functions instead.
+
+ :param str,list target: path or list of paths for files or directories to be read from
+ :param bool validate: checks the validity of the descriptor's content if
+ **True**, skips these checks otherwise
+ :param bool follow_links: determines if we'll follow symlinks when traversing
+ directories (requires python 2.6)
+ :param int buffer_size: descriptors we'll buffer before waiting for some to
+ be read, this is unbounded if zero
+ :param str persistence_path: if set we will load and save processed file
+ listings from this path, errors are ignored
+ :param stem.descriptor.__init__.DocumentHandler document_handler: method in
+ which to parse :class:`~stem.descriptor.networkstatus.NetworkStatusDocument`
+ :param dict kwargs: additional arguments for the descriptor constructor
+ """
+
+ def __init__(self, target, validate = False, follow_links = False, buffer_size = 100, persistence_path = None, document_handler = stem.descriptor.DocumentHandler.ENTRIES, **kwargs):
+ if isinstance(target, (bytes, str_type)):
+ self._targets = [target]
+ else:
+ self._targets = target
+
+ # expand any relative paths we got
+
+ self._targets = list(map(os.path.abspath, self._targets))
+
+ self._validate = validate
+ self._follow_links = follow_links
+ self._persistence_path = persistence_path
+ self._document_handler = document_handler
+ self._kwargs = kwargs
+ self._read_listeners = []
+ self._skip_listeners = []
+ self._processed_files = {}
+
+ self._reader_thread = None
+ self._reader_thread_lock = threading.RLock()
+
+ self._iter_lock = threading.RLock()
+ self._iter_notice = threading.Event()
+
+ self._is_stopped = threading.Event()
+ self._is_stopped.set()
+
+ # Descriptors that we have read but not yet provided to the caller. A
+ # FINISHED entry is used by the reading thread to indicate the end.
+
+ self._unreturned_descriptors = queue.Queue(buffer_size)
+
+ if self._persistence_path:
+ try:
+ processed_files = load_processed_files(self._persistence_path)
+ self.set_processed_files(processed_files)
+ except:
+ pass
+
+ def get_processed_files(self):
+ """
+ For each file that we have read descriptor data from this provides a
+ mapping of the form...
+
+ ::
+
+ absolute path (str) => last modified unix timestamp (int)
+
+ This includes entries set through the
+ :func:`~stem.descriptor.reader.DescriptorReader.set_processed_files`
+ method. Each run resets this to only the files that were present during
+ that run.
+
+ :returns: **dict** with the absolute paths and unix timestamp for the last
+ modified times of the files we have processed
+ """
+
+ # make sure that we only provide back absolute paths
+ return dict((os.path.abspath(k), v) for (k, v) in list(self._processed_files.items()))
+
+ def set_processed_files(self, processed_files):
+ """
+ Sets the listing of the files we have processed. Most often this is used
+ with a newly created :class:`~stem.descriptor.reader.DescriptorReader` to
+ pre-populate the listing of descriptor files that we have seen.
+
+ :param dict processed_files: mapping of absolute paths (**str**) to unix
+ timestamps for the last modified time (**int**)
+ """
+
+ self._processed_files = dict(processed_files)
+
+ def register_read_listener(self, listener):
+ """
+ Registers a listener for when files are read. This is executed prior to
+ processing files. Listeners are expected to be of the form...
+
+ ::
+
+ my_listener(path)
+
+ :param functor listener: functor to be notified when files are read
+ """
+
+ self._read_listeners.append(listener)
+
+ def register_skip_listener(self, listener):
+ """
+ Registers a listener for files that are skipped. This listener is expected
+ to be a functor of the form...
+
+ ::
+
+ my_listener(path, exception)
+
+ :param functor listener: functor to be notified of files that are skipped
+ to read errors or because they couldn't be parsed as valid descriptor data
+ """
+
+ self._skip_listeners.append(listener)
+
+ def get_buffered_descriptor_count(self):
+ """
+ Provides the number of descriptors that are waiting to be iterated over.
+ This is limited to the buffer_size that we were constructed with.
+
+ :returns: **int** for the estimated number of currently enqueued
+ descriptors, this is not entirely reliable
+ """
+
+ return self._unreturned_descriptors.qsize()
+
+ def start(self):
+ """
+ Starts reading our descriptor files.
+
+ :raises: **ValueError** if we're already reading the descriptor files
+ """
+
+ with self._reader_thread_lock:
+ if self._reader_thread:
+ raise ValueError('Already running, you need to call stop() first')
+ else:
+ self._is_stopped.clear()
+ self._reader_thread = threading.Thread(target = self._read_descriptor_files, name='Descriptor Reader')
+ self._reader_thread.setDaemon(True)
+ self._reader_thread.start()
+
+ def stop(self):
+ """
+ Stops further reading of descriptor files.
+ """
+
+ with self._reader_thread_lock:
+ self._is_stopped.set()
+ self._iter_notice.set()
+
+ # clears our queue to unblock enqueue calls
+
+ try:
+ while True:
+ self._unreturned_descriptors.get_nowait()
+ except queue.Empty:
+ pass
+
+ self._reader_thread.join()
+ self._reader_thread = None
+
+ if self._persistence_path:
+ try:
+ processed_files = self.get_processed_files()
+ save_processed_files(self._persistence_path, processed_files)
+ except:
+ pass
+
+ def _read_descriptor_files(self):
+ new_processed_files = {}
+ remaining_files = list(self._targets)
+
+ while remaining_files and not self._is_stopped.is_set():
+ target = remaining_files.pop(0)
+
+ if not os.path.exists(target):
+ self._notify_skip_listeners(target, FileMissing())
+ continue
+
+ if os.path.isdir(target):
+ walker = os.walk(target, followlinks = self._follow_links)
+ self._handle_walker(walker, new_processed_files)
+ else:
+ self._handle_file(target, new_processed_files)
+
+ self._processed_files = new_processed_files
+
+ if not self._is_stopped.is_set():
+ self._unreturned_descriptors.put(FINISHED)
+
+ self._iter_notice.set()
+
+ def __iter__(self):
+ with self._iter_lock:
+ while not self._is_stopped.is_set():
+ try:
+ descriptor = self._unreturned_descriptors.get_nowait()
+
+ if descriptor == FINISHED:
+ break
+ else:
+ yield descriptor
+ except queue.Empty:
+ self._iter_notice.wait()
+ self._iter_notice.clear()
+
+ def _handle_walker(self, walker, new_processed_files):
+ for root, _, files in walker:
+ for filename in files:
+ self._handle_file(os.path.join(root, filename), new_processed_files)
+
+ # this can take a while if, say, we're including the root directory
+ if self._is_stopped.is_set():
+ return
+
+ def _handle_file(self, target, new_processed_files):
+ # This is a file. Register its last modified timestamp and check if
+ # it's a file that we should skip.
+
+ try:
+ last_modified = int(os.stat(target).st_mtime)
+ last_used = self._processed_files.get(target)
+ new_processed_files[target] = last_modified
+ except OSError as exc:
+ self._notify_skip_listeners(target, ReadFailed(exc))
+ return
+
+ if last_used and last_used >= last_modified:
+ self._notify_skip_listeners(target, AlreadyRead(last_modified, last_used))
+ return
+
+ # Block devices and such are never descriptors, and can cause us to block
+ # for quite a while so skipping anything that isn't a regular file.
+
+ if not os.path.isfile(target):
+ return
+
+ # The mimetypes module only checks the file extension. To actually
+ # check the content (like the 'file' command) we'd need something like
+ # pymagic (https://github.com/cloudburst/pymagic).
+
+ target_type = mimetypes.guess_type(target)
+
+ if target_type[0] in (None, 'text/plain'):
+ # either '.txt' or an unknown type
+ self._handle_descriptor_file(target, target_type)
+ elif stem.util.system.is_tarfile(target):
+ # handles gzip, bz2, and decompressed tarballs among others
+ self._handle_archive(target)
+ else:
+ self._notify_skip_listeners(target, UnrecognizedType(target_type))
+
+ def _handle_descriptor_file(self, target, mime_type):
+ try:
+ self._notify_read_listeners(target)
+
+ with open(target, 'rb') as target_file:
+ for desc in stem.descriptor.parse_file(target_file, validate = self._validate, document_handler = self._document_handler, **self._kwargs):
+ if self._is_stopped.is_set():
+ return
+
+ self._unreturned_descriptors.put(desc)
+ self._iter_notice.set()
+ except TypeError as exc:
+ self._notify_skip_listeners(target, UnrecognizedType(mime_type))
+ except ValueError as exc:
+ self._notify_skip_listeners(target, ParsingFailure(exc))
+ except IOError as exc:
+ self._notify_skip_listeners(target, ReadFailed(exc))
+
+ def _handle_archive(self, target):
+ # TODO: When dropping python 2.6 support go back to using 'with' for
+ # tarfiles...
+ #
+ # http://bugs.python.org/issue7232
+
+ tar_file = None
+
+ try:
+ self._notify_read_listeners(target)
+ tar_file = tarfile.open(target)
+
+ for tar_entry in tar_file:
+ if tar_entry.isfile():
+ entry = tar_file.extractfile(tar_entry)
+
+ try:
+ for desc in stem.descriptor.parse_file(entry, validate = self._validate, document_handler = self._document_handler, **self._kwargs):
+ if self._is_stopped.is_set():
+ return
+
+ desc._set_path(os.path.abspath(target))
+ desc._set_archive_path(tar_entry.name)
+ self._unreturned_descriptors.put(desc)
+ self._iter_notice.set()
+ except TypeError as exc:
+ self._notify_skip_listeners(target, ParsingFailure(exc))
+ except ValueError as exc:
+ self._notify_skip_listeners(target, ParsingFailure(exc))
+ finally:
+ entry.close()
+ except IOError as exc:
+ self._notify_skip_listeners(target, ReadFailed(exc))
+ finally:
+ if tar_file:
+ tar_file.close()
+
+ def _notify_read_listeners(self, path):
+ for listener in self._read_listeners:
+ listener(path)
+
+ def _notify_skip_listeners(self, path, exception):
+ for listener in self._skip_listeners:
+ listener(path, exception)
+
+ def __enter__(self):
+ self.start()
+ return self
+
+ def __exit__(self, exit_type, value, traceback):
+ self.stop()
diff --git a/ext/stem/descriptor/remote.py b/ext/stem/descriptor/remote.py
new file mode 100644
index 00000000..4d3423d0
--- /dev/null
+++ b/ext/stem/descriptor/remote.py
@@ -0,0 +1,777 @@
+# Copyright 2013-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Module for remotely retrieving descriptors from directory authorities and
+mirrors. This is most easily done through the
+:class:`~stem.descriptor.remote.DescriptorDownloader` class, which issues
+:class:`~stem.descriptor.remote.Query` instances to get you the descriptor
+content. For example...
+
+::
+
+ from stem.descriptor.remote import DescriptorDownloader
+
+ downloader = DescriptorDownloader(
+ use_mirrors = True,
+ timeout = 10,
+ )
+
+ query = downloader.get_server_descriptors()
+
+ print 'Exit Relays:'
+
+ try:
+ for desc in query.run():
+ if desc.exit_policy.is_exiting_allowed():
+ print ' %s (%s)' % (desc.nickname, desc.fingerprint)
+
+ print
+ print 'Query took %0.2f seconds' % query.runtime
+ except Exception as exc:
+ print 'Unable to retrieve the server descriptors: %s' % exc
+
+If you don't care about errors then you can also simply iterate over the query
+itself...
+
+::
+
+ for desc in downloader.get_server_descriptors():
+ if desc.exit_policy.is_exiting_allowed():
+ print ' %s (%s)' % (desc.nickname, desc.fingerprint)
+
+::
+
+ get_authorities - Provides tor directory information.
+
+ DirectoryAuthority - Information about a tor directory authority.
+
+ Query - Asynchronous request to download tor descriptors
+ |- start - issues the query if it isn't already running
+ +- run - blocks until the request is finished and provides the results
+
+ DescriptorDownloader - Configurable class for issuing queries
+ |- use_directory_mirrors - use directory mirrors to download future descriptors
+ |- get_server_descriptors - provides present server descriptors
+ |- get_extrainfo_descriptors - provides present extrainfo descriptors
+ |- get_microdescriptors - provides present microdescriptors
+ |- get_consensus - provides the present consensus or router status entries
+ |- get_key_certificates - provides present authority key certificates
+ +- query - request an arbitrary descriptor resource
+
+.. versionadded:: 1.1.0
+
+.. data:: MAX_FINGERPRINTS
+
+ Maximum number of descriptors that can requested at a time by their
+ fingerprints.
+
+.. data:: MAX_MICRODESCRIPTOR_HASHES
+
+ Maximum number of microdescriptors that can requested at a time by their
+ hashes.
+"""
+
+import io
+import random
+import sys
+import threading
+import time
+import zlib
+
+try:
+ import urllib.request as urllib
+except ImportError:
+ import urllib2 as urllib
+
+import stem.descriptor
+
+from stem import Flag
+from stem.util import log
+
+# Tor has a limited number of descriptors we can fetch explicitly by their
+# fingerprint or hashes due to a limit on the url length by squid proxies.
+
+MAX_FINGERPRINTS = 96
+MAX_MICRODESCRIPTOR_HASHES = 92
+
+# We commonly only want authorities that vote in the consensus, and hence have
+# a v3ident.
+
+HAS_V3IDENT = lambda auth: auth.v3ident is not None
+
+
+def _guess_descriptor_type(resource):
+ # Attempts to determine the descriptor type based on the resource url. This
+ # raises a ValueError if the resource isn't recognized.
+
+ if resource.startswith('/tor/server/'):
+ return 'server-descriptor 1.0'
+ elif resource.startswith('/tor/extra/'):
+ return 'extra-info 1.0'
+ elif resource.startswith('/tor/micro/'):
+ return 'microdescriptor 1.0'
+ elif resource.startswith('/tor/status-vote/'):
+ return 'network-status-consensus-3 1.0'
+ elif resource.startswith('/tor/keys/'):
+ return 'dir-key-certificate-3 1.0'
+ else:
+ raise ValueError("Unable to determine the descriptor type for '%s'" % resource)
+
+
+class Query(object):
+ """
+ Asynchronous request for descriptor content from a directory authority or
+ mirror. These can either be made through the
+ :class:`~stem.descriptor.remote.DescriptorDownloader` or directly for more
+ advanced usage.
+
+ To block on the response and get results either call
+ :func:`~stem.descriptor.remote.Query.run` or iterate over the Query. The
+ :func:`~stem.descriptor.remote.Query.run` method pass along any errors that
+ arise...
+
+ ::
+
+ from stem.descriptor.remote import Query
+
+ query = Query(
+ '/tor/server/all.z',
+ block = True,
+ timeout = 30,
+ )
+
+ print 'Current relays:'
+
+ if not query.error:
+ for desc in query:
+ print desc.fingerprint
+ else:
+ print 'Unable to retrieve the server descriptors: %s' % query.error
+
+ ... while iterating fails silently...
+
+ ::
+
+ print 'Current relays:'
+
+ for desc in Query('/tor/server/all.z', 'server-descriptor 1.0'):
+ print desc.fingerprint
+
+ In either case exceptions are available via our 'error' attribute.
+
+ Tor provides quite a few different descriptor resources via its directory
+ protocol (see section 4.2 and later of the `dir-spec
+ <https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt>`_).
+ Commonly useful ones include...
+
+ ===================================== ===========
+ Resource Description
+ ===================================== ===========
+ /tor/server/all.z all present server descriptors
+ /tor/server/fp/<fp1>+<fp2>+<fp3>.z server descriptors with the given fingerprints
+ /tor/extra/all.z all present extrainfo descriptors
+ /tor/extra/fp/<fp1>+<fp2>+<fp3>.z extrainfo descriptors with the given fingerprints
+ /tor/micro/d/<hash1>-<hash2>.z microdescriptors with the given hashes
+ /tor/status-vote/current/consensus.z present consensus
+ /tor/keys/all.z key certificates for the authorities
+ /tor/keys/fp/<v3ident1>+<v3ident2>.z key certificates for specific authorities
+ ===================================== ===========
+
+ The '.z' suffix can be excluded to get a plaintext rather than compressed
+ response. Compression is handled transparently, so this shouldn't matter to
+ the caller.
+
+ :var str resource: resource being fetched, such as '/tor/server/all.z'
+ :var str descriptor_type: type of descriptors being fetched (for options see
+ :func:`~stem.descriptor.__init__.parse_file`), this is guessed from the
+ resource if **None**
+
+ :var list endpoints: (address, dirport) tuples of the authority or mirror
+ we're querying, this uses authorities if undefined
+ :var int retries: number of times to attempt the request if downloading it
+ fails
+ :var bool fall_back_to_authority: when retrying request issues the last
+ request to a directory authority if **True**
+
+ :var str content: downloaded descriptor content
+ :var Exception error: exception if a problem occured
+ :var bool is_done: flag that indicates if our request has finished
+ :var str download_url: last url used to download the descriptor, this is
+ unset until we've actually made a download attempt
+
+ :var float start_time: unix timestamp when we first started running
+ :var float timeout: duration before we'll time out our request
+ :var float runtime: time our query took, this is **None** if it's not yet
+ finished
+
+ :var bool validate: checks the validity of the descriptor's content if
+ **True**, skips these checks otherwise
+ :var stem.descriptor.__init__.DocumentHandler document_handler: method in
+ which to parse a :class:`~stem.descriptor.networkstatus.NetworkStatusDocument`
+ :var dict kwargs: additional arguments for the descriptor constructor
+
+ :param bool start: start making the request when constructed (default is **True**)
+ :param bool block: only return after the request has been completed, this is
+ the same as running **query.run(True)** (default is **False**)
+ """
+
+ def __init__(self, resource, descriptor_type = None, endpoints = None, retries = 2, fall_back_to_authority = False, timeout = None, start = True, block = False, validate = False, document_handler = stem.descriptor.DocumentHandler.ENTRIES, **kwargs):
+ if not resource.startswith('/'):
+ raise ValueError("Resources should start with a '/': %s" % resource)
+
+ self.resource = resource
+
+ if descriptor_type:
+ self.descriptor_type = descriptor_type
+ else:
+ self.descriptor_type = _guess_descriptor_type(resource)
+
+ self.endpoints = endpoints if endpoints else []
+ self.retries = retries
+ self.fall_back_to_authority = fall_back_to_authority
+
+ self.content = None
+ self.error = None
+ self.is_done = False
+ self.download_url = None
+
+ self.start_time = None
+ self.timeout = timeout
+ self.runtime = None
+
+ self.validate = validate
+ self.document_handler = document_handler
+ self.kwargs = kwargs
+
+ self._downloader_thread = None
+ self._downloader_thread_lock = threading.RLock()
+
+ if start:
+ self.start()
+
+ if block:
+ self.run(True)
+
+ def start(self):
+ """
+ Starts downloading the scriptors if we haven't started already.
+ """
+
+ with self._downloader_thread_lock:
+ if self._downloader_thread is None:
+ self._downloader_thread = threading.Thread(
+ name = 'Descriptor Query',
+ target = self._download_descriptors,
+ args = (self.retries,)
+ )
+
+ self._downloader_thread.setDaemon(True)
+ self._downloader_thread.start()
+
+ def run(self, suppress = False):
+ """
+ Blocks until our request is complete then provides the descriptors. If we
+ haven't yet started our request then this does so.
+
+ :param bool suppress: avoids raising exceptions if **True**
+
+ :returns: list for the requested :class:`~stem.descriptor.__init__.Descriptor` instances
+
+ :raises:
+ Using the iterator can fail with the following if **suppress** is
+ **False**...
+
+ * **ValueError** if the descriptor contents is malformed
+ * **socket.timeout** if our request timed out
+ * **urllib2.URLError** for most request failures
+
+ Note that the urllib2 module may fail with other exception types, in
+ which case we'll pass it along.
+ """
+
+ return list(self._run(suppress))
+
+ def _run(self, suppress):
+ with self._downloader_thread_lock:
+ self.start()
+ self._downloader_thread.join()
+
+ if self.error:
+ if suppress:
+ return
+
+ raise self.error
+ else:
+ if self.content is None:
+ if suppress:
+ return
+
+ raise ValueError('BUG: _download_descriptors() finished without either results or an error')
+
+ try:
+ results = stem.descriptor.parse_file(
+ io.BytesIO(self.content),
+ self.descriptor_type,
+ validate = self.validate,
+ document_handler = self.document_handler,
+ **self.kwargs
+ )
+
+ for desc in results:
+ yield desc
+ except ValueError as exc:
+ self.error = exc # encountered a parsing error
+
+ if suppress:
+ return
+
+ raise self.error
+
+ def __iter__(self):
+ for desc in self._run(True):
+ yield desc
+
+ def _pick_url(self, use_authority = False):
+ """
+ Provides a url that can be queried. If we have multiple endpoints then one
+ will be picked randomly.
+
+ :param bool use_authority: ignores our endpoints and uses a directory
+ authority instead
+
+ :returns: **str** for the url being queried by this request
+ """
+
+ if use_authority or not self.endpoints:
+ authority = random.choice(filter(HAS_V3IDENT, get_authorities().values()))
+ address, dirport = authority.address, authority.dir_port
+ else:
+ address, dirport = random.choice(self.endpoints)
+
+ return 'http://%s:%i/%s' % (address, dirport, self.resource.lstrip('/'))
+
+ def _download_descriptors(self, retries):
+ try:
+ use_authority = retries == 0 and self.fall_back_to_authority
+ self.download_url = self._pick_url(use_authority)
+
+ self.start_time = time.time()
+ response = urllib.urlopen(self.download_url, timeout = self.timeout).read()
+
+ if self.download_url.endswith('.z'):
+ response = zlib.decompress(response)
+
+ self.content = response.strip()
+
+ self.runtime = time.time() - self.start_time
+ log.trace("Descriptors retrieved from '%s' in %0.2fs" % (self.download_url, self.runtime))
+ except:
+ exc = sys.exc_info()[1]
+
+ if retries > 0:
+ log.debug("Unable to download descriptors from '%s' (%i retries remaining): %s" % (self.download_url, retries, exc))
+ return self._download_descriptors(retries - 1)
+ else:
+ log.debug("Unable to download descriptors from '%s': %s" % (self.download_url, exc))
+ self.error = exc
+ finally:
+ self.is_done = True
+
+
+class DescriptorDownloader(object):
+ """
+ Configurable class that issues :class:`~stem.descriptor.remote.Query`
+ instances on your behalf.
+
+ :param bool use_mirrors: downloads the present consensus and uses the directory
+ mirrors to fetch future requests, this fails silently if the consensus
+ cannot be downloaded
+ :param default_args: default arguments for the
+ :class:`~stem.descriptor.remote.Query` constructor
+ """
+
+ def __init__(self, use_mirrors = False, **default_args):
+ self._default_args = default_args
+
+ authorities = filter(HAS_V3IDENT, get_authorities().values())
+ self._endpoints = [(auth.address, auth.dir_port) for auth in authorities]
+
+ if use_mirrors:
+ try:
+ start_time = time.time()
+ self.use_directory_mirrors()
+ log.debug('Retrieved directory mirrors (took %0.2fs)' % (time.time() - start_time))
+ except Exception as exc:
+ log.debug('Unable to retrieve directory mirrors: %s' % exc)
+
+ def use_directory_mirrors(self):
+ """
+ Downloads the present consensus and configures ourselves to use directory
+ mirrors, in addition to authorities.
+
+ :returns: :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV3`
+ from which we got the directory mirrors
+
+ :raises: **Exception** if unable to determine the directory mirrors
+ """
+
+ authorities = filter(HAS_V3IDENT, get_authorities().values())
+ new_endpoints = set([(auth.address, auth.dir_port) for auth in authorities])
+
+ consensus = list(self.get_consensus(document_handler = stem.descriptor.DocumentHandler.DOCUMENT).run())[0]
+
+ for desc in consensus.routers.values():
+ if Flag.V2DIR in desc.flags:
+ new_endpoints.add((desc.address, desc.dir_port))
+
+ # we need our endpoints to be a list rather than set for random.choice()
+
+ self._endpoints = list(new_endpoints)
+
+ return consensus
+
+ def get_server_descriptors(self, fingerprints = None, **query_args):
+ """
+ Provides the server descriptors with the given fingerprints. If no
+ fingerprints are provided then this returns all descriptors in the present
+ consensus.
+
+ :param str,list fingerprints: fingerprint or list of fingerprints to be
+ retrieved, gets all descriptors if **None**
+ :param query_args: additional arguments for the
+ :class:`~stem.descriptor.remote.Query` constructor
+
+ :returns: :class:`~stem.descriptor.remote.Query` for the server descriptors
+
+ :raises: **ValueError** if we request more than 96 descriptors by their
+ fingerprints (this is due to a limit on the url length by squid proxies).
+ """
+
+ resource = '/tor/server/all.z'
+
+ if isinstance(fingerprints, str):
+ fingerprints = [fingerprints]
+
+ if fingerprints:
+ if len(fingerprints) > MAX_FINGERPRINTS:
+ raise ValueError('Unable to request more than %i descriptors at a time by their fingerprints' % MAX_FINGERPRINTS)
+
+ resource = '/tor/server/fp/%s.z' % '+'.join(fingerprints)
+
+ return self.query(resource, **query_args)
+
+ def get_extrainfo_descriptors(self, fingerprints = None, **query_args):
+ """
+ Provides the extrainfo descriptors with the given fingerprints. If no
+ fingerprints are provided then this returns all descriptors in the present
+ consensus.
+
+ :param str,list fingerprints: fingerprint or list of fingerprints to be
+ retrieved, gets all descriptors if **None**
+ :param query_args: additional arguments for the
+ :class:`~stem.descriptor.remote.Query` constructor
+
+ :returns: :class:`~stem.descriptor.remote.Query` for the extrainfo descriptors
+
+ :raises: **ValueError** if we request more than 96 descriptors by their
+ fingerprints (this is due to a limit on the url length by squid proxies).
+ """
+
+ resource = '/tor/extra/all.z'
+
+ if isinstance(fingerprints, str):
+ fingerprints = [fingerprints]
+
+ if fingerprints:
+ if len(fingerprints) > MAX_FINGERPRINTS:
+ raise ValueError('Unable to request more than %i descriptors at a time by their fingerprints' % MAX_FINGERPRINTS)
+
+ resource = '/tor/extra/fp/%s.z' % '+'.join(fingerprints)
+
+ return self.query(resource, **query_args)
+
+ def get_microdescriptors(self, hashes, **query_args):
+ """
+ Provides the microdescriptors with the given hashes. To get these see the
+ 'microdescriptor_hashes' attribute of
+ :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3`. Note
+ that these are only provided via a microdescriptor consensus (such as
+ 'cached-microdesc-consensus' in your data directory).
+
+ :param str,list hashes: microdescriptor hash or list of hashes to be
+ retrieved
+ :param query_args: additional arguments for the
+ :class:`~stem.descriptor.remote.Query` constructor
+
+ :returns: :class:`~stem.descriptor.remote.Query` for the microdescriptors
+
+ :raises: **ValueError** if we request more than 92 microdescriptors by their
+ hashes (this is due to a limit on the url length by squid proxies).
+ """
+
+ if isinstance(hashes, str):
+ hashes = [hashes]
+
+ if len(hashes) > MAX_MICRODESCRIPTOR_HASHES:
+ raise ValueError('Unable to request more than %i microdescriptors at a time by their hashes' % MAX_MICRODESCRIPTOR_HASHES)
+
+ return self.query('/tor/micro/d/%s.z' % '-'.join(hashes), **query_args)
+
+ def get_consensus(self, authority_v3ident = None, **query_args):
+ """
+ Provides the present router status entries.
+
+ :param str authority_v3ident: fingerprint of the authority key for which
+ to get the consensus, see `'v3ident' in tor's config.c
+ <https://gitweb.torproject.org/tor.git/tree/src/or/config.c#n819>`_
+ for the values.
+ :param query_args: additional arguments for the
+ :class:`~stem.descriptor.remote.Query` constructor
+
+ :returns: :class:`~stem.descriptor.remote.Query` for the router status
+ entries
+ """
+
+ resource = '/tor/status-vote/current/consensus'
+
+ if authority_v3ident:
+ resource += '/%s' % authority_v3ident
+
+ return self.query(resource + '.z', **query_args)
+
+ def get_vote(self, authority, **query_args):
+ """
+ Provides the present vote for a given directory authority.
+
+ :param stem.descriptor.remote.DirectoryAuthority authority: authority for which to retrieve a vote for
+ :param query_args: additional arguments for the
+ :class:`~stem.descriptor.remote.Query` constructor
+
+ :returns: :class:`~stem.descriptor.remote.Query` for the router status
+ entries
+ """
+
+ resource = '/tor/status-vote/current/authority'
+
+ if 'endpoint' not in query_args:
+ query_args['endpoints'] = [(authority.address, authority.dir_port)]
+
+ return self.query(resource + '.z', **query_args)
+
+ def get_key_certificates(self, authority_v3idents = None, **query_args):
+ """
+ Provides the key certificates for authorities with the given fingerprints.
+ If no fingerprints are provided then this returns all present key
+ certificates.
+
+ :param str authority_v3idents: fingerprint or list of fingerprints of the
+ authority keys, see `'v3ident' in tor's config.c
+ <https://gitweb.torproject.org/tor.git/tree/src/or/config.c#n819>`_
+ for the values.
+ :param query_args: additional arguments for the
+ :class:`~stem.descriptor.remote.Query` constructor
+
+ :returns: :class:`~stem.descriptor.remote.Query` for the key certificates
+
+ :raises: **ValueError** if we request more than 96 key certificates by
+ their identity fingerprints (this is due to a limit on the url length by
+ squid proxies).
+ """
+
+ resource = '/tor/keys/all.z'
+
+ if isinstance(authority_v3idents, str):
+ authority_v3idents = [authority_v3idents]
+
+ if authority_v3idents:
+ if len(authority_v3idents) > MAX_FINGERPRINTS:
+ raise ValueError('Unable to request more than %i key certificates at a time by their identity fingerprints' % MAX_FINGERPRINTS)
+
+ resource = '/tor/keys/fp/%s.z' % '+'.join(authority_v3idents)
+
+ return self.query(resource, **query_args)
+
+ def query(self, resource, **query_args):
+ """
+ Issues a request for the given resource.
+
+ :param str resource: resource being fetched, such as '/tor/server/all.z'
+ :param query_args: additional arguments for the
+ :class:`~stem.descriptor.remote.Query` constructor
+
+ :returns: :class:`~stem.descriptor.remote.Query` for the descriptors
+
+ :raises: **ValueError** if resource is clearly invalid or the descriptor
+ type can't be determined when 'descriptor_type' is **None**
+ """
+
+ args = dict(self._default_args)
+ args.update(query_args)
+
+ if 'endpoints' not in args:
+ args['endpoints'] = self._endpoints
+
+ if 'fall_back_to_authority' not in args:
+ args['fall_back_to_authority'] = True
+
+ return Query(
+ resource,
+ **args
+ )
+
+
+class DirectoryAuthority(object):
+ """
+ Tor directory authority, a special type of relay `hardcoded into tor
+ <https://gitweb.torproject.org/tor.git/tree/src/or/config.c#n819>`_
+ that enumerates the other relays within the network.
+
+ At a very high level tor works as follows...
+
+ 1. A volunteer starts up a new tor relay, during which it sends a `server
+ descriptor <server_descriptor.html>`_ to each of the directory
+ authorities.
+
+ 2. Each hour the directory authorities make a `vote <networkstatus.html>`_
+ that says who they think the active relays are in the network and some
+ attributes about them.
+
+ 3. The directory authorities send each other their votes, and compile that
+ into the `consensus <networkstatus.html>`_. This document is very similar
+ to the votes, the only difference being that the majority of the
+ authorities agree upon and sign this document. The idividual relay entries
+ in the vote or consensus is called `router status entries
+ <router_status_entry.html>`_.
+
+ 4. Tor clients (people using the service) download the consensus from one of
+ the authorities or a mirror to determine the active relays within the
+ network. They in turn use this to construct their circuits and use the
+ network.
+
+ .. versionchanged:: 1.3.0
+ Added the is_bandwidth_authority attribute.
+
+ :var str nickname: nickname of the authority
+ :var str address: IP address of the authority, currently they're all IPv4 but
+ this may not always be the case
+ :var int or_port: port on which the relay services relay traffic
+ :var int dir_port: port on which directory information is available
+ :var str fingerprint: relay fingerprint
+ :var str v3ident: identity key fingerprint used to sign votes and consensus
+ """
+
+ def __init__(self, nickname = None, address = None, or_port = None, dir_port = None, is_bandwidth_authority = False, fingerprint = None, v3ident = None):
+ self.nickname = nickname
+ self.address = address
+ self.or_port = or_port
+ self.dir_port = dir_port
+ self.is_bandwidth_authority = is_bandwidth_authority
+ self.fingerprint = fingerprint
+ self.v3ident = v3ident
+
+
+DIRECTORY_AUTHORITIES = {
+ 'moria1': DirectoryAuthority(
+ nickname = 'moria1',
+ address = '128.31.0.39',
+ or_port = 9101,
+ dir_port = 9131,
+ is_bandwidth_authority = True,
+ fingerprint = '9695DFC35FFEB861329B9F1AB04C46397020CE31',
+ v3ident = 'D586D18309DED4CD6D57C18FDB97EFA96D330566',
+ ),
+ 'tor26': DirectoryAuthority(
+ nickname = 'tor26',
+ address = '86.59.21.38',
+ or_port = 443,
+ dir_port = 80,
+ is_bandwidth_authority = True,
+ fingerprint = '847B1F850344D7876491A54892F904934E4EB85D',
+ v3ident = '14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4',
+ ),
+ 'dizum': DirectoryAuthority(
+ nickname = 'dizum',
+ address = '194.109.206.212',
+ or_port = 443,
+ dir_port = 80,
+ is_bandwidth_authority = False,
+ fingerprint = '7EA6EAD6FD83083C538F44038BBFA077587DD755',
+ v3ident = 'E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58',
+ ),
+ 'Tonga': DirectoryAuthority(
+ nickname = 'Tonga',
+ address = '82.94.251.203',
+ or_port = 443,
+ dir_port = 80,
+ is_bandwidth_authority = False,
+ fingerprint = '4A0CCD2DDC7995083D73F5D667100C8A5831F16D',
+ v3ident = None, # does not vote in the consensus
+ ),
+ 'gabelmoo': DirectoryAuthority(
+ nickname = 'gabelmoo',
+ address = '131.188.40.189',
+ or_port = 443,
+ dir_port = 80,
+ is_bandwidth_authority = True,
+ fingerprint = 'F2044413DAC2E02E3D6BCF4735A19BCA1DE97281',
+ v3ident = 'ED03BB616EB2F60BEC80151114BB25CEF515B226',
+ ),
+ 'dannenberg': DirectoryAuthority(
+ nickname = 'dannenberg',
+ address = '193.23.244.244',
+ or_port = 443,
+ dir_port = 80,
+ is_bandwidth_authority = False,
+ fingerprint = '7BE683E65D48141321C5ED92F075C55364AC7123',
+ v3ident = '585769C78764D58426B8B52B6651A5A71137189A',
+ ),
+ 'urras': DirectoryAuthority(
+ nickname = 'urras',
+ address = '208.83.223.34',
+ or_port = 80,
+ dir_port = 443,
+ is_bandwidth_authority = False,
+ fingerprint = '0AD3FA884D18F89EEA2D89C019379E0E7FD94417',
+ v3ident = '80550987E1D626E3EBA5E5E75A458DE0626D088C',
+ ),
+ 'maatuska': DirectoryAuthority(
+ nickname = 'maatuska',
+ address = '171.25.193.9',
+ or_port = 80,
+ dir_port = 443,
+ is_bandwidth_authority = True,
+ fingerprint = 'BD6A829255CB08E66FBE7D3748363586E46B3810',
+ v3ident = '49015F787433103580E3B66A1707A00E60F2D15B',
+ ),
+ 'Faravahar': DirectoryAuthority(
+ nickname = 'Faravahar',
+ address = '154.35.175.225',
+ or_port = 443,
+ dir_port = 80,
+ is_bandwidth_authority = False,
+ fingerprint = 'CF6D0AAFB385BE71B8E111FC5CFF4B47923733BC',
+ v3ident = 'EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97',
+ ),
+ 'longclaw': DirectoryAuthority(
+ nickname = 'longclaw',
+ address = '199.254.238.52',
+ or_port = 443,
+ dir_port = 80,
+ is_bandwidth_authority = True,
+ fingerprint = '74A910646BCEEFBCD2E874FC1DC997430F968145',
+ v3ident = '23D15D965BC35114467363C165C4F724B64B4F66',
+ ),
+}
+
+
+def get_authorities():
+ """
+ Provides the Tor directory authority information as of **Tor on 11/21/14**.
+ The directory information hardcoded into Tor and occasionally changes, so the
+ information this provides might not necessarily match your version of tor.
+
+ :returns: dict of str nicknames to :class:`~stem.descriptor.remote.DirectoryAuthority` instances
+ """
+
+ return dict(DIRECTORY_AUTHORITIES)
diff --git a/ext/stem/descriptor/router_status_entry.py b/ext/stem/descriptor/router_status_entry.py
new file mode 100644
index 00000000..c33baa39
--- /dev/null
+++ b/ext/stem/descriptor/router_status_entry.py
@@ -0,0 +1,635 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Parsing for router status entries, the information for individual routers
+within a network status document. This information is provided from a few
+sources...
+
+* control port via 'GETINFO ns/\*' and 'GETINFO md/\*' queries
+* router entries in a network status document, like the cached-consensus
+
+**Module Overview:**
+
+::
+
+ RouterStatusEntry - Common parent for router status entries
+ |- RouterStatusEntryV2 - Entry for a network status v2 document
+ |- RouterStatusEntryV3 - Entry for a network status v3 document
+ +- RouterStatusEntryMicroV3 - Entry for a microdescriptor flavored v3 document
+"""
+
+import base64
+import binascii
+
+import stem.exit_policy
+import stem.prereq
+import stem.util.str_tools
+
+from stem.descriptor import (
+ KEYWORD_LINE,
+ Descriptor,
+ _value,
+ _values,
+ _get_descriptor_components,
+ _read_until_keywords,
+)
+
+
+def _parse_file(document_file, validate, entry_class, entry_keyword = 'r', start_position = None, end_position = None, section_end_keywords = (), extra_args = ()):
+ """
+ Reads a range of the document_file containing some number of entry_class
+ instances. We deliminate the entry_class entries by the keyword on their
+ first line (entry_keyword). When finished the document is left at the
+ end_position.
+
+ Either an end_position or section_end_keywords must be provided.
+
+ :param file document_file: file with network status document content
+ :param bool validate: checks the validity of the document's contents if
+ **True**, skips these checks otherwise
+ :param class entry_class: class to construct instance for
+ :param str entry_keyword: first keyword for the entry instances
+ :param int start_position: start of the section, default is the current position
+ :param int end_position: end of the section
+ :param tuple section_end_keywords: keyword(s) that deliminate the end of the
+ section if no end_position was provided
+ :param tuple extra_args: extra arguments for the entry_class (after the
+ content and validate flag)
+
+ :returns: iterator over entry_class instances
+
+ :raises:
+ * **ValueError** if the contents is malformed and validate is **True**
+ * **IOError** if the file can't be read
+ """
+
+ if start_position:
+ document_file.seek(start_position)
+ else:
+ start_position = document_file.tell()
+
+ # check if we're starting at the end of the section (ie, there's no entries to read)
+ if section_end_keywords:
+ first_keyword = None
+ line_match = KEYWORD_LINE.match(stem.util.str_tools._to_unicode(document_file.readline()))
+
+ if line_match:
+ first_keyword = line_match.groups()[0]
+
+ document_file.seek(start_position)
+
+ if first_keyword in section_end_keywords:
+ return
+
+ while end_position is None or document_file.tell() < end_position:
+ desc_lines, ending_keyword = _read_until_keywords(
+ (entry_keyword,) + section_end_keywords,
+ document_file,
+ ignore_first = True,
+ end_position = end_position,
+ include_ending_keyword = True
+ )
+
+ desc_content = bytes.join(b'', desc_lines)
+
+ if desc_content:
+ yield entry_class(desc_content, validate, *extra_args)
+
+ # check if we stopped at the end of the section
+ if ending_keyword in section_end_keywords:
+ break
+ else:
+ break
+
+
+def _parse_r_line(descriptor, entries):
+ # Parses a RouterStatusEntry's 'r' line. They're very nearly identical for
+ # all current entry types (v2, v3, and microdescriptor v3) with one little
+ # wrinkle: only the microdescriptor flavor excludes a 'digest' field.
+ #
+ # For v2 and v3 router status entries:
+ # "r" nickname identity digest publication IP ORPort DirPort
+ # example: r mauer BD7xbfsCFku3+tgybEZsg8Yjhvw itcuKQ6PuPLJ7m/Oi928WjO2j8g 2012-06-22 13:19:32 80.101.105.103 9001 0
+ #
+ # For v3 microdescriptor router status entries:
+ # "r" nickname identity publication IP ORPort DirPort
+ # example: r Konata ARIJF2zbqirB9IwsW0mQznccWww 2012-09-24 13:40:40 69.64.48.168 9001 9030
+
+ value = _value('r', entries)
+ include_digest = not isinstance(descriptor, RouterStatusEntryMicroV3)
+
+ r_comp = value.split(' ')
+
+ # inject a None for the digest to normalize the field positioning
+ if not include_digest:
+ r_comp.insert(2, None)
+
+ if len(r_comp) < 8:
+ expected_field_count = 'eight' if include_digest else 'seven'
+ raise ValueError("%s 'r' line must have %s values: r %s" % (descriptor._name(), expected_field_count, value))
+
+ if not stem.util.tor_tools.is_valid_nickname(r_comp[0]):
+ raise ValueError("%s nickname isn't valid: %s" % (descriptor._name(), r_comp[0]))
+ elif not stem.util.connection.is_valid_ipv4_address(r_comp[5]):
+ raise ValueError("%s address isn't a valid IPv4 address: %s" % (descriptor._name(), r_comp[5]))
+ elif not stem.util.connection.is_valid_port(r_comp[6]):
+ raise ValueError('%s ORPort is invalid: %s' % (descriptor._name(), r_comp[6]))
+ elif not stem.util.connection.is_valid_port(r_comp[7], allow_zero = True):
+ raise ValueError('%s DirPort is invalid: %s' % (descriptor._name(), r_comp[7]))
+
+ descriptor.nickname = r_comp[0]
+ descriptor.fingerprint = _base64_to_hex(r_comp[1])
+
+ if include_digest:
+ descriptor.digest = _base64_to_hex(r_comp[2])
+
+ descriptor.address = r_comp[5]
+ descriptor.or_port = int(r_comp[6])
+ descriptor.dir_port = None if r_comp[7] == '0' else int(r_comp[7])
+
+ try:
+ published = '%s %s' % (r_comp[3], r_comp[4])
+ descriptor.published = stem.util.str_tools._parse_timestamp(published)
+ except ValueError:
+ raise ValueError("Publication time time wasn't parsable: r %s" % value)
+
+
+def _parse_a_line(descriptor, entries):
+ # "a" SP address ":" portlist
+ # example: a [2001:888:2133:0:82:94:251:204]:9001
+
+ or_addresses = []
+
+ for value in _values('a', entries):
+ if ':' not in value:
+ raise ValueError("%s 'a' line must be of the form '[address]:[ports]': a %s" % (descriptor._name(), value))
+
+ address, port = value.rsplit(':', 1)
+ is_ipv6 = address.startswith('[') and address.endswith(']')
+
+ if is_ipv6:
+ address = address[1:-1] # remove brackets
+
+ if not ((not is_ipv6 and stem.util.connection.is_valid_ipv4_address(address)) or
+ (is_ipv6 and stem.util.connection.is_valid_ipv6_address(address))):
+ raise ValueError("%s 'a' line must start with an IPv6 address: a %s" % (descriptor._name(), value))
+
+ if stem.util.connection.is_valid_port(port):
+ or_addresses.append((address, int(port), is_ipv6))
+ else:
+ raise ValueError("%s 'a' line had an invalid port (%s): a %s" % (descriptor._name(), port, value))
+
+ descriptor.or_addresses = or_addresses
+
+
+def _parse_s_line(descriptor, entries):
+ # "s" Flags
+ # example: s Named Running Stable Valid
+
+ value = _value('s', entries)
+ flags = [] if value == '' else value.split(' ')
+ descriptor.flags = flags
+
+ for flag in flags:
+ if flags.count(flag) > 1:
+ raise ValueError('%s had duplicate flags: s %s' % (descriptor._name(), value))
+ elif flag == '':
+ raise ValueError("%s had extra whitespace on its 's' line: s %s" % (descriptor._name(), value))
+
+
+def _parse_v_line(descriptor, entries):
+ # "v" version
+ # example: v Tor 0.2.2.35
+ #
+ # The spec says that if this starts with "Tor " then what follows is a
+ # tor version. If not then it has "upgraded to a more sophisticated
+ # protocol versioning system".
+
+ value = _value('v', entries)
+ descriptor.version_line = value
+
+ if value.startswith('Tor '):
+ try:
+ descriptor.version = stem.version._get_version(value[4:])
+ except ValueError as exc:
+ raise ValueError('%s has a malformed tor version (%s): v %s' % (descriptor._name(), exc, value))
+
+
+def _parse_w_line(descriptor, entries):
+ # "w" "Bandwidth=" INT ["Measured=" INT] ["Unmeasured=1"]
+ # example: w Bandwidth=7980
+
+ value = _value('w', entries)
+ w_comp = value.split(' ')
+
+ if len(w_comp) < 1:
+ raise ValueError("%s 'w' line is blank: w %s" % (descriptor._name(), value))
+ elif not w_comp[0].startswith('Bandwidth='):
+ raise ValueError("%s 'w' line needs to start with a 'Bandwidth=' entry: w %s" % (descriptor._name(), value))
+
+ bandwidth = None
+ measured = None
+ is_unmeasured = False
+ unrecognized_bandwidth_entries = []
+
+ for w_entry in w_comp:
+ if '=' in w_entry:
+ w_key, w_value = w_entry.split('=', 1)
+ else:
+ w_key, w_value = w_entry, None
+
+ if w_key == 'Bandwidth':
+ if not (w_value and w_value.isdigit()):
+ raise ValueError("%s 'Bandwidth=' entry needs to have a numeric value: w %s" % (descriptor._name(), value))
+
+ bandwidth = int(w_value)
+ elif w_key == 'Measured':
+ if not (w_value and w_value.isdigit()):
+ raise ValueError("%s 'Measured=' entry needs to have a numeric value: w %s" % (descriptor._name(), value))
+
+ measured = int(w_value)
+ elif w_key == 'Unmeasured':
+ if w_value != '1':
+ raise ValueError("%s 'Unmeasured=' should only have the value of '1': w %s" % (descriptor._name(), value))
+
+ is_unmeasured = True
+ else:
+ unrecognized_bandwidth_entries.append(w_entry)
+
+ descriptor.bandwidth = bandwidth
+ descriptor.measured = measured
+ descriptor.is_unmeasured = is_unmeasured
+ descriptor.unrecognized_bandwidth_entries = unrecognized_bandwidth_entries
+
+
+def _parse_p_line(descriptor, entries):
+ # "p" ("accept" / "reject") PortList
+ # p reject 1-65535
+ # example: p accept 80,110,143,443,993,995,6660-6669,6697,7000-7001
+
+ value = _value('p', entries)
+
+ try:
+ descriptor.exit_policy = stem.exit_policy.MicroExitPolicy(value)
+ except ValueError as exc:
+ raise ValueError('%s exit policy is malformed (%s): p %s' % (descriptor._name(), exc, value))
+
+
+def _parse_m_line(descriptor, entries):
+ # "m" methods 1*(algorithm "=" digest)
+ # example: m 8,9,10,11,12 sha256=g1vx9si329muxV3tquWIXXySNOIwRGMeAESKs/v4DWs
+
+ all_hashes = []
+
+ for value in _values('m', entries):
+ m_comp = value.split(' ')
+
+ if not (descriptor.document and descriptor.document.is_vote):
+ vote_status = 'vote' if descriptor.document else '<undefined document>'
+ raise ValueError("%s 'm' line should only appear in votes (appeared in a %s): m %s" % (descriptor._name(), vote_status, value))
+ elif len(m_comp) < 1:
+ raise ValueError("%s 'm' line needs to start with a series of methods: m %s" % (descriptor._name(), value))
+
+ try:
+ methods = [int(entry) for entry in m_comp[0].split(',')]
+ except ValueError:
+ raise ValueError('%s microdescriptor methods should be a series of comma separated integers: m %s' % (descriptor._name(), value))
+
+ hashes = {}
+
+ for entry in m_comp[1:]:
+ if '=' not in entry:
+ raise ValueError("%s can only have a series of 'algorithm=digest' mappings after the methods: m %s" % (descriptor._name(), value))
+
+ hash_name, digest = entry.split('=', 1)
+ hashes[hash_name] = digest
+
+ all_hashes.append((methods, hashes))
+
+ descriptor.microdescriptor_hashes = all_hashes
+
+
+def _parse_microdescriptor_m_line(descriptor, entries):
+ # "m" digest
+ # example: m aiUklwBrua82obG5AsTX+iEpkjQA2+AQHxZ7GwMfY70
+
+ descriptor.digest = _base64_to_hex(_value('m', entries), check_if_fingerprint = False)
+
+
+def _base64_to_hex(identity, check_if_fingerprint = True):
+ """
+ Decodes a base64 value to hex. For example...
+
+ ::
+
+ >>> _base64_to_hex('p1aag7VwarGxqctS7/fS0y5FU+s')
+ 'A7569A83B5706AB1B1A9CB52EFF7D2D32E4553EB'
+
+ :param str identity: encoded fingerprint from the consensus
+ :param bool check_if_fingerprint: asserts that the result is a fingerprint if **True**
+
+ :returns: **str** with the uppercase hex encoding of the relay's fingerprint
+
+ :raises: **ValueError** if the result isn't a valid fingerprint
+ """
+
+ # trailing equal signs were stripped from the identity
+ missing_padding = len(identity) % 4
+ identity += '=' * missing_padding
+
+ try:
+ identity_decoded = base64.b64decode(stem.util.str_tools._to_bytes(identity))
+ except (TypeError, binascii.Error):
+ raise ValueError("Unable to decode identity string '%s'" % identity)
+
+ fingerprint = binascii.b2a_hex(identity_decoded).upper()
+
+ if stem.prereq.is_python_3():
+ fingerprint = stem.util.str_tools._to_unicode(fingerprint)
+
+ if check_if_fingerprint:
+ if not stem.util.tor_tools.is_valid_fingerprint(fingerprint):
+ raise ValueError("Decoded '%s' to be '%s', which isn't a valid fingerprint" % (identity, fingerprint))
+
+ return fingerprint
+
+
+class RouterStatusEntry(Descriptor):
+ """
+ Information about an individual router stored within a network status
+ document. This is the common parent for concrete status entry types.
+
+ :var stem.descriptor.networkstatus.NetworkStatusDocument document: **\*** document that this descriptor came from
+
+ :var str nickname: **\*** router's nickname
+ :var str fingerprint: **\*** router's fingerprint
+ :var datetime published: **\*** router's publication
+ :var str address: **\*** router's IP address
+ :var int or_port: **\*** router's ORPort
+ :var int dir_port: **\*** router's DirPort
+
+ :var list flags: **\*** list of :data:`~stem.Flag` associated with the relay
+
+ :var stem.version.Version version: parsed version of tor, this is **None** if
+ the relay's using a new versioning scheme
+ :var str version_line: versioning information reported by the relay
+ """
+
+ ATTRIBUTES = {
+ 'nickname': (None, _parse_r_line),
+ 'fingerprint': (None, _parse_r_line),
+ 'published': (None, _parse_r_line),
+ 'address': (None, _parse_r_line),
+ 'or_port': (None, _parse_r_line),
+ 'dir_port': (None, _parse_r_line),
+
+ 'flags': (None, _parse_s_line),
+
+ 'version_line': (None, _parse_v_line),
+ 'version': (None, _parse_v_line),
+ }
+
+ PARSER_FOR_LINE = {
+ 'r': _parse_r_line,
+ 's': _parse_s_line,
+ 'v': _parse_v_line,
+ }
+
+ def __init__(self, content, validate = False, document = None):
+ """
+ Parse a router descriptor in a network status document.
+
+ :param str content: router descriptor content to be parsed
+ :param NetworkStatusDocument document: document this descriptor came from
+ :param bool validate: checks the validity of the content if **True**, skips
+ these checks otherwise
+
+ :raises: **ValueError** if the descriptor data is invalid
+ """
+
+ super(RouterStatusEntry, self).__init__(content, lazy_load = not validate)
+ self.document = document
+ entries = _get_descriptor_components(content, validate)
+
+ if validate:
+ for keyword in self._required_fields():
+ if keyword not in entries:
+ raise ValueError("%s must have a '%s' line:\n%s" % (self._name(True), keyword, str(self)))
+
+ for keyword in self._single_fields():
+ if keyword in entries and len(entries[keyword]) > 1:
+ raise ValueError("%s can only have a single '%s' line, got %i:\n%s" % (self._name(True), keyword, len(entries[keyword]), str(self)))
+
+ if 'r' != list(entries.keys())[0]:
+ raise ValueError("%s are expected to start with a 'r' line:\n%s" % (self._name(True), str(self)))
+
+ self._parse(entries, validate)
+ else:
+ self._entries = entries
+
+ def _name(self, is_plural = False):
+ """
+ Name for this descriptor type.
+ """
+
+ return 'Router status entries' if is_plural else 'Router status entry'
+
+ def _required_fields(self):
+ """
+ Provides lines that must appear in the descriptor.
+ """
+
+ return ()
+
+ def _single_fields(self):
+ """
+ Provides lines that can only appear in the descriptor once.
+ """
+
+ return ()
+
+ def _compare(self, other, method):
+ if not isinstance(other, RouterStatusEntry):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
+
+
+class RouterStatusEntryV2(RouterStatusEntry):
+ """
+ Information about an individual router stored within a version 2 network
+ status document.
+
+ :var str digest: **\*** router's upper-case hex digest
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as **None** if undefined
+ """
+
+ ATTRIBUTES = dict(RouterStatusEntry.ATTRIBUTES, **{
+ 'digest': (None, _parse_r_line),
+ })
+
+ def _name(self, is_plural = False):
+ return 'Router status entries (v2)' if is_plural else 'Router status entry (v2)'
+
+ def _required_fields(self):
+ return ('r')
+
+ def _single_fields(self):
+ return ('r', 's', 'v')
+
+ def _compare(self, other, method):
+ if not isinstance(other, RouterStatusEntryV2):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
+
+
+class RouterStatusEntryV3(RouterStatusEntry):
+ """
+ Information about an individual router stored within a version 3 network
+ status document.
+
+ :var list or_addresses: **\*** relay's OR addresses, this is a tuple listing
+ of the form (address (**str**), port (**int**), is_ipv6 (**bool**))
+ :var str digest: **\*** router's upper-case hex digest
+
+ :var int bandwidth: bandwidth claimed by the relay (in kb/s)
+ :var int measured: bandwidth measured to be available by the relay, this is a
+ unit-less heuristic generated by the Bandwidth authoritites to weight relay
+ selection
+ :var bool is_unmeasured: bandwidth measurement isn't based on three or more
+ measurements
+ :var list unrecognized_bandwidth_entries: **\*** bandwidth weighting
+ information that isn't yet recognized
+
+ :var stem.exit_policy.MicroExitPolicy exit_policy: router's exit policy
+
+ :var list microdescriptor_hashes: **\*** tuples of two values, the list of
+ consensus methods for generating a set of digests and the 'algorithm =>
+ digest' mappings
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as **None** if undefined
+ """
+
+ ATTRIBUTES = dict(RouterStatusEntry.ATTRIBUTES, **{
+ 'digest': (None, _parse_r_line),
+ 'or_addresses': ([], _parse_a_line),
+
+ 'bandwidth': (None, _parse_w_line),
+ 'measured': (None, _parse_w_line),
+ 'is_unmeasured': (False, _parse_w_line),
+ 'unrecognized_bandwidth_entries': ([], _parse_w_line),
+
+ 'exit_policy': (None, _parse_p_line),
+ 'microdescriptor_hashes': ([], _parse_m_line),
+ })
+
+ PARSER_FOR_LINE = dict(RouterStatusEntry.PARSER_FOR_LINE, **{
+ 'a': _parse_a_line,
+ 'w': _parse_w_line,
+ 'p': _parse_p_line,
+ 'm': _parse_m_line,
+ })
+
+ def _name(self, is_plural = False):
+ return 'Router status entries (v3)' if is_plural else 'Router status entry (v3)'
+
+ def _required_fields(self):
+ return ('r', 's')
+
+ def _single_fields(self):
+ return ('r', 's', 'v', 'w', 'p')
+
+ def _compare(self, other, method):
+ if not isinstance(other, RouterStatusEntryV3):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
+
+
+class RouterStatusEntryMicroV3(RouterStatusEntry):
+ """
+ Information about an individual router stored within a microdescriptor
+ flavored network status document.
+
+ :var int bandwidth: bandwidth claimed by the relay (in kb/s)
+ :var int measured: bandwidth measured to be available by the relay
+ :var bool is_unmeasured: bandwidth measurement isn't based on three or more
+ measurements
+ :var list unrecognized_bandwidth_entries: **\*** bandwidth weighting
+ information that isn't yet recognized
+
+ :var str digest: **\*** router's hex encoded digest of our corresponding microdescriptor
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as **None** if undefined
+ """
+
+ ATTRIBUTES = dict(RouterStatusEntry.ATTRIBUTES, **{
+ 'bandwidth': (None, _parse_w_line),
+ 'measured': (None, _parse_w_line),
+ 'is_unmeasured': (False, _parse_w_line),
+ 'unrecognized_bandwidth_entries': ([], _parse_w_line),
+
+ 'digest': (None, _parse_microdescriptor_m_line),
+ })
+
+ PARSER_FOR_LINE = dict(RouterStatusEntry.PARSER_FOR_LINE, **{
+ 'w': _parse_w_line,
+ 'm': _parse_microdescriptor_m_line,
+ })
+
+ def _name(self, is_plural = False):
+ return 'Router status entries (micro v3)' if is_plural else 'Router status entry (micro v3)'
+
+ def _required_fields(self):
+ return ('r', 's', 'm')
+
+ def _single_fields(self):
+ return ('r', 's', 'v', 'w', 'm')
+
+ def _compare(self, other, method):
+ if not isinstance(other, RouterStatusEntryMicroV3):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
diff --git a/ext/stem/descriptor/server_descriptor.py b/ext/stem/descriptor/server_descriptor.py
new file mode 100644
index 00000000..b375066f
--- /dev/null
+++ b/ext/stem/descriptor/server_descriptor.py
@@ -0,0 +1,822 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Parsing for Tor server descriptors, which contains the infrequently changing
+information about a Tor relay (contact information, exit policy, public keys,
+etc). This information is provided from a few sources...
+
+* The control port via 'GETINFO desc/\*' queries.
+
+* The 'cached-descriptors' file in Tor's data directory.
+
+* Archived descriptors provided by CollecTor
+ (https://collector.torproject.org/).
+
+* Directory authorities and mirrors via their DirPort.
+
+**Module Overview:**
+
+::
+
+ ServerDescriptor - Tor server descriptor.
+ |- RelayDescriptor - Server descriptor for a relay.
+ |
+ |- BridgeDescriptor - Scrubbed server descriptor for a bridge.
+ | |- is_scrubbed - checks if our content has been properly scrubbed
+ | +- get_scrubbing_issues - description of issues with our scrubbing
+ |
+ |- digest - calculates the upper-case hex digest value for our content
+ |- get_annotations - dictionary of content prior to the descriptor entry
+ +- get_annotation_lines - lines that provided the annotations
+"""
+
+import functools
+import hashlib
+import re
+
+import stem.descriptor.extrainfo_descriptor
+import stem.exit_policy
+import stem.prereq
+import stem.util.connection
+import stem.util.str_tools
+import stem.util.tor_tools
+import stem.version
+
+from stem import str_type
+
+from stem.descriptor import (
+ PGP_BLOCK_END,
+ Descriptor,
+ _get_descriptor_components,
+ _read_until_keywords,
+ _bytes_for_block,
+ _value,
+ _values,
+ _parse_simple_line,
+ _parse_bytes_line,
+ _parse_timestamp_line,
+ _parse_forty_character_hex,
+ _parse_key_block,
+)
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+# relay descriptors must have exactly one of the following
+REQUIRED_FIELDS = (
+ 'router',
+ 'bandwidth',
+ 'published',
+ 'onion-key',
+ 'signing-key',
+ 'router-signature',
+)
+
+# optional entries that can appear at most once
+SINGLE_FIELDS = (
+ 'platform',
+ 'fingerprint',
+ 'hibernating',
+ 'uptime',
+ 'contact',
+ 'read-history',
+ 'write-history',
+ 'eventdns',
+ 'family',
+ 'caches-extra-info',
+ 'extra-info-digest',
+ 'hidden-service-dir',
+ 'protocols',
+ 'allow-single-hop-exits',
+ 'ntor-onion-key',
+)
+
+DEFAULT_IPV6_EXIT_POLICY = stem.exit_policy.MicroExitPolicy('reject 1-65535')
+REJECT_ALL_POLICY = stem.exit_policy.ExitPolicy('reject *:*')
+
+
+def _parse_file(descriptor_file, is_bridge = False, validate = False, **kwargs):
+ """
+ Iterates over the server descriptors in a file.
+
+ :param file descriptor_file: file with descriptor content
+ :param bool is_bridge: parses the file as being a bridge descriptor
+ :param bool validate: checks the validity of the descriptor's content if
+ **True**, skips these checks otherwise
+ :param dict kwargs: additional arguments for the descriptor constructor
+
+ :returns: iterator for ServerDescriptor instances in the file
+
+ :raises:
+ * **ValueError** if the contents is malformed and validate is True
+ * **IOError** if the file can't be read
+ """
+
+ # Handler for relay descriptors
+ #
+ # Cached descriptors consist of annotations followed by the descriptor
+ # itself. For instance...
+ #
+ # @downloaded-at 2012-03-14 16:31:05
+ # @source "145.53.65.130"
+ # router caerSidi 71.35.143.157 9001 0 0
+ # platform Tor 0.2.1.30 on Linux x86_64
+ # <rest of the descriptor content>
+ # router-signature
+ # -----BEGIN SIGNATURE-----
+ # <signature for the above descriptor>
+ # -----END SIGNATURE-----
+ #
+ # Metrics descriptor files are the same, but lack any annotations. The
+ # following simply does the following...
+ #
+ # - parse as annotations until we get to 'router'
+ # - parse as descriptor content until we get to 'router-signature' followed
+ # by the end of the signature block
+ # - construct a descriptor and provide it back to the caller
+ #
+ # Any annotations after the last server descriptor is ignored (never provided
+ # to the caller).
+
+ while True:
+ annotations = _read_until_keywords('router', descriptor_file)
+
+ if not is_bridge:
+ descriptor_content = _read_until_keywords('router-signature', descriptor_file)
+
+ # we've reached the 'router-signature', now include the pgp style block
+
+ block_end_prefix = PGP_BLOCK_END.split(' ', 1)[0]
+ descriptor_content += _read_until_keywords(block_end_prefix, descriptor_file, True)
+ else:
+ descriptor_content = _read_until_keywords('router-digest', descriptor_file, True)
+
+ if descriptor_content:
+ if descriptor_content[0].startswith(b'@type'):
+ descriptor_content = descriptor_content[1:]
+
+ # strip newlines from annotations
+ annotations = list(map(bytes.strip, annotations))
+
+ descriptor_text = bytes.join(b'', descriptor_content)
+
+ if is_bridge:
+ yield BridgeDescriptor(descriptor_text, validate, annotations, **kwargs)
+ else:
+ yield RelayDescriptor(descriptor_text, validate, annotations, **kwargs)
+ else:
+ if validate and annotations:
+ orphaned_annotations = stem.util.str_tools._to_unicode(b'\n'.join(annotations))
+ raise ValueError('Content conform to being a server descriptor:\n%s' % orphaned_annotations)
+
+ break # done parsing descriptors
+
+
+def _parse_router_line(descriptor, entries):
+ # "router" nickname address ORPort SocksPort DirPort
+
+ value = _value('router', entries)
+ router_comp = value.split()
+
+ if len(router_comp) < 5:
+ raise ValueError('Router line must have five values: router %s' % value)
+ elif not stem.util.tor_tools.is_valid_nickname(router_comp[0]):
+ raise ValueError("Router line entry isn't a valid nickname: %s" % router_comp[0])
+ elif not stem.util.connection.is_valid_ipv4_address(router_comp[1]):
+ raise ValueError("Router line entry isn't a valid IPv4 address: %s" % router_comp[1])
+ elif not stem.util.connection.is_valid_port(router_comp[2], allow_zero = True):
+ raise ValueError("Router line's ORPort is invalid: %s" % router_comp[2])
+ elif not stem.util.connection.is_valid_port(router_comp[3], allow_zero = True):
+ raise ValueError("Router line's SocksPort is invalid: %s" % router_comp[3])
+ elif not stem.util.connection.is_valid_port(router_comp[4], allow_zero = True):
+ raise ValueError("Router line's DirPort is invalid: %s" % router_comp[4])
+
+ descriptor.nickname = router_comp[0]
+ descriptor.address = router_comp[1]
+ descriptor.or_port = int(router_comp[2])
+ descriptor.socks_port = None if router_comp[3] == '0' else int(router_comp[3])
+ descriptor.dir_port = None if router_comp[4] == '0' else int(router_comp[4])
+
+
+def _parse_bandwidth_line(descriptor, entries):
+ # "bandwidth" bandwidth-avg bandwidth-burst bandwidth-observed
+
+ value = _value('bandwidth', entries)
+ bandwidth_comp = value.split()
+
+ if len(bandwidth_comp) < 3:
+ raise ValueError('Bandwidth line must have three values: bandwidth %s' % value)
+ elif not bandwidth_comp[0].isdigit():
+ raise ValueError("Bandwidth line's average rate isn't numeric: %s" % bandwidth_comp[0])
+ elif not bandwidth_comp[1].isdigit():
+ raise ValueError("Bandwidth line's burst rate isn't numeric: %s" % bandwidth_comp[1])
+ elif not bandwidth_comp[2].isdigit():
+ raise ValueError("Bandwidth line's observed rate isn't numeric: %s" % bandwidth_comp[2])
+
+ descriptor.average_bandwidth = int(bandwidth_comp[0])
+ descriptor.burst_bandwidth = int(bandwidth_comp[1])
+ descriptor.observed_bandwidth = int(bandwidth_comp[2])
+
+
+def _parse_platform_line(descriptor, entries):
+ # "platform" string
+
+ _parse_bytes_line('platform', 'platform')(descriptor, entries)
+
+ # The platform attribute was set earlier. This line can contain any
+ # arbitrary data, but tor seems to report its version followed by the
+ # os like the following...
+ #
+ # platform Tor 0.2.2.35 (git-73ff13ab3cc9570d) on Linux x86_64
+ #
+ # There's no guarantee that we'll be able to pick these out the
+ # version, but might as well try to save our caller the effort.
+
+ value = _value('platform', entries)
+ platform_match = re.match('^(?:node-)?Tor (\S*).* on (.*)$', value)
+
+ if platform_match:
+ version_str, descriptor.operating_system = platform_match.groups()
+
+ try:
+ descriptor.tor_version = stem.version._get_version(version_str)
+ except ValueError:
+ pass
+
+
+def _parse_fingerprint_line(descriptor, entries):
+ # This is forty hex digits split into space separated groups of four.
+ # Checking that we match this pattern.
+
+ value = _value('fingerprint', entries)
+ fingerprint = value.replace(' ', '')
+
+ for grouping in value.split(' '):
+ if len(grouping) != 4:
+ raise ValueError('Fingerprint line should have groupings of four hex digits: %s' % value)
+
+ if not stem.util.tor_tools.is_valid_fingerprint(fingerprint):
+ raise ValueError('Tor relay fingerprints consist of forty hex digits: %s' % value)
+
+ descriptor.fingerprint = fingerprint
+
+
+def _parse_hibernating_line(descriptor, entries):
+ # "hibernating" 0|1 (in practice only set if one)
+
+ value = _value('hibernating', entries)
+
+ if value not in ('0', '1'):
+ raise ValueError('Hibernating line had an invalid value, must be zero or one: %s' % value)
+
+ descriptor.hibernating = value == '1'
+
+
+def _parse_hidden_service_dir_line(descriptor, entries):
+ value = _value('hidden-service-dir', entries)
+
+ if value:
+ descriptor.hidden_service_dir = value.split(' ')
+ else:
+ descriptor.hidden_service_dir = ['2']
+
+
+def _parse_uptime_line(descriptor, entries):
+ # We need to be tolerant of negative uptimes to accommodate a past tor
+ # bug...
+ #
+ # Changes in version 0.1.2.7-alpha - 2007-02-06
+ # - If our system clock jumps back in time, don't publish a negative
+ # uptime in the descriptor. Also, don't let the global rate limiting
+ # buckets go absurdly negative.
+ #
+ # After parsing all of the attributes we'll double check that negative
+ # uptimes only occurred prior to this fix.
+
+ value = _value('uptime', entries)
+
+ try:
+ descriptor.uptime = int(value)
+ except ValueError:
+ raise ValueError('Uptime line must have an integer value: %s' % value)
+
+
+def _parse_protocols_line(descriptor, entries):
+ value = _value('protocols', entries)
+ protocols_match = re.match('^Link (.*) Circuit (.*)$', value)
+
+ if not protocols_match:
+ raise ValueError('Protocols line did not match the expected pattern: protocols %s' % value)
+
+ link_versions, circuit_versions = protocols_match.groups()
+ descriptor.link_protocols = link_versions.split(' ')
+ descriptor.circuit_protocols = circuit_versions.split(' ')
+
+
+def _parse_or_address_line(descriptor, entries):
+ all_values = _values('or-address', entries)
+ or_addresses = []
+
+ for entry in all_values:
+ line = 'or-address %s' % entry
+
+ if ':' not in entry:
+ raise ValueError('or-address line missing a colon: %s' % line)
+
+ address, port = entry.rsplit(':', 1)
+ is_ipv6 = address.startswith('[') and address.endswith(']')
+
+ if is_ipv6:
+ address = address[1:-1] # remove brackets
+
+ if not ((not is_ipv6 and stem.util.connection.is_valid_ipv4_address(address)) or
+ (is_ipv6 and stem.util.connection.is_valid_ipv6_address(address))):
+ raise ValueError('or-address line has a malformed address: %s' % line)
+
+ if not stem.util.connection.is_valid_port(port):
+ raise ValueError('or-address line has a malformed port: %s' % line)
+
+ or_addresses.append((address, int(port), is_ipv6))
+
+ descriptor.or_addresses = or_addresses
+
+
+def _parse_history_line(keyword, history_end_attribute, history_interval_attribute, history_values_attribute, descriptor, entries):
+ value = _value(keyword, entries)
+ timestamp, interval, remainder = stem.descriptor.extrainfo_descriptor._parse_timestamp_and_interval(keyword, value)
+
+ try:
+ if remainder:
+ history_values = [int(entry) for entry in remainder.split(',')]
+ else:
+ history_values = []
+ except ValueError:
+ raise ValueError('%s line has non-numeric values: %s %s' % (keyword, keyword, value))
+
+ setattr(descriptor, history_end_attribute, timestamp)
+ setattr(descriptor, history_interval_attribute, interval)
+ setattr(descriptor, history_values_attribute, history_values)
+
+
+def _parse_exit_policy(descriptor, entries):
+ if hasattr(descriptor, '_unparsed_exit_policy'):
+ if descriptor._unparsed_exit_policy == [str_type('reject *:*')]:
+ descriptor.exit_policy = REJECT_ALL_POLICY
+ else:
+ descriptor.exit_policy = stem.exit_policy.ExitPolicy(*descriptor._unparsed_exit_policy)
+
+ del descriptor._unparsed_exit_policy
+
+
+_parse_contact_line = _parse_bytes_line('contact', 'contact')
+_parse_published_line = _parse_timestamp_line('published', 'published')
+_parse_extrainfo_digest_line = _parse_forty_character_hex('extra-info-digest', 'extra_info_digest')
+_parse_read_history_line = functools.partial(_parse_history_line, 'read-history', 'read_history_end', 'read_history_interval', 'read_history_values')
+_parse_write_history_line = functools.partial(_parse_history_line, 'write-history', 'write_history_end', 'write_history_interval', 'write_history_values')
+_parse_ipv6_policy_line = lambda descriptor, entries: setattr(descriptor, 'exit_policy_v6', stem.exit_policy.MicroExitPolicy(_value('ipv6-policy', entries)))
+_parse_allow_single_hop_exits_line = lambda descriptor, entries: setattr(descriptor, 'allow_single_hop_exits', 'allow_single_hop_exits' in entries)
+_parse_caches_extra_info_line = lambda descriptor, entries: setattr(descriptor, 'extra_info_cache', 'extra_info_cache' in entries)
+_parse_family_line = lambda descriptor, entries: setattr(descriptor, 'family', set(_value('family', entries).split(' ')))
+_parse_eventdns_line = lambda descriptor, entries: setattr(descriptor, 'eventdns', _value('eventdns', entries) == '1')
+_parse_onion_key_line = _parse_key_block('onion-key', 'onion_key', 'RSA PUBLIC KEY')
+_parse_signing_key_line = _parse_key_block('signing-key', 'signing_key', 'RSA PUBLIC KEY')
+_parse_router_signature_line = _parse_key_block('router-signature', 'signature', 'SIGNATURE')
+_parse_ntor_onion_key_line = _parse_simple_line('ntor-onion-key', 'ntor_onion_key')
+_parse_router_digest_line = _parse_forty_character_hex('router-digest', '_digest')
+
+
+class ServerDescriptor(Descriptor):
+ """
+ Common parent for server descriptors.
+
+ :var str nickname: **\*** relay's nickname
+ :var str fingerprint: identity key fingerprint
+ :var datetime published: **\*** time in UTC when this descriptor was made
+
+ :var str address: **\*** IPv4 address of the relay
+ :var int or_port: **\*** port used for relaying
+ :var int socks_port: **\*** port used as client (deprecated, always **None**)
+ :var int dir_port: **\*** port used for descriptor mirroring
+
+ :var bytes platform: line with operating system and tor version
+ :var stem.version.Version tor_version: version of tor
+ :var str operating_system: operating system
+ :var int uptime: uptime when published in seconds
+ :var bytes contact: contact information
+ :var stem.exit_policy.ExitPolicy exit_policy: **\*** stated exit policy
+ :var stem.exit_policy.MicroExitPolicy exit_policy_v6: **\*** exit policy for IPv6
+ :var set family: **\*** nicknames or fingerprints of declared family
+
+ :var int average_bandwidth: **\*** average rate it's willing to relay in bytes/s
+ :var int burst_bandwidth: **\*** burst rate it's willing to relay in bytes/s
+ :var int observed_bandwidth: **\*** estimated capacity based on usage in bytes/s
+
+ :var list link_protocols: link protocols supported by the relay
+ :var list circuit_protocols: circuit protocols supported by the relay
+ :var bool hibernating: **\*** hibernating when published
+ :var bool allow_single_hop_exits: **\*** flag if single hop exiting is allowed
+ :var bool extra_info_cache: **\*** flag if a mirror for extra-info documents
+ :var str extra_info_digest: upper-case hex encoded digest of our extra-info document
+ :var bool eventdns: flag for evdns backend (deprecated, always unset)
+ :var list or_addresses: **\*** alternative for our address/or_port
+ attributes, each entry is a tuple of the form (address (**str**), port
+ (**int**), is_ipv6 (**bool**))
+
+ Deprecated, moved to extra-info descriptor...
+
+ :var datetime read_history_end: end of the sampling interval
+ :var int read_history_interval: seconds per interval
+ :var list read_history_values: bytes read during each interval
+
+ :var datetime write_history_end: end of the sampling interval
+ :var int write_history_interval: seconds per interval
+ :var list write_history_values: bytes written during each interval
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as **None** if undefined
+ """
+
+ ATTRIBUTES = {
+ 'nickname': (None, _parse_router_line),
+ 'fingerprint': (None, _parse_fingerprint_line),
+ 'contact': (None, _parse_contact_line),
+ 'published': (None, _parse_published_line),
+ 'exit_policy': (None, _parse_exit_policy),
+
+ 'address': (None, _parse_router_line),
+ 'or_port': (None, _parse_router_line),
+ 'socks_port': (None, _parse_router_line),
+ 'dir_port': (None, _parse_router_line),
+
+ 'platform': (None, _parse_platform_line),
+ 'tor_version': (None, _parse_platform_line),
+ 'operating_system': (None, _parse_platform_line),
+ 'uptime': (None, _parse_uptime_line),
+ 'exit_policy_v6': (DEFAULT_IPV6_EXIT_POLICY, _parse_ipv6_policy_line),
+ 'family': (set(), _parse_family_line),
+
+ 'average_bandwidth': (None, _parse_bandwidth_line),
+ 'burst_bandwidth': (None, _parse_bandwidth_line),
+ 'observed_bandwidth': (None, _parse_bandwidth_line),
+
+ 'link_protocols': (None, _parse_protocols_line),
+ 'circuit_protocols': (None, _parse_protocols_line),
+ 'hibernating': (False, _parse_hibernating_line),
+ 'allow_single_hop_exits': (False, _parse_allow_single_hop_exits_line),
+ 'extra_info_cache': (False, _parse_caches_extra_info_line),
+ 'extra_info_digest': (None, _parse_extrainfo_digest_line),
+ 'hidden_service_dir': (None, _parse_hidden_service_dir_line),
+ 'eventdns': (None, _parse_eventdns_line),
+ 'or_addresses': ([], _parse_or_address_line),
+
+ 'read_history_end': (None, _parse_read_history_line),
+ 'read_history_interval': (None, _parse_read_history_line),
+ 'read_history_values': (None, _parse_read_history_line),
+
+ 'write_history_end': (None, _parse_write_history_line),
+ 'write_history_interval': (None, _parse_write_history_line),
+ 'write_history_values': (None, _parse_write_history_line),
+ }
+
+ PARSER_FOR_LINE = {
+ 'router': _parse_router_line,
+ 'bandwidth': _parse_bandwidth_line,
+ 'platform': _parse_platform_line,
+ 'published': _parse_published_line,
+ 'fingerprint': _parse_fingerprint_line,
+ 'contact': _parse_contact_line,
+ 'hibernating': _parse_hibernating_line,
+ 'extra-info-digest': _parse_extrainfo_digest_line,
+ 'hidden-service-dir': _parse_hidden_service_dir_line,
+ 'uptime': _parse_uptime_line,
+ 'protocols': _parse_protocols_line,
+ 'or-address': _parse_or_address_line,
+ 'read-history': _parse_read_history_line,
+ 'write-history': _parse_write_history_line,
+ 'ipv6-policy': _parse_ipv6_policy_line,
+ 'allow-single-hop-exits': _parse_allow_single_hop_exits_line,
+ 'caches-extra-info': _parse_caches_extra_info_line,
+ 'family': _parse_family_line,
+ 'eventdns': _parse_eventdns_line,
+ }
+
+ def __init__(self, raw_contents, validate = False, annotations = None):
+ """
+ Server descriptor constructor, created from an individual relay's
+ descriptor content (as provided by 'GETINFO desc/*', cached descriptors,
+ and metrics).
+
+ By default this validates the descriptor's content as it's parsed. This
+ validation can be disables to either improve performance or be accepting of
+ malformed data.
+
+ :param str raw_contents: descriptor content provided by the relay
+ :param bool validate: checks the validity of the descriptor's content if
+ **True**, skips these checks otherwise
+ :param list annotations: lines that appeared prior to the descriptor
+
+ :raises: **ValueError** if the contents is malformed and validate is True
+ """
+
+ super(ServerDescriptor, self).__init__(raw_contents, lazy_load = not validate)
+ self._annotation_lines = annotations if annotations else []
+
+ # A descriptor contains a series of 'keyword lines' which are simply a
+ # keyword followed by an optional value. Lines can also be followed by a
+ # signature block.
+ #
+ # We care about the ordering of 'accept' and 'reject' entries because this
+ # influences the resulting exit policy, but for everything else the order
+ # does not matter so breaking it into key / value pairs.
+
+ entries, self._unparsed_exit_policy = _get_descriptor_components(stem.util.str_tools._to_unicode(raw_contents), validate, ('accept', 'reject'))
+
+ if validate:
+ self._parse(entries, validate)
+
+ _parse_exit_policy(self, entries)
+
+ # if we have a negative uptime and a tor version that shouldn't exhibit
+ # this bug then fail validation
+
+ if validate and self.uptime and self.tor_version:
+ if self.uptime < 0 and self.tor_version >= stem.version.Version('0.1.2.7'):
+ raise ValueError("Descriptor for version '%s' had a negative uptime value: %i" % (self.tor_version, self.uptime))
+
+ self._check_constraints(entries)
+ else:
+ self._entries = entries
+
+ def digest(self):
+ """
+ Provides the hex encoded sha1 of our content. This value is part of the
+ network status entry for this relay.
+
+ :returns: **unicode** with the upper-case hex digest value for this server descriptor
+ """
+
+ raise NotImplementedError('Unsupported Operation: this should be implemented by the ServerDescriptor subclass')
+
+ @lru_cache()
+ def get_annotations(self):
+ """
+ Provides content that appeared prior to the descriptor. If this comes from
+ the cached-descriptors file then this commonly contains content like...
+
+ ::
+
+ @downloaded-at 2012-03-18 21:18:29
+ @source "173.254.216.66"
+
+ :returns: **dict** with the key/value pairs in our annotations
+ """
+
+ annotation_dict = {}
+
+ for line in self._annotation_lines:
+ if b' ' in line:
+ key, value = line.split(b' ', 1)
+ annotation_dict[key] = value
+ else:
+ annotation_dict[line] = None
+
+ return annotation_dict
+
+ def get_annotation_lines(self):
+ """
+ Provides the lines of content that appeared prior to the descriptor. This
+ is the same as the
+ :func:`~stem.descriptor.server_descriptor.ServerDescriptor.get_annotations`
+ results, but with the unparsed lines and ordering retained.
+
+ :returns: **list** with the lines of annotation that came before this descriptor
+ """
+
+ return self._annotation_lines
+
+ def _check_constraints(self, entries):
+ """
+ Does a basic check that the entries conform to this descriptor type's
+ constraints.
+
+ :param dict entries: keyword => (value, pgp key) entries
+
+ :raises: **ValueError** if an issue arises in validation
+ """
+
+ for keyword in self._required_fields():
+ if keyword not in entries:
+ raise ValueError("Descriptor must have a '%s' entry" % keyword)
+
+ for keyword in self._single_fields():
+ if keyword in entries and len(entries[keyword]) > 1:
+ raise ValueError("The '%s' entry can only appear once in a descriptor" % keyword)
+
+ expected_first_keyword = self._first_keyword()
+ if expected_first_keyword and expected_first_keyword != list(entries.keys())[0]:
+ raise ValueError("Descriptor must start with a '%s' entry" % expected_first_keyword)
+
+ expected_last_keyword = self._last_keyword()
+ if expected_last_keyword and expected_last_keyword != list(entries.keys())[-1]:
+ raise ValueError("Descriptor must end with a '%s' entry" % expected_last_keyword)
+
+ if not self.exit_policy:
+ raise ValueError("Descriptor must have at least one 'accept' or 'reject' entry")
+
+ # Constraints that the descriptor must meet to be valid. These can be None if
+ # not applicable.
+
+ def _required_fields(self):
+ return REQUIRED_FIELDS
+
+ def _single_fields(self):
+ return REQUIRED_FIELDS + SINGLE_FIELDS
+
+ def _first_keyword(self):
+ return 'router'
+
+ def _last_keyword(self):
+ return 'router-signature'
+
+
+class RelayDescriptor(ServerDescriptor):
+ """
+ Server descriptor (`descriptor specification
+ <https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt>`_)
+
+ :var str onion_key: **\*** key used to encrypt EXTEND cells
+ :var str ntor_onion_key: base64 key used to encrypt EXTEND in the ntor protocol
+ :var str signing_key: **\*** relay's long-term identity key
+ :var str signature: **\*** signature for this descriptor
+
+ **\*** attribute is required when we're parsed with validation
+ """
+
+ ATTRIBUTES = dict(ServerDescriptor.ATTRIBUTES, **{
+ 'onion_key': (None, _parse_onion_key_line),
+ 'ntor_onion_key': (None, _parse_ntor_onion_key_line),
+ 'signing_key': (None, _parse_signing_key_line),
+ 'signature': (None, _parse_router_signature_line),
+ })
+
+ PARSER_FOR_LINE = dict(ServerDescriptor.PARSER_FOR_LINE, **{
+ 'onion-key': _parse_onion_key_line,
+ 'ntor-onion-key': _parse_ntor_onion_key_line,
+ 'signing-key': _parse_signing_key_line,
+ 'router-signature': _parse_router_signature_line,
+ })
+
+ def __init__(self, raw_contents, validate = False, annotations = None):
+ super(RelayDescriptor, self).__init__(raw_contents, validate, annotations)
+
+ if validate:
+ if self.fingerprint:
+ key_hash = hashlib.sha1(_bytes_for_block(self.signing_key)).hexdigest()
+
+ if key_hash != self.fingerprint.lower():
+ raise ValueError('Fingerprint does not match the hash of our signing key (fingerprint: %s, signing key hash: %s)' % (self.fingerprint.lower(), key_hash))
+
+ if stem.prereq.is_crypto_available():
+ signed_digest = self._digest_for_signature(self.signing_key, self.signature)
+
+ if signed_digest != self.digest():
+ raise ValueError('Decrypted digest does not match local digest (calculated: %s, local: %s)' % (signed_digest, self.digest()))
+
+ @lru_cache()
+ def digest(self):
+ """
+ Provides the digest of our descriptor's content.
+
+ :returns: the digest string encoded in uppercase hex
+
+ :raises: ValueError if the digest canot be calculated
+ """
+
+ return self._digest_for_content(b'router ', b'\nrouter-signature\n')
+
+ def _compare(self, other, method):
+ if not isinstance(other, RelayDescriptor):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __hash__(self):
+ return hash(str(self).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
+
+
+class BridgeDescriptor(ServerDescriptor):
+ """
+ Bridge descriptor (`bridge descriptor specification
+ <https://collector.torproject.org/formats.html#bridge-descriptors>`_)
+ """
+
+ ATTRIBUTES = dict(ServerDescriptor.ATTRIBUTES, **{
+ '_digest': (None, _parse_router_digest_line),
+ })
+
+ PARSER_FOR_LINE = dict(ServerDescriptor.PARSER_FOR_LINE, **{
+ 'router-digest': _parse_router_digest_line,
+ })
+
+ def digest(self):
+ return self._digest
+
+ def is_scrubbed(self):
+ """
+ Checks if we've been properly scrubbed in accordance with the `bridge
+ descriptor specification
+ <https://collector.torproject.org/formats.html#bridge-descriptors>`_.
+ Validation is a moving target so this may not be fully up to date.
+
+ :returns: **True** if we're scrubbed, **False** otherwise
+ """
+
+ return self.get_scrubbing_issues() == []
+
+ @lru_cache()
+ def get_scrubbing_issues(self):
+ """
+ Provides issues with our scrubbing.
+
+ :returns: **list** of strings which describe issues we have with our
+ scrubbing, this list is empty if we're properly scrubbed
+ """
+
+ issues = []
+
+ if not self.address.startswith('10.'):
+ issues.append("Router line's address should be scrubbed to be '10.x.x.x': %s" % self.address)
+
+ if self.contact and self.contact != 'somebody':
+ issues.append("Contact line should be scrubbed to be 'somebody', but instead had '%s'" % self.contact)
+
+ for address, _, is_ipv6 in self.or_addresses:
+ if not is_ipv6 and not address.startswith('10.'):
+ issues.append("or-address line's address should be scrubbed to be '10.x.x.x': %s" % address)
+ elif is_ipv6 and not address.startswith('fd9f:2e19:3bcf::'):
+ # TODO: this check isn't quite right because we aren't checking that
+ # the next grouping of hex digits contains 1-2 digits
+ issues.append("or-address line's address should be scrubbed to be 'fd9f:2e19:3bcf::xx:xxxx': %s" % address)
+
+ for line in self.get_unrecognized_lines():
+ if line.startswith('onion-key '):
+ issues.append('Bridge descriptors should have their onion-key scrubbed: %s' % line)
+ elif line.startswith('signing-key '):
+ issues.append('Bridge descriptors should have their signing-key scrubbed: %s' % line)
+ elif line.startswith('router-signature '):
+ issues.append('Bridge descriptors should have their signature scrubbed: %s' % line)
+
+ return issues
+
+ def _required_fields(self):
+ # bridge required fields are the same as a relay descriptor, minus items
+ # excluded according to the format page
+
+ excluded_fields = [
+ 'onion-key',
+ 'signing-key',
+ 'router-signature',
+ ]
+
+ included_fields = [
+ 'router-digest',
+ ]
+
+ return tuple(included_fields + [f for f in REQUIRED_FIELDS if f not in excluded_fields])
+
+ def _single_fields(self):
+ return self._required_fields() + SINGLE_FIELDS
+
+ def _last_keyword(self):
+ return None
+
+ def _compare(self, other, method):
+ if not isinstance(other, BridgeDescriptor):
+ return False
+
+ return method(str(self).strip(), str(other).strip())
+
+ def __hash__(self):
+ return hash(str(self).strip())
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __lt__(self, other):
+ return self._compare(other, lambda s, o: s < o)
+
+ def __le__(self, other):
+ return self._compare(other, lambda s, o: s <= o)
diff --git a/ext/stem/descriptor/tordnsel.py b/ext/stem/descriptor/tordnsel.py
new file mode 100644
index 00000000..75a252bd
--- /dev/null
+++ b/ext/stem/descriptor/tordnsel.py
@@ -0,0 +1,117 @@
+# Copyright 2013-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Parsing for `TorDNSEL <https://www.torproject.org/projects/tordnsel.html.en>`_
+exit list files.
+
+::
+
+ TorDNSEL - Exit list provided by TorDNSEL
+"""
+
+import stem.util.connection
+import stem.util.str_tools
+import stem.util.tor_tools
+
+from stem.descriptor import (
+ Descriptor,
+ _read_until_keywords,
+ _get_descriptor_components,
+)
+
+
+def _parse_file(tordnsel_file, validate = False, **kwargs):
+ """
+ Iterates over a tordnsel file.
+
+ :returns: iterator for :class:`~stem.descriptor.tordnsel.TorDNSEL`
+ instances in the file
+
+ :raises:
+ * **ValueError** if the contents is malformed and validate is **True**
+ * **IOError** if the file can't be read
+ """
+
+ # skip content prior to the first ExitNode
+ _read_until_keywords('ExitNode', tordnsel_file, skip = True)
+
+ while True:
+ contents = _read_until_keywords('ExitAddress', tordnsel_file)
+ contents += _read_until_keywords('ExitNode', tordnsel_file)
+
+ if contents:
+ yield TorDNSEL(bytes.join(b'', contents), validate, **kwargs)
+ else:
+ break # done parsing file
+
+
+class TorDNSEL(Descriptor):
+ """
+ TorDNSEL descriptor (`exitlist specification
+ <https://www.torproject.org/tordnsel/exitlist-spec.txt>`_)
+
+ :var str fingerprint: **\*** authority's fingerprint
+ :var datetime published: **\*** time in UTC when this descriptor was made
+ :var datetime last_status: **\*** time in UTC when the relay was seen in a v2 network status
+ :var list exit_addresses: **\*** list of (str address, datetime date) tuples consisting of the found IPv4 exit address and the time
+
+ **\*** attribute is either required when we're parsed with validation or has
+ a default value, others are left as **None** if undefined
+ """
+
+ def __init__(self, raw_contents, validate):
+ super(TorDNSEL, self).__init__(raw_contents)
+ raw_contents = stem.util.str_tools._to_unicode(raw_contents)
+ entries = _get_descriptor_components(raw_contents, validate)
+
+ self.fingerprint = None
+ self.published = None
+ self.last_status = None
+ self.exit_addresses = []
+
+ self._parse(entries, validate)
+
+ def _parse(self, entries, validate):
+
+ for keyword, values in list(entries.items()):
+ value, block_type, block_content = values[0]
+
+ if validate and block_content:
+ raise ValueError('Unexpected block content: %s' % block_content)
+
+ if keyword == 'ExitNode':
+ if validate and not stem.util.tor_tools.is_valid_fingerprint(value):
+ raise ValueError('Tor relay fingerprints consist of forty hex digits: %s' % value)
+
+ self.fingerprint = value
+ elif keyword == 'Published':
+ try:
+ self.published = stem.util.str_tools._parse_timestamp(value)
+ except ValueError:
+ if validate:
+ raise ValueError("Published time wasn't parsable: %s" % value)
+ elif keyword == 'LastStatus':
+ try:
+ self.last_status = stem.util.str_tools._parse_timestamp(value)
+ except ValueError:
+ if validate:
+ raise ValueError("LastStatus time wasn't parsable: %s" % value)
+ elif keyword == 'ExitAddress':
+ for value, block_type, block_content in values:
+ address, date = value.split(' ', 1)
+
+ if validate:
+ if not stem.util.connection.is_valid_ipv4_address(address):
+ raise ValueError("ExitAddress isn't a valid IPv4 address: %s" % address)
+ elif block_content:
+ raise ValueError('Unexpected block content: %s' % block_content)
+
+ try:
+ date = stem.util.str_tools._parse_timestamp(date)
+ self.exit_addresses.append((address, date))
+ except ValueError:
+ if validate:
+ raise ValueError("ExitAddress found time wasn't parsable: %s" % value)
+ elif validate:
+ raise ValueError('Unrecognized keyword: %s' % keyword)
diff --git a/ext/stem/exit_policy.py b/ext/stem/exit_policy.py
new file mode 100644
index 00000000..62b9a12f
--- /dev/null
+++ b/ext/stem/exit_policy.py
@@ -0,0 +1,1094 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Representation of tor exit policies. These can be easily used to check if
+exiting to a destination is permissible or not. For instance...
+
+::
+
+ >>> from stem.exit_policy import ExitPolicy, MicroExitPolicy
+ >>> policy = ExitPolicy('accept *:80', 'accept *:443', 'reject *:*')
+ >>> print(policy)
+ accept *:80, accept *:443, reject *:*
+ >>> print(policy.summary())
+ accept 80, 443
+ >>> policy.can_exit_to('75.119.206.243', 80)
+ True
+
+ >>> policy = MicroExitPolicy('accept 80,443')
+ >>> print(policy)
+ accept 80,443
+ >>> policy.can_exit_to('75.119.206.243', 80)
+ True
+
+::
+
+ ExitPolicy - Exit policy for a Tor relay
+ |- MicroExitPolicy - Microdescriptor exit policy
+ |
+ |- can_exit_to - check if exiting to this destination is allowed or not
+ |- is_exiting_allowed - check if any exiting is allowed
+ |- summary - provides a short label, similar to a microdescriptor
+ |- has_private - checks if policy has anything expanded from the 'private' keyword
+ |- strip_private - provides a copy of the policy without 'private' entries
+ |- has_default - checks if policy ends with the defaultly appended suffix
+ |- strip_default - provides a copy of the policy without the default suffix
+ |- __str__ - string representation
+ +- __iter__ - ExitPolicyRule entries that this contains
+
+ ExitPolicyRule - Single rule of an exit policy chain
+ |- MicroExitPolicyRule - Single rule for a microdescriptor policy
+ |
+ |- is_address_wildcard - checks if we'll accept any address
+ |- is_port_wildcard - checks if we'll accept any port
+ |- get_address_type - provides the protocol our ip address belongs to
+ |- is_match - checks if we match a given destination
+ |- get_mask - provides the address representation of our mask
+ |- get_masked_bits - provides the bit representation of our mask
+ |- is_default - flag indicating if this was part of the default end of a policy
+ |- is_private - flag indicating if this was expanded from a 'private' keyword
+ +- __str__ - string representation for this rule
+
+ get_config_policy - provides the ExitPolicy based on torrc rules
+
+.. data:: AddressType (enum)
+
+ Enumerations for IP address types that can be in an exit policy.
+
+ ============ ===========
+ AddressType Description
+ ============ ===========
+ **WILDCARD** any address of either IPv4 or IPv6
+ **IPv4** IPv4 address
+ **IPv6** IPv6 address
+ ============ ===========
+"""
+
+from __future__ import absolute_import
+
+import socket
+import zlib
+
+import stem.prereq
+import stem.util.connection
+import stem.util.enum
+import stem.util.str_tools
+
+from stem import str_type
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+AddressType = stem.util.enum.Enum(('WILDCARD', 'Wildcard'), ('IPv4', 'IPv4'), ('IPv6', 'IPv6'))
+
+# Addresses aliased by the 'private' policy. From the tor man page...
+#
+# To specify all internal and link-local networks (including 0.0.0.0/8,
+# 169.254.0.0/16, 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, and 172.16.0.0/12),
+# you can use the 'private' alias instead of an address.
+
+PRIVATE_ADDRESSES = (
+ '0.0.0.0/8',
+ '169.254.0.0/16',
+ '127.0.0.0/8',
+ '192.168.0.0/16',
+ '10.0.0.0/8',
+ '172.16.0.0/12',
+)
+
+
+def get_config_policy(rules, ip_address = None):
+ """
+ Converts an ExitPolicy found in a torrc to a proper exit pattern. This
+ accounts for...
+
+ * ports being optional
+ * the 'private' keyword
+
+ :param str,list rules: comma separated rules or list to be converted
+ :param str ip_address: this relay's IP address for the 'private' policy if
+ it's present, this defaults to the local address
+
+ :returns: :class:`~stem.exit_policy.ExitPolicy` reflected by the rules
+
+ :raises: **ValueError** if input isn't a valid tor exit policy
+ """
+
+ if ip_address and not (stem.util.connection.is_valid_ipv4_address(ip_address) or stem.util.connection.is_valid_ipv6_address(ip_address)):
+ raise ValueError("%s isn't a valid IP address" % ip_address)
+
+ if isinstance(rules, (bytes, str_type)):
+ rules = rules.split(',')
+
+ result = []
+
+ for rule in rules:
+ rule = rule.strip()
+
+ if not rule:
+ continue
+
+ if ':' not in rule:
+ rule = '%s:*' % rule
+
+ if 'private' in rule:
+ acceptance = rule.split(' ', 1)[0]
+ port = rule.split(':', 1)[1]
+ addresses = list(PRIVATE_ADDRESSES)
+
+ if ip_address:
+ addresses.append(ip_address)
+ else:
+ try:
+ addresses.append(socket.gethostbyname(socket.gethostname()))
+ except:
+ pass # we might not have a network connection
+
+ for private_addr in addresses:
+ result.append(ExitPolicyRule('%s %s:%s' % (acceptance, private_addr, port)))
+ else:
+ result.append(ExitPolicyRule(rule))
+
+ # torrc policies can apply to IPv4 or IPv6, so we need to make sure /0
+ # addresses aren't treated as being a full wildcard
+
+ for rule in result:
+ rule._submask_wildcard = False
+
+ return ExitPolicy(*result)
+
+
+def _flag_private_rules(rules):
+ """
+ Determine if part of our policy was expanded from the 'private' keyword. This
+ doesn't differentiate if this actually came from the 'private' keyword or a
+ series of rules exactly matching it.
+ """
+
+ matches = []
+
+ for i, rule in enumerate(rules):
+ if i + len(PRIVATE_ADDRESSES) + 1 > len(rules):
+ break
+
+ rule_str = '%s/%s' % (rule.address, rule.get_masked_bits())
+
+ if rule_str == PRIVATE_ADDRESSES[0]:
+ matches.append(i)
+
+ for start_index in matches:
+ # To match the private policy the following must all be true...
+ #
+ # * series of addresses and bit masks match PRIVATE_ADDRESSES
+ # * all rules have the same port range and acceptance
+ # * all rules have the same acceptance (all accept or reject entries)
+
+ rule_set = rules[start_index:start_index + len(PRIVATE_ADDRESSES) + 1]
+ is_match = True
+
+ min_port, max_port = rule_set[0].min_port, rule_set[0].max_port
+ is_accept = rule_set[0].is_accept
+
+ for i, rule in enumerate(rule_set[:-1]):
+ rule_str = '%s/%s' % (rule.address, rule.get_masked_bits())
+
+ if rule_str != PRIVATE_ADDRESSES[i] or rule.min_port != min_port or rule.max_port != max_port or rule.is_accept != is_accept:
+ is_match = False
+ break
+
+ # The last rule is for the relay's public address, so it's dynamic.
+
+ last_rule = rule_set[-1]
+
+ if last_rule.is_address_wildcard() or last_rule.min_port != min_port or last_rule.max_port != max_port or last_rule.is_accept != is_accept:
+ is_match = False
+ if is_match:
+ for rule in rule_set:
+ rule._is_private = True
+
+
+def _flag_default_rules(rules):
+ """
+ Determine if part of our policy ends with the defaultly appended suffix.
+ """
+
+ if len(rules) >= len(DEFAULT_POLICY_RULES):
+ rules_suffix = tuple(rules[-len(DEFAULT_POLICY_RULES):])
+
+ if rules_suffix == DEFAULT_POLICY_RULES:
+ for rule in rules_suffix:
+ rule._is_default_suffix = True
+
+
+class ExitPolicy(object):
+ """
+ Policy for the destinations that a relay allows or denies exiting to. This
+ is, in effect, just a list of :class:`~stem.exit_policy.ExitPolicyRule`
+ entries.
+
+ :param list rules: **str** or :class:`~stem.exit_policy.ExitPolicyRule`
+ entries that make up this policy
+ """
+
+ def __init__(self, *rules):
+ # sanity check the types
+
+ for rule in rules:
+ if not isinstance(rule, (bytes, str_type, ExitPolicyRule)):
+ raise TypeError('Exit policy rules can only contain strings or ExitPolicyRules, got a %s (%s)' % (type(rule), rules))
+
+ # Unparsed representation of the rules we were constructed with. Our
+ # _get_rules() method consumes this to provide ExitPolicyRule instances.
+ # This is lazily evaluated so we don't need to actually parse the exit
+ # policy if it's never used.
+
+ is_all_str = True
+
+ for rule in rules:
+ if not isinstance(rule, (bytes, str_type)):
+ is_all_str = False
+
+ if rules and is_all_str:
+ byte_rules = [stem.util.str_tools._to_bytes(r) for r in rules]
+ self._input_rules = zlib.compress(b','.join(byte_rules))
+ else:
+ self._input_rules = rules
+
+ self._rules = None
+ self._hash = None
+
+ # Result when no rules apply. According to the spec policies default to 'is
+ # allowed', but our microdescriptor policy subclass might want to change
+ # this.
+
+ self._is_allowed_default = True
+
+ @lru_cache()
+ def can_exit_to(self, address = None, port = None, strict = False):
+ """
+ Checks if this policy allows exiting to a given destination or not. If the
+ address or port is omitted then this will check if we're allowed to exit to
+ any instances of the defined address or port.
+
+ :param str address: IPv4 or IPv6 address (with or without brackets)
+ :param int port: port number
+ :param bool strict: if the address or port is excluded then check if we can
+ exit to **all** instances of the defined address or port
+
+ :returns: **True** if exiting to this destination is allowed, **False** otherwise
+ """
+
+ for rule in self._get_rules():
+ if rule.is_match(address, port, strict):
+ return rule.is_accept
+
+ return self._is_allowed_default
+
+ @lru_cache()
+ def is_exiting_allowed(self):
+ """
+ Provides **True** if the policy allows exiting whatsoever, **False**
+ otherwise.
+ """
+
+ rejected_ports = set()
+
+ for rule in self._get_rules():
+ if rule.is_accept:
+ for port in range(rule.min_port, rule.max_port + 1):
+ if port not in rejected_ports:
+ return True
+ elif rule.is_address_wildcard():
+ if rule.is_port_wildcard():
+ return False
+ else:
+ rejected_ports.update(range(rule.min_port, rule.max_port + 1))
+
+ return self._is_allowed_default
+
+ @lru_cache()
+ def summary(self):
+ """
+ Provides a short description of our policy chain, similar to a
+ microdescriptor. This excludes entries that don't cover all IP
+ addresses, and is either white-list or blacklist policy based on
+ the final entry. For instance...
+
+ ::
+
+ >>> policy = ExitPolicy('accept *:80', 'accept *:443', 'reject *:*')
+ >>> policy.summary()
+ 'accept 80, 443'
+
+ >>> policy = ExitPolicy('accept *:443', 'reject *:1-1024', 'accept *:*')
+ >>> policy.summary()
+ 'reject 1-442, 444-1024'
+
+ :returns: **str** with a concise summary for our policy
+ """
+
+ # determines if we're a white-list or blacklist
+ is_whitelist = not self._is_allowed_default
+
+ for rule in self._get_rules():
+ if rule.is_address_wildcard() and rule.is_port_wildcard():
+ is_whitelist = not rule.is_accept
+ break
+
+ # Iterates over the policies and adds the the ports we'll return (ie,
+ # allows if a white-list and rejects if a blacklist). Regardless of a
+ # port's allow/reject policy, all further entries with that port are
+ # ignored since policies respect the first matching policy.
+
+ display_ports, skip_ports = [], set()
+
+ for rule in self._get_rules():
+ if not rule.is_address_wildcard():
+ continue
+ elif rule.is_port_wildcard():
+ break
+
+ for port in range(rule.min_port, rule.max_port + 1):
+ if port in skip_ports:
+ continue
+
+ # if accept + white-list or reject + blacklist then add
+ if rule.is_accept == is_whitelist:
+ display_ports.append(port)
+
+ # all further entries with this port should be ignored
+ skip_ports.add(port)
+
+ # convert port list to a list of ranges (ie, ['1-3'] rather than [1, 2, 3])
+ if display_ports:
+ display_ranges, temp_range = [], []
+ display_ports.sort()
+ display_ports.append(None) # ending item to include last range in loop
+
+ for port in display_ports:
+ if not temp_range or temp_range[-1] + 1 == port:
+ temp_range.append(port)
+ else:
+ if len(temp_range) > 1:
+ display_ranges.append('%i-%i' % (temp_range[0], temp_range[-1]))
+ else:
+ display_ranges.append(str(temp_range[0]))
+
+ temp_range = [port]
+ else:
+ # everything for the inverse
+ is_whitelist = not is_whitelist
+ display_ranges = ['1-65535']
+
+ # constructs the summary string
+ label_prefix = 'accept ' if is_whitelist else 'reject '
+
+ return (label_prefix + ', '.join(display_ranges)).strip()
+
+ def has_private(self):
+ """
+ Checks if we have any rules expanded from the 'private' keyword. Tor
+ appends these by default to the start of the policy and includes a dynamic
+ address (the relay's public IP).
+
+ .. versionadded:: 1.3.0
+
+ :returns: **True** if we have any private rules expanded from the 'private'
+ keyword, **False** otherwise
+ """
+
+ for rule in self._get_rules():
+ if rule.is_private():
+ return True
+
+ return False
+
+ def strip_private(self):
+ """
+ Provides a copy of this policy without 'private' policy entries.
+
+ .. versionadded:: 1.3.0
+
+ :returns: **ExitPolicy** without private rules
+ """
+
+ return ExitPolicy(*[rule for rule in self._get_rules() if not rule.is_private()])
+
+ def has_default(self):
+ """
+ Checks if we have the default policy suffix.
+
+ .. versionadded:: 1.3.0
+
+ :returns: **True** if we have the default policy suffix, **False** otherwise
+ """
+
+ for rule in self._get_rules():
+ if rule.is_default():
+ return True
+
+ return False
+
+ def strip_default(self):
+ """
+ Provides a copy of this policy without the default policy suffix.
+
+ .. versionadded:: 1.3.0
+
+ :returns: **ExitPolicy** without default rules
+ """
+
+ return ExitPolicy(*[rule for rule in self._get_rules() if not rule.is_default()])
+
+ def _get_rules(self):
+ if self._rules is None:
+ rules = []
+ is_all_accept, is_all_reject = True, True
+
+ if isinstance(self._input_rules, bytes):
+ decompressed_rules = zlib.decompress(self._input_rules).split(b',')
+ else:
+ decompressed_rules = self._input_rules
+
+ for rule in decompressed_rules:
+ if isinstance(rule, bytes):
+ rule = stem.util.str_tools._to_unicode(rule)
+
+ if isinstance(rule, str_type):
+ rule = ExitPolicyRule(rule.strip())
+
+ if rule.is_accept:
+ is_all_reject = False
+ else:
+ is_all_accept = False
+
+ rules.append(rule)
+
+ if rule.is_address_wildcard() and rule.is_port_wildcard():
+ break # this is a catch-all, no reason to include more
+
+ # If we only have one kind of entry *and* end with a wildcard then
+ # we might as well use the simpler version. For instance...
+ #
+ # reject *:80, reject *:443, reject *:*
+ #
+ # ... could also be represented as simply...
+ #
+ # reject *:*
+ #
+ # This mostly comes up with reject-all policies because the
+ # 'reject private:*' appends an extra seven rules that have no
+ # effect.
+
+ if rules and (rules[-1].is_address_wildcard() and rules[-1].is_port_wildcard()):
+ if is_all_accept:
+ rules = [ExitPolicyRule('accept *:*')]
+ elif is_all_reject:
+ rules = [ExitPolicyRule('reject *:*')]
+
+ _flag_private_rules(rules)
+ _flag_default_rules(rules)
+
+ self._rules = rules
+ self._input_rules = None
+
+ return self._rules
+
+ def __len__(self):
+ return len(self._get_rules())
+
+ def __iter__(self):
+ for rule in self._get_rules():
+ yield rule
+
+ @lru_cache()
+ def __str__(self):
+ return ', '.join([str(rule) for rule in self._get_rules()])
+
+ def __hash__(self):
+ if self._hash is None:
+ my_hash = 0
+
+ for rule in self._get_rules():
+ my_hash *= 1024
+ my_hash += hash(rule)
+
+ self._hash = my_hash
+
+ return self._hash
+
+ def __eq__(self, other):
+ if isinstance(other, ExitPolicy):
+ return self._get_rules() == list(other)
+ else:
+ return False
+
+
+class MicroExitPolicy(ExitPolicy):
+ """
+ Exit policy provided by the microdescriptors. This is a distilled version of
+ a normal :class:`~stem.exit_policy.ExitPolicy` contains, just consisting of a
+ list of ports that are either accepted or rejected. For instance...
+
+ ::
+
+ accept 80,443 # only accepts common http ports
+ reject 1-1024 # only accepts non-privileged ports
+
+ Since these policies are a subset of the exit policy information (lacking IP
+ ranges) clients can only use them to guess if a relay will accept traffic or
+ not. To quote the `dir-spec <https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt>`_ (section 3.2.1)...
+
+ ::
+
+ With microdescriptors, clients don't learn exact exit policies:
+ clients can only guess whether a relay accepts their request, try the
+ BEGIN request, and might get end-reason-exit-policy if they guessed
+ wrong, in which case they'll have to try elsewhere.
+
+ :var bool is_accept: **True** if these are ports that we accept, **False** if
+ they're ports that we reject
+
+ :param str policy: policy string that describes this policy
+ """
+
+ def __init__(self, policy):
+ # Microdescriptor policies are of the form...
+ #
+ # MicrodescriptrPolicy ::= ("accept" / "reject") SP PortList NL
+ # PortList ::= PortOrRange
+ # PortList ::= PortList "," PortOrRange
+ # PortOrRange ::= INT "-" INT / INT
+
+ self._policy = policy
+
+ if policy.startswith('accept'):
+ self.is_accept = True
+ elif policy.startswith('reject'):
+ self.is_accept = False
+ else:
+ raise ValueError("A microdescriptor exit policy must start with either 'accept' or 'reject': %s" % policy)
+
+ policy = policy[6:]
+
+ if not policy.startswith(' ') or (len(policy) - 1 != len(policy.lstrip())):
+ raise ValueError('A microdescriptor exit policy should have a space separating accept/reject from its port list: %s' % self._policy)
+
+ policy = policy[1:]
+
+ # convert our port list into MicroExitPolicyRule
+ rules = []
+
+ for port_entry in policy.split(','):
+ if '-' in port_entry:
+ min_port, max_port = port_entry.split('-', 1)
+ else:
+ min_port = max_port = port_entry
+
+ if not stem.util.connection.is_valid_port(min_port) or \
+ not stem.util.connection.is_valid_port(max_port):
+ raise ValueError("'%s' is an invalid port range" % port_entry)
+
+ rules.append(MicroExitPolicyRule(self.is_accept, int(min_port), int(max_port)))
+
+ super(MicroExitPolicy, self).__init__(*rules)
+ self._is_allowed_default = not self.is_accept
+
+ def __str__(self):
+ return self._policy
+
+ def __hash__(self):
+ return hash(str(self))
+
+ def __eq__(self, other):
+ if isinstance(other, MicroExitPolicy):
+ return str(self) == str(other)
+ else:
+ return False
+
+
+class ExitPolicyRule(object):
+ """
+ Single rule from the user's exit policy. These rules are chained together to
+ form complete policies that describe where a relay will and will not allow
+ traffic to exit.
+
+ The format of these rules are formally described in the `dir-spec
+ <https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt>`_ as an
+ 'exitpattern'. Note that while these are similar to tor's man page entry for
+ ExitPolicies, it's not the exact same. An exitpattern is better defined and
+ stricter in what it'll accept. For instance, ports are not optional and it
+ does not contain the 'private' alias.
+
+ This should be treated as an immutable object.
+
+ :var bool is_accept: indicates if exiting is allowed or disallowed
+
+ :var str address: address that this rule is for
+
+ :var int min_port: lower end of the port range that we include (inclusive)
+ :var int max_port: upper end of the port range that we include (inclusive)
+
+ :param str rule: exit policy rule to be parsed
+
+ :raises: **ValueError** if input isn't a valid tor exit policy rule
+ """
+
+ def __init__(self, rule):
+ # policy ::= "accept" exitpattern | "reject" exitpattern
+ # exitpattern ::= addrspec ":" portspec
+
+ if rule.startswith('accept'):
+ self.is_accept = True
+ elif rule.startswith('reject'):
+ self.is_accept = False
+ else:
+ raise ValueError("An exit policy must start with either 'accept' or 'reject': %s" % rule)
+
+ exitpattern = rule[6:]
+
+ if not exitpattern.startswith(' ') or (len(exitpattern) - 1 != len(exitpattern.lstrip())):
+ raise ValueError('An exit policy should have a space separating its accept/reject from the exit pattern: %s' % rule)
+
+ exitpattern = exitpattern[1:]
+
+ if ':' not in exitpattern:
+ raise ValueError("An exitpattern must be of the form 'addrspec:portspec': %s" % rule)
+
+ self.address = None
+ self._address_type = None
+ self._masked_bits = None
+ self.min_port = self.max_port = None
+ self._hash = None
+
+ # Our mask in ip notation (ex. '255.255.255.0'). This is only set if we
+ # either have a custom mask that can't be represented by a number of bits,
+ # or the user has called mask(), lazily loading this.
+
+ self._mask = None
+
+ addrspec, portspec = exitpattern.rsplit(':', 1)
+ self._apply_addrspec(rule, addrspec)
+ self._apply_portspec(rule, portspec)
+
+ # If true then a submask of /0 is treated by is_address_wildcard() as being
+ # a wildcard.
+
+ self._submask_wildcard = True
+
+ # Flags to indicate if this rule seems to be expanded from the 'private'
+ # keyword or tor's default policy suffix.
+
+ self._is_private = False
+ self._is_default_suffix = False
+
+ def is_address_wildcard(self):
+ """
+ **True** if we'll match against any address, **False** otherwise.
+
+ Note that if this policy can apply to both IPv4 and IPv6 then this is
+ different from being for a /0 (since, for instance, 0.0.0.0/0 wouldn't
+ match against an IPv6 address). That said, /0 addresses are highly unusual
+ and most things citing exit policies are IPv4 specific anyway, making this
+ moot.
+
+ :returns: **bool** for if our address matching is a wildcard
+ """
+
+ if self._submask_wildcard and self.get_masked_bits() == 0:
+ return True
+
+ return self._address_type == _address_type_to_int(AddressType.WILDCARD)
+
+ def is_port_wildcard(self):
+ """
+ **True** if we'll match against any port, **False** otherwise.
+
+ :returns: **bool** for if our port matching is a wildcard
+ """
+
+ return self.min_port in (0, 1) and self.max_port == 65535
+
+ def is_match(self, address = None, port = None, strict = False):
+ """
+ **True** if we match against the given destination, **False** otherwise. If
+ the address or port is omitted then this will check if we're allowed to
+ exit to any instances of the defined address or port.
+
+ :param str address: IPv4 or IPv6 address (with or without brackets)
+ :param int port: port number
+ :param bool strict: if the address or port is excluded then check if we can
+ exit to **all** instances of the defined address or port
+
+ :returns: **bool** indicating if we match against this destination
+
+ :raises: **ValueError** if provided with a malformed address or port
+ """
+
+ # validate our input and check if the argument doesn't match our address type
+
+ if address is not None:
+ address_type = self.get_address_type()
+
+ if stem.util.connection.is_valid_ipv4_address(address):
+ if address_type == AddressType.IPv6:
+ return False
+ elif stem.util.connection.is_valid_ipv6_address(address, allow_brackets = True):
+ if address_type == AddressType.IPv4:
+ return False
+
+ address = address.lstrip('[').rstrip(']')
+ else:
+ raise ValueError("'%s' isn't a valid IPv4 or IPv6 address" % address)
+
+ if port is not None and not stem.util.connection.is_valid_port(port):
+ raise ValueError("'%s' isn't a valid port" % port)
+
+ # If we're not matching against an address or port but the rule has one
+ # then we're a fuzzy match. When that happens...
+ #
+ # * If strict and a reject rule then we're a match ('can exit to *all* instances').
+ # * If not strict and an accept rule then match ('an exit ot *any* instance').
+
+ fuzzy_match = False
+
+ if not self.is_address_wildcard():
+ # Already got the integer representation of our mask and our address
+ # with the mask applied. Just need to check if this address with the
+ # mask applied matches.
+
+ if address is None:
+ fuzzy_match = True
+ else:
+ comparison_addr_bin = int(stem.util.connection._get_address_binary(address), 2)
+ comparison_addr_bin &= self._get_mask_bin()
+
+ if self._get_address_bin() != comparison_addr_bin:
+ return False
+
+ if not self.is_port_wildcard():
+ if port is None:
+ fuzzy_match = True
+ elif port < self.min_port or port > self.max_port:
+ return False
+
+ if fuzzy_match:
+ return strict != self.is_accept
+ else:
+ return True
+
+ def get_address_type(self):
+ """
+ Provides the :data:`~stem.exit_policy.AddressType` for our policy.
+
+ :returns: :data:`~stem.exit_policy.AddressType` for the type of address that we have
+ """
+
+ return _int_to_address_type(self._address_type)
+
+ def get_mask(self, cache = True):
+ """
+ Provides the address represented by our mask. This is **None** if our
+ address type is a wildcard.
+
+ :param bool cache: caches the result if **True**
+
+ :returns: str of our subnet mask for the address (ex. '255.255.255.0')
+ """
+
+ # Lazy loading our mask because it very infrequently requested. There's
+ # no reason to usually usse memory for it.
+
+ if not self._mask:
+ address_type = self.get_address_type()
+
+ if address_type == AddressType.WILDCARD:
+ mask = None
+ elif address_type == AddressType.IPv4:
+ mask = stem.util.connection.get_mask_ipv4(self._masked_bits)
+ elif address_type == AddressType.IPv6:
+ mask = stem.util.connection.get_mask_ipv6(self._masked_bits)
+
+ if not cache:
+ return mask
+
+ self._mask = mask
+
+ return self._mask
+
+ def get_masked_bits(self):
+ """
+ Provides the number of bits our subnet mask represents. This is **None** if
+ our mask can't have a bit representation.
+
+ :returns: int with the bit representation of our mask
+ """
+
+ return self._masked_bits
+
+ def is_private(self):
+ """
+ Checks if this rule was expanded from the 'private' policy keyword.
+
+ .. versionadded:: 1.3.0
+
+ :returns: **True** if this rule was expanded from the 'private' keyword, **False** otherwise.
+ """
+
+ return self._is_private
+
+ def is_default(self):
+ """
+ Checks if this rule belongs to the default exit policy suffix.
+
+ .. versionadded:: 1.3.0
+
+ :returns: **True** if this rule was part of the default end of a policy, **False** otherwise.
+ """
+
+ return self._is_default_suffix
+
+ @lru_cache()
+ def __str__(self):
+ """
+ Provides the string representation of our policy. This does not
+ necessarily match the rule that we were constructed from (due to things
+ like IPv6 address collapsing or the multiple representations that our mask
+ can have). However, it is a valid that would be accepted by our constructor
+ to re-create this rule.
+ """
+
+ label = 'accept ' if self.is_accept else 'reject '
+
+ if self.is_address_wildcard():
+ label += '*:'
+ else:
+ address_type = self.get_address_type()
+
+ if address_type == AddressType.IPv4:
+ label += self.address
+ else:
+ label += '[%s]' % self.address
+
+ # Including our mask label as follows...
+ # - exclude our mask if it doesn't do anything
+ # - use our masked bit count if we can
+ # - use the mask itself otherwise
+
+ if (address_type == AddressType.IPv4 and self._masked_bits == 32) or \
+ (address_type == AddressType.IPv6 and self._masked_bits == 128):
+ label += ':'
+ elif self._masked_bits is not None:
+ label += '/%i:' % self._masked_bits
+ else:
+ label += '/%s:' % self.get_mask()
+
+ if self.is_port_wildcard():
+ label += '*'
+ elif self.min_port == self.max_port:
+ label += str(self.min_port)
+ else:
+ label += '%i-%i' % (self.min_port, self.max_port)
+
+ return label
+
+ def __hash__(self):
+ if self._hash is None:
+ my_hash = 0
+
+ for attr in ('is_accept', 'address', 'min_port', 'max_port'):
+ my_hash *= 1024
+
+ attr_value = getattr(self, attr)
+
+ if attr_value is not None:
+ my_hash += hash(attr_value)
+
+ my_hash *= 1024
+ my_hash += hash(self.get_mask(False))
+
+ self._hash = my_hash
+
+ return self._hash
+
+ @lru_cache()
+ def _get_mask_bin(self):
+ # provides an integer representation of our mask
+
+ return int(stem.util.connection._get_address_binary(self.get_mask(False)), 2)
+
+ @lru_cache()
+ def _get_address_bin(self):
+ # provides an integer representation of our address
+
+ return int(stem.util.connection._get_address_binary(self.address), 2) & self._get_mask_bin()
+
+ def _apply_addrspec(self, rule, addrspec):
+ # Parses the addrspec...
+ # addrspec ::= "*" | ip4spec | ip6spec
+
+ if '/' in addrspec:
+ self.address, addr_extra = addrspec.split('/', 1)
+ else:
+ self.address, addr_extra = addrspec, None
+
+ if addrspec == '*':
+ self._address_type = _address_type_to_int(AddressType.WILDCARD)
+ self.address = self._masked_bits = None
+ elif stem.util.connection.is_valid_ipv4_address(self.address):
+ # ipv4spec ::= ip4 | ip4 "/" num_ip4_bits | ip4 "/" ip4mask
+ # ip4 ::= an IPv4 address in dotted-quad format
+ # ip4mask ::= an IPv4 mask in dotted-quad format
+ # num_ip4_bits ::= an integer between 0 and 32
+
+ self._address_type = _address_type_to_int(AddressType.IPv4)
+
+ if addr_extra is None:
+ self._masked_bits = 32
+ elif stem.util.connection.is_valid_ipv4_address(addr_extra):
+ # provided with an ip4mask
+ try:
+ self._masked_bits = stem.util.connection._get_masked_bits(addr_extra)
+ except ValueError:
+ # mask can't be represented as a number of bits (ex. '255.255.0.255')
+ self._mask = addr_extra
+ self._masked_bits = None
+ elif addr_extra.isdigit():
+ # provided with a num_ip4_bits
+ self._masked_bits = int(addr_extra)
+
+ if self._masked_bits < 0 or self._masked_bits > 32:
+ raise ValueError('IPv4 masks must be in the range of 0-32 bits')
+ else:
+ raise ValueError("The '%s' isn't a mask nor number of bits: %s" % (addr_extra, rule))
+ elif self.address.startswith('[') and self.address.endswith(']') and \
+ stem.util.connection.is_valid_ipv6_address(self.address[1:-1]):
+ # ip6spec ::= ip6 | ip6 "/" num_ip6_bits
+ # ip6 ::= an IPv6 address, surrounded by square brackets.
+ # num_ip6_bits ::= an integer between 0 and 128
+
+ self.address = stem.util.connection.expand_ipv6_address(self.address[1:-1].upper())
+ self._address_type = _address_type_to_int(AddressType.IPv6)
+
+ if addr_extra is None:
+ self._masked_bits = 128
+ elif addr_extra.isdigit():
+ # provided with a num_ip6_bits
+ self._masked_bits = int(addr_extra)
+
+ if self._masked_bits < 0 or self._masked_bits > 128:
+ raise ValueError('IPv6 masks must be in the range of 0-128 bits')
+ else:
+ raise ValueError("The '%s' isn't a number of bits: %s" % (addr_extra, rule))
+ else:
+ raise ValueError("Address isn't a wildcard, IPv4, or IPv6 address: %s" % rule)
+
+ def _apply_portspec(self, rule, portspec):
+ # Parses the portspec...
+ # portspec ::= "*" | port | port "-" port
+ # port ::= an integer between 1 and 65535, inclusive.
+ #
+ # Due to a tor bug the spec says that we should accept port of zero, but
+ # connections to port zero are never permitted.
+
+ if portspec == '*':
+ self.min_port, self.max_port = 1, 65535
+ elif portspec.isdigit():
+ # provided with a single port
+ if stem.util.connection.is_valid_port(portspec, allow_zero = True):
+ self.min_port = self.max_port = int(portspec)
+ else:
+ raise ValueError("'%s' isn't within a valid port range: %s" % (portspec, rule))
+ elif '-' in portspec:
+ # provided with a port range
+ port_comp = portspec.split('-', 1)
+
+ if stem.util.connection.is_valid_port(port_comp, allow_zero = True):
+ self.min_port = int(port_comp[0])
+ self.max_port = int(port_comp[1])
+
+ if self.min_port > self.max_port:
+ raise ValueError("Port range has a lower bound that's greater than its upper bound: %s" % rule)
+ else:
+ raise ValueError('Malformed port range: %s' % rule)
+ else:
+ raise ValueError("Port value isn't a wildcard, integer, or range: %s" % rule)
+
+ def __eq__(self, other):
+ if isinstance(other, ExitPolicyRule):
+ # Our string representation encompasses our effective policy. Technically
+ # this isn't quite right since our rule attribute may differ (ie, 'accept
+ # 0.0.0.0/0' == 'accept 0.0.0.0/0.0.0.0' will be True), but these
+ # policies are effectively equivalent.
+
+ return hash(self) == hash(other)
+ else:
+ return False
+
+
+def _address_type_to_int(address_type):
+ return AddressType.index_of(address_type)
+
+
+def _int_to_address_type(address_type_int):
+ return list(AddressType)[address_type_int]
+
+
+class MicroExitPolicyRule(ExitPolicyRule):
+ """
+ Lighter weight ExitPolicyRule derivative for microdescriptors.
+ """
+
+ def __init__(self, is_accept, min_port, max_port):
+ self.is_accept = is_accept
+ self.address = None # wildcard address
+ self.min_port = min_port
+ self.max_port = max_port
+ self._hash = None
+
+ def is_address_wildcard(self):
+ return True
+
+ def get_address_type(self):
+ return AddressType.WILDCARD
+
+ def get_mask(self, cache = True):
+ return None
+
+ def get_masked_bits(self):
+ return None
+
+ def __hash__(self):
+ if self._hash is None:
+ my_hash = 0
+
+ for attr in ('is_accept', 'min_port', 'max_port'):
+ my_hash *= 1024
+
+ attr_value = getattr(self, attr)
+
+ if attr_value is not None:
+ my_hash += hash(attr_value)
+
+ self._hash = my_hash
+
+ return self._hash
+
+
+DEFAULT_POLICY_RULES = tuple([ExitPolicyRule(rule) for rule in (
+ 'reject *:25',
+ 'reject *:119',
+ 'reject *:135-139',
+ 'reject *:445',
+ 'reject *:563',
+ 'reject *:1214',
+ 'reject *:4661-4666',
+ 'reject *:6346-6429',
+ 'reject *:6699',
+ 'reject *:6881-6999',
+ 'accept *:*',
+)])
diff --git a/ext/stem/interpreter/__init__.py b/ext/stem/interpreter/__init__.py
new file mode 100644
index 00000000..cf69d63a
--- /dev/null
+++ b/ext/stem/interpreter/__init__.py
@@ -0,0 +1,141 @@
+# Copyright 2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Interactive interpreter for interacting with Tor directly. This adds usability
+features such as tab completion, history, and IRC-style functions (like /help).
+"""
+
+__all__ = [
+ 'arguments',
+ 'autocomplete',
+ 'commands',
+ 'help',
+]
+
+import os
+import sys
+
+import stem
+import stem.connection
+import stem.prereq
+import stem.process
+import stem.util.conf
+import stem.util.system
+import stem.util.term
+
+from stem.util.term import Attr, Color, format
+
+PROMPT = format('>>> ', Color.GREEN, Attr.BOLD, Attr.READLINE_ESCAPE)
+
+STANDARD_OUTPUT = (Color.BLUE, )
+BOLD_OUTPUT = (Color.BLUE, Attr.BOLD)
+HEADER_OUTPUT = (Color.GREEN, )
+HEADER_BOLD_OUTPUT = (Color.GREEN, Attr.BOLD)
+ERROR_OUTPUT = (Attr.BOLD, Color.RED)
+
+settings_path = os.path.join(os.path.dirname(__file__), 'settings.cfg')
+uses_settings = stem.util.conf.uses_settings('stem_interpreter', settings_path)
+
+
+@uses_settings
+def msg(message, config, **attr):
+ return config.get(message).format(**attr)
+
+
+def main():
+ import readline
+
+ import stem.interpreter.arguments
+ import stem.interpreter.autocomplete
+ import stem.interpreter.commands
+
+ try:
+ args = stem.interpreter.arguments.parse(sys.argv[1:])
+ except ValueError as exc:
+ print(exc)
+ sys.exit(1)
+
+ if args.print_help:
+ print(stem.interpreter.arguments.get_help())
+ sys.exit()
+
+ if args.disable_color:
+ global PROMPT
+ stem.util.term.DISABLE_COLOR_SUPPORT = True
+ PROMPT = '>>> '
+
+ # If the user isn't connecting to something in particular then offer to start
+ # tor if it isn't running.
+
+ if not (args.user_provided_port or args.user_provided_socket):
+ is_tor_running = stem.util.system.is_running('tor') or stem.util.system.is_running('tor.real')
+
+ if not is_tor_running:
+ if not stem.util.system.is_available('tor'):
+ print(format(msg('msg.tor_unavailable'), *ERROR_OUTPUT))
+ sys.exit(1)
+ else:
+ print(format(msg('msg.starting_tor'), *HEADER_OUTPUT))
+
+ stem.process.launch_tor_with_config(
+ config = {
+ 'SocksPort': '0',
+ 'ControlPort': str(args.control_port),
+ 'CookieAuthentication': '1',
+ 'ExitPolicy': 'reject *:*',
+ },
+ completion_percent = 5,
+ take_ownership = True,
+ )
+
+ control_port = (args.control_address, args.control_port)
+ control_socket = args.control_socket
+
+ # If the user explicitely specified an endpoint then just try to connect to
+ # that.
+
+ if args.user_provided_socket and not args.user_provided_port:
+ control_port = None
+ elif args.user_provided_port and not args.user_provided_socket:
+ control_socket = None
+
+ controller = stem.connection.connect(
+ control_port = control_port,
+ control_socket = control_socket,
+ password_prompt = True,
+ )
+
+ if controller is None:
+ sys.exit(1)
+
+ with controller:
+ autocompleter = stem.interpreter.autocomplete.Autocompleter(controller)
+ readline.parse_and_bind('tab: complete')
+ readline.set_completer(autocompleter.complete)
+ readline.set_completer_delims('\n')
+
+ interpreter = stem.interpreter.commands.ControlInterpretor(controller)
+
+ for line in msg('msg.startup_banner').splitlines():
+ line_format = HEADER_BOLD_OUTPUT if line.startswith(' ') else HEADER_OUTPUT
+ print(format(line, *line_format))
+
+ print('')
+
+ while True:
+ try:
+ prompt = '... ' if interpreter.is_multiline_context else PROMPT
+
+ if stem.prereq.is_python_3():
+ user_input = input(prompt)
+ else:
+ user_input = raw_input(prompt)
+
+ response = interpreter.run_command(user_input)
+
+ if response is not None:
+ print(response)
+ except (KeyboardInterrupt, EOFError, stem.SocketClosed) as exc:
+ print('') # move cursor to the following line
+ break
diff --git a/ext/stem/interpreter/arguments.py b/ext/stem/interpreter/arguments.py
new file mode 100644
index 00000000..eadd0436
--- /dev/null
+++ b/ext/stem/interpreter/arguments.py
@@ -0,0 +1,94 @@
+# Copyright 2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Commandline argument parsing for our interpreter prompt.
+"""
+
+import collections
+import getopt
+
+import stem.interpreter
+import stem.util.connection
+
+DEFAULT_ARGS = {
+ 'control_address': '127.0.0.1',
+ 'control_port': 9051,
+ 'user_provided_port': False,
+ 'control_socket': '/var/run/tor/control',
+ 'user_provided_socket': False,
+ 'disable_color': False,
+ 'print_help': False,
+}
+
+OPT = 'i:s:h'
+OPT_EXPANDED = ['interface=', 'socket=', 'no-color', 'help']
+
+
+def parse(argv):
+ """
+ Parses our arguments, providing a named tuple with their values.
+
+ :param list argv: input arguments to be parsed
+
+ :returns: a **named tuple** with our parsed arguments
+
+ :raises: **ValueError** if we got an invalid argument
+ """
+
+ args = dict(DEFAULT_ARGS)
+
+ try:
+ recognized_args, unrecognized_args = getopt.getopt(argv, OPT, OPT_EXPANDED)
+
+ if unrecognized_args:
+ error_msg = "aren't recognized arguments" if len(unrecognized_args) > 1 else "isn't a recognized argument"
+ raise getopt.GetoptError("'%s' %s" % ("', '".join(unrecognized_args), error_msg))
+ except Exception as exc:
+ raise ValueError('%s (for usage provide --help)' % exc)
+
+ for opt, arg in recognized_args:
+ if opt in ('-i', '--interface'):
+ if ':' in arg:
+ address, port = arg.split(':', 1)
+ else:
+ address, port = None, arg
+
+ if address is not None:
+ if not stem.util.connection.is_valid_ipv4_address(address):
+ raise ValueError("'%s' isn't a valid IPv4 address" % address)
+
+ args['control_address'] = address
+
+ if not stem.util.connection.is_valid_port(port):
+ raise ValueError("'%s' isn't a valid port number" % port)
+
+ args['control_port'] = int(port)
+ args['user_provided_port'] = True
+ elif opt in ('-s', '--socket'):
+ args['control_socket'] = arg
+ args['user_provided_socket'] = True
+ elif opt == '--no-color':
+ args['disable_color'] = True
+ elif opt in ('-h', '--help'):
+ args['print_help'] = True
+
+ # translates our args dict into a named tuple
+
+ Args = collections.namedtuple('Args', args.keys())
+ return Args(**args)
+
+
+def get_help():
+ """
+ Provides our --help usage information.
+
+ :returns: **str** with our usage information
+ """
+
+ return stem.interpreter.msg(
+ 'msg.help',
+ address = DEFAULT_ARGS['control_address'],
+ port = DEFAULT_ARGS['control_port'],
+ socket = DEFAULT_ARGS['control_socket'],
+ )
diff --git a/ext/stem/interpreter/autocomplete.py b/ext/stem/interpreter/autocomplete.py
new file mode 100644
index 00000000..a6d940db
--- /dev/null
+++ b/ext/stem/interpreter/autocomplete.py
@@ -0,0 +1,115 @@
+# Copyright 2014-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Tab completion for our interpreter prompt.
+"""
+
+from stem.interpreter import uses_settings
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+
+@uses_settings
+def _get_commands(controller, config):
+ """
+ Provides commands recognized by tor.
+ """
+
+ commands = config.get('autocomplete', [])
+
+ if controller is None:
+ return commands
+
+ # GETINFO commands. Lines are of the form '[option] -- [description]'. This
+ # strips '*' from options that accept values.
+
+ results = controller.get_info('info/names', None)
+
+ if results:
+ for line in results.splitlines():
+ option = line.split(' ', 1)[0].rstrip('*')
+ commands.append('GETINFO %s' % option)
+ else:
+ commands.append('GETINFO ')
+
+ # GETCONF, SETCONF, and RESETCONF commands. Lines are of the form
+ # '[option] [type]'.
+
+ results = controller.get_info('config/names', None)
+
+ if results:
+ for line in results.splitlines():
+ option = line.split(' ', 1)[0]
+
+ commands.append('GETCONF %s' % option)
+ commands.append('SETCONF %s' % option)
+ commands.append('RESETCONF %s' % option)
+ else:
+ commands += ['GETCONF ', 'SETCONF ', 'RESETCONF ']
+
+ # SETEVENT, USEFEATURE, and SIGNAL commands. For each of these the GETINFO
+ # results are simply a space separated lists of the values they can have.
+
+ options = (
+ ('SETEVENTS ', 'events/names'),
+ ('USEFEATURE ', 'features/names'),
+ ('SIGNAL ', 'signal/names'),
+ )
+
+ for prefix, getinfo_cmd in options:
+ results = controller.get_info(getinfo_cmd, None)
+
+ if results:
+ commands += [prefix + value for value in results.split()]
+ else:
+ commands.append(prefix)
+
+ # Adds /help commands.
+
+ usage_info = config.get('help.usage', {})
+
+ for cmd in usage_info.keys():
+ commands.append('/help ' + cmd)
+
+ return commands
+
+
+class Autocompleter(object):
+ def __init__(self, controller):
+ self._commands = _get_commands(controller)
+
+ @lru_cache()
+ def matches(self, text):
+ """
+ Provides autocompletion matches for the given text.
+
+ :param str text: text to check for autocompletion matches with
+
+ :returns: **list** with possible matches
+ """
+
+ lowercase_text = text.lower()
+ return [cmd for cmd in self._commands if cmd.lower().startswith(lowercase_text)]
+
+ def complete(self, text, state):
+ """
+ Provides case insensetive autocompletion options, acting as a functor for
+ the readlines set_completer function.
+
+ :param str text: text to check for autocompletion matches with
+ :param int state: index of result to be provided, readline fetches matches
+ until this function provides None
+
+ :returns: **str** with the autocompletion match, **None** if eithe none
+ exists or state is higher than our number of matches
+ """
+
+ try:
+ return self.matches(text)[state]
+ except IndexError:
+ return None
diff --git a/ext/stem/interpreter/commands.py b/ext/stem/interpreter/commands.py
new file mode 100644
index 00000000..40475178
--- /dev/null
+++ b/ext/stem/interpreter/commands.py
@@ -0,0 +1,354 @@
+# Copyright 2014-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Handles making requests and formatting the responses.
+"""
+
+import code
+import socket
+
+import stem
+import stem.control
+import stem.descriptor.remote
+import stem.interpreter.help
+import stem.util.connection
+import stem.util.str_tools
+import stem.util.tor_tools
+
+from stem.interpreter import STANDARD_OUTPUT, BOLD_OUTPUT, ERROR_OUTPUT, uses_settings, msg
+from stem.util.term import format
+
+
+def _get_fingerprint(arg, controller):
+ """
+ Resolves user input into a relay fingerprint. This accepts...
+
+ * Fingerprints
+ * Nicknames
+ * IPv4 addresses, either with or without an ORPort
+ * Empty input, which is resolved to ourselves if we're a relay
+
+ :param str arg: input to be resolved to a relay fingerprint
+ :param stem.control.Controller controller: tor control connection
+
+ :returns: **str** for the relay fingerprint
+
+ :raises: **ValueError** if we're unable to resolve the input to a relay
+ """
+
+ if not arg:
+ try:
+ return controller.get_info('fingerprint')
+ except:
+ raise ValueError("We aren't a relay, no information to provide")
+ elif stem.util.tor_tools.is_valid_fingerprint(arg):
+ return arg
+ elif stem.util.tor_tools.is_valid_nickname(arg):
+ try:
+ return controller.get_network_status(arg).fingerprint
+ except:
+ raise ValueError("Unable to find a relay with the nickname of '%s'" % arg)
+ elif ':' in arg or stem.util.connection.is_valid_ipv4_address(arg):
+ if ':' in arg:
+ address, port = arg.split(':', 1)
+
+ if not stem.util.connection.is_valid_ipv4_address(address):
+ raise ValueError("'%s' isn't a valid IPv4 address" % address)
+ elif port and not stem.util.connection.is_valid_port(port):
+ raise ValueError("'%s' isn't a valid port" % port)
+
+ port = int(port)
+ else:
+ address, port = arg, None
+
+ matches = {}
+
+ for desc in controller.get_network_statuses():
+ if desc.address == address:
+ if not port or desc.or_port == port:
+ matches[desc.or_port] = desc.fingerprint
+
+ if len(matches) == 0:
+ raise ValueError('No relays found at %s' % arg)
+ elif len(matches) == 1:
+ return list(matches.values())[0]
+ else:
+ response = "There's multiple relays at %s, include a port to specify which.\n\n" % arg
+
+ for i, or_port in enumerate(matches):
+ response += ' %i. %s:%s, fingerprint: %s\n' % (i + 1, address, or_port, matches[or_port])
+
+ raise ValueError(response)
+ else:
+ raise ValueError("'%s' isn't a fingerprint, nickname, or IP address" % arg)
+
+
+class ControlInterpretor(code.InteractiveConsole):
+ """
+ Handles issuing requests and providing nicely formed responses, with support
+ for special irc style subcommands.
+ """
+
+ def __init__(self, controller):
+ self._received_events = []
+
+ code.InteractiveConsole.__init__(self, {
+ 'stem': stem,
+ 'stem.control': stem.control,
+ 'controller': controller,
+ 'events': self.get_events,
+ })
+
+ self._controller = controller
+ self._run_python_commands = True
+
+ # Indicates if we're processing a multiline command, such as conditional
+ # block or loop.
+
+ self.is_multiline_context = False
+
+ # Intercept events our controller hears about at a pretty low level since
+ # the user will likely be requesting them by direct 'SETEVENTS' calls.
+
+ handle_event_real = self._controller._handle_event
+
+ def handle_event_wrapper(event_message):
+ handle_event_real(event_message)
+ self._received_events.append(event_message)
+
+ self._controller._handle_event = handle_event_wrapper
+
+ def get_events(self, *event_types):
+ events = list(self._received_events)
+ event_types = list(map(str.upper, event_types)) # make filtering case insensitive
+
+ if event_types:
+ events = [e for e in events if e.type in event_types]
+
+ return events
+
+ def do_help(self, arg):
+ """
+ Performs the '/help' operation, giving usage information for the given
+ argument or a general summary if there wasn't one.
+ """
+
+ return stem.interpreter.help.response(self._controller, arg)
+
+ def do_events(self, arg):
+ """
+ Performs the '/events' operation, dumping the events that we've received
+ belonging to the given types. If no types are specified then this provides
+ all buffered events.
+
+ If the user runs '/events clear' then this clears the list of events we've
+ received.
+ """
+
+ event_types = arg.upper().split()
+
+ if 'CLEAR' in event_types:
+ del self._received_events[:]
+ return format('cleared event backlog', *STANDARD_OUTPUT)
+
+ return '\n'.join([format(str(e), *STANDARD_OUTPUT) for e in self.get_events(*event_types)])
+
+ def do_info(self, arg):
+ """
+ Performs the '/info' operation, looking up a relay by fingerprint, IP
+ address, or nickname and printing its descriptor and consensus entries in a
+ pretty fashion.
+ """
+
+ try:
+ fingerprint = _get_fingerprint(arg, self._controller)
+ except ValueError as exc:
+ return format(str(exc), *ERROR_OUTPUT)
+
+ ns_desc = self._controller.get_network_status(fingerprint, None)
+ server_desc = self._controller.get_server_descriptor(fingerprint, None)
+ extrainfo_desc = None
+ micro_desc = self._controller.get_microdescriptor(fingerprint, None)
+
+ # We'll mostly rely on the router status entry. Either the server
+ # descriptor or microdescriptor will be missing, so we'll treat them as
+ # being optional.
+
+ if not ns_desc:
+ return format('Unable to find consensus information for %s' % fingerprint, *ERROR_OUTPUT)
+
+ # More likely than not we'll have the microdescriptor but not server and
+ # extrainfo descriptors. If so then fetching them.
+
+ downloader = stem.descriptor.remote.DescriptorDownloader(timeout = 5)
+ server_desc_query = downloader.get_server_descriptors(fingerprint)
+ extrainfo_desc_query = downloader.get_extrainfo_descriptors(fingerprint)
+
+ for desc in server_desc_query:
+ server_desc = desc
+
+ for desc in extrainfo_desc_query:
+ extrainfo_desc = desc
+
+ address_extrainfo = []
+
+ try:
+ address_extrainfo.append(socket.gethostbyaddr(ns_desc.address)[0])
+ except:
+ pass
+
+ try:
+ address_extrainfo.append(self._controller.get_info('ip-to-country/%s' % ns_desc.address))
+ except:
+ pass
+
+ address_extrainfo_label = ' (%s)' % ', '.join(address_extrainfo) if address_extrainfo else ''
+
+ if server_desc:
+ exit_policy_label = str(server_desc.exit_policy)
+ elif micro_desc:
+ exit_policy_label = str(micro_desc.exit_policy)
+ else:
+ exit_policy_label = 'Unknown'
+
+ lines = [
+ '%s (%s)' % (ns_desc.nickname, fingerprint),
+ format('address: ', *BOLD_OUTPUT) + '%s:%s%s' % (ns_desc.address, ns_desc.or_port, address_extrainfo_label),
+ ]
+
+ if server_desc:
+ lines.append(format('tor version: ', *BOLD_OUTPUT) + str(server_desc.tor_version))
+
+ lines.append(format('flags: ', *BOLD_OUTPUT) + ', '.join(ns_desc.flags))
+ lines.append(format('exit policy: ', *BOLD_OUTPUT) + exit_policy_label)
+
+ if server_desc and server_desc.contact:
+ contact = stem.util.str_tools._to_unicode(server_desc.contact)
+
+ # clears up some highly common obscuring
+
+ for alias in (' at ', ' AT '):
+ contact = contact.replace(alias, '@')
+
+ for alias in (' dot ', ' DOT '):
+ contact = contact.replace(alias, '.')
+
+ lines.append(format('contact: ', *BOLD_OUTPUT) + contact)
+
+ descriptor_section = [
+ ('Server Descriptor:', server_desc),
+ ('Extrainfo Descriptor:', extrainfo_desc),
+ ('Microdescriptor:', micro_desc),
+ ('Router Status Entry:', ns_desc),
+ ]
+
+ div = format('-' * 80, *STANDARD_OUTPUT)
+
+ for label, desc in descriptor_section:
+ if desc:
+ lines += ['', div, format(label, *BOLD_OUTPUT), div, '']
+ lines += [format(l, *STANDARD_OUTPUT) for l in str(desc).splitlines()]
+
+ return '\n'.join(lines)
+
+ def do_python(self, arg):
+ """
+ Performs the '/python' operation, toggling if we accept python commands or
+ not.
+ """
+
+ if not arg:
+ status = 'enabled' if self._run_python_commands else 'disabled'
+ return format('Python support is currently %s.' % status, *STANDARD_OUTPUT)
+ elif arg.lower() == 'enable':
+ self._run_python_commands = True
+ elif arg.lower() == 'disable':
+ self._run_python_commands = False
+ else:
+ return format("'%s' is not recognized. Please run either '/python enable' or '/python disable'." % arg, *ERROR_OUTPUT)
+
+ if self._run_python_commands:
+ response = "Python support enabled, we'll now run non-interpreter commands as python."
+ else:
+ response = "Python support disabled, we'll now pass along all commands to tor."
+
+ return format(response, *STANDARD_OUTPUT)
+
+ @uses_settings
+ def run_command(self, command, config):
+ """
+ Runs the given command. Requests starting with a '/' are special commands
+ to the interpreter, and anything else is sent to the control port.
+
+ :param stem.control.Controller controller: tor control connection
+ :param str command: command to be processed
+
+ :returns: **list** out output lines, each line being a list of
+ (msg, format) tuples
+
+ :raises: **stem.SocketClosed** if the control connection has been severed
+ """
+
+ if not self._controller.is_alive():
+ raise stem.SocketClosed()
+
+ # Commands fall into three categories:
+ #
+ # * Interpretor commands. These start with a '/'.
+ #
+ # * Controller commands stem knows how to handle. We use our Controller's
+ # methods for these to take advantage of caching and present nicer
+ # output.
+ #
+ # * Other tor commands. We pass these directly on to the control port.
+
+ cmd, arg = command.strip(), ''
+
+ if ' ' in cmd:
+ cmd, arg = cmd.split(' ', 1)
+
+ output = ''
+
+ if cmd.startswith('/'):
+ cmd = cmd.lower()
+
+ if cmd == '/quit':
+ raise stem.SocketClosed()
+ elif cmd == '/events':
+ output = self.do_events(arg)
+ elif cmd == '/info':
+ output = self.do_info(arg)
+ elif cmd == '/python':
+ output = self.do_python(arg)
+ elif cmd == '/help':
+ output = self.do_help(arg)
+ else:
+ output = format("'%s' isn't a recognized command" % command, *ERROR_OUTPUT)
+ else:
+ cmd = cmd.upper() # makes commands uppercase to match the spec
+
+ if cmd.replace('+', '') in ('LOADCONF', 'POSTDESCRIPTOR'):
+ # provides a notice that multi-line controller input isn't yet implemented
+ output = format(msg('msg.multiline_unimplemented_notice'), *ERROR_OUTPUT)
+ elif cmd == 'QUIT':
+ self._controller.msg(command)
+ raise stem.SocketClosed()
+ else:
+ is_tor_command = cmd in config.get('help.usage', {}) and cmd.lower() != 'events'
+
+ if self._run_python_commands and not is_tor_command:
+ self.is_multiline_context = code.InteractiveConsole.push(self, command)
+ return
+ else:
+ try:
+ output = format(self._controller.msg(command).raw_content().strip(), *STANDARD_OUTPUT)
+ except stem.ControllerError as exc:
+ if isinstance(exc, stem.SocketClosed):
+ raise exc
+ else:
+ output = format(str(exc), *ERROR_OUTPUT)
+
+ output += '\n' # give ourselves an extra line before the next prompt
+
+ return output
diff --git a/ext/stem/interpreter/help.py b/ext/stem/interpreter/help.py
new file mode 100644
index 00000000..83db0b1b
--- /dev/null
+++ b/ext/stem/interpreter/help.py
@@ -0,0 +1,145 @@
+# Copyright 2014-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Provides our /help responses.
+"""
+
+from stem.interpreter import (
+ STANDARD_OUTPUT,
+ BOLD_OUTPUT,
+ ERROR_OUTPUT,
+ msg,
+ uses_settings,
+)
+
+from stem.util.term import format
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+
+def response(controller, arg):
+ """
+ Provides our /help response.
+
+ :param stem.control.Controller controller: tor control connection
+ :param str arg: controller or interpreter command to provide help output for
+
+ :returns: **str** with our help response
+ """
+
+ # Normalizing inputs first so we can better cache responses.
+
+ return _response(controller, _normalize(arg))
+
+
+def _normalize(arg):
+ arg = arg.upper()
+
+ # If there's multiple arguments then just take the first. This is
+ # particularly likely if they're trying to query a full command (for
+ # instance "/help GETINFO version")
+
+ arg = arg.split(' ')[0]
+
+ # strip slash if someone enters an interpreter command (ex. "/help /help")
+
+ if arg.startswith('/'):
+ arg = arg[1:]
+
+ return arg
+
+
+@lru_cache()
+@uses_settings
+def _response(controller, arg, config):
+ if not arg:
+ return _general_help()
+
+ usage_info = config.get('help.usage', {})
+
+ if arg not in usage_info:
+ return format("No help information available for '%s'..." % arg, *ERROR_OUTPUT)
+
+ output = format(usage_info[arg] + '\n', *BOLD_OUTPUT)
+
+ description = config.get('help.description.%s' % arg.lower(), '')
+
+ for line in description.splitlines():
+ output += format(' ' + line, *STANDARD_OUTPUT) + '\n'
+
+ output += '\n'
+
+ if arg == 'GETINFO':
+ results = controller.get_info('info/names', None)
+
+ if results:
+ for line in results.splitlines():
+ if ' -- ' in line:
+ opt, summary = line.split(' -- ', 1)
+
+ output += format('%-33s' % opt, *BOLD_OUTPUT)
+ output += format(' - %s' % summary, *STANDARD_OUTPUT) + '\n'
+ elif arg == 'GETCONF':
+ results = controller.get_info('config/names', None)
+
+ if results:
+ options = [opt.split(' ', 1)[0] for opt in results.splitlines()]
+
+ for i in range(0, len(options), 2):
+ line = ''
+
+ for entry in options[i:i + 2]:
+ line += '%-42s' % entry
+
+ output += format(line.rstrip(), *STANDARD_OUTPUT) + '\n'
+ elif arg == 'SIGNAL':
+ signal_options = config.get('help.signal.options', {})
+
+ for signal, summary in signal_options.items():
+ output += format('%-15s' % signal, *BOLD_OUTPUT)
+ output += format(' - %s' % summary, *STANDARD_OUTPUT) + '\n'
+ elif arg == 'SETEVENTS':
+ results = controller.get_info('events/names', None)
+
+ if results:
+ entries = results.split()
+
+ # displays four columns of 20 characters
+
+ for i in range(0, len(entries), 4):
+ line = ''
+
+ for entry in entries[i:i + 4]:
+ line += '%-20s' % entry
+
+ output += format(line.rstrip(), *STANDARD_OUTPUT) + '\n'
+ elif arg == 'USEFEATURE':
+ results = controller.get_info('features/names', None)
+
+ if results:
+ output += format(results, *STANDARD_OUTPUT) + '\n'
+ elif arg in ('LOADCONF', 'POSTDESCRIPTOR'):
+ # gives a warning that this option isn't yet implemented
+ output += format(msg('msg.multiline_unimplemented_notice'), *ERROR_OUTPUT) + '\n'
+
+ return output.rstrip()
+
+
+def _general_help():
+ lines = []
+
+ for line in msg('help.general').splitlines():
+ div = line.find(' - ')
+
+ if div != -1:
+ cmd, description = line[:div], line[div:]
+ lines.append(format(cmd, *BOLD_OUTPUT) + format(description, *STANDARD_OUTPUT))
+ else:
+ lines.append(format(line, *BOLD_OUTPUT))
+
+ return '\n'.join(lines)
diff --git a/ext/stem/interpreter/settings.cfg b/ext/stem/interpreter/settings.cfg
new file mode 100644
index 00000000..1bacf1cb
--- /dev/null
+++ b/ext/stem/interpreter/settings.cfg
@@ -0,0 +1,326 @@
+################################################################################
+#
+# Configuration data used by Stem's interpreter prompt.
+#
+################################################################################
+
+ ##################
+# GENERAL MESSAGES #
+ ##################
+
+msg.multiline_unimplemented_notice Multi-line control options like this are not yet implemented.
+
+msg.help
+|Interactive interpreter for Tor. This provides you with direct access
+|to Tor's control interface via either python or direct requests.
+|
+| -i, --interface [ADDRESS:]PORT change control interface from {address}:{port}
+| -s, --socket SOCKET_PATH attach using unix domain socket if present,
+| SOCKET_PATH defaults to: {socket}
+| --no-color disables colorized output
+| -h, --help presents this help
+|
+
+msg.startup_banner
+|Welcome to Stem's interpreter prompt. This provides you with direct access to
+|Tor's control interface.
+|
+|This acts like a standard python interpreter with a Tor connection available
+|via your 'controller' variable...
+|
+| >>> controller.get_info('version')
+| '0.2.5.1-alpha-dev (git-245ecfff36c0cecc)'
+|
+|You can also issue requests directly to Tor...
+|
+| >>> GETINFO version
+| 250-version=0.2.5.1-alpha-dev (git-245ecfff36c0cecc)
+| 250 OK
+|
+|For more information run '/help'.
+|
+
+msg.tor_unavailable Tor isn't running and the command currently isn't in your PATH.
+
+msg.starting_tor
+|Tor isn't running. Starting a temporary Tor instance for our interpreter to
+|interact with. This will have a minimal non-relaying configuration, and be
+|shut down when you're done.
+|
+|--------------------------------------------------------------------------------
+|
+
+ #################
+# OUTPUT OF /HELP #
+ #################
+
+# Response for the '/help' command without any arguments.
+
+help.general
+|Interpretor commands include:
+| /help - provides information for interpreter and tor commands
+| /events - prints events that we've received
+| /info - general information for a relay
+| /python - enable or disable support for running python commands
+| /quit - shuts down the interpreter
+|
+|Tor commands include:
+| GETINFO - queries information from tor
+| GETCONF, SETCONF, RESETCONF - show or edit a configuration option
+| SIGNAL - issues control signal to the process (for resetting, stopping, etc)
+| SETEVENTS - configures the events tor will notify us of
+|
+| USEFEATURE - enables custom behavior for the controller
+| SAVECONF - writes tor's current configuration to our torrc
+| LOADCONF - loads the given input like it was part of our torrc
+| MAPADDRESS - replaces requests for one address with another
+| POSTDESCRIPTOR - adds a relay descriptor to our cache
+| EXTENDCIRCUIT - create or extend a tor circuit
+| SETCIRCUITPURPOSE - configures the purpose associated with a circuit
+| CLOSECIRCUIT - closes the given circuit
+| ATTACHSTREAM - associates an application's stream with a tor circuit
+| REDIRECTSTREAM - sets a stream's destination
+| CLOSESTREAM - closes the given stream
+| ADD_ONION - create a new hidden service
+| DEL_ONION - delete a hidden service that was created with ADD_ONION
+| HSFETCH - retrieve a hidden service descriptor, providing it in a HS_DESC_CONTENT event
+| HSPOST - uploads a hidden service descriptor
+| RESOLVE - issues an asynchronous dns or rdns request over tor
+| TAKEOWNERSHIP - instructs tor to quit when this control connection is closed
+| PROTOCOLINFO - queries version and controller authentication information
+| QUIT - disconnect the control connection
+|
+|For more information use '/help [OPTION]'.
+
+# Usage of tor and interpreter commands.
+
+help.usage HELP => /help [OPTION]
+help.usage EVENTS => /events [types]
+help.usage INFO => /info [relay fingerprint, nickname, or IP address]
+help.usage PYTHON => /python [enable,disable]
+help.usage QUIT => /quit
+help.usage GETINFO => GETINFO OPTION
+help.usage GETCONF => GETCONF OPTION
+help.usage SETCONF => SETCONF PARAM[=VALUE]
+help.usage RESETCONF => RESETCONF PARAM[=VALUE]
+help.usage SIGNAL => SIGNAL SIG
+help.usage SETEVENTS => SETEVENTS [EXTENDED] [EVENTS]
+help.usage USEFEATURE => USEFEATURE OPTION
+help.usage SAVECONF => SAVECONF
+help.usage LOADCONF => LOADCONF...
+help.usage MAPADDRESS => MAPADDRESS SOURCE_ADDR=DESTINATION_ADDR
+help.usage POSTDESCRIPTOR => POSTDESCRIPTOR [purpose=general/controller/bridge] [cache=yes/no]...
+help.usage EXTENDCIRCUIT => EXTENDCIRCUIT CircuitID [PATH] [purpose=general/controller]
+help.usage SETCIRCUITPURPOSE => SETCIRCUITPURPOSE CircuitID purpose=general/controller
+help.usage CLOSECIRCUIT => CLOSECIRCUIT CircuitID [IfUnused]
+help.usage ATTACHSTREAM => ATTACHSTREAM StreamID CircuitID [HOP=HopNum]
+help.usage REDIRECTSTREAM => REDIRECTSTREAM StreamID Address [Port]
+help.usage CLOSESTREAM => CLOSESTREAM StreamID Reason [Flag]
+help.usage ADD_ONION => KeyType:KeyBlob [Flags=Flag] (Port=Port [,Target])...
+help.usage DEL_ONION => ServiceID
+help.usage HSFETCH => HSFETCH (HSAddress/v2-DescId) [SERVER=Server]...
+help.usage HSPOST => [SERVER=Server] DESCRIPTOR
+help.usage RESOLVE => RESOLVE [mode=reverse] address
+help.usage TAKEOWNERSHIP => TAKEOWNERSHIP
+help.usage PROTOCOLINFO => PROTOCOLINFO [ProtocolVersion]
+
+# Longer description of what tor and interpreter commands do.
+
+help.description.help
+|Provides usage information for the given interpreter, tor command, or tor
+|configuration option.
+|
+|Example:
+| /help info # provides a description of the '/info' option
+| /help GETINFO # usage information for tor's GETINFO controller option
+
+help.description.events
+|Provides events that we've received belonging to the given event types. If
+|no types are specified then this provides all the messages that we've
+|received.
+|
+|You can also run '/events clear' to clear the backlog of events we've
+|received.
+
+help.description.info
+|Provides information for a relay that's currently in the consensus. If no
+|relay is specified then this provides information on ourselves.
+
+help.description.python
+|Enables or disables support for running python commands. This determines how
+|we treat commands this interpreter doesn't recognize...
+|
+|* If enabled then unrecognized commands are executed as python.
+|* If disabled then unrecognized commands are passed along to tor.
+
+help.description.quit
+|Terminates the interpreter.
+
+help.description.getinfo
+|Queries the tor process for information. Options are...
+|
+
+help.description.getconf
+|Provides the current value for a given configuration value. Options include...
+|
+
+help.description.setconf
+|Sets the given configuration parameters. Values can be quoted or non-quoted
+|strings, and reverts the option to 0 or NULL if not provided.
+|
+|Examples:
+| * Sets a contact address and resets our family to NULL
+| SETCONF MyFamily [email protected]
+|
+| * Sets an exit policy that only includes port 80/443
+| SETCONF ExitPolicy=\"accept *:80, accept *:443, reject *:*\"\
+
+help.description.resetconf
+|Reverts the given configuration options to their default values. If a value
+|is provided then this behaves in the same way as SETCONF.
+|
+|Examples:
+| * Returns both of our accounting parameters to their defaults
+| RESETCONF AccountingMax AccountingStart
+|
+| * Uses the default exit policy and sets our nickname to be 'Goomba'
+| RESETCONF ExitPolicy Nickname=Goomba
+
+help.description.signal
+|Issues a signal that tells the tor process to reload its torrc, dump its
+|stats, halt, etc.
+
+help.description.setevents
+|Sets the events that we will receive. This turns off any events that aren't
+|listed so sending 'SETEVENTS' without any values will turn off all event reporting.
+|
+|For Tor versions between 0.1.1.9 and 0.2.2.1 adding 'EXTENDED' causes some
+|events to give us additional information. After version 0.2.2.1 this is
+|always on.
+|
+|Events include...
+|
+
+help.description.usefeature
+|Customizes the behavior of the control port. Options include...
+|
+
+help.description.saveconf
+|Writes Tor's current configuration to its torrc.
+
+help.description.loadconf
+|Reads the given text like it belonged to our torrc.
+|
+|Example:
+| +LOADCONF
+| # sets our exit policy to just accept ports 80 and 443
+| ExitPolicy accept *:80
+| ExitPolicy accept *:443
+| ExitPolicy reject *:*
+| .
+
+help.description.mapaddress
+|Replaces future requests for one address with another.
+|
+|Example:
+| MAPADDRESS 0.0.0.0=torproject.org 1.2.3.4=tor.freehaven.net
+
+help.description.postdescriptor
+|Simulates getting a new relay descriptor.
+
+help.description.extendcircuit
+|Extends the given circuit or create a new one if the CircuitID is zero. The
+|PATH is a comma separated list of fingerprints. If it isn't set then this
+|uses Tor's normal path selection.
+
+help.description.setcircuitpurpose
+|Sets the purpose attribute for a circuit.
+
+help.description.closecircuit
+|Closes the given circuit. If "IfUnused" is included then this only closes
+|the circuit if it isn't currently being used.
+
+help.description.attachstream
+|Attaches a stream with the given built circuit (tor picks one on its own if
+|CircuitID is zero). If HopNum is given then this hop is used to exit the
+|circuit, otherwise the last relay is used.
+
+help.description.redirectstream
+|Sets the destination for a given stream. This can only be done after a
+|stream is created but before it's attached to a circuit.
+
+help.description.closestream
+|Closes the given stream, the reason being an integer matching a reason as
+|per section 6.3 of the tor-spec.
+
+help.description.add_onion
+|Creates a new hidden service. Unlike 'SETCONF HiddenServiceDir...' this
+|doesn't persist the service to disk.
+
+help.description.del_onion
+|Delete a hidden service that was created with ADD_ONION.
+
+help.description.hsfetch
+|Retrieves the descriptor for a hidden service. This is an asynchronous
+|request, with the descriptor provided by a HS_DESC_CONTENT event.
+
+help.description.hspost
+|Uploads a descriptor to a hidden service directory.
+
+help.description.resolve
+|Performs IPv4 DNS resolution over tor, doing a reverse lookup instead if
+|"mode=reverse" is included. This request is processed in the background and
+|results in a ADDRMAP event with the response.
+
+help.description.takeownership
+|Instructs Tor to gracefully shut down when this control connection is closed.
+
+help.description.protocolinfo
+|Provides bootstrapping information that a controller might need when first
+|starting, like Tor's version and controller authentication. This can be done
+|before authenticating to the control port.
+
+help.signal.options RELOAD / HUP => reload our torrc
+help.signal.options SHUTDOWN / INT => gracefully shut down, waiting 30 seconds if we're a relay
+help.signal.options DUMP / USR1 => logs information about open connections and circuits
+help.signal.options DEBUG / USR2 => makes us log at the DEBUG runlevel
+help.signal.options HALT / TERM => immediately shut down
+help.signal.options CLEARDNSCACHE => clears any cached DNS results
+help.signal.options NEWNYM => clears the DNS cache and uses new circuits for future connections
+
+ ################
+# TAB COMPLETION #
+ ################
+
+# Commands we'll autocomplete when the user hits tab. This is just the start of
+# our autocompletion list - more are determined dynamically by checking what
+# tor supports.
+
+autocomplete /help
+autocomplete /events
+autocomplete /info
+autocomplete /quit
+autocomplete SAVECONF
+autocomplete MAPADDRESS
+autocomplete EXTENDCIRCUIT
+autocomplete SETCIRCUITPURPOSE
+autocomplete SETROUTERPURPOSE
+autocomplete ATTACHSTREAM
+#autocomplete +POSTDESCRIPTOR # TODO: needs multi-line support
+autocomplete REDIRECTSTREAM
+autocomplete CLOSESTREAM
+autocomplete CLOSECIRCUIT
+autocomplete QUIT
+autocomplete RESOLVE
+autocomplete PROTOCOLINFO
+#autocomplete +LOADCONF # TODO: needs multi-line support
+autocomplete TAKEOWNERSHIP
+autocomplete AUTHCHALLENGE
+autocomplete DROPGUARDS
+autocomplete ADD_ONION NEW:BEST
+autocomplete ADD_ONION NEW:RSA1024
+autocomplete ADD_ONION RSA1024:
+autocomplete DEL_ONION
+autocomplete HSFETCH
+autocomplete HSPOST
+
diff --git a/ext/stem/prereq.py b/ext/stem/prereq.py
new file mode 100644
index 00000000..5ed87efc
--- /dev/null
+++ b/ext/stem/prereq.py
@@ -0,0 +1,132 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Checks for stem dependencies. We require python 2.6 or greater (including the
+3.x series). Other requirements for complete functionality are...
+
+* pycrypto module
+
+ * validating descriptor signature integrity
+
+::
+
+ check_requirements - checks for minimum requirements for running stem
+
+ is_python_27 - checks if python 2.7 or later is available
+ is_python_3 - checks if python 3.0 or later is available
+
+ is_crypto_available - checks if the pycrypto module is available
+"""
+
+import inspect
+import sys
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+CRYPTO_UNAVAILABLE = "Unable to import the pycrypto module. Because of this we'll be unable to verify descriptor signature integrity. You can get pycrypto from: https://www.dlitz.net/software/pycrypto/"
+
+
+def check_requirements():
+ """
+ Checks that we meet the minimum requirements to run stem. If we don't then
+ this raises an ImportError with the issue.
+
+ :raises: **ImportError** with the problem if we don't meet stem's
+ requirements
+ """
+
+ major_version, minor_version = sys.version_info[0:2]
+
+ if major_version < 2 or (major_version == 2 and minor_version < 6):
+ raise ImportError('stem requires python version 2.6 or greater')
+
+
+def is_python_27():
+ """
+ Checks if we're running python 2.7 or above (including the 3.x series).
+
+ :returns: **True** if we meet this requirement and **False** otherwise
+ """
+
+ major_version, minor_version = sys.version_info[0:2]
+
+ return major_version > 2 or (major_version == 2 and minor_version >= 7)
+
+
+def is_python_3():
+ """
+ Checks if we're in the 3.0 - 3.x range.
+
+ :returns: **True** if we meet this requirement and **False** otherwise
+ """
+
+ return sys.version_info[0] == 3
+
+
+@lru_cache()
+def is_crypto_available():
+ """
+ Checks if the pycrypto functions we use are available. This is used for
+ verifying relay descriptor signatures.
+
+ :returns: **True** if we can use pycrypto and **False** otherwise
+ """
+
+ from stem.util import log
+
+ try:
+ from Crypto.PublicKey import RSA
+ from Crypto.Util import asn1
+ from Crypto.Util.number import long_to_bytes
+ return True
+ except ImportError:
+ log.log_once('stem.prereq.is_crypto_available', log.INFO, CRYPTO_UNAVAILABLE)
+ return False
+
+
+@lru_cache()
+def is_mock_available():
+ """
+ Checks if the mock module is available. In python 3.3 and up it is a builtin
+ unittest module, but before this it needed to be `installed separately
+ <https://pypi.python.org/pypi/mock/>`_. Imports should be as follows....
+
+ ::
+
+ try:
+ # added in python 3.3
+ from unittest.mock import Mock
+ except ImportError:
+ from mock import Mock
+
+ :returns: **True** if the mock module is available and **False** otherwise
+ """
+
+ try:
+ # checks for python 3.3 version
+ import unittest.mock
+ return True
+ except ImportError:
+ pass
+
+ try:
+ import mock
+
+ # check for mock's patch.dict() which was introduced in version 0.7.0
+
+ if not hasattr(mock.patch, 'dict'):
+ raise ImportError()
+
+ # check for mock's new_callable argument for patch() which was introduced in version 0.8.0
+
+ if 'new_callable' not in inspect.getargspec(mock.patch).args:
+ raise ImportError()
+
+ return True
+ except ImportError:
+ return False
diff --git a/ext/stem/process.py b/ext/stem/process.py
new file mode 100644
index 00000000..b8b1f92f
--- /dev/null
+++ b/ext/stem/process.py
@@ -0,0 +1,272 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Helper functions for working with tor as a process.
+
+:NO_TORRC:
+ when provided as a torrc_path tor is ran with a blank configuration
+
+:DEFAULT_INIT_TIMEOUT:
+ number of seconds before we time out our attempt to start a tor instance
+
+**Module Overview:**
+
+::
+
+ launch_tor - starts up a tor process
+ launch_tor_with_config - starts a tor process with a custom torrc
+"""
+
+import os
+import re
+import signal
+import subprocess
+import tempfile
+
+import stem.prereq
+import stem.util.str_tools
+import stem.util.system
+import stem.version
+
+NO_TORRC = '<no torrc>'
+DEFAULT_INIT_TIMEOUT = 90
+
+
+def launch_tor(tor_cmd = 'tor', args = None, torrc_path = None, completion_percent = 100, init_msg_handler = None, timeout = DEFAULT_INIT_TIMEOUT, take_ownership = False, stdin = None):
+ """
+ Initializes a tor process. This blocks until initialization completes or we
+ error out.
+
+ If tor's data directory is missing or stale then bootstrapping will include
+ making several requests to the directory authorities which can take a little
+ while. Usually this is done in 50 seconds or so, but occasionally calls seem
+ to get stuck, taking well over the default timeout.
+
+ **To work to must log at NOTICE runlevel to stdout.** It does this by
+ default, but if you have a 'Log' entry in your torrc then you'll also need
+ 'Log NOTICE stdout'.
+
+ Note: The timeout argument does not work on Windows, and relies on the global
+ state of the signal module.
+
+ :param str tor_cmd: command for starting tor
+ :param list args: additional arguments for tor
+ :param str torrc_path: location of the torrc for us to use
+ :param int completion_percent: percent of bootstrap completion at which
+ this'll return
+ :param functor init_msg_handler: optional functor that will be provided with
+ tor's initialization stdout as we get it
+ :param int timeout: time after which the attempt to start tor is aborted, no
+ timeouts are applied if **None**
+ :param bool take_ownership: asserts ownership over the tor process so it
+ aborts if this python process terminates or a :class:`~stem.control.Controller`
+ we establish to it disconnects
+ :param str stdin: content to provide on stdin
+
+ :returns: **subprocess.Popen** instance for the tor subprocess
+
+ :raises: **OSError** if we either fail to create the tor process or reached a
+ timeout without success
+ """
+
+ if stem.util.system.is_windows():
+ timeout = None
+
+ # sanity check that we got a tor binary
+
+ if os.path.sep in tor_cmd:
+ # got a path (either relative or absolute), check what it leads to
+
+ if os.path.isdir(tor_cmd):
+ raise OSError("'%s' is a directory, not the tor executable" % tor_cmd)
+ elif not os.path.isfile(tor_cmd):
+ raise OSError("'%s' doesn't exist" % tor_cmd)
+ elif not stem.util.system.is_available(tor_cmd):
+ raise OSError("'%s' isn't available on your system. Maybe it's not in your PATH?" % tor_cmd)
+
+ # double check that we have a torrc to work with
+ if torrc_path not in (None, NO_TORRC) and not os.path.exists(torrc_path):
+ raise OSError("torrc doesn't exist (%s)" % torrc_path)
+
+ # starts a tor subprocess, raising an OSError if it fails
+ runtime_args, temp_file = [tor_cmd], None
+
+ if args:
+ runtime_args += args
+
+ if torrc_path:
+ if torrc_path == NO_TORRC:
+ temp_file = tempfile.mkstemp(prefix = 'empty-torrc-', text = True)[1]
+ runtime_args += ['-f', temp_file]
+ else:
+ runtime_args += ['-f', torrc_path]
+
+ if take_ownership:
+ runtime_args += ['__OwningControllerProcess', str(os.getpid())]
+
+ tor_process = subprocess.Popen(runtime_args, stdout = subprocess.PIPE, stdin = subprocess.PIPE, stderr = subprocess.PIPE)
+
+ if stdin:
+ tor_process.stdin.write(stem.util.str_tools._to_bytes(stdin))
+ tor_process.stdin.close()
+
+ if timeout:
+ def timeout_handler(signum, frame):
+ # terminates the uninitialized tor process and raise on timeout
+
+ tor_process.kill()
+ raise OSError('reached a %i second timeout without success' % timeout)
+
+ signal.signal(signal.SIGALRM, timeout_handler)
+ signal.alarm(timeout)
+
+ bootstrap_line = re.compile('Bootstrapped ([0-9]+)%: ')
+ problem_line = re.compile('\[(warn|err)\] (.*)$')
+ last_problem = 'Timed out'
+
+ try:
+ while True:
+ # Tor's stdout will be read as ASCII bytes. This is fine for python 2, but
+ # in python 3 that means it'll mismatch with other operations (for instance
+ # the bootstrap_line.search() call later will fail).
+ #
+ # It seems like python 2.x is perfectly happy for this to be unicode, so
+ # normalizing to that.
+
+ init_line = tor_process.stdout.readline().decode('utf-8', 'replace').strip()
+
+ # this will provide empty results if the process is terminated
+
+ if not init_line:
+ tor_process.kill() # ... but best make sure
+ raise OSError('Process terminated: %s' % last_problem)
+
+ # provide the caller with the initialization message if they want it
+
+ if init_msg_handler:
+ init_msg_handler(init_line)
+
+ # return the process if we're done with bootstrapping
+
+ bootstrap_match = bootstrap_line.search(init_line)
+ problem_match = problem_line.search(init_line)
+
+ if bootstrap_match and int(bootstrap_match.group(1)) >= completion_percent:
+ return tor_process
+ elif problem_match:
+ runlevel, msg = problem_match.groups()
+
+ if 'see warnings above' not in msg:
+ if ': ' in msg:
+ msg = msg.split(': ')[-1].strip()
+
+ last_problem = msg
+ finally:
+ if timeout:
+ signal.alarm(0) # stop alarm
+
+ tor_process.stdout.close()
+ tor_process.stderr.close()
+
+ if temp_file:
+ try:
+ os.remove(temp_file)
+ except:
+ pass
+
+
+def launch_tor_with_config(config, tor_cmd = 'tor', completion_percent = 100, init_msg_handler = None, timeout = DEFAULT_INIT_TIMEOUT, take_ownership = False):
+ """
+ Initializes a tor process, like :func:`~stem.process.launch_tor`, but with a
+ customized configuration. This writes a temporary torrc to disk, launches
+ tor, then deletes the torrc.
+
+ For example...
+
+ ::
+
+ tor_process = stem.process.launch_tor_with_config(
+ config = {
+ 'ControlPort': '2778',
+ 'Log': [
+ 'NOTICE stdout',
+ 'ERR file /tmp/tor_error_log',
+ ],
+ },
+ )
+
+ :param dict config: configuration options, such as "{'ControlPort': '9051'}",
+ values can either be a **str** or **list of str** if for multiple values
+ :param str tor_cmd: command for starting tor
+ :param int completion_percent: percent of bootstrap completion at which
+ this'll return
+ :param functor init_msg_handler: optional functor that will be provided with
+ tor's initialization stdout as we get it
+ :param int timeout: time after which the attempt to start tor is aborted, no
+ timeouts are applied if **None**
+ :param bool take_ownership: asserts ownership over the tor process so it
+ aborts if this python process terminates or a :class:`~stem.control.Controller`
+ we establish to it disconnects
+
+ :returns: **subprocess.Popen** instance for the tor subprocess
+
+ :raises: **OSError** if we either fail to create the tor process or reached a
+ timeout without success
+ """
+
+ # TODO: Drop this version check when tor 0.2.6.3 or higher is the only game
+ # in town.
+
+ try:
+ use_stdin = stem.version.get_system_tor_version(tor_cmd) >= stem.version.Requirement.TORRC_VIA_STDIN
+ except IOError:
+ use_stdin = False
+
+ # we need to be sure that we're logging to stdout to figure out when we're
+ # done bootstrapping
+
+ if 'Log' in config:
+ stdout_options = ['DEBUG stdout', 'INFO stdout', 'NOTICE stdout']
+
+ if isinstance(config['Log'], str):
+ config['Log'] = [config['Log']]
+
+ has_stdout = False
+
+ for log_config in config['Log']:
+ if log_config in stdout_options:
+ has_stdout = True
+ break
+
+ if not has_stdout:
+ config['Log'].append('NOTICE stdout')
+
+ config_str = ''
+
+ for key, values in list(config.items()):
+ if isinstance(values, str):
+ config_str += '%s %s\n' % (key, values)
+ else:
+ for value in values:
+ config_str += '%s %s\n' % (key, value)
+
+ if use_stdin:
+ return launch_tor(tor_cmd, ['-f', '-'], None, completion_percent, init_msg_handler, timeout, take_ownership, stdin = config_str)
+ else:
+ torrc_descriptor, torrc_path = tempfile.mkstemp(prefix = 'torrc-', text = True)
+
+ try:
+ with open(torrc_path, 'w') as torrc_file:
+ torrc_file.write(config_str)
+
+ # prevents tor from erroring out due to a missing torrc if it gets a sighup
+ args = ['__ReloadTorrcOnSIGHUP', '0']
+
+ return launch_tor(tor_cmd, args, torrc_path, completion_percent, init_msg_handler, timeout, take_ownership)
+ finally:
+ try:
+ os.close(torrc_descriptor)
+ os.remove(torrc_path)
+ except:
+ pass
diff --git a/ext/stem/response/__init__.py b/ext/stem/response/__init__.py
new file mode 100644
index 00000000..df534a91
--- /dev/null
+++ b/ext/stem/response/__init__.py
@@ -0,0 +1,588 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Parses replies from the control socket.
+
+**Module Overview:**
+
+::
+
+ convert - translates a ControlMessage into a particular response subclass
+
+ ControlMessage - Message that's read from the control socket.
+ |- SingleLineResponse - Simple tor response only including a single line of information.
+ |
+ |- from_str - provides a ControlMessage for the given string
+ |- is_ok - response had a 250 status
+ |- content - provides the parsed message content
+ |- raw_content - unparsed socket data
+ |- __str__ - content stripped of protocol formatting
+ +- __iter__ - ControlLine entries for the content of the message
+
+ ControlLine - String subclass with methods for parsing controller responses.
+ |- remainder - provides the unparsed content
+ |- is_empty - checks if the remaining content is empty
+ |- is_next_quoted - checks if the next entry is a quoted value
+ |- is_next_mapping - checks if the next entry is a KEY=VALUE mapping
+ |- peek_key - provides the key of the next entry
+ |- pop - removes and returns the next entry
+ +- pop_mapping - removes and returns the next entry as a KEY=VALUE mapping
+"""
+
+__all__ = [
+ 'add_onion',
+ 'events',
+ 'getinfo',
+ 'getconf',
+ 'protocolinfo',
+ 'authchallenge',
+ 'convert',
+ 'ControlMessage',
+ 'ControlLine',
+ 'SingleLineResponse',
+]
+
+import re
+import threading
+
+try:
+ from StringIO import StringIO
+except ImportError:
+ from io import StringIO
+
+import stem.socket
+
+KEY_ARG = re.compile('^(\S+)=')
+
+# Escape sequences from the 'esc_for_log' function of tor's 'common/util.c'.
+# It's hard to tell what controller functions use this in practice, but direct
+# users are...
+# - 'COOKIEFILE' field of PROTOCOLINFO responses
+# - logged messages about bugs
+# - the 'getinfo_helper_listeners' function of control.c
+
+CONTROL_ESCAPES = {r'\\': '\\', r'\"': '\"', r'\'': '\'',
+ r'\r': '\r', r'\n': '\n', r'\t': '\t'}
+
+
+def convert(response_type, message, **kwargs):
+ """
+ Converts a :class:`~stem.response.ControlMessage` into a particular kind of
+ tor response. This does an in-place conversion of the message from being a
+ :class:`~stem.response.ControlMessage` to a subclass for its response type.
+ Recognized types include...
+
+ =================== =====
+ response_type Class
+ =================== =====
+ **GETINFO** :class:`stem.response.getinfo.GetInfoResponse`
+ **GETCONF** :class:`stem.response.getconf.GetConfResponse`
+ **MAPADDRESS** :class:`stem.response.mapaddress.MapAddressResponse`
+ **EVENT** :class:`stem.response.events.Event` subclass
+ **PROTOCOLINFO** :class:`stem.response.protocolinfo.ProtocolInfoResponse`
+ **AUTHCHALLENGE** :class:`stem.response.authchallenge.AuthChallengeResponse`
+ **SINGLELINE** :class:`stem.response.SingleLineResponse`
+ =================== =====
+
+ :param str response_type: type of tor response to convert to
+ :param stem.response.ControlMessage message: message to be converted
+ :param kwargs: optional keyword arguments to be passed to the parser method
+
+ :raises:
+ * :class:`stem.ProtocolError` the message isn't a proper response of
+ that type
+ * :class:`stem.InvalidArguments` the arguments given as input are
+ invalid, this is can only be raised if the response_type is: **GETINFO**,
+ **GETCONF**
+ * :class:`stem.InvalidRequest` the arguments given as input are
+ invalid, this is can only be raised if the response_type is:
+ **MAPADDRESS**
+ * :class:`stem.OperationFailed` if the action the event represents failed,
+ this is can only be raised if the response_type is: **MAPADDRESS**
+ * **TypeError** if argument isn't a :class:`~stem.response.ControlMessage`
+ or response_type isn't supported
+ """
+
+ import stem.response.add_onion
+ import stem.response.authchallenge
+ import stem.response.events
+ import stem.response.getinfo
+ import stem.response.getconf
+ import stem.response.mapaddress
+ import stem.response.protocolinfo
+
+ if not isinstance(message, ControlMessage):
+ raise TypeError('Only able to convert stem.response.ControlMessage instances')
+
+ response_types = {
+ 'ADD_ONION': stem.response.add_onion.AddOnionResponse,
+ 'AUTHCHALLENGE': stem.response.authchallenge.AuthChallengeResponse,
+ 'EVENT': stem.response.events.Event,
+ 'GETINFO': stem.response.getinfo.GetInfoResponse,
+ 'GETCONF': stem.response.getconf.GetConfResponse,
+ 'MAPADDRESS': stem.response.mapaddress.MapAddressResponse,
+ 'SINGLELINE': SingleLineResponse,
+ 'PROTOCOLINFO': stem.response.protocolinfo.ProtocolInfoResponse,
+ }
+
+ try:
+ response_class = response_types[response_type]
+ except TypeError:
+ raise TypeError('Unsupported response type: %s' % response_type)
+
+ message.__class__ = response_class
+ message._parse_message(**kwargs)
+
+
+class ControlMessage(object):
+ """
+ Message from the control socket. This is iterable and can be stringified for
+ individual message components stripped of protocol formatting. Messages are
+ never empty.
+ """
+
+ @staticmethod
+ def from_str(content, msg_type = None, **kwargs):
+ """
+ Provides a ControlMessage for the given content.
+
+ .. versionadded:: 1.1.0
+
+ :param str content: message to construct the message from
+ :param str msg_type: type of tor reply to parse the content as
+ :param kwargs: optional keyword arguments to be passed to the parser method
+
+ :returns: stem.response.ControlMessage instance
+ """
+
+ msg = stem.socket.recv_message(StringIO(content))
+
+ if msg_type is not None:
+ convert(msg_type, msg, **kwargs)
+
+ return msg
+
+ def __init__(self, parsed_content, raw_content):
+ if not parsed_content:
+ raise ValueError("ControlMessages can't be empty")
+
+ self._parsed_content = parsed_content
+ self._raw_content = raw_content
+
+ def is_ok(self):
+ """
+ Checks if any of our lines have a 250 response.
+
+ :returns: **True** if any lines have a 250 response code, **False** otherwise
+ """
+
+ for code, _, _ in self._parsed_content:
+ if code == '250':
+ return True
+
+ return False
+
+ def content(self, get_bytes = False):
+ """
+ Provides the parsed message content. These are entries of the form...
+
+ ::
+
+ (status_code, divider, content)
+
+ **status_code**
+ Three character code for the type of response (defined in section 4 of
+ the control-spec).
+
+ **divider**
+ Single character to indicate if this is mid-reply, data, or an end to the
+ message (defined in section 2.3 of the control-spec).
+
+ **content**
+ The following content is the actual payload of the line.
+
+ For data entries the content is the full multi-line payload with newline
+ linebreaks and leading periods unescaped.
+
+ The **status_code** and **divider** are both strings (**bytes** in python
+ 2.x and **unicode** in python 3.x). The **content** however is **bytes** if
+ **get_bytes** is **True**.
+
+ .. versionchanged:: 1.1.0
+ Added the get_bytes argument.
+
+ :param bool get_bytes: provides **bytes** for the **content** rather than a **str**
+
+ :returns: **list** of (str, str, str) tuples for the components of this message
+ """
+
+ if stem.prereq.is_python_3() and not get_bytes:
+ return [(code, div, stem.util.str_tools._to_unicode(content)) for (code, div, content) in self._parsed_content]
+ else:
+ return list(self._parsed_content)
+
+ def raw_content(self, get_bytes = False):
+ """
+ Provides the unparsed content read from the control socket.
+
+ .. versionchanged:: 1.1.0
+ Added the get_bytes argument.
+
+ :param bool get_bytes: if **True** then this provides **bytes** rather than a **str**
+
+ :returns: **str** of the socket data used to generate this message
+ """
+
+ if stem.prereq.is_python_3() and not get_bytes:
+ return stem.util.str_tools._to_unicode(self._raw_content)
+ else:
+ return self._raw_content
+
+ def __str__(self):
+ """
+ Content of the message, stripped of status code and divider protocol
+ formatting.
+ """
+
+ return '\n'.join(list(self))
+
+ def __iter__(self):
+ """
+ Provides :class:`~stem.response.ControlLine` instances for the content of
+ the message. This is stripped of status codes and dividers, for instance...
+
+ ::
+
+ 250+info/names=
+ desc/id/* -- Router descriptors by ID.
+ desc/name/* -- Router descriptors by nickname.
+ .
+ 250 OK
+
+ Would provide two entries...
+
+ ::
+
+ 1st - "info/names=
+ desc/id/* -- Router descriptors by ID.
+ desc/name/* -- Router descriptors by nickname."
+ 2nd - "OK"
+ """
+
+ for _, _, content in self._parsed_content:
+ if stem.prereq.is_python_3():
+ content = stem.util.str_tools._to_unicode(content)
+
+ yield ControlLine(content)
+
+ def __len__(self):
+ """
+ :returns: number of ControlLines
+ """
+
+ return len(self._parsed_content)
+
+ def __getitem__(self, index):
+ """
+ :returns: :class:`~stem.response.ControlLine` at the index
+ """
+
+ content = self._parsed_content[index][2]
+
+ if stem.prereq.is_python_3():
+ content = stem.util.str_tools._to_unicode(content)
+
+ return ControlLine(content)
+
+
+class ControlLine(str):
+ """
+ String subclass that represents a line of controller output. This behaves as
+ a normal string with additional methods for parsing and popping entries from
+ a space delimited series of elements like a stack.
+
+ None of these additional methods effect ourselves as a string (which is still
+ immutable). All methods are thread safe.
+ """
+
+ def __new__(self, value):
+ return str.__new__(self, value)
+
+ def __init__(self, value):
+ self._remainder = value
+ self._remainder_lock = threading.RLock()
+
+ def remainder(self):
+ """
+ Provides our unparsed content. This is an empty string after we've popped
+ all entries.
+
+ :returns: **str** of the unparsed content
+ """
+
+ return self._remainder
+
+ def is_empty(self):
+ """
+ Checks if we have further content to pop or not.
+
+ :returns: **True** if we have additional content, **False** otherwise
+ """
+
+ return self._remainder == ''
+
+ def is_next_quoted(self, escaped = False):
+ """
+ Checks if our next entry is a quoted value or not.
+
+ :param bool escaped: unescapes the CONTROL_ESCAPES escape sequences
+
+ :returns: **True** if the next entry can be parsed as a quoted value, **False** otherwise
+ """
+
+ start_quote, end_quote = _get_quote_indices(self._remainder, escaped)
+ return start_quote == 0 and end_quote != -1
+
+ def is_next_mapping(self, key = None, quoted = False, escaped = False):
+ """
+ Checks if our next entry is a KEY=VALUE mapping or not.
+
+ :param str key: checks that the key matches this value, skipping the check if **None**
+ :param bool quoted: checks that the mapping is to a quoted value
+ :param bool escaped: unescapes the CONTROL_ESCAPES escape sequences
+
+ :returns: **True** if the next entry can be parsed as a key=value mapping,
+ **False** otherwise
+ """
+
+ remainder = self._remainder # temp copy to avoid locking
+ key_match = KEY_ARG.match(remainder)
+
+ if key_match:
+ if key and key != key_match.groups()[0]:
+ return False
+
+ if quoted:
+ # checks that we have a quoted value and that it comes after the 'key='
+ start_quote, end_quote = _get_quote_indices(remainder, escaped)
+ return start_quote == key_match.end() and end_quote != -1
+ else:
+ return True # we just needed to check for the key
+ else:
+ return False # doesn't start with a key
+
+ def peek_key(self):
+ """
+ Provides the key of the next entry, providing **None** if it isn't a
+ key/value mapping.
+
+ :returns: **str** with the next entry's key
+ """
+
+ remainder = self._remainder
+ key_match = KEY_ARG.match(remainder)
+
+ if key_match:
+ return key_match.groups()[0]
+ else:
+ return None
+
+ def pop(self, quoted = False, escaped = False):
+ """
+ Parses the next space separated entry, removing it and the space from our
+ remaining content. Examples...
+
+ ::
+
+ >>> line = ControlLine("\\"We're all mad here.\\" says the grinning cat.")
+ >>> print line.pop(True)
+ "We're all mad here."
+ >>> print line.pop()
+ "says"
+ >>> print line.remainder()
+ "the grinning cat."
+
+ >>> line = ControlLine("\\"this has a \\\\\\" and \\\\\\\\ in it\\" foo=bar more_data")
+ >>> print line.pop(True, True)
+ "this has a \\" and \\\\ in it"
+
+ :param bool quoted: parses the next entry as a quoted value, removing the quotes
+ :param bool escaped: unescapes the CONTROL_ESCAPES escape sequences
+
+ :returns: **str** of the next space separated entry
+
+ :raises:
+ * **ValueError** if quoted is True without the value being quoted
+ * **IndexError** if we don't have any remaining content left to parse
+ """
+
+ with self._remainder_lock:
+ next_entry, remainder = _parse_entry(self._remainder, quoted, escaped)
+ self._remainder = remainder
+ return next_entry
+
+ def pop_mapping(self, quoted = False, escaped = False):
+ """
+ Parses the next space separated entry as a KEY=VALUE mapping, removing it
+ and the space from our remaining content.
+
+ :param bool quoted: parses the value as being quoted, removing the quotes
+ :param bool escaped: unescapes the CONTROL_ESCAPES escape sequences
+
+ :returns: **tuple** of the form (key, value)
+
+ :raises: **ValueError** if this isn't a KEY=VALUE mapping or if quoted is
+ **True** without the value being quoted
+ :raises: **IndexError** if there's nothing to parse from the line
+ """
+
+ with self._remainder_lock:
+ if self.is_empty():
+ raise IndexError('no remaining content to parse')
+
+ key_match = KEY_ARG.match(self._remainder)
+
+ if not key_match:
+ raise ValueError("the next entry isn't a KEY=VALUE mapping: " + self._remainder)
+
+ # parse off the key
+ key = key_match.groups()[0]
+ remainder = self._remainder[key_match.end():]
+
+ next_entry, remainder = _parse_entry(remainder, quoted, escaped)
+ self._remainder = remainder
+ return (key, next_entry)
+
+
+def _parse_entry(line, quoted, escaped):
+ """
+ Parses the next entry from the given space separated content.
+
+ :param str line: content to be parsed
+ :param bool quoted: parses the next entry as a quoted value, removing the quotes
+ :param bool escaped: unescapes the CONTROL_ESCAPES escape sequences
+
+ :returns: **tuple** of the form (entry, remainder)
+
+ :raises:
+ * **ValueError** if quoted is True without the next value being quoted
+ * **IndexError** if there's nothing to parse from the line
+ """
+
+ if line == '':
+ raise IndexError('no remaining content to parse')
+
+ next_entry, remainder = '', line
+
+ if quoted:
+ # validate and parse the quoted value
+ start_quote, end_quote = _get_quote_indices(remainder, escaped)
+
+ if start_quote != 0 or end_quote == -1:
+ raise ValueError("the next entry isn't a quoted value: " + line)
+
+ next_entry, remainder = remainder[1:end_quote], remainder[end_quote + 1:]
+ else:
+ # non-quoted value, just need to check if there's more data afterward
+ if ' ' in remainder:
+ next_entry, remainder = remainder.split(' ', 1)
+ else:
+ next_entry, remainder = remainder, ''
+
+ if escaped:
+ next_entry = _unescape(next_entry)
+
+ return (next_entry, remainder.lstrip())
+
+
+def _get_quote_indices(line, escaped):
+ """
+ Provides the indices of the next two quotes in the given content.
+
+ :param str line: content to be parsed
+ :param bool escaped: unescapes the CONTROL_ESCAPES escape sequences
+
+ :returns: **tuple** of two ints, indices being -1 if a quote doesn't exist
+ """
+
+ indices, quote_index = [], -1
+
+ for _ in range(2):
+ quote_index = line.find('"', quote_index + 1)
+
+ # if we have escapes then we need to skip any r'\"' entries
+ if escaped:
+ # skip check if index is -1 (no match) or 0 (first character)
+ while quote_index >= 1 and line[quote_index - 1] == '\\':
+ quote_index = line.find('"', quote_index + 1)
+
+ indices.append(quote_index)
+
+ return tuple(indices)
+
+
+def _unescape(entry):
+ # Unescapes the given string with the mappings in CONTROL_ESCAPES.
+ #
+ # This can't be a simple series of str.replace() calls because replacements
+ # need to be excluded from consideration for further unescaping. For
+ # instance, '\\t' should be converted to '\t' rather than a tab.
+
+ def _pop_with_unescape(entry):
+ # Pop either the first character or the escape sequence conversion the
+ # entry starts with. This provides a tuple of...
+ #
+ # (unescaped prefix, remaining entry)
+
+ for esc_sequence, replacement in CONTROL_ESCAPES.items():
+ if entry.startswith(esc_sequence):
+ return (replacement, entry[len(esc_sequence):])
+
+ return (entry[0], entry[1:])
+
+ result = []
+
+ while entry:
+ prefix, entry = _pop_with_unescape(entry)
+ result.append(prefix)
+
+ return ''.join(result)
+
+
+class SingleLineResponse(ControlMessage):
+ """
+ Reply to a request that performs an action rather than querying data. These
+ requests only contain a single line, which is 'OK' if successful, and a
+ description of the problem if not.
+
+ :var str code: status code for our line
+ :var str message: content of the line
+ """
+
+ def is_ok(self, strict = False):
+ """
+ Checks if the response code is "250". If strict is **True** then this
+ checks if the response is "250 OK"
+
+ :param bool strict: checks for a "250 OK" message if **True**
+
+ :returns:
+ * If strict is **False**: **True** if the response code is "250", **False** otherwise
+ * If strict is **True**: **True** if the response is "250 OK", **False** otherwise
+ """
+
+ if strict:
+ return self.content()[0] == ('250', ' ', 'OK')
+
+ return self.content()[0][0] == '250'
+
+ def _parse_message(self):
+ content = self.content()
+
+ if len(content) > 1:
+ raise stem.ProtocolError('Received multi-line response')
+ elif len(content) == 0:
+ raise stem.ProtocolError('Received empty response')
+ else:
+ self.code, _, self.message = content[0]
diff --git a/ext/stem/response/add_onion.py b/ext/stem/response/add_onion.py
new file mode 100644
index 00000000..1472668a
--- /dev/null
+++ b/ext/stem/response/add_onion.py
@@ -0,0 +1,43 @@
+# Copyright 2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+import stem.response
+
+
+class AddOnionResponse(stem.response.ControlMessage):
+ """
+ ADD_ONION response.
+
+ :var str service_id: hidden service address without the '.onion' suffix
+ :var str private_key: base64 encoded hidden service private key
+ :var str private_key_type: crypto used to generate the hidden service private
+ key (such as RSA1024)
+ """
+
+ def _parse_message(self):
+ # Example:
+ # 250-ServiceID=gfzprpioee3hoppz
+ # 250-PrivateKey=RSA1024:MIICXgIBAAKBgQDZvYVxv...
+ # 250 OK
+
+ self.service_id = None
+ self.private_key = None
+ self.private_key_type = None
+
+ if not self.is_ok():
+ raise stem.ProtocolError("ADD_ONION response didn't have an OK status: %s" % self)
+
+ if not str(self).startswith('ServiceID='):
+ raise stem.ProtocolError('ADD_ONION response should start with the service id: %s' % self)
+
+ for line in list(self):
+ if '=' in line:
+ key, value = line.split('=', 1)
+
+ if key == 'ServiceID':
+ self.service_id = value
+ elif key == 'PrivateKey':
+ if ':' not in value:
+ raise stem.ProtocolError("ADD_ONION PrivateKey lines should be of the form 'PrivateKey=[type]:[key]: %s" % self)
+
+ self.private_key_type, self.private_key = value.split(':', 1)
diff --git a/ext/stem/response/authchallenge.py b/ext/stem/response/authchallenge.py
new file mode 100644
index 00000000..60f39970
--- /dev/null
+++ b/ext/stem/response/authchallenge.py
@@ -0,0 +1,56 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+import binascii
+
+import stem.response
+import stem.socket
+import stem.util.str_tools
+import stem.util.tor_tools
+
+
+class AuthChallengeResponse(stem.response.ControlMessage):
+ """
+ AUTHCHALLENGE query response.
+
+ :var str server_hash: server hash provided by tor
+ :var str server_nonce: server nonce provided by tor
+ """
+
+ def _parse_message(self):
+ # Example:
+ # 250 AUTHCHALLENGE SERVERHASH=680A73C9836C4F557314EA1C4EDE54C285DB9DC89C83627401AEF9D7D27A95D5 SERVERNONCE=F8EA4B1F2C8B40EF1AF68860171605B910E3BBCABADF6FC3DB1FA064F4690E85
+
+ self.server_hash = None
+ self.server_nonce = None
+
+ if not self.is_ok():
+ raise stem.ProtocolError("AUTHCHALLENGE response didn't have an OK status:\n%s" % self)
+ elif len(self) > 1:
+ raise stem.ProtocolError('Received multiline AUTHCHALLENGE response:\n%s' % self)
+
+ line = self[0]
+
+ # sanity check that we're a AUTHCHALLENGE response
+ if not line.pop() == 'AUTHCHALLENGE':
+ raise stem.ProtocolError('Message is not an AUTHCHALLENGE response (%s)' % self)
+
+ if line.is_next_mapping('SERVERHASH'):
+ value = line.pop_mapping()[1]
+
+ if not stem.util.tor_tools.is_hex_digits(value, 64):
+ raise stem.ProtocolError('SERVERHASH has an invalid value: %s' % value)
+
+ self.server_hash = binascii.a2b_hex(stem.util.str_tools._to_bytes(value))
+ else:
+ raise stem.ProtocolError('Missing SERVERHASH mapping: %s' % line)
+
+ if line.is_next_mapping('SERVERNONCE'):
+ value = line.pop_mapping()[1]
+
+ if not stem.util.tor_tools.is_hex_digits(value, 64):
+ raise stem.ProtocolError('SERVERNONCE has an invalid value: %s' % value)
+
+ self.server_nonce = binascii.a2b_hex(stem.util.str_tools._to_bytes(value))
+ else:
+ raise stem.ProtocolError('Missing SERVERNONCE mapping: %s' % line)
diff --git a/ext/stem/response/events.py b/ext/stem/response/events.py
new file mode 100644
index 00000000..9c386495
--- /dev/null
+++ b/ext/stem/response/events.py
@@ -0,0 +1,1331 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+import io
+import re
+import time
+
+import stem
+import stem.control
+import stem.descriptor.router_status_entry
+import stem.response
+import stem.version
+
+from stem import str_type, int_type
+from stem.util import connection, log, str_tools, tor_tools
+
+# Matches keyword=value arguments. This can't be a simple "(.*)=(.*)" pattern
+# because some positional arguments, like circuit paths, can have an equal
+# sign.
+
+KW_ARG = re.compile('^(.*) ([A-Za-z0-9_]+)=(\S*)$')
+QUOTED_KW_ARG = re.compile('^(.*) ([A-Za-z0-9_]+)="(.*)"$')
+CELL_TYPE = re.compile('^[a-z0-9_]+$')
+
+
+class Event(stem.response.ControlMessage):
+ """
+ Base for events we receive asynchronously, as described in section 4.1 of the
+ `control-spec
+ <https://gitweb.torproject.org/torspec.git/tree/control-spec.txt>`_.
+
+ :var str type: event type
+ :var int arrived_at: unix timestamp for when the message arrived
+ :var list positional_args: positional arguments of the event
+ :var dict keyword_args: key/value arguments of the event
+ """
+
+ _POSITIONAL_ARGS = () # attribute names for recognized positional arguments
+ _KEYWORD_ARGS = {} # map of 'keyword => attribute' for recognized attributes
+ _QUOTED = () # positional arguments that are quoted
+ _OPTIONALLY_QUOTED = () # positional arguments that may or may not be quoted
+ _SKIP_PARSING = False # skip parsing contents into our positional_args and keyword_args
+ _VERSION_ADDED = stem.version.Version('0.1.1.1-alpha') # minimum version with control-spec V1 event support
+
+ def _parse_message(self, arrived_at = None):
+ if arrived_at is None:
+ arrived_at = int(time.time())
+
+ if not str(self).strip():
+ raise stem.ProtocolError('Received a blank tor event. Events must at the very least have a type.')
+
+ self.type = str(self).split()[0]
+ self.arrived_at = arrived_at
+
+ # if we're a recognized event type then translate ourselves into that subclass
+
+ if self.type in EVENT_TYPE_TO_CLASS:
+ self.__class__ = EVENT_TYPE_TO_CLASS[self.type]
+
+ self.positional_args = []
+ self.keyword_args = {}
+
+ if not self._SKIP_PARSING:
+ self._parse_standard_attr()
+
+ self._parse()
+
+ def _parse_standard_attr(self):
+ """
+ Most events are of the form...
+ 650 *( positional_args ) *( key "=" value )
+
+ This parses this standard format, populating our **positional_args** and
+ **keyword_args** attributes and creating attributes if it's in our event's
+ **_POSITIONAL_ARGS** and **_KEYWORD_ARGS**.
+ """
+
+ # Tor events contain some number of positional arguments followed by
+ # key/value mappings. Parsing keyword arguments from the end until we hit
+ # something that isn't a key/value mapping. The rest are positional.
+
+ content = str(self)
+
+ while True:
+ match = QUOTED_KW_ARG.match(content)
+
+ if not match:
+ match = KW_ARG.match(content)
+
+ if match:
+ content, keyword, value = match.groups()
+ self.keyword_args[keyword] = value
+ else:
+ break
+
+ # Setting attributes for the fields that we recognize.
+
+ self.positional_args = content.split()[1:]
+ positional = list(self.positional_args)
+
+ for attr_name in self._POSITIONAL_ARGS:
+ attr_value = None
+
+ if positional:
+ if attr_name in self._QUOTED or (attr_name in self._OPTIONALLY_QUOTED and positional[0].startswith('"')):
+ attr_values = [positional.pop(0)]
+
+ if not attr_values[0].startswith('"'):
+ raise stem.ProtocolError("The %s value should be quoted, but didn't have a starting quote: %s" % (attr_name, self))
+
+ while True:
+ if not positional:
+ raise stem.ProtocolError("The %s value should be quoted, but didn't have an ending quote: %s" % (attr_name, self))
+
+ attr_values.append(positional.pop(0))
+
+ if attr_values[-1].endswith('"'):
+ break
+
+ attr_value = ' '.join(attr_values)[1:-1]
+ else:
+ attr_value = positional.pop(0)
+
+ setattr(self, attr_name, attr_value)
+
+ for controller_attr_name, attr_name in self._KEYWORD_ARGS.items():
+ setattr(self, attr_name, self.keyword_args.get(controller_attr_name))
+
+ # method overwritten by our subclasses for special handling that they do
+ def _parse(self):
+ pass
+
+ def _log_if_unrecognized(self, attr, attr_enum):
+ """
+ Checks if an attribute exists in a given enumeration, logging a message if
+ it isn't. Attributes can either be for a string or collection of strings
+
+ :param str attr: name of the attribute to check
+ :param stem.util.enum.Enum enum: enumeration to check against
+ """
+
+ attr_values = getattr(self, attr)
+
+ if attr_values:
+ if isinstance(attr_values, (bytes, str_type)):
+ attr_values = [attr_values]
+
+ for value in attr_values:
+ if value not in attr_enum:
+ log_id = 'event.%s.unknown_%s.%s' % (self.type.lower(), attr, value)
+ unrecognized_msg = "%s event had an unrecognized %s (%s). Maybe a new addition to the control protocol? Full Event: '%s'" % (self.type, attr, value, self)
+ log.log_once(log_id, log.INFO, unrecognized_msg)
+
+
+class AddrMapEvent(Event):
+ """
+ Event that indicates a new address mapping.
+
+ The ADDRMAP event was one of the first Control Protocol V1 events and was
+ introduced in tor version 0.1.1.1-alpha.
+
+ .. versionchanged:: 1.1.0
+ Added the cached attribute.
+
+ :var str hostname: address being resolved
+ :var str destination: destionation of the resolution, this is usually an ip,
+ but could be a hostname if TrackHostExits is enabled or **NONE** if the
+ resolution failed
+ :var datetime expiry: expiration time of the resolution in local time
+ :var str error: error code if the resolution failed
+ :var datetime utc_expiry: expiration time of the resolution in UTC
+ :var bool cached: **True** if the resolution will be kept until it expires,
+ **False** otherwise or **None** if undefined
+ """
+
+ _POSITIONAL_ARGS = ('hostname', 'destination', 'expiry')
+ _KEYWORD_ARGS = {
+ 'error': 'error',
+ 'EXPIRES': 'utc_expiry',
+ 'CACHED': 'cached',
+ }
+ _OPTIONALLY_QUOTED = ('expiry')
+
+ def _parse(self):
+ if self.destination == '<error>':
+ self.destination = None
+
+ if self.expiry is not None:
+ if self.expiry == 'NEVER':
+ self.expiry = None
+ else:
+ try:
+ self.expiry = stem.util.str_tools._parse_timestamp(self.expiry)
+ except ValueError:
+ raise stem.ProtocolError('Unable to parse date in ADDRMAP event: %s' % self)
+
+ if self.utc_expiry is not None:
+ self.utc_expiry = stem.util.str_tools._parse_timestamp(self.utc_expiry)
+
+ if self.cached is not None:
+ if self.cached == 'YES':
+ self.cached = True
+ elif self.cached == 'NO':
+ self.cached = False
+ else:
+ raise stem.ProtocolError("An ADDRMAP event's CACHED mapping can only be 'YES' or 'NO': %s" % self)
+
+
+class AuthDirNewDescEvent(Event):
+ """
+ Event specific to directory authorities, indicating that we just received new
+ descriptors. The descriptor type contained within this event is unspecified
+ so the descriptor contents are left unparsed.
+
+ The AUTHDIR_NEWDESCS event was introduced in tor version 0.1.1.10-alpha.
+
+ :var stem.AuthDescriptorAction action: what is being done with the descriptor
+ :var str message: explanation of why we chose this action
+ :var str descriptor: content of the descriptor
+ """
+
+ _SKIP_PARSING = True
+ _VERSION_ADDED = stem.version.Requirement.EVENT_AUTHDIR_NEWDESCS
+
+ def _parse(self):
+ lines = str(self).split('\n')
+
+ if len(lines) < 5:
+ raise stem.ProtocolError("AUTHDIR_NEWDESCS events must contain lines for at least the type, action, message, descriptor, and terminating 'OK'")
+ elif lines[-1] != 'OK':
+ raise stem.ProtocolError("AUTHDIR_NEWDESCS doesn't end with an 'OK'")
+
+ # TODO: For stem 2.0.0 we should consider changing 'descriptor' to a
+ # ServerDescriptor instance.
+
+ self.action = lines[1]
+ self.message = lines[2]
+ self.descriptor = '\n'.join(lines[3:-1])
+
+
+class BandwidthEvent(Event):
+ """
+ Event emitted every second with the bytes sent and received by tor.
+
+ The BW event was one of the first Control Protocol V1 events and was
+ introduced in tor version 0.1.1.1-alpha.
+
+ :var long read: bytes received by tor that second
+ :var long written: bytes sent by tor that second
+ """
+
+ _POSITIONAL_ARGS = ('read', 'written')
+
+ def _parse(self):
+ if not self.read:
+ raise stem.ProtocolError('BW event is missing its read value')
+ elif not self.written:
+ raise stem.ProtocolError('BW event is missing its written value')
+ elif not self.read.isdigit() or not self.written.isdigit():
+ raise stem.ProtocolError("A BW event's bytes sent and received should be a positive numeric value, received: %s" % self)
+
+ self.read = int_type(self.read)
+ self.written = int_type(self.written)
+
+
+class BuildTimeoutSetEvent(Event):
+ """
+ Event indicating that the timeout value for a circuit has changed. This was
+ first added in tor version 0.2.2.7.
+
+ The BUILDTIMEOUT_SET event was introduced in tor version 0.2.2.7-alpha.
+
+ :var stem.TimeoutSetType set_type: way in which the timeout is changing
+ :var int total_times: circuit build times tor used to determine the timeout
+ :var int timeout: circuit timeout value in milliseconds
+ :var int xm: Pareto parameter Xm in milliseconds
+ :var float alpha: Pareto parameter alpha
+ :var float quantile: CDF quantile cutoff point
+ :var float timeout_rate: ratio of circuits that have time out
+ :var int close_timeout: duration to keep measurement circuits in milliseconds
+ :var float close_rate: ratio of measurement circuits that are closed
+ """
+
+ _POSITIONAL_ARGS = ('set_type',)
+ _KEYWORD_ARGS = {
+ 'TOTAL_TIMES': 'total_times',
+ 'TIMEOUT_MS': 'timeout',
+ 'XM': 'xm',
+ 'ALPHA': 'alpha',
+ 'CUTOFF_QUANTILE': 'quantile',
+ 'TIMEOUT_RATE': 'timeout_rate',
+ 'CLOSE_MS': 'close_timeout',
+ 'CLOSE_RATE': 'close_rate',
+ }
+ _VERSION_ADDED = stem.version.Requirement.EVENT_BUILDTIMEOUT_SET
+
+ def _parse(self):
+ # convert our integer and float parameters
+
+ for param in ('total_times', 'timeout', 'xm', 'close_timeout'):
+ param_value = getattr(self, param)
+
+ if param_value is not None:
+ try:
+ setattr(self, param, int(param_value))
+ except ValueError:
+ raise stem.ProtocolError('The %s of a BUILDTIMEOUT_SET should be an integer: %s' % (param, self))
+
+ for param in ('alpha', 'quantile', 'timeout_rate', 'close_rate'):
+ param_value = getattr(self, param)
+
+ if param_value is not None:
+ try:
+ setattr(self, param, float(param_value))
+ except ValueError:
+ raise stem.ProtocolError('The %s of a BUILDTIMEOUT_SET should be a float: %s' % (param, self))
+
+ self._log_if_unrecognized('set_type', stem.TimeoutSetType)
+
+
+class CircuitEvent(Event):
+ """
+ Event that indicates that a circuit has changed.
+
+ The fingerprint or nickname values in our 'path' may be **None** if the
+ VERBOSE_NAMES feature isn't enabled. The option was first introduced in tor
+ version 0.1.2.2, and on by default after 0.2.2.1.
+
+ The CIRC event was one of the first Control Protocol V1 events and was
+ introduced in tor version 0.1.1.1-alpha.
+
+ .. versionchanged:: 1.4.0
+ Added the socks_username and socks_password attributes which is used for
+ `stream isolation
+ <https://gitweb.torproject.org/torspec.git/tree/proposals/171-separate-streams.txt>`_.
+
+ :var str id: circuit identifier
+ :var stem.CircStatus status: reported status for the circuit
+ :var tuple path: relays involved in the circuit, these are
+ **(fingerprint, nickname)** tuples
+ :var tuple build_flags: :data:`~stem.CircBuildFlag` attributes
+ governing how the circuit is built
+ :var stem.CircPurpose purpose: purpose that the circuit is intended for
+ :var stem.HiddenServiceState hs_state: status if this is a hidden service circuit
+ :var str rend_query: circuit's rendezvous-point if this is hidden service related
+ :var datetime created: time when the circuit was created or cannibalized
+ :var stem.CircClosureReason reason: reason for the circuit to be closed
+ :var stem.CircClosureReason remote_reason: remote side's reason for the circuit to be closed
+ :var str socks_username: username for using this circuit
+ :var str socks_password: password for using this circuit
+ """
+
+ _POSITIONAL_ARGS = ('id', 'status', 'path')
+ _KEYWORD_ARGS = {
+ 'BUILD_FLAGS': 'build_flags',
+ 'PURPOSE': 'purpose',
+ 'HS_STATE': 'hs_state',
+ 'REND_QUERY': 'rend_query',
+ 'TIME_CREATED': 'created',
+ 'REASON': 'reason',
+ 'REMOTE_REASON': 'remote_reason',
+ 'SOCKS_USERNAME': 'socks_username',
+ 'SOCKS_PASSWORD': 'socks_password',
+ }
+
+ def _parse(self):
+ self.path = tuple(stem.control._parse_circ_path(self.path))
+
+ if self.build_flags is not None:
+ self.build_flags = tuple(self.build_flags.split(','))
+
+ if self.created is not None:
+ try:
+ self.created = str_tools._parse_iso_timestamp(self.created)
+ except ValueError as exc:
+ raise stem.ProtocolError('Unable to parse create date (%s): %s' % (exc, self))
+
+ if not tor_tools.is_valid_circuit_id(self.id):
+ raise stem.ProtocolError("Circuit IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.id, self))
+
+ self._log_if_unrecognized('status', stem.CircStatus)
+ self._log_if_unrecognized('build_flags', stem.CircBuildFlag)
+ self._log_if_unrecognized('purpose', stem.CircPurpose)
+ self._log_if_unrecognized('hs_state', stem.HiddenServiceState)
+ self._log_if_unrecognized('reason', stem.CircClosureReason)
+ self._log_if_unrecognized('remote_reason', stem.CircClosureReason)
+
+ def _compare(self, other, method):
+ if not isinstance(other, CircuitEvent):
+ return False
+
+ for attr in ('id', 'status', 'path', 'build_flags', 'purpose', 'hs_state', 'rend_query', 'created', 'reason', 'remote_reason', 'socks_username', 'socks_port'):
+ my_attr = getattr(self, attr)
+ other_attr = getattr(other, attr)
+
+ # Our id attribute is technically a string, but Tor conventionally uses
+ # ints. Attempt to handle as ints if that's the case so we get numeric
+ # ordering.
+
+ if attr == 'id' and my_attr and other_attr:
+ if my_attr.isdigit() and other_attr.isdigit():
+ my_attr = int(my_attr)
+ other_attr = int(other_attr)
+
+ if my_attr is None:
+ my_attr = ''
+
+ if other_attr is None:
+ other_attr = ''
+
+ if my_attr != other_attr:
+ return method(my_attr, other_attr)
+
+ return True
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __gt__(self, other):
+ return self._compare(other, lambda s, o: s > o)
+
+ def __ge__(self, other):
+ return self._compare(other, lambda s, o: s >= o)
+
+
+class CircMinorEvent(Event):
+ """
+ Event providing information about minor changes in our circuits. This was
+ first added in tor version 0.2.3.11.
+
+ The CIRC_MINOR event was introduced in tor version 0.2.3.11-alpha.
+
+ :var str id: circuit identifier
+ :var stem.CircEvent event: type of change in the circuit
+ :var tuple path: relays involved in the circuit, these are
+ **(fingerprint, nickname)** tuples
+ :var tuple build_flags: :data:`~stem.CircBuildFlag` attributes
+ governing how the circuit is built
+ :var stem.CircPurpose purpose: purpose that the circuit is intended for
+ :var stem.HiddenServiceState hs_state: status if this is a hidden service circuit
+ :var str rend_query: circuit's rendezvous-point if this is hidden service related
+ :var datetime created: time when the circuit was created or cannibalized
+ :var stem.CircPurpose old_purpose: prior purpose for the circuit
+ :var stem.HiddenServiceState old_hs_state: prior status as a hidden service circuit
+ """
+
+ _POSITIONAL_ARGS = ('id', 'event', 'path')
+ _KEYWORD_ARGS = {
+ 'BUILD_FLAGS': 'build_flags',
+ 'PURPOSE': 'purpose',
+ 'HS_STATE': 'hs_state',
+ 'REND_QUERY': 'rend_query',
+ 'TIME_CREATED': 'created',
+ 'OLD_PURPOSE': 'old_purpose',
+ 'OLD_HS_STATE': 'old_hs_state',
+ }
+ _VERSION_ADDED = stem.version.Requirement.EVENT_CIRC_MINOR
+
+ def _parse(self):
+ self.path = tuple(stem.control._parse_circ_path(self.path))
+
+ if self.build_flags is not None:
+ self.build_flags = tuple(self.build_flags.split(','))
+
+ if self.created is not None:
+ try:
+ self.created = str_tools._parse_iso_timestamp(self.created)
+ except ValueError as exc:
+ raise stem.ProtocolError('Unable to parse create date (%s): %s' % (exc, self))
+
+ if not tor_tools.is_valid_circuit_id(self.id):
+ raise stem.ProtocolError("Circuit IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.id, self))
+
+ self._log_if_unrecognized('event', stem.CircEvent)
+ self._log_if_unrecognized('build_flags', stem.CircBuildFlag)
+ self._log_if_unrecognized('purpose', stem.CircPurpose)
+ self._log_if_unrecognized('hs_state', stem.HiddenServiceState)
+ self._log_if_unrecognized('old_purpose', stem.CircPurpose)
+ self._log_if_unrecognized('old_hs_state', stem.HiddenServiceState)
+
+
+class ClientsSeenEvent(Event):
+ """
+ Periodic event on bridge relays that provides a summary of our users.
+
+ The CLIENTS_SEEN event was introduced in tor version 0.2.1.10-alpha.
+
+ :var datetime start_time: time in UTC that we started collecting these stats
+ :var dict locales: mapping of country codes to a rounded count for the number of users
+ :var dict ip_versions: mapping of ip protocols to a rounded count for the number of users
+ """
+
+ _KEYWORD_ARGS = {
+ 'TimeStarted': 'start_time',
+ 'CountrySummary': 'locales',
+ 'IPVersions': 'ip_versions',
+ }
+ _VERSION_ADDED = stem.version.Requirement.EVENT_CLIENTS_SEEN
+
+ def _parse(self):
+ if self.start_time is not None:
+ self.start_time = stem.util.str_tools._parse_timestamp(self.start_time)
+
+ if self.locales is not None:
+ locale_to_count = {}
+
+ for entry in self.locales.split(','):
+ if '=' not in entry:
+ raise stem.ProtocolError("The CLIENTS_SEEN's CountrySummary should be a comma separated listing of '<locale>=<count>' mappings: %s" % self)
+
+ locale, count = entry.split('=', 1)
+
+ if len(locale) != 2:
+ raise stem.ProtocolError("Locales should be a two character code, got '%s': %s" % (locale, self))
+ elif not count.isdigit():
+ raise stem.ProtocolError('Locale count was non-numeric (%s): %s' % (count, self))
+ elif locale in locale_to_count:
+ raise stem.ProtocolError("CountrySummary had multiple mappings for '%s': %s" % (locale, self))
+
+ locale_to_count[locale] = int(count)
+
+ self.locales = locale_to_count
+
+ if self.ip_versions is not None:
+ protocol_to_count = {}
+
+ for entry in self.ip_versions.split(','):
+ if '=' not in entry:
+ raise stem.ProtocolError("The CLIENTS_SEEN's IPVersions should be a comma separated listing of '<protocol>=<count>' mappings: %s" % self)
+
+ protocol, count = entry.split('=', 1)
+
+ if not count.isdigit():
+ raise stem.ProtocolError('IP protocol count was non-numeric (%s): %s' % (count, self))
+
+ protocol_to_count[protocol] = int(count)
+
+ self.ip_versions = protocol_to_count
+
+
+class ConfChangedEvent(Event):
+ """
+ Event that indicates that our configuration changed, either in response to a
+ SETCONF or RELOAD signal.
+
+ The CONF_CHANGED event was introduced in tor version 0.2.3.3-alpha.
+
+ :var dict config: mapping of configuration options to their new values
+ (**None** if the option is being unset)
+ """
+
+ _SKIP_PARSING = True
+ _VERSION_ADDED = stem.version.Requirement.EVENT_CONF_CHANGED
+
+ def _parse(self):
+ self.config = {}
+
+ # Skip first and last line since they're the header and footer. For
+ # instance...
+ #
+ # 650-CONF_CHANGED
+ # 650-ExitNodes=caerSidi
+ # 650-ExitPolicy
+ # 650-MaxCircuitDirtiness=20
+ # 650 OK
+
+ for line in str(self).splitlines()[1:-1]:
+ if '=' in line:
+ key, value = line.split('=', 1)
+ else:
+ key, value = line, None
+
+ self.config[key] = value
+
+
+class DescChangedEvent(Event):
+ """
+ Event that indicates that our descriptor has changed.
+
+ The DESCCHANGED event was introduced in tor version 0.1.2.2-alpha.
+ """
+
+ _VERSION_ADDED = stem.version.Requirement.EVENT_DESCCHANGED
+
+
+class GuardEvent(Event):
+ """
+ Event that indicates that our guard relays have changed. The 'endpoint' could
+ be either a...
+
+ * fingerprint
+ * 'fingerprint=nickname' pair
+
+ The derived 'endpoint_*' attributes are generally more useful.
+
+ The GUARD event was introduced in tor version 0.1.2.5-alpha.
+
+ :var stem.GuardType guard_type: purpose the guard relay is for
+ :var str endpoint: relay that the event concerns
+ :var str endpoint_fingerprint: endpoint's finterprint
+ :var str endpoint_nickname: endpoint's nickname if it was provided
+ :var stem.GuardStatus status: status of the guard relay
+ """
+
+ _VERSION_ADDED = stem.version.Requirement.EVENT_GUARD
+ _POSITIONAL_ARGS = ('guard_type', 'endpoint', 'status')
+
+ def _parse(self):
+ self.endpoint_fingerprint = None
+ self.endpoint_nickname = None
+
+ try:
+ self.endpoint_fingerprint, self.endpoint_nickname = \
+ stem.control._parse_circ_entry(self.endpoint)
+ except stem.ProtocolError:
+ raise stem.ProtocolError("GUARD's endpoint doesn't match a ServerSpec: %s" % self)
+
+ self._log_if_unrecognized('guard_type', stem.GuardType)
+ self._log_if_unrecognized('status', stem.GuardStatus)
+
+
+class HSDescEvent(Event):
+ """
+ Event triggered when we fetch a hidden service descriptor that currently isn't in our cache.
+
+ The HS_DESC event was introduced in tor version 0.2.5.2-alpha.
+
+ .. versionadded:: 1.2.0
+
+ .. versionchanged:: 1.3.0
+ Added the reason attribute.
+
+ :var stem.HSDescAction action: what is happening with the descriptor
+ :var str address: hidden service address
+ :var stem.HSAuth authentication: service's authentication method
+ :var str directory: hidden service directory servicing the request
+ :var str directory_fingerprint: hidden service directory's finterprint
+ :var str directory_nickname: hidden service directory's nickname if it was provided
+ :var str descriptor_id: descriptor identifier
+ :var stem.HSDescReason reason: reason the descriptor failed to be fetched
+ """
+
+ _VERSION_ADDED = stem.version.Requirement.EVENT_HS_DESC
+ _POSITIONAL_ARGS = ('action', 'address', 'authentication', 'directory', 'descriptor_id')
+ _KEYWORD_ARGS = {'REASON': 'reason'}
+
+ def _parse(self):
+ self.directory_fingerprint = None
+ self.directory_nickname = None
+
+ try:
+ self.directory_fingerprint, self.directory_nickname = \
+ stem.control._parse_circ_entry(self.directory)
+ except stem.ProtocolError:
+ raise stem.ProtocolError("HS_DESC's directory doesn't match a ServerSpec: %s" % self)
+
+ self._log_if_unrecognized('action', stem.HSDescAction)
+ self._log_if_unrecognized('authentication', stem.HSAuth)
+
+
+class HSDescContentEvent(Event):
+ """
+ Provides the content of hidden service descriptors we fetch.
+
+ The HS_DESC_CONTENT event was introduced in tor version 0.2.7.1-alpha.
+
+ .. versionadded:: 1.4.0
+
+ :var str address: hidden service address
+ :var str descriptor_id: descriptor identifier
+ :var str directory: hidden service directory servicing the request
+ :var str directory_fingerprint: hidden service directory's finterprint
+ :var str directory_nickname: hidden service directory's nickname if it was provided
+ :var stem.descriptor.hidden_service_descriptor.HiddenServiceDescriptor descriptor: descriptor that was retrieved
+ """
+
+ _VERSION_ADDED = stem.version.Requirement.EVENT_HS_DESC_CONTENT
+ _POSITIONAL_ARGS = ('address', 'descriptor_id', 'directory')
+
+ def _parse(self):
+ if self.address == 'UNKNOWN':
+ self.address = None
+
+ self.directory_fingerprint = None
+ self.directory_nickname = None
+
+ try:
+ self.directory_fingerprint, self.directory_nickname = \
+ stem.control._parse_circ_entry(self.directory)
+ except stem.ProtocolError:
+ raise stem.ProtocolError("HS_DESC_CONTENT's directory doesn't match a ServerSpec: %s" % self)
+
+ # skip the first line (our positional arguments) and last ('OK')
+
+ desc_content = str_tools._to_bytes('\n'.join(str(self).splitlines()[1:-1]))
+ self.descriptor = None
+
+ if desc_content:
+ self.descriptor = list(stem.descriptor.hidden_service_descriptor._parse_file(io.BytesIO(desc_content)))[0]
+
+
+class LogEvent(Event):
+ """
+ Tor logging event. These are the most visible kind of event since, by
+ default, tor logs at the NOTICE :data:`~stem.Runlevel` to stdout.
+
+ The logging events were some of the first Control Protocol V1 events
+ and were introduced in tor version 0.1.1.1-alpha.
+
+ :var stem.Runlevel runlevel: runlevel of the logged message
+ :var str message: logged message
+ """
+
+ _SKIP_PARSING = True
+
+ def _parse(self):
+ self.runlevel = self.type
+ self._log_if_unrecognized('runlevel', stem.Runlevel)
+
+ # message is our content, minus the runlevel and ending "OK" if a
+ # multi-line message
+
+ self.message = str(self)[len(self.runlevel) + 1:].rstrip('\nOK')
+
+
+class NetworkStatusEvent(Event):
+ """
+ Event for when our copy of the consensus has changed. This was introduced in
+ tor version 0.1.2.3.
+
+ The NS event was introduced in tor version 0.1.2.3-alpha.
+
+ :var list desc: :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` for the changed descriptors
+ """
+
+ _SKIP_PARSING = True
+ _VERSION_ADDED = stem.version.Requirement.EVENT_NS
+
+ def _parse(self):
+ content = str(self).lstrip('NS\n').rstrip('\nOK')
+
+ # TODO: For stem 2.0.0 consider changing 'desc' to 'descriptors' to match
+ # our other events.
+
+ self.desc = list(stem.descriptor.router_status_entry._parse_file(
+ io.BytesIO(str_tools._to_bytes(content)),
+ True,
+ entry_class = stem.descriptor.router_status_entry.RouterStatusEntryV3,
+ ))
+
+
+class NewConsensusEvent(Event):
+ """
+ Event for when we have a new consensus. This is similar to
+ :class:`~stem.response.events.NetworkStatusEvent`, except that it contains
+ the whole consensus so anything not listed is implicitly no longer
+ recommended.
+
+ The NEWCONSENSUS event was introduced in tor version 0.2.1.13-alpha.
+
+ :var list desc: :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` for the changed descriptors
+ """
+
+ _SKIP_PARSING = True
+ _VERSION_ADDED = stem.version.Requirement.EVENT_NEWCONSENSUS
+
+ def _parse(self):
+ content = str(self).lstrip('NEWCONSENSUS\n').rstrip('\nOK')
+
+ # TODO: For stem 2.0.0 consider changing 'desc' to 'descriptors' to match
+ # our other events.
+
+ self.desc = list(stem.descriptor.router_status_entry._parse_file(
+ io.BytesIO(str_tools._to_bytes(content)),
+ True,
+ entry_class = stem.descriptor.router_status_entry.RouterStatusEntryV3,
+ ))
+
+
+class NewDescEvent(Event):
+ """
+ Event that indicates that a new descriptor is available.
+
+ The fingerprint or nickname values in our 'relays' may be **None** if the
+ VERBOSE_NAMES feature isn't enabled. The option was first introduced in tor
+ version 0.1.2.2, and on by default after 0.2.2.1.
+
+ The NEWDESC event was one of the first Control Protocol V1 events and was
+ introduced in tor version 0.1.1.1-alpha.
+
+ :var tuple relays: **(fingerprint, nickname)** tuples for the relays with
+ new descriptors
+ """
+
+ def _parse(self):
+ self.relays = tuple([stem.control._parse_circ_entry(entry) for entry in str(self).split()[1:]])
+
+
+class ORConnEvent(Event):
+ """
+ Event that indicates a change in a relay connection. The 'endpoint' could be
+ any of several things including a...
+
+ * fingerprint
+ * nickname
+ * 'fingerprint=nickname' pair
+ * address:port
+
+ The derived 'endpoint_*' attributes are generally more useful.
+
+ The ORCONN event was one of the first Control Protocol V1 events and was
+ introduced in tor version 0.1.1.1-alpha. Its id attribute was added in
+ version 0.2.5.2-alpha.
+
+ .. versionchanged:: 1.2.0
+ Added the id attribute.
+
+ :var str id: connection identifier
+ :var str endpoint: relay that the event concerns
+ :var str endpoint_fingerprint: endpoint's finterprint if it was provided
+ :var str endpoint_nickname: endpoint's nickname if it was provided
+ :var str endpoint_address: endpoint's address if it was provided
+ :var int endpoint_port: endpoint's port if it was provided
+ :var stem.ORStatus status: state of the connection
+ :var stem.ORClosureReason reason: reason for the connection to be closed
+ :var int circ_count: number of established and pending circuits
+ """
+
+ _POSITIONAL_ARGS = ('endpoint', 'status')
+ _KEYWORD_ARGS = {
+ 'REASON': 'reason',
+ 'NCIRCS': 'circ_count',
+ 'ID': 'id',
+ }
+
+ def _parse(self):
+ self.endpoint_fingerprint = None
+ self.endpoint_nickname = None
+ self.endpoint_address = None
+ self.endpoint_port = None
+
+ try:
+ self.endpoint_fingerprint, self.endpoint_nickname = \
+ stem.control._parse_circ_entry(self.endpoint)
+ except stem.ProtocolError:
+ if ':' not in self.endpoint:
+ raise stem.ProtocolError("ORCONN endpoint is neither a relay nor 'address:port': %s" % self)
+
+ address, port = self.endpoint.split(':', 1)
+
+ if not connection.is_valid_port(port):
+ raise stem.ProtocolError("ORCONN's endpoint location's port is invalid: %s" % self)
+
+ self.endpoint_address = address
+ self.endpoint_port = int(port)
+
+ if self.circ_count is not None:
+ if not self.circ_count.isdigit():
+ raise stem.ProtocolError('ORCONN event got a non-numeric circuit count (%s): %s' % (self.circ_count, self))
+
+ self.circ_count = int(self.circ_count)
+
+ if self.id and not tor_tools.is_valid_connection_id(self.id):
+ raise stem.ProtocolError("Connection IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.id, self))
+
+ self._log_if_unrecognized('status', stem.ORStatus)
+ self._log_if_unrecognized('reason', stem.ORClosureReason)
+
+
+class SignalEvent(Event):
+ """
+ Event that indicates that tor has received and acted upon a signal being sent
+ to the process. As of tor version 0.2.4.6 the only signals conveyed by this
+ event are...
+
+ * RELOAD
+ * DUMP
+ * DEBUG
+ * NEWNYM
+ * CLEARDNSCACHE
+
+ The SIGNAL event was introduced in tor version 0.2.3.1-alpha.
+
+ :var stem.Signal signal: signal that tor received
+ """
+
+ _POSITIONAL_ARGS = ('signal',)
+ _VERSION_ADDED = stem.version.Requirement.EVENT_SIGNAL
+
+ def _parse(self):
+ # log if we recieved an unrecognized signal
+ expected_signals = (
+ stem.Signal.RELOAD,
+ stem.Signal.DUMP,
+ stem.Signal.DEBUG,
+ stem.Signal.NEWNYM,
+ stem.Signal.CLEARDNSCACHE,
+ )
+
+ self._log_if_unrecognized('signal', expected_signals)
+
+
+class StatusEvent(Event):
+ """
+ Notification of a change in tor's state. These are generally triggered for
+ the same sort of things as log messages of the NOTICE level or higher.
+ However, unlike :class:`~stem.response.events.LogEvent` these contain well
+ formed data.
+
+ The STATUS_GENERAL, STATUS_CLIENT, STATUS_SERVER events were introduced
+ in tor version 0.1.2.3-alpha.
+
+ :var stem.StatusType status_type: category of the status event
+ :var stem.Runlevel runlevel: runlevel of the logged message
+ :var str action: activity that caused this message
+ :var dict arguments: attributes about the event
+ """
+
+ _POSITIONAL_ARGS = ('runlevel', 'action')
+ _VERSION_ADDED = stem.version.Requirement.EVENT_STATUS
+
+ def _parse(self):
+ if self.type == 'STATUS_GENERAL':
+ self.status_type = stem.StatusType.GENERAL
+ elif self.type == 'STATUS_CLIENT':
+ self.status_type = stem.StatusType.CLIENT
+ elif self.type == 'STATUS_SERVER':
+ self.status_type = stem.StatusType.SERVER
+ else:
+ raise ValueError("BUG: Unrecognized status type (%s), likely an EVENT_TYPE_TO_CLASS addition without revising how 'status_type' is assigned." % self.type)
+
+ # Just an alias for our parent class' keyword_args since that already
+ # parses these for us. Unlike our other event types Tor commonly supplies
+ # arbitrary key/value pairs for these, so making an alias here to better
+ # draw attention that the StatusEvent will likely have them.
+
+ self.arguments = self.keyword_args
+
+ self._log_if_unrecognized('runlevel', stem.Runlevel)
+
+
+class StreamEvent(Event):
+ """
+ Event that indicates that a stream has changed.
+
+ The STREAM event was one of the first Control Protocol V1 events and was
+ introduced in tor version 0.1.1.1-alpha.
+
+ :var str id: stream identifier
+ :var stem.StreamStatus status: reported status for the stream
+ :var str circ_id: circuit that the stream is attached to, this is **None** of
+ the stream is unattached
+ :var str target: destination of the stream
+ :var str target_address: destination address (ip, hostname, or '(Tor_internal)')
+ :var int target_port: destination port
+ :var stem.StreamClosureReason reason: reason for the stream to be closed
+ :var stem.StreamClosureReason remote_reason: remote side's reason for the stream to be closed
+ :var stem.StreamSource source: origin of the REMAP request
+ :var str source_addr: requester of the connection
+ :var str source_address: requester address (ip or hostname)
+ :var int source_port: requester port
+ :var stem.StreamPurpose purpose: purpose for the stream
+ """
+
+ _POSITIONAL_ARGS = ('id', 'status', 'circ_id', 'target')
+ _KEYWORD_ARGS = {
+ 'REASON': 'reason',
+ 'REMOTE_REASON': 'remote_reason',
+ 'SOURCE': 'source',
+ 'SOURCE_ADDR': 'source_addr',
+ 'PURPOSE': 'purpose',
+ }
+
+ def _parse(self):
+ if self.target is None:
+ raise stem.ProtocolError("STREAM event didn't have a target: %s" % self)
+ else:
+ if ':' not in self.target:
+ raise stem.ProtocolError("Target location must be of the form 'address:port': %s" % self)
+
+ address, port = self.target.rsplit(':', 1)
+
+ if not connection.is_valid_port(port, allow_zero = True):
+ raise stem.ProtocolError("Target location's port is invalid: %s" % self)
+
+ self.target_address = address
+ self.target_port = int(port)
+
+ if self.source_addr is None:
+ self.source_address = None
+ self.source_port = None
+ else:
+ if ':' not in self.source_addr:
+ raise stem.ProtocolError("Source location must be of the form 'address:port': %s" % self)
+
+ address, port = self.source_addr.split(':', 1)
+
+ if not connection.is_valid_port(port, allow_zero = True):
+ raise stem.ProtocolError("Source location's port is invalid: %s" % self)
+
+ self.source_address = address
+ self.source_port = int(port)
+
+ # spec specifies a circ_id of zero if the stream is unattached
+
+ if self.circ_id == '0':
+ self.circ_id = None
+
+ self._log_if_unrecognized('reason', stem.StreamClosureReason)
+ self._log_if_unrecognized('remote_reason', stem.StreamClosureReason)
+ self._log_if_unrecognized('purpose', stem.StreamPurpose)
+
+
+class StreamBwEvent(Event):
+ """
+ Event (emitted approximately every second) with the bytes sent and received
+ by the application since the last such event on this stream.
+
+ The STREAM_BW event was introduced in tor version 0.1.2.8-beta.
+
+ :var str id: stream identifier
+ :var long written: bytes sent by the application
+ :var long read: bytes received by the application
+ """
+
+ _POSITIONAL_ARGS = ('id', 'written', 'read')
+ _VERSION_ADDED = stem.version.Requirement.EVENT_STREAM_BW
+
+ def _parse(self):
+ if not tor_tools.is_valid_stream_id(self.id):
+ raise stem.ProtocolError("Stream IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.id, self))
+ elif not self.written:
+ raise stem.ProtocolError('STREAM_BW event is missing its written value')
+ elif not self.read:
+ raise stem.ProtocolError('STREAM_BW event is missing its read value')
+ elif not self.read.isdigit() or not self.written.isdigit():
+ raise stem.ProtocolError("A STREAM_BW event's bytes sent and received should be a positive numeric value, received: %s" % self)
+
+ self.read = int_type(self.read)
+ self.written = int_type(self.written)
+
+
+class TransportLaunchedEvent(Event):
+ """
+ Event triggered when a pluggable transport is launched.
+
+ The TRANSPORT_LAUNCHED event was introduced in tor version 0.2.5.0-alpha.
+
+ .. versionadded:: 1.1.0
+
+ :var str type: 'server' or 'client'
+ :var str name: name of the pluggable transport
+ :var str address: IPv4 or IPv6 address where the transport is listening for
+ connections
+ :var int port: port where the transport is listening for connections
+ """
+
+ _POSITIONAL_ARGS = ('type', 'name', 'address', 'port')
+ _VERSION_ADDED = stem.version.Requirement.EVENT_TRANSPORT_LAUNCHED
+
+ def _parse(self):
+ if self.type not in ('server', 'client'):
+ raise stem.ProtocolError("Transport type should either be 'server' or 'client': %s" % self)
+
+ if not connection.is_valid_ipv4_address(self.address) and \
+ not connection.is_valid_ipv6_address(self.address):
+ raise stem.ProtocolError("Transport address isn't a valid IPv4 or IPv6 address: %s" % self)
+
+ if not connection.is_valid_port(self.port):
+ raise stem.ProtocolError('Transport port is invalid: %s' % self)
+
+ self.port = int(self.port)
+
+
+class ConnectionBandwidthEvent(Event):
+ """
+ Event emitted every second with the bytes sent and received by tor on a
+ per-connection basis.
+
+ The CONN_BW event was introduced in tor version 0.2.5.2-alpha.
+
+ .. versionadded:: 1.2.0
+
+ :var str id: connection identifier
+ :var stem.ConnectionType type: connection type
+ :var long read: bytes received by tor that second
+ :var long written: bytes sent by tor that second
+ """
+
+ _KEYWORD_ARGS = {
+ 'ID': 'id',
+ 'TYPE': 'type',
+ 'READ': 'read',
+ 'WRITTEN': 'written',
+ }
+
+ _VERSION_ADDED = stem.version.Requirement.EVENT_CONN_BW
+
+ def _parse(self):
+ if not self.id:
+ raise stem.ProtocolError('CONN_BW event is missing its id')
+ elif not self.type:
+ raise stem.ProtocolError('CONN_BW event is missing its type')
+ elif not self.read:
+ raise stem.ProtocolError('CONN_BW event is missing its read value')
+ elif not self.written:
+ raise stem.ProtocolError('CONN_BW event is missing its written value')
+ elif not self.read.isdigit() or not self.written.isdigit():
+ raise stem.ProtocolError("A CONN_BW event's bytes sent and received should be a positive numeric value, received: %s" % self)
+ elif not tor_tools.is_valid_connection_id(self.id):
+ raise stem.ProtocolError("Connection IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.id, self))
+
+ self.read = int_type(self.read)
+ self.written = int_type(self.written)
+
+ self._log_if_unrecognized('type', stem.ConnectionType)
+
+
+class CircuitBandwidthEvent(Event):
+ """
+ Event emitted every second with the bytes sent and received by tor on a
+ per-circuit basis.
+
+ The CIRC_BW event was introduced in tor version 0.2.5.2-alpha.
+
+ .. versionadded:: 1.2.0
+
+ :var str id: circuit identifier
+ :var long read: bytes received by tor that second
+ :var long written: bytes sent by tor that second
+ """
+
+ _KEYWORD_ARGS = {
+ 'ID': 'id',
+ 'READ': 'read',
+ 'WRITTEN': 'written',
+ }
+
+ _VERSION_ADDED = stem.version.Requirement.EVENT_CIRC_BW
+
+ def _parse(self):
+ if not self.id:
+ raise stem.ProtocolError('CIRC_BW event is missing its id')
+ elif not self.read:
+ raise stem.ProtocolError('CIRC_BW event is missing its read value')
+ elif not self.written:
+ raise stem.ProtocolError('CIRC_BW event is missing its written value')
+ elif not self.read.isdigit() or not self.written.isdigit():
+ raise stem.ProtocolError("A CIRC_BW event's bytes sent and received should be a positive numeric value, received: %s" % self)
+ elif not tor_tools.is_valid_circuit_id(self.id):
+ raise stem.ProtocolError("Circuit IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.id, self))
+
+ self.read = int_type(self.read)
+ self.written = int_type(self.written)
+
+
+class CellStatsEvent(Event):
+ """
+ Event emitted every second with a count of the number of cells types broken
+ down by the circuit. **These events are only emitted if TestingTorNetwork is
+ set.**
+
+ The CELL_STATS event was introduced in tor version 0.2.5.2-alpha.
+
+ .. versionadded:: 1.2.0
+
+ :var str id: circuit identifier
+ :var str inbound_queue: inbound queue identifier
+ :var str inbound_connection: inbound connection identifier
+ :var dict inbound_added: mapping of added inbound cell types to their count
+ :var dict inbound_removed: mapping of removed inbound cell types to their count
+ :var dict inbound_time: mapping of inbound cell types to the time they took to write in milliseconds
+ :var str outbound_queue: outbound queue identifier
+ :var str outbound_connection: outbound connection identifier
+ :var dict outbound_added: mapping of added outbound cell types to their count
+ :var dict outbound_removed: mapping of removed outbound cell types to their count
+ :var dict outbound_time: mapping of outbound cell types to the time they took to write in milliseconds
+ """
+
+ _KEYWORD_ARGS = {
+ 'ID': 'id',
+ 'InboundQueue': 'inbound_queue',
+ 'InboundConn': 'inbound_connection',
+ 'InboundAdded': 'inbound_added',
+ 'InboundRemoved': 'inbound_removed',
+ 'InboundTime': 'inbound_time',
+ 'OutboundQueue': 'outbound_queue',
+ 'OutboundConn': 'outbound_connection',
+ 'OutboundAdded': 'outbound_added',
+ 'OutboundRemoved': 'outbound_removed',
+ 'OutboundTime': 'outbound_time',
+ }
+
+ _VERSION_ADDED = stem.version.Requirement.EVENT_CELL_STATS
+
+ def _parse(self):
+ if self.id and not tor_tools.is_valid_circuit_id(self.id):
+ raise stem.ProtocolError("Circuit IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.id, self))
+ elif self.inbound_queue and not tor_tools.is_valid_circuit_id(self.inbound_queue):
+ raise stem.ProtocolError("Queue IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.inbound_queue, self))
+ elif self.inbound_connection and not tor_tools.is_valid_connection_id(self.inbound_connection):
+ raise stem.ProtocolError("Connection IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.inbound_connection, self))
+ elif self.outbound_queue and not tor_tools.is_valid_circuit_id(self.outbound_queue):
+ raise stem.ProtocolError("Queue IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.outbound_queue, self))
+ elif self.outbound_connection and not tor_tools.is_valid_connection_id(self.outbound_connection):
+ raise stem.ProtocolError("Connection IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.outbound_connection, self))
+
+ self.inbound_added = _parse_cell_type_mapping(self.inbound_added)
+ self.inbound_removed = _parse_cell_type_mapping(self.inbound_removed)
+ self.inbound_time = _parse_cell_type_mapping(self.inbound_time)
+ self.outbound_added = _parse_cell_type_mapping(self.outbound_added)
+ self.outbound_removed = _parse_cell_type_mapping(self.outbound_removed)
+ self.outbound_time = _parse_cell_type_mapping(self.outbound_time)
+
+
+class TokenBucketEmptyEvent(Event):
+ """
+ Event emitted when refilling an empty token bucket. **These events are only
+ emitted if TestingTorNetwork is set.**
+
+ The TB_EMPTY event was introduced in tor version 0.2.5.2-alpha.
+
+ .. versionadded:: 1.2.0
+
+ :var stem.TokenBucket bucket: bucket being refilled
+ :var str id: connection identifier
+ :var int read: time in milliseconds since the read bucket was last refilled
+ :var int written: time in milliseconds since the write bucket was last refilled
+ :var int last_refill: time in milliseconds the bucket has been empty since last refilled
+ """
+
+ _POSITIONAL_ARGS = ('bucket',)
+ _KEYWORD_ARGS = {
+ 'ID': 'id',
+ 'READ': 'read',
+ 'WRITTEN': 'written',
+ 'LAST': 'last_refill',
+ }
+
+ _VERSION_ADDED = stem.version.Requirement.EVENT_TB_EMPTY
+
+ def _parse(self):
+ if self.id and not tor_tools.is_valid_connection_id(self.id):
+ raise stem.ProtocolError("Connection IDs must be one to sixteen alphanumeric characters, got '%s': %s" % (self.id, self))
+ elif not self.read.isdigit():
+ raise stem.ProtocolError("A TB_EMPTY's READ value should be a positive numeric value, received: %s" % self)
+ elif not self.written.isdigit():
+ raise stem.ProtocolError("A TB_EMPTY's WRITTEN value should be a positive numeric value, received: %s" % self)
+ elif not self.last_refill.isdigit():
+ raise stem.ProtocolError("A TB_EMPTY's LAST value should be a positive numeric value, received: %s" % self)
+
+ self.read = int(self.read)
+ self.written = int(self.written)
+ self.last_refill = int(self.last_refill)
+
+ self._log_if_unrecognized('bucket', stem.TokenBucket)
+
+
+def _parse_cell_type_mapping(mapping):
+ """
+ Parses a mapping of the form...
+
+ key1:value1,key2:value2...
+
+ ... in which keys are strings and values are integers.
+
+ :param str mapping: value to be parsed
+
+ :returns: dict of **str => int** mappings
+
+ :rasies: **stem.ProtocolError** if unable to parse the mapping
+ """
+
+ if mapping is None:
+ return None
+
+ results = {}
+
+ for entry in mapping.split(','):
+ if ':' not in entry:
+ raise stem.ProtocolError("Mappings are expected to be of the form 'key:value', got '%s': %s" % (entry, mapping))
+
+ key, value = entry.split(':', 1)
+
+ if not CELL_TYPE.match(key):
+ raise stem.ProtocolError("Key had invalid characters, got '%s': %s" % (key, mapping))
+ elif not value.isdigit():
+ raise stem.ProtocolError("Values should just be integers, got '%s': %s" % (value, mapping))
+
+ results[key] = int(value)
+
+ return results
+
+
+EVENT_TYPE_TO_CLASS = {
+ 'ADDRMAP': AddrMapEvent,
+ 'AUTHDIR_NEWDESCS': AuthDirNewDescEvent,
+ 'BUILDTIMEOUT_SET': BuildTimeoutSetEvent,
+ 'BW': BandwidthEvent,
+ 'CELL_STATS': CellStatsEvent,
+ 'CIRC': CircuitEvent,
+ 'CIRC_BW': CircuitBandwidthEvent,
+ 'CIRC_MINOR': CircMinorEvent,
+ 'CLIENTS_SEEN': ClientsSeenEvent,
+ 'CONF_CHANGED': ConfChangedEvent,
+ 'CONN_BW': ConnectionBandwidthEvent,
+ 'DEBUG': LogEvent,
+ 'DESCCHANGED': DescChangedEvent,
+ 'ERR': LogEvent,
+ 'GUARD': GuardEvent,
+ 'HS_DESC': HSDescEvent,
+ 'HS_DESC_CONTENT': HSDescContentEvent,
+ 'INFO': LogEvent,
+ 'NEWCONSENSUS': NewConsensusEvent,
+ 'NEWDESC': NewDescEvent,
+ 'NOTICE': LogEvent,
+ 'NS': NetworkStatusEvent,
+ 'ORCONN': ORConnEvent,
+ 'SIGNAL': SignalEvent,
+ 'STATUS_CLIENT': StatusEvent,
+ 'STATUS_GENERAL': StatusEvent,
+ 'STATUS_SERVER': StatusEvent,
+ 'STREAM': StreamEvent,
+ 'STREAM_BW': StreamBwEvent,
+ 'TB_EMPTY': TokenBucketEmptyEvent,
+ 'TRANSPORT_LAUNCHED': TransportLaunchedEvent,
+ 'WARN': LogEvent,
+
+ # accounting for a bug in tor 0.2.0.22
+ 'STATUS_SEVER': StatusEvent,
+}
diff --git a/ext/stem/response/getconf.py b/ext/stem/response/getconf.py
new file mode 100644
index 00000000..ce145532
--- /dev/null
+++ b/ext/stem/response/getconf.py
@@ -0,0 +1,55 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+import stem.response
+import stem.socket
+
+
+class GetConfResponse(stem.response.ControlMessage):
+ """
+ Reply for a GETCONF query.
+
+ Note that configuration parameters won't match what we queried for if it's one
+ of the special mapping options (ex. 'HiddenServiceOptions').
+
+ :var dict entries: mapping between the config parameter (**str**) and their
+ values (**list** of **str**)
+ """
+
+ def _parse_message(self):
+ # Example:
+ # 250-CookieAuthentication=0
+ # 250-ControlPort=9100
+ # 250-DataDirectory=/home/neena/.tor
+ # 250 DirPort
+
+ self.entries = {}
+ remaining_lines = list(self)
+
+ if self.content() == [('250', ' ', 'OK')]:
+ return
+
+ if not self.is_ok():
+ unrecognized_keywords = []
+ for code, _, line in self.content():
+ if code == '552' and line.startswith('Unrecognized configuration key "') and line.endswith('"'):
+ unrecognized_keywords.append(line[32:-1])
+
+ if unrecognized_keywords:
+ raise stem.InvalidArguments('552', 'GETCONF request contained unrecognized keywords: %s' % ', '.join(unrecognized_keywords), unrecognized_keywords)
+ else:
+ raise stem.ProtocolError('GETCONF response contained a non-OK status code:\n%s' % self)
+
+ while remaining_lines:
+ line = remaining_lines.pop(0)
+
+ if line.is_next_mapping():
+ key, value = line.split('=', 1)
+ else:
+ key, value = (line.pop(), None)
+
+ if key not in self.entries:
+ self.entries[key] = []
+
+ if value is not None:
+ self.entries[key].append(value)
diff --git a/ext/stem/response/getinfo.py b/ext/stem/response/getinfo.py
new file mode 100644
index 00000000..07985934
--- /dev/null
+++ b/ext/stem/response/getinfo.py
@@ -0,0 +1,78 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+import stem.response
+import stem.socket
+
+
+class GetInfoResponse(stem.response.ControlMessage):
+ """
+ Reply for a GETINFO query.
+
+ :var dict entries: mapping between the queried options and their bytes values
+ """
+
+ def _parse_message(self):
+ # Example:
+ # 250-version=0.2.3.11-alpha-dev (git-ef0bc7f8f26a917c)
+ # 250+config-text=
+ # ControlPort 9051
+ # DataDirectory /home/atagar/.tor
+ # ExitPolicy reject *:*
+ # Log notice stdout
+ # Nickname Unnamed
+ # ORPort 9050
+ # .
+ # 250 OK
+
+ self.entries = {}
+ remaining_lines = [content for (code, div, content) in self.content(get_bytes = True)]
+
+ if not self.is_ok() or not remaining_lines.pop() == b'OK':
+ unrecognized_keywords = []
+ for code, _, line in self.content():
+ if code == '552' and line.startswith('Unrecognized key "') and line.endswith('"'):
+ unrecognized_keywords.append(line[18:-1])
+
+ if unrecognized_keywords:
+ raise stem.InvalidArguments('552', 'GETINFO request contained unrecognized keywords: %s\n' % ', '.join(unrecognized_keywords), unrecognized_keywords)
+ else:
+ raise stem.ProtocolError("GETINFO response didn't have an OK status:\n%s" % self)
+
+ while remaining_lines:
+ try:
+ key, value = remaining_lines.pop(0).split(b'=', 1)
+ except ValueError:
+ raise stem.ProtocolError('GETINFO replies should only contain parameter=value mappings:\n%s' % self)
+
+ if stem.prereq.is_python_3():
+ key = stem.util.str_tools._to_unicode(key)
+
+ # if the value is a multiline value then it *must* be of the form
+ # '<key>=\n<value>'
+
+ if b'\n' in value:
+ if not value.startswith(b'\n'):
+ raise stem.ProtocolError("GETINFO response contained a multi-line value that didn't start with a newline:\n%s" % self)
+
+ value = value[1:]
+
+ self.entries[key] = value
+
+ def _assert_matches(self, params):
+ """
+ Checks if we match a given set of parameters, and raise a ProtocolError if not.
+
+ :param set params: parameters to assert that we contain
+
+ :raises:
+ * :class:`stem.ProtocolError` if parameters don't match this response
+ """
+
+ reply_params = set(self.entries.keys())
+
+ if params != reply_params:
+ requested_label = ', '.join(params)
+ reply_label = ', '.join(reply_params)
+
+ raise stem.ProtocolError("GETINFO reply doesn't match the parameters that we requested. Queried '%s' but got '%s'." % (requested_label, reply_label))
diff --git a/ext/stem/response/mapaddress.py b/ext/stem/response/mapaddress.py
new file mode 100644
index 00000000..5d2b418e
--- /dev/null
+++ b/ext/stem/response/mapaddress.py
@@ -0,0 +1,42 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+import stem.response
+import stem.socket
+
+
+class MapAddressResponse(stem.response.ControlMessage):
+ """
+ Reply for a MAPADDRESS query.
+ Doesn't raise an exception unless no addresses were mapped successfully.
+
+ :var dict entries: mapping between the original and replacement addresses
+
+ :raises:
+ * :class:`stem.OperationFailed` if Tor was unable to satisfy the request
+ * :class:`stem.InvalidRequest` if the addresses provided were invalid
+ """
+
+ def _parse_message(self):
+ # Example:
+ # 250-127.192.10.10=torproject.org
+ # 250 1.2.3.4=tor.freehaven.net
+
+ if not self.is_ok():
+ for code, _, message in self.content():
+ if code == '512':
+ raise stem.InvalidRequest(code, message)
+ elif code == '451':
+ raise stem.OperationFailed(code, message)
+ else:
+ raise stem.ProtocolError('MAPADDRESS returned unexpected response code: %s', code)
+
+ self.entries = {}
+
+ for code, _, message in self.content():
+ if code == '250':
+ try:
+ key, value = message.split('=', 1)
+ self.entries[key] = value
+ except ValueError:
+ raise stem.ProtocolError(None, "MAPADDRESS returned '%s', which isn't a mapping" % message)
diff --git a/ext/stem/response/protocolinfo.py b/ext/stem/response/protocolinfo.py
new file mode 100644
index 00000000..97201de5
--- /dev/null
+++ b/ext/stem/response/protocolinfo.py
@@ -0,0 +1,122 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+import stem.response
+import stem.socket
+import stem.version
+
+from stem.connection import AuthMethod
+from stem.util import log
+
+
+class ProtocolInfoResponse(stem.response.ControlMessage):
+ """
+ Version one PROTOCOLINFO query response.
+
+ The protocol_version is the only mandatory data for a valid PROTOCOLINFO
+ response, so all other values are None if undefined or empty if a collection.
+
+ :var int protocol_version: protocol version of the response
+ :var stem.version.Version tor_version: version of the tor process
+ :var tuple auth_methods: :data:`stem.connection.AuthMethod` types that tor will accept
+ :var tuple unknown_auth_methods: strings of unrecognized auth methods
+ :var str cookie_path: path of tor's authentication cookie
+ """
+
+ def _parse_message(self):
+ # Example:
+ # 250-PROTOCOLINFO 1
+ # 250-AUTH METHODS=COOKIE COOKIEFILE="/home/atagar/.tor/control_auth_cookie"
+ # 250-VERSION Tor="0.2.1.30"
+ # 250 OK
+
+ self.protocol_version = None
+ self.tor_version = None
+ self.auth_methods = ()
+ self.unknown_auth_methods = ()
+ self.cookie_path = None
+
+ auth_methods, unknown_auth_methods = [], []
+ remaining_lines = list(self)
+
+ if not self.is_ok() or not remaining_lines.pop() == 'OK':
+ raise stem.ProtocolError("PROTOCOLINFO response didn't have an OK status:\n%s" % self)
+
+ # sanity check that we're a PROTOCOLINFO response
+ if not remaining_lines[0].startswith('PROTOCOLINFO'):
+ raise stem.ProtocolError('Message is not a PROTOCOLINFO response:\n%s' % self)
+
+ while remaining_lines:
+ line = remaining_lines.pop(0)
+ line_type = line.pop()
+
+ if line_type == 'PROTOCOLINFO':
+ # Line format:
+ # FirstLine = "PROTOCOLINFO" SP PIVERSION CRLF
+ # PIVERSION = 1*DIGIT
+
+ if line.is_empty():
+ raise stem.ProtocolError("PROTOCOLINFO response's initial line is missing the protocol version: %s" % line)
+
+ try:
+ self.protocol_version = int(line.pop())
+ except ValueError:
+ raise stem.ProtocolError('PROTOCOLINFO response version is non-numeric: %s' % line)
+
+ # The piversion really should be '1' but, according to the spec, tor
+ # does not necessarily need to provide the PROTOCOLINFO version that we
+ # requested. Log if it's something we aren't expecting but still make
+ # an effort to parse like a v1 response.
+
+ if self.protocol_version != 1:
+ log.info("We made a PROTOCOLINFO version 1 query but got a version %i response instead. We'll still try to use it, but this may cause problems." % self.protocol_version)
+ elif line_type == 'AUTH':
+ # Line format:
+ # AuthLine = "250-AUTH" SP "METHODS=" AuthMethod *("," AuthMethod)
+ # *(SP "COOKIEFILE=" AuthCookieFile) CRLF
+ # AuthMethod = "NULL" / "HASHEDPASSWORD" / "COOKIE"
+ # AuthCookieFile = QuotedString
+
+ # parse AuthMethod mapping
+ if not line.is_next_mapping('METHODS'):
+ raise stem.ProtocolError("PROTOCOLINFO response's AUTH line is missing its mandatory 'METHODS' mapping: %s" % line)
+
+ for method in line.pop_mapping()[1].split(','):
+ if method == 'NULL':
+ auth_methods.append(AuthMethod.NONE)
+ elif method == 'HASHEDPASSWORD':
+ auth_methods.append(AuthMethod.PASSWORD)
+ elif method == 'COOKIE':
+ auth_methods.append(AuthMethod.COOKIE)
+ elif method == 'SAFECOOKIE':
+ auth_methods.append(AuthMethod.SAFECOOKIE)
+ else:
+ unknown_auth_methods.append(method)
+ message_id = 'stem.response.protocolinfo.unknown_auth_%s' % method
+ log.log_once(message_id, log.INFO, "PROTOCOLINFO response included a type of authentication that we don't recognize: %s" % method)
+
+ # our auth_methods should have a single AuthMethod.UNKNOWN entry if
+ # any unknown authentication methods exist
+ if AuthMethod.UNKNOWN not in auth_methods:
+ auth_methods.append(AuthMethod.UNKNOWN)
+
+ # parse optional COOKIEFILE mapping (quoted and can have escapes)
+ if line.is_next_mapping('COOKIEFILE', True, True):
+ self.cookie_path = line.pop_mapping(True, True)[1]
+ elif line_type == 'VERSION':
+ # Line format:
+ # VersionLine = "250-VERSION" SP "Tor=" TorVersion OptArguments CRLF
+ # TorVersion = QuotedString
+
+ if not line.is_next_mapping('Tor', True):
+ raise stem.ProtocolError("PROTOCOLINFO response's VERSION line is missing its mandatory tor version mapping: %s" % line)
+
+ try:
+ self.tor_version = stem.version.Version(line.pop_mapping(True)[1])
+ except ValueError as exc:
+ raise stem.ProtocolError(exc)
+ else:
+ log.debug("Unrecognized PROTOCOLINFO line type '%s', ignoring it: %s" % (line_type, line))
+
+ self.auth_methods = tuple(auth_methods)
+ self.unknown_auth_methods = tuple(unknown_auth_methods)
diff --git a/ext/stem/socket.py b/ext/stem/socket.py
new file mode 100644
index 00000000..8024098e
--- /dev/null
+++ b/ext/stem/socket.py
@@ -0,0 +1,663 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Supports communication with sockets speaking the Tor control protocol. This
+allows us to send messages as basic strings, and receive responses as
+:class:`~stem.response.ControlMessage` instances.
+
+**This module only consists of low level components, and is not intended for
+users.** See our `tutorials <../tutorials.html>`_ and `Control Module
+<control.html>`_ if you're new to Stem and looking to get started.
+
+With that aside, these can still be used for raw socket communication with
+Tor...
+
+::
+
+ import stem
+ import stem.connection
+ import stem.socket
+
+ if __name__ == '__main__':
+ try:
+ control_socket = stem.socket.ControlPort(port = 9051)
+ stem.connection.authenticate(control_socket)
+ except stem.SocketError as exc:
+ print 'Unable to connect to tor on port 9051: %s' % exc
+ sys.exit(1)
+ except stem.connection.AuthenticationFailure as exc:
+ print 'Unable to authenticate: %s' % exc
+ sys.exit(1)
+
+ print "Issuing 'GETINFO version' query...\\n"
+ control_socket.send('GETINFO version')
+ print control_socket.recv()
+
+::
+
+ % python example.py
+ Issuing 'GETINFO version' query...
+
+ version=0.2.4.10-alpha-dev (git-8be6058d8f31e578)
+ OK
+
+**Module Overview:**
+
+::
+
+ ControlSocket - Socket wrapper that speaks the tor control protocol.
+ |- ControlPort - Control connection via a port.
+ | |- get_address - provides the ip address of our socket
+ | +- get_port - provides the port of our socket
+ |
+ |- ControlSocketFile - Control connection via a local file socket.
+ | +- get_socket_path - provides the path of the socket we connect to
+ |
+ |- send - sends a message to the socket
+ |- recv - receives a ControlMessage from the socket
+ |- is_alive - reports if the socket is known to be closed
+ |- is_localhost - returns if the socket is for the local system or not
+ |- connect - connects a new socket
+ |- close - shuts down the socket
+ +- __enter__ / __exit__ - manages socket connection
+
+ send_message - Writes a message to a control socket.
+ recv_message - Reads a ControlMessage from a control socket.
+ send_formatting - Performs the formatting expected from sent messages.
+"""
+
+from __future__ import absolute_import
+
+import re
+import socket
+import threading
+import time
+
+import stem.prereq
+import stem.response
+import stem.util.str_tools
+
+from stem.util import log
+
+
+class ControlSocket(object):
+ """
+ Wrapper for a socket connection that speaks the Tor control protocol. To the
+ better part this transparently handles the formatting for sending and
+ receiving complete messages. All methods are thread safe.
+
+ Callers should not instantiate this class directly, but rather use subclasses
+ which are expected to implement the **_make_socket()** method.
+ """
+
+ def __init__(self):
+ self._socket, self._socket_file = None, None
+ self._is_alive = False
+ self._connection_time = 0.0 # time when we last connected or disconnected
+
+ # Tracks sending and receiving separately. This should be safe, and doing
+ # so prevents deadlock where we block writes because we're waiting to read
+ # a message that isn't coming.
+
+ self._send_lock = threading.RLock()
+ self._recv_lock = threading.RLock()
+
+ def send(self, message, raw = False):
+ """
+ Formats and sends a message to the control socket. For more information see
+ the :func:`~stem.socket.send_message` function.
+
+ :param str message: message to be formatted and sent to the socket
+ :param bool raw: leaves the message formatting untouched, passing it to the socket as-is
+
+ :raises:
+ * :class:`stem.SocketError` if a problem arises in using the socket
+ * :class:`stem.SocketClosed` if the socket is known to be shut down
+ """
+
+ with self._send_lock:
+ try:
+ if not self.is_alive():
+ raise stem.SocketClosed()
+
+ send_message(self._socket_file, message, raw)
+ except stem.SocketClosed as exc:
+ # if send_message raises a SocketClosed then we should properly shut
+ # everything down
+
+ if self.is_alive():
+ self.close()
+
+ raise exc
+
+ def recv(self):
+ """
+ Receives a message from the control socket, blocking until we've received
+ one. For more information see the :func:`~stem.socket.recv_message` function.
+
+ :returns: :class:`~stem.response.ControlMessage` for the message received
+
+ :raises:
+ * :class:`stem.ProtocolError` the content from the socket is malformed
+ * :class:`stem.SocketClosed` if the socket closes before we receive a complete message
+ """
+
+ with self._recv_lock:
+ try:
+ # makes a temporary reference to the _socket_file because connect()
+ # and close() may set or unset it
+
+ socket_file = self._socket_file
+
+ if not socket_file:
+ raise stem.SocketClosed()
+
+ return recv_message(socket_file)
+ except stem.SocketClosed as exc:
+ # If recv_message raises a SocketClosed then we should properly shut
+ # everything down. However, there's a couple cases where this will
+ # cause deadlock...
+ #
+ # * this socketClosed was *caused by* a close() call, which is joining
+ # on our thread
+ #
+ # * a send() call that's currently in flight is about to call close(),
+ # also attempting to join on us
+ #
+ # To resolve this we make a non-blocking call to acquire the send lock.
+ # If we get it then great, we can close safely. If not then one of the
+ # above are in progress and we leave the close to them.
+
+ if self.is_alive():
+ if self._send_lock.acquire(False):
+ self.close()
+ self._send_lock.release()
+
+ raise exc
+
+ def is_alive(self):
+ """
+ Checks if the socket is known to be closed. We won't be aware if it is
+ until we either use it or have explicitily shut it down.
+
+ In practice a socket derived from a port knows about its disconnection
+ after a failed :func:`~stem.socket.ControlSocket.recv` call. Socket file
+ derived connections know after either a
+ :func:`~stem.socket.ControlSocket.send` or
+ :func:`~stem.socket.ControlSocket.recv`.
+
+ This means that to have reliable detection for when we're disconnected
+ you need to continually pull from the socket (which is part of what the
+ :class:`~stem.control.BaseController` does).
+
+ :returns: **bool** that's **True** if our socket is connected and **False** otherwise
+ """
+
+ return self._is_alive
+
+ def is_localhost(self):
+ """
+ Returns if the connection is for the local system or not.
+
+ :returns: **bool** that's **True** if the connection is for the local host and **False** otherwise
+ """
+
+ return False
+
+ def connection_time(self):
+ """
+ Provides the unix timestamp for when our socket was either connected or
+ disconnected. That is to say, the time we connected if we're currently
+ connected and the time we disconnected if we're not connected.
+
+ .. versionadded:: 1.3.0
+
+ :returns: **float** for when we last connected or disconnected, zero if
+ we've never connected
+ """
+
+ return self._connection_time
+
+ def connect(self):
+ """
+ Connects to a new socket, closing our previous one if we're already
+ attached.
+
+ :raises: :class:`stem.SocketError` if unable to make a socket
+ """
+
+ with self._send_lock:
+ # Closes the socket if we're currently attached to one. Once we're no
+ # longer alive it'll be safe to acquire the recv lock because recv()
+ # calls no longer block (raising SocketClosed instead).
+
+ if self.is_alive():
+ self.close()
+
+ with self._recv_lock:
+ self._socket = self._make_socket()
+ self._socket_file = self._socket.makefile(mode = 'rwb')
+ self._is_alive = True
+ self._connection_time = time.time()
+
+ # It's possible for this to have a transient failure...
+ # SocketError: [Errno 4] Interrupted system call
+ #
+ # It's safe to retry, so give it another try if it fails.
+
+ try:
+ self._connect()
+ except stem.SocketError:
+ self._connect() # single retry
+
+ def close(self):
+ """
+ Shuts down the socket. If it's already closed then this is a no-op.
+ """
+
+ with self._send_lock:
+ # Function is idempotent with one exception: we notify _close() if this
+ # is causing our is_alive() state to change.
+
+ is_change = self.is_alive()
+
+ if self._socket:
+ # if we haven't yet established a connection then this raises an error
+ # socket.error: [Errno 107] Transport endpoint is not connected
+
+ try:
+ self._socket.shutdown(socket.SHUT_RDWR)
+ except socket.error:
+ pass
+
+ # Suppressing unexpected exceptions from close. For instance, if the
+ # socket's file has already been closed then with python 2.7 that raises
+ # with...
+ # error: [Errno 32] Broken pipe
+
+ try:
+ self._socket.close()
+ except:
+ pass
+
+ if self._socket_file:
+ try:
+ self._socket_file.close()
+ except:
+ pass
+
+ self._socket = None
+ self._socket_file = None
+ self._is_alive = False
+ self._connection_time = time.time()
+
+ if is_change:
+ self._close()
+
+ def _get_send_lock(self):
+ """
+ The send lock is useful to classes that interact with us at a deep level
+ because it's used to lock :func:`stem.socket.ControlSocket.connect` /
+ :func:`stem.socket.ControlSocket.close`, and by extension our
+ :func:`stem.socket.ControlSocket.is_alive` state changes.
+
+ :returns: **threading.RLock** that governs sending messages to our socket
+ and state changes
+ """
+
+ return self._send_lock
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exit_type, value, traceback):
+ self.close()
+
+ def _connect(self):
+ """
+ Connection callback that can be overwritten by subclasses and wrappers.
+ """
+
+ pass
+
+ def _close(self):
+ """
+ Disconnection callback that can be overwritten by subclasses and wrappers.
+ """
+
+ pass
+
+ def _make_socket(self):
+ """
+ Constructs and connects new socket. This is implemented by subclasses.
+
+ :returns: **socket.socket** for our configuration
+
+ :raises:
+ * :class:`stem.SocketError` if unable to make a socket
+ * **NotImplementedError** if not implemented by a subclass
+ """
+
+ raise NotImplementedError('Unsupported Operation: this should be implemented by the ControlSocket subclass')
+
+
+class ControlPort(ControlSocket):
+ """
+ Control connection to tor. For more information see tor's ControlPort torrc
+ option.
+ """
+
+ def __init__(self, address = '127.0.0.1', port = 9051, connect = True):
+ """
+ ControlPort constructor.
+
+ :param str address: ip address of the controller
+ :param int port: port number of the controller
+ :param bool connect: connects to the socket if True, leaves it unconnected otherwise
+
+ :raises: :class:`stem.SocketError` if connect is **True** and we're
+ unable to establish a connection
+ """
+
+ super(ControlPort, self).__init__()
+ self._control_addr = address
+ self._control_port = port
+
+ if connect:
+ self.connect()
+
+ def get_address(self):
+ """
+ Provides the ip address our socket connects to.
+
+ :returns: str with the ip address of our socket
+ """
+
+ return self._control_addr
+
+ def get_port(self):
+ """
+ Provides the port our socket connects to.
+
+ :returns: int with the port of our socket
+ """
+
+ return self._control_port
+
+ def is_localhost(self):
+ return self._control_addr == '127.0.0.1'
+
+ def _make_socket(self):
+ try:
+ control_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ control_socket.connect((self._control_addr, self._control_port))
+ return control_socket
+ except socket.error as exc:
+ raise stem.SocketError(exc)
+
+
+class ControlSocketFile(ControlSocket):
+ """
+ Control connection to tor. For more information see tor's ControlSocket torrc
+ option.
+ """
+
+ def __init__(self, path = '/var/run/tor/control', connect = True):
+ """
+ ControlSocketFile constructor.
+
+ :param str socket_path: path where the control socket is located
+ :param bool connect: connects to the socket if True, leaves it unconnected otherwise
+
+ :raises: :class:`stem.SocketError` if connect is **True** and we're
+ unable to establish a connection
+ """
+
+ super(ControlSocketFile, self).__init__()
+ self._socket_path = path
+
+ if connect:
+ self.connect()
+
+ def get_socket_path(self):
+ """
+ Provides the path our socket connects to.
+
+ :returns: str with the path for our control socket
+ """
+
+ return self._socket_path
+
+ def is_localhost(self):
+ return True
+
+ def _make_socket(self):
+ try:
+ control_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ control_socket.connect(self._socket_path)
+ return control_socket
+ except socket.error as exc:
+ raise stem.SocketError(exc)
+
+
+def send_message(control_file, message, raw = False):
+ """
+ Sends a message to the control socket, adding the expected formatting for
+ single verses multi-line messages. Neither message type should contain an
+ ending newline (if so it'll be treated as a multi-line message with a blank
+ line at the end). If the message doesn't contain a newline then it's sent
+ as...
+
+ ::
+
+ <message>\\r\\n
+
+ and if it does contain newlines then it's split on ``\\n`` and sent as...
+
+ ::
+
+ +<line 1>\\r\\n
+ <line 2>\\r\\n
+ <line 3>\\r\\n
+ .\\r\\n
+
+ :param file control_file: file derived from the control socket (see the
+ socket's makefile() method for more information)
+ :param str message: message to be sent on the control socket
+ :param bool raw: leaves the message formatting untouched, passing it to the
+ socket as-is
+
+ :raises:
+ * :class:`stem.SocketError` if a problem arises in using the socket
+ * :class:`stem.SocketClosed` if the socket is known to be shut down
+ """
+
+ if not raw:
+ message = send_formatting(message)
+
+ try:
+ control_file.write(stem.util.str_tools._to_bytes(message))
+ control_file.flush()
+
+ log_message = message.replace('\r\n', '\n').rstrip()
+ log.trace('Sent to tor:\n' + log_message)
+ except socket.error as exc:
+ log.info('Failed to send message: %s' % exc)
+
+ # When sending there doesn't seem to be a reliable method for
+ # distinguishing between failures from a disconnect verses other things.
+ # Just accounting for known disconnection responses.
+
+ if str(exc) == '[Errno 32] Broken pipe':
+ raise stem.SocketClosed(exc)
+ else:
+ raise stem.SocketError(exc)
+ except AttributeError:
+ # if the control_file has been closed then flush will receive:
+ # AttributeError: 'NoneType' object has no attribute 'sendall'
+
+ log.info('Failed to send message: file has been closed')
+ raise stem.SocketClosed('file has been closed')
+
+
+def recv_message(control_file):
+ """
+ Pulls from a control socket until we either have a complete message or
+ encounter a problem.
+
+ :param file control_file: file derived from the control socket (see the
+ socket's makefile() method for more information)
+
+ :returns: :class:`~stem.response.ControlMessage` read from the socket
+
+ :raises:
+ * :class:`stem.ProtocolError` the content from the socket is malformed
+ * :class:`stem.SocketClosed` if the socket closes before we receive
+ a complete message
+ """
+
+ parsed_content, raw_content = [], b''
+ logging_prefix = 'Error while receiving a control message (%s): '
+
+ while True:
+ try:
+ # From a real socket readline() would always provide bytes, but during
+ # tests we might be given a StringIO in which case it's unicode under
+ # python 3.x.
+
+ line = stem.util.str_tools._to_bytes(control_file.readline())
+ except AttributeError:
+ # if the control_file has been closed then we will receive:
+ # AttributeError: 'NoneType' object has no attribute 'recv'
+
+ prefix = logging_prefix % 'SocketClosed'
+ log.info(prefix + 'socket file has been closed')
+ raise stem.SocketClosed('socket file has been closed')
+ except (socket.error, ValueError) as exc:
+ # When disconnected we get...
+ #
+ # Python 2:
+ # socket.error: [Errno 107] Transport endpoint is not connected
+ #
+ # Python 3:
+ # ValueError: I/O operation on closed file.
+
+ prefix = logging_prefix % 'SocketClosed'
+ log.info(prefix + 'received exception "%s"' % exc)
+ raise stem.SocketClosed(exc)
+
+ raw_content += line
+
+ # Parses the tor control lines. These are of the form...
+ # <status code><divider><content>\r\n
+
+ if len(line) == 0:
+ # if the socket is disconnected then the readline() method will provide
+ # empty content
+
+ prefix = logging_prefix % 'SocketClosed'
+ log.info(prefix + 'empty socket content')
+ raise stem.SocketClosed('Received empty socket content.')
+ elif len(line) < 4:
+ prefix = logging_prefix % 'ProtocolError'
+ log.info(prefix + 'line too short, "%s"' % log.escape(line))
+ raise stem.ProtocolError('Badly formatted reply line: too short')
+ elif not re.match(b'^[a-zA-Z0-9]{3}[-+ ]', line):
+ prefix = logging_prefix % 'ProtocolError'
+ log.info(prefix + 'malformed status code/divider, "%s"' % log.escape(line))
+ raise stem.ProtocolError('Badly formatted reply line: beginning is malformed')
+ elif not line.endswith(b'\r\n'):
+ prefix = logging_prefix % 'ProtocolError'
+ log.info(prefix + 'no CRLF linebreak, "%s"' % log.escape(line))
+ raise stem.ProtocolError('All lines should end with CRLF')
+
+ line = line[:-2] # strips off the CRLF
+ status_code, divider, content = line[:3], line[3:4], line[4:]
+
+ if stem.prereq.is_python_3():
+ status_code = stem.util.str_tools._to_unicode(status_code)
+ divider = stem.util.str_tools._to_unicode(divider)
+
+ if divider == '-':
+ # mid-reply line, keep pulling for more content
+ parsed_content.append((status_code, divider, content))
+ elif divider == ' ':
+ # end of the message, return the message
+ parsed_content.append((status_code, divider, content))
+
+ log_message = raw_content.replace(b'\r\n', b'\n').rstrip()
+ log.trace('Received from tor:\n' + stem.util.str_tools._to_unicode(log_message))
+
+ return stem.response.ControlMessage(parsed_content, raw_content)
+ elif divider == '+':
+ # data entry, all of the following lines belong to the content until we
+ # get a line with just a period
+
+ while True:
+ try:
+ line = stem.util.str_tools._to_bytes(control_file.readline())
+ except socket.error as exc:
+ prefix = logging_prefix % 'SocketClosed'
+ log.info(prefix + 'received an exception while mid-way through a data reply (exception: "%s", read content: "%s")' % (exc, log.escape(raw_content)))
+ raise stem.SocketClosed(exc)
+
+ raw_content += line
+
+ if not line.endswith(b'\r\n'):
+ prefix = logging_prefix % 'ProtocolError'
+ log.info(prefix + 'CRLF linebreaks missing from a data reply, "%s"' % log.escape(raw_content))
+ raise stem.ProtocolError('All lines should end with CRLF')
+ elif line == b'.\r\n':
+ break # data block termination
+
+ line = line[:-2] # strips off the CRLF
+
+ # lines starting with a period are escaped by a second period (as per
+ # section 2.4 of the control-spec)
+
+ if line.startswith(b'..'):
+ line = line[1:]
+
+ # appends to previous content, using a newline rather than CRLF
+ # separator (more conventional for multi-line string content outside
+ # the windows world)
+
+ content += b'\n' + line
+
+ parsed_content.append((status_code, divider, content))
+ else:
+ # this should never be reached due to the prefix regex, but might as well
+ # be safe...
+ prefix = logging_prefix % 'ProtocolError'
+ log.warn(prefix + "\"%s\" isn't a recognized divider type" % divider)
+ raise stem.ProtocolError("Unrecognized divider type '%s': %s" % (divider, stem.util.str_tools._to_unicode(line)))
+
+
+def send_formatting(message):
+ """
+ Performs the formatting expected from sent control messages. For more
+ information see the :func:`~stem.socket.send_message` function.
+
+ :param str message: message to be formatted
+
+ :returns: **str** of the message wrapped by the formatting expected from
+ controllers
+ """
+
+ # From control-spec section 2.2...
+ # Command = Keyword OptArguments CRLF / "+" Keyword OptArguments CRLF CmdData
+ # Keyword = 1*ALPHA
+ # OptArguments = [ SP *(SP / VCHAR) ]
+ #
+ # A command is either a single line containing a Keyword and arguments, or a
+ # multiline command whose initial keyword begins with +, and whose data
+ # section ends with a single "." on a line of its own.
+
+ # if we already have \r\n entries then standardize on \n to start with
+ message = message.replace('\r\n', '\n')
+
+ if '\n' in message:
+ return '+%s\r\n.\r\n' % message.replace('\n', '\r\n')
+ else:
+ return message + '\r\n'
diff --git a/ext/stem/util/__init__.py b/ext/stem/util/__init__.py
new file mode 100644
index 00000000..1c18df3a
--- /dev/null
+++ b/ext/stem/util/__init__.py
@@ -0,0 +1,20 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Utility functions used by the stem library.
+"""
+
+__all__ = [
+ 'conf',
+ 'connection',
+ 'enum',
+ 'log',
+ 'lru_cache',
+ 'ordereddict',
+ 'proc',
+ 'system',
+ 'term',
+ 'test_tools',
+ 'tor_tools',
+]
diff --git a/ext/stem/util/conf.py b/ext/stem/util/conf.py
new file mode 100644
index 00000000..6b0efd71
--- /dev/null
+++ b/ext/stem/util/conf.py
@@ -0,0 +1,745 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Handlers for text configuration files. Configurations are simple string to
+string mappings, with the configuration files using the following rules...
+
+* the key/value is separated by a space
+* anything after a '#' is ignored as a comment
+* excess whitespace is trimmed
+* empty lines are ignored
+* multi-line values can be defined by following the key with lines starting
+ with a '|'
+
+For instance...
+
+::
+
+ # This is my sample config
+ user.name Galen
+ user.password yabba1234 # here's an inline comment
+ user.notes takes a fancy to pepperjack cheese
+ blankEntry.example
+
+ msg.greeting
+ |Multi-line message exclaiming of the
+ |wonder and awe that is pepperjack!
+
+... would be loaded as...
+
+::
+
+ config = {
+ 'user.name': 'Galen',
+ 'user.password': 'yabba1234',
+ 'user.notes': 'takes a fancy to pepperjack cheese',
+ 'blankEntry.example': '',
+ 'msg.greeting': 'Multi-line message exclaiming of the\\nwonder and awe that is pepperjack!',
+ }
+
+Configurations are managed via the :class:`~stem.util.conf.Config` class. The
+:class:`~stem.util.conf.Config` can be be used directly with its
+:func:`~stem.util.conf.Config.get` and :func:`~stem.util.conf.Config.set`
+methods, but usually modules will want a local dictionary with just the
+configurations that it cares about.
+
+To do this use the :func:`~stem.util.conf.config_dict` function. For example...
+
+::
+
+ import getpass
+ from stem.util import conf, connection
+
+ def config_validator(key, value):
+ if key == 'timeout':
+ # require at least a one second timeout
+ return max(1, value)
+ elif key == 'endpoint':
+ if not connection.is_valid_ipv4_address(value):
+ raise ValueError("'%s' isn't a valid IPv4 address" % value)
+ elif key == 'port':
+ if not connection.is_valid_port(value):
+ raise ValueError("'%s' isn't a valid port" % value)
+ elif key == 'retries':
+ # negative retries really don't make sense
+ return max(0, value)
+
+ CONFIG = conf.config_dict('ssh_login', {
+ 'username': getpass.getuser(),
+ 'password': '',
+ 'timeout': 10,
+ 'endpoint': '263.12.8.0',
+ 'port': 22,
+ 'reconnect': False,
+ 'retries': 3,
+ }, config_validator)
+
+There's several things going on here so lets take it step by step...
+
+* The :func:`~stem.util.conf.config_dict` provides a dictionary that's bound
+ to a given configuration. If the "ssh_proxy_config" configuration changes
+ then so will the contents of CONFIG.
+
+* The dictionary we're passing to :func:`~stem.util.conf.config_dict` provides
+ two important pieces of information: default values and their types. See the
+ Config's :func:`~stem.util.conf.Config.get` method for how these type
+ inferences work.
+
+* The config_validator is a hook we're adding to make sure CONFIG only gets
+ values we think are valid. In this case it ensures that our timeout value
+ is at least one second, and rejects endpoints or ports that are invalid.
+
+Now lets say our user has the following configuration file...
+
+::
+
+ username waddle_doo
+ password jabberwocky
+ timeout -15
+ port 9000000
+ retries lots
+ reconnect true
+ logging debug
+
+... and we load it as follows...
+
+::
+
+ >>> from stem.util import conf
+ >>> our_config = conf.get_config('ssh_login')
+ >>> our_config.load('/home/atagar/user_config')
+ >>> print CONFIG # doctest: +SKIP
+ {
+ "username": "waddle_doo",
+ "password": "jabberwocky",
+ "timeout": 1,
+ "endpoint": "263.12.8.0",
+ "port": 22,
+ "reconnect": True,
+ "retries": 3,
+ }
+
+Here's an expanation of what happened...
+
+* the username, password, and reconnect attributes took the values in the
+ configuration file
+
+* the 'config_validator' we added earlier allows for a minimum timeout of one
+ and rejected the invalid port (with a log message)
+
+* we weren't able to convert the retries' "lots" value to an integer so it kept
+ its default value and logged a warning
+
+* the user didn't supply an endpoint so that remained unchanged
+
+* our CONFIG didn't have a 'logging' attribute so it was ignored
+
+**Module Overview:**
+
+::
+
+ config_dict - provides a dictionary that's kept in sync with our config
+ get_config - singleton for getting configurations
+ uses_settings - provides an annotation for functions that use configurations
+ parse_enum_csv - helper funcion for parsing confguration entries for enums
+
+ Config - Custom configuration
+ |- load - reads a configuration file
+ |- save - writes the current configuration to a file
+ |- clear - empties our loaded configuration contents
+ |- add_listener - notifies the given listener when an update occurs
+ |- clear_listeners - removes any attached listeners
+ |- keys - provides keys in the loaded configuration
+ |- set - sets the given key/value pair
+ |- unused_keys - provides keys that have never been requested
+ |- get - provides the value for a given key, with type inference
+ +- get_value - provides the value for a given key as a string
+"""
+
+import inspect
+import os
+import threading
+
+from stem.util import log
+
+try:
+ # added in python 2.7
+ from collections import OrderedDict
+except ImportError:
+ from stem.util.ordereddict import OrderedDict
+
+CONFS = {} # mapping of identifier to singleton instances of configs
+
+
+class _SyncListener(object):
+ def __init__(self, config_dict, interceptor):
+ self.config_dict = config_dict
+ self.interceptor = interceptor
+
+ def update(self, config, key):
+ if key in self.config_dict:
+ new_value = config.get(key, self.config_dict[key])
+
+ if new_value == self.config_dict[key]:
+ return # no change
+
+ if self.interceptor:
+ interceptor_value = self.interceptor(key, new_value)
+
+ if interceptor_value:
+ new_value = interceptor_value
+
+ self.config_dict[key] = new_value
+
+
+def config_dict(handle, conf_mappings, handler = None):
+ """
+ Makes a dictionary that stays synchronized with a configuration.
+
+ This takes a dictionary of 'config_key => default_value' mappings and
+ changes the values to reflect our current configuration. This will leave
+ the previous values alone if...
+
+ * we don't have a value for that config_key
+ * we can't convert our value to be the same type as the default_value
+
+ If a handler is provided then this is called just prior to assigning new
+ values to the config_dict. The handler function is expected to accept the
+ (key, value) for the new values and return what we should actually insert
+ into the dictionary. If this returns None then the value is updated as
+ normal.
+
+ For more information about how we convert types see our
+ :func:`~stem.util.conf.Config.get` method.
+
+ **The dictionary you get from this is manged by the
+ :class:`~stem.util.conf.Config` class and should be treated as being
+ read-only.**
+
+ :param str handle: unique identifier for a config instance
+ :param dict conf_mappings: config key/value mappings used as our defaults
+ :param functor handler: function referred to prior to assigning values
+ """
+
+ selected_config = get_config(handle)
+ selected_config.add_listener(_SyncListener(conf_mappings, handler).update)
+ return conf_mappings
+
+
+def get_config(handle):
+ """
+ Singleton constructor for configuration file instances. If a configuration
+ already exists for the handle then it's returned. Otherwise a fresh instance
+ is constructed.
+
+ :param str handle: unique identifier used to access this config instance
+ """
+
+ if handle not in CONFS:
+ CONFS[handle] = Config()
+
+ return CONFS[handle]
+
+
+def uses_settings(handle, path, lazy_load = True):
+ """
+ Provides a function that can be used as a decorator for other functions that
+ require settings to be loaded. Functions with this decorator will be provided
+ with the configuration as its 'config' keyword argument.
+
+ .. versionchanged:: 1.3.0
+ Omits the 'config' argument if the funcion we're decorating doesn't accept
+ it.
+
+ ::
+
+ uses_settings = stem.util.conf.uses_settings('my_app', '/path/to/settings.cfg')
+
+ @uses_settings
+ def my_function(config):
+ print 'hello %s!' % config.get('username', '')
+
+ :param str handle: hande for the configuration
+ :param str path: path where the configuration should be loaded from
+ :param bool lazy_load: loads the configuration file when the decorator is
+ used if true, otherwise it's loaded right away
+
+ :returns: **function** that can be used as a decorator to provide the
+ configuration
+
+ :raises: **IOError** if we fail to read the configuration file, if
+ **lazy_load** is true then this arises when we use the decorator
+ """
+
+ config = get_config(handle)
+
+ if not lazy_load and not config.get('settings_loaded', False):
+ config.load(path)
+ config.set('settings_loaded', 'true')
+
+ def decorator(func):
+ def wrapped(*args, **kwargs):
+ if lazy_load and not config.get('settings_loaded', False):
+ config.load(path)
+ config.set('settings_loaded', 'true')
+
+ if 'config' in inspect.getargspec(func).args:
+ return func(*args, config = config, **kwargs)
+ else:
+ return func(*args, **kwargs)
+
+ return wrapped
+
+ return decorator
+
+
+def parse_enum(key, value, enumeration):
+ """
+ Provides the enumeration value for a given key. This is a case insensitive
+ lookup and raises an exception if the enum key doesn't exist.
+
+ :param str key: configuration key being looked up
+ :param str value: value to be parsed
+ :param stem.util.enum.Enum enumeration: enumeration the values should be in
+
+ :returns: enumeration value
+
+ :raises: **ValueError** if the **value** isn't among the enumeration keys
+ """
+
+ return parse_enum_csv(key, value, enumeration, 1)[0]
+
+
+def parse_enum_csv(key, value, enumeration, count = None):
+ """
+ Parses a given value as being a comma separated listing of enumeration keys,
+ returning the corresponding enumeration values. This is intended to be a
+ helper for config handlers. The checks this does are case insensitive.
+
+ The **count** attribute can be used to make assertions based on the number of
+ values. This can be...
+
+ * None to indicate that there's no restrictions.
+ * An int to indicate that we should have this many values.
+ * An (int, int) tuple to indicate the range that values can be in. This range
+ is inclusive and either can be None to indicate the lack of a lower or
+ upper bound.
+
+ :param str key: configuration key being looked up
+ :param str value: value to be parsed
+ :param stem.util.enum.Enum enumeration: enumeration the values should be in
+ :param int,tuple count: validates that we have this many items
+
+ :returns: list with the enumeration values
+
+ :raises: **ValueError** if the count assertion fails or the **value** entries
+ don't match the enumeration keys
+ """
+
+ values = [val.upper().strip() for val in value.split(',')]
+
+ if values == ['']:
+ return []
+
+ if count is None:
+ pass # no count validateion checks to do
+ elif isinstance(count, int):
+ if len(values) != count:
+ raise ValueError("Config entry '%s' is expected to be %i comma separated values, got '%s'" % (key, count, value))
+ elif isinstance(count, tuple) and len(count) == 2:
+ minimum, maximum = count
+
+ if minimum is not None and len(values) < minimum:
+ raise ValueError("Config entry '%s' must have at least %i comma separated values, got '%s'" % (key, minimum, value))
+
+ if maximum is not None and len(values) > maximum:
+ raise ValueError("Config entry '%s' can have at most %i comma separated values, got '%s'" % (key, maximum, value))
+ else:
+ raise ValueError("The count must be None, an int, or two value tuple. Got '%s' (%s)'" % (count, type(count)))
+
+ result = []
+ enum_keys = [k.upper() for k in list(enumeration.keys())]
+ enum_values = list(enumeration)
+
+ for val in values:
+ if val in enum_keys:
+ result.append(enum_values[enum_keys.index(val)])
+ else:
+ raise ValueError("The '%s' entry of config entry '%s' wasn't in the enumeration (expected %s)" % (val, key, ', '.join(enum_keys)))
+
+ return result
+
+
+class Config(object):
+ """
+ Handler for easily working with custom configurations, providing persistence
+ to and from files. All operations are thread safe.
+
+ **Example usage:**
+
+ User has a file at '/home/atagar/myConfig' with...
+
+ ::
+
+ destination.ip 1.2.3.4
+ destination.port blarg
+
+ startup.run export PATH=$PATH:~/bin
+ startup.run alias l=ls
+
+ And they have a script with...
+
+ ::
+
+ from stem.util import conf
+
+ # Configuration values we'll use in this file. These are mappings of
+ # configuration keys to the default values we'll use if the user doesn't
+ # have something different in their config file (or it doesn't match this
+ # type).
+
+ ssh_config = conf.config_dict('ssh_login', {
+ 'login.user': 'atagar',
+ 'login.password': 'pepperjack_is_awesome!',
+ 'destination.ip': '127.0.0.1',
+ 'destination.port': 22,
+ 'startup.run': [],
+ })
+
+ # Makes an empty config instance with the handle of 'ssh_login'. This is
+ # a singleton so other classes can fetch this same configuration from
+ # this handle.
+
+ user_config = conf.get_config('ssh_login')
+
+ # Loads the user's configuration file, warning if this fails.
+
+ try:
+ user_config.load("/home/atagar/myConfig")
+ except IOError as exc:
+ print "Unable to load the user's config: %s" % exc
+
+ # This replace the contents of ssh_config with the values from the user's
+ # config file if...
+ #
+ # * the key is present in the config file
+ # * we're able to convert the configuration file's value to the same type
+ # as what's in the mapping (see the Config.get() method for how these
+ # type inferences work)
+ #
+ # For instance in this case...
+ #
+ # * the login values are left alone because they aren't in the user's
+ # config file
+ #
+ # * the 'destination.port' is also left with the value of 22 because we
+ # can't turn "blarg" into an integer
+ #
+ # The other values are replaced, so ssh_config now becomes...
+ #
+ # {'login.user': 'atagar',
+ # 'login.password': 'pepperjack_is_awesome!',
+ # 'destination.ip': '1.2.3.4',
+ # 'destination.port': 22,
+ # 'startup.run': ['export PATH=$PATH:~/bin', 'alias l=ls']}
+ #
+ # Information for what values fail to load and why are reported to
+ # 'stem.util.log'.
+ """
+
+ def __init__(self):
+ self._path = None # location we last loaded from or saved to
+ self._contents = {} # configuration key/value pairs
+ self._listeners = [] # functors to be notified of config changes
+
+ # used for accessing _contents
+ self._contents_lock = threading.RLock()
+
+ # keys that have been requested (used to provide unused config contents)
+ self._requested_keys = set()
+
+ def load(self, path = None):
+ """
+ Reads in the contents of the given path, adding its configuration values
+ to our current contents. If the path is a directory then this loads each
+ of the files, recursively.
+
+ .. versionchanged:: 1.3.0
+ Added support for directories.
+
+ :param str path: file or directory path to be loaded, this uses the last
+ loaded path if not provided
+
+ :raises:
+ * **IOError** if we fail to read the file (it doesn't exist, insufficient
+ permissions, etc)
+ * **ValueError** if no path was provided and we've never been provided one
+ """
+
+ if path:
+ self._path = path
+ elif not self._path:
+ raise ValueError('Unable to load configuration: no path provided')
+
+ if os.path.isdir(self._path):
+ for root, dirnames, filenames in os.walk(self._path):
+ for filename in filenames:
+ self.load(os.path.join(root, filename))
+
+ return
+
+ with open(self._path, 'r') as config_file:
+ read_contents = config_file.readlines()
+
+ with self._contents_lock:
+ while read_contents:
+ line = read_contents.pop(0)
+
+ # strips any commenting or excess whitespace
+ comment_start = line.find('#')
+
+ if comment_start != -1:
+ line = line[:comment_start]
+
+ line = line.strip()
+
+ # parse the key/value pair
+ if line:
+ try:
+ key, value = line.split(' ', 1)
+ value = value.strip()
+ except ValueError:
+ log.debug("Config entry '%s' is expected to be of the format 'Key Value', defaulting to '%s' -> ''" % (line, line))
+ key, value = line, ''
+
+ if not value:
+ # this might be a multi-line entry, try processing it as such
+ multiline_buffer = []
+
+ while read_contents and read_contents[0].lstrip().startswith('|'):
+ content = read_contents.pop(0).lstrip()[1:] # removes '\s+|' prefix
+ content = content.rstrip('\n') # trailing newline
+ multiline_buffer.append(content)
+
+ if multiline_buffer:
+ self.set(key, '\n'.join(multiline_buffer), False)
+ continue
+
+ self.set(key, value, False)
+
+ def save(self, path = None):
+ """
+ Saves configuration contents to disk. If a path is provided then it
+ replaces the configuration location that we track.
+
+ :param str path: location to be saved to
+
+ :raises: **ValueError** if no path was provided and we've never been provided one
+ """
+
+ if path:
+ self._path = path
+ elif not self._path:
+ raise ValueError('Unable to save configuration: no path provided')
+
+ with self._contents_lock:
+ with open(self._path, 'w') as output_file:
+ for entry_key in sorted(self.keys()):
+ for entry_value in self.get_value(entry_key, multiple = True):
+ # check for multi line entries
+ if '\n' in entry_value:
+ entry_value = '\n|' + entry_value.replace('\n', '\n|')
+
+ output_file.write('%s %s\n' % (entry_key, entry_value))
+
+ def clear(self):
+ """
+ Drops the configuration contents and reverts back to a blank, unloaded
+ state.
+ """
+
+ with self._contents_lock:
+ self._contents.clear()
+ self._requested_keys = set()
+
+ def add_listener(self, listener, backfill = True):
+ """
+ Registers the function to be notified of configuration updates. Listeners
+ are expected to be functors which accept (config, key).
+
+ :param functor listener: function to be notified when our configuration is changed
+ :param bool backfill: calls the function with our current values if **True**
+ """
+
+ with self._contents_lock:
+ self._listeners.append(listener)
+
+ if backfill:
+ for key in self.keys():
+ listener(self, key)
+
+ def clear_listeners(self):
+ """
+ Removes all attached listeners.
+ """
+
+ self._listeners = []
+
+ def keys(self):
+ """
+ Provides all keys in the currently loaded configuration.
+
+ :returns: **list** if strings for the configuration keys we've loaded
+ """
+
+ return list(self._contents.keys())
+
+ def unused_keys(self):
+ """
+ Provides the configuration keys that have never been provided to a caller
+ via :func:`~stem.util.conf.config_dict` or the
+ :func:`~stem.util.conf.Config.get` and
+ :func:`~stem.util.conf.Config.get_value` methods.
+
+ :returns: **set** of configuration keys we've loaded but have never been requested
+ """
+
+ return set(self.keys()).difference(self._requested_keys)
+
+ def set(self, key, value, overwrite = True):
+ """
+ Appends the given key/value configuration mapping, behaving the same as if
+ we'd loaded this from a configuration file.
+
+ :param str key: key for the configuration mapping
+ :param str,list value: value we're setting the mapping to
+ :param bool overwrite: replaces the previous value if **True**, otherwise
+ the values are appended
+ """
+
+ with self._contents_lock:
+ if isinstance(value, str):
+ if not overwrite and key in self._contents:
+ self._contents[key].append(value)
+ else:
+ self._contents[key] = [value]
+
+ for listener in self._listeners:
+ listener(self, key)
+ elif isinstance(value, (list, tuple)):
+ if not overwrite and key in self._contents:
+ self._contents[key] += value
+ else:
+ self._contents[key] = value
+
+ for listener in self._listeners:
+ listener(self, key)
+ else:
+ raise ValueError("Config.set() only accepts str, list, or tuple. Provided value was a '%s'" % type(value))
+
+ def get(self, key, default = None):
+ """
+ Fetches the given configuration, using the key and default value to
+ determine the type it should be. Recognized inferences are:
+
+ * **default is a boolean => boolean**
+
+ * values are case insensitive
+ * provides the default if the value isn't "true" or "false"
+
+ * **default is an integer => int**
+
+ * provides the default if the value can't be converted to an int
+
+ * **default is a float => float**
+
+ * provides the default if the value can't be converted to a float
+
+ * **default is a list => list**
+
+ * string contents for all configuration values with this key
+
+ * **default is a tuple => tuple**
+
+ * string contents for all configuration values with this key
+
+ * **default is a dictionary => dict**
+
+ * values without "=>" in them are ignored
+ * values are split into key/value pairs on "=>" with extra whitespace
+ stripped
+
+ :param str key: config setting to be fetched
+ :param default object: value provided if no such key exists or fails to be converted
+
+ :returns: given configuration value with its type inferred with the above rules
+ """
+
+ is_multivalue = isinstance(default, (list, tuple, dict))
+ val = self.get_value(key, default, is_multivalue)
+
+ if val == default:
+ return val # don't try to infer undefined values
+
+ if isinstance(default, bool):
+ if val.lower() == 'true':
+ val = True
+ elif val.lower() == 'false':
+ val = False
+ else:
+ log.debug("Config entry '%s' is expected to be a boolean, defaulting to '%s'" % (key, str(default)))
+ val = default
+ elif isinstance(default, int):
+ try:
+ val = int(val)
+ except ValueError:
+ log.debug("Config entry '%s' is expected to be an integer, defaulting to '%i'" % (key, default))
+ val = default
+ elif isinstance(default, float):
+ try:
+ val = float(val)
+ except ValueError:
+ log.debug("Config entry '%s' is expected to be a float, defaulting to '%f'" % (key, default))
+ val = default
+ elif isinstance(default, list):
+ val = list(val) # make a shallow copy
+ elif isinstance(default, tuple):
+ val = tuple(val)
+ elif isinstance(default, dict):
+ val_map = OrderedDict()
+ for entry in val:
+ if '=>' in entry:
+ entry_key, entry_val = entry.split('=>', 1)
+ val_map[entry_key.strip()] = entry_val.strip()
+ else:
+ log.debug('Ignoring invalid %s config entry (expected a mapping, but "%s" was missing "=>")' % (key, entry))
+ val = val_map
+
+ return val
+
+ def get_value(self, key, default = None, multiple = False):
+ """
+ This provides the current value associated with a given key.
+
+ :param str key: config setting to be fetched
+ :param object default: value provided if no such key exists
+ :param bool multiple: provides back a list of all values if **True**,
+ otherwise this returns the last loaded configuration value
+
+ :returns: **str** or **list** of string configuration values associated
+ with the given key, providing the default if no such key exists
+ """
+
+ with self._contents_lock:
+ if key in self._contents:
+ self._requested_keys.add(key)
+
+ if multiple:
+ return self._contents[key]
+ else:
+ return self._contents[key][-1]
+ else:
+ message_id = 'stem.util.conf.missing_config_key_%s' % key
+ log.log_once(message_id, log.TRACE, "config entry '%s' not found, defaulting to '%s'" % (key, default))
+ return default
diff --git a/ext/stem/util/connection.py b/ext/stem/util/connection.py
new file mode 100644
index 00000000..88d70d58
--- /dev/null
+++ b/ext/stem/util/connection.py
@@ -0,0 +1,651 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Connection and networking based utility functions.
+
+**Module Overview:**
+
+::
+
+ get_connections - quieries the connections belonging to a given process
+ system_resolvers - provides connection resolution methods that are likely to be available
+ port_usage - brief description of the common usage for a port
+
+ is_valid_ipv4_address - checks if a string is a valid IPv4 address
+ is_valid_ipv6_address - checks if a string is a valid IPv6 address
+ is_valid_port - checks if something is a valid representation for a port
+ is_private_address - checks if an IPv4 address belongs to a private range or not
+
+ expand_ipv6_address - provides an IPv6 address with its collapsed portions expanded
+ get_mask_ipv4 - provides the mask representation for a given number of bits
+ get_mask_ipv6 - provides the IPv6 mask representation for a given number of bits
+
+.. data:: Resolver (enum)
+
+ Method for resolving a process' connections.
+
+ .. versionadded:: 1.1.0
+ .. versionchanged:: 1.4.0
+ Added **NETSTAT_WINDOWS**.
+
+ ==================== ===========
+ Resolver Description
+ ==================== ===========
+ **PROC** /proc contents
+ **NETSTAT** netstat
+ **NETSTAT_WINDOWS** netstat command under Windows
+ **SS** ss command
+ **LSOF** lsof command
+ **SOCKSTAT** sockstat command under *nix
+ **BSD_SOCKSTAT** sockstat command under FreeBSD
+ **BSD_PROCSTAT** procstat command under FreeBSD
+ ==================== ===========
+"""
+
+import collections
+import hashlib
+import hmac
+import os
+import platform
+import re
+
+import stem.util.proc
+import stem.util.system
+
+from stem import str_type
+from stem.util import conf, enum, log
+
+# Connection resolution is risky to log about since it's highly likely to
+# contain sensitive information. That said, it's also difficult to get right in
+# a platform independent fashion. To opt into the logging requried to
+# troubleshoot connection resolution set the following...
+
+LOG_CONNECTION_RESOLUTION = False
+
+Resolver = enum.Enum(
+ ('PROC', 'proc'),
+ ('NETSTAT', 'netstat'),
+ ('NETSTAT_WINDOWS', 'netstat (windows)'),
+ ('SS', 'ss'),
+ ('LSOF', 'lsof'),
+ ('SOCKSTAT', 'sockstat'),
+ ('BSD_SOCKSTAT', 'sockstat (bsd)'),
+ ('BSD_PROCSTAT', 'procstat (bsd)')
+)
+
+Connection = collections.namedtuple('Connection', [
+ 'local_address',
+ 'local_port',
+ 'remote_address',
+ 'remote_port',
+ 'protocol',
+])
+
+FULL_IPv4_MASK = '255.255.255.255'
+FULL_IPv6_MASK = 'FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF'
+
+CRYPTOVARIABLE_EQUALITY_COMPARISON_NONCE = os.urandom(32)
+
+PORT_USES = None # port number => description
+
+RESOLVER_COMMAND = {
+ Resolver.PROC: '',
+
+ # -n = prevents dns lookups, -p = include process
+ Resolver.NETSTAT: 'netstat -np',
+
+ # -a = show all TCP/UDP connections, -n = numeric addresses and ports, -o = include pid
+ Resolver.NETSTAT_WINDOWS: 'netstat -ano',
+
+ # -n = numeric ports, -p = include process, -t = tcp sockets, -u = udp sockets
+ Resolver.SS: 'ss -nptu',
+
+ # -n = prevent dns lookups, -P = show port numbers (not names), -i = ip only, -w = no warnings
+ # (lsof provides a '-p <pid>' but oddly in practice it seems to be ~11-28% slower)
+ Resolver.LSOF: 'lsof -wnPi',
+
+ Resolver.SOCKSTAT: 'sockstat',
+
+ # -4 = IPv4, -c = connected sockets
+ Resolver.BSD_SOCKSTAT: 'sockstat -4c',
+
+ # -f <pid> = process pid
+ Resolver.BSD_PROCSTAT: 'procstat -f {pid}',
+}
+
+RESOLVER_FILTER = {
+ Resolver.PROC: '',
+
+ # tcp 0 586 192.168.0.1:44284 38.229.79.2:443 ESTABLISHED 15843/tor
+ Resolver.NETSTAT: '^{protocol}\s+.*\s+{local_address}:{local_port}\s+{remote_address}:{remote_port}\s+ESTABLISHED\s+{pid}/{name}\s*$',
+
+ # tcp 586 192.168.0.1:44284 38.229.79.2:443 ESTABLISHED 15843
+ Resolver.NETSTAT_WINDOWS: '^\s*{protocol}\s+{local_address}:{local_port}\s+{remote_address}:{remote_port}\s+ESTABLISHED\s+{pid}\s*$',
+
+ # tcp ESTAB 0 0 192.168.0.20:44415 38.229.79.2:443 users:(("tor",15843,9))
+ Resolver.SS: '^{protocol}\s+ESTAB\s+.*\s+{local_address}:{local_port}\s+{remote_address}:{remote_port}\s+users:\(\("{name}",{pid},[0-9]+\)\)$',
+
+ # tor 3873 atagar 45u IPv4 40994 0t0 TCP 10.243.55.20:45724->194.154.227.109:9001 (ESTABLISHED)
+ Resolver.LSOF: '^{name}\s+{pid}\s+.*\s+{protocol}\s+{local_address}:{local_port}->{remote_address}:{remote_port} \(ESTABLISHED\)$',
+
+ # atagar tor 15843 tcp4 192.168.0.20:44092 68.169.35.102:443 ESTABLISHED
+ Resolver.SOCKSTAT: '^\S+\s+{name}\s+{pid}\s+{protocol}4\s+{local_address}:{local_port}\s+{remote_address}:{remote_port}\s+ESTABLISHED$',
+
+ # _tor tor 4397 12 tcp4 172.27.72.202:54011 127.0.0.1:9001
+ Resolver.BSD_SOCKSTAT: '^\S+\s+{name}\s+{pid}\s+\S+\s+{protocol}4\s+{local_address}:{local_port}\s+{remote_address}:{remote_port}$',
+
+ # 3561 tor 4 s - rw---n-- 2 0 TCP 10.0.0.2:9050 10.0.0.1:22370
+ Resolver.BSD_PROCSTAT: '^\s*{pid}\s+{name}\s+.*\s+{protocol}\s+{local_address}:{local_port}\s+{remote_address}:{remote_port}$',
+}
+
+
+def get_connections(resolver, process_pid = None, process_name = None):
+ """
+ Retrieves a list of the current connections for a given process. This
+ provides a list of Connection instances, which have five attributes...
+
+ * **local_address** (str)
+ * **local_port** (int)
+ * **remote_address** (str)
+ * **remote_port** (int)
+ * **protocol** (str, generally either 'tcp' or 'udp')
+
+ .. versionadded:: 1.1.0
+
+ :param Resolver resolver: method of connection resolution to use
+ :param int process_pid: pid of the process to retrieve
+ :param str process_name: name of the process to retrieve
+
+ :returns: **list** of Connection instances
+
+ :raises:
+ * **ValueError** if using **Resolver.PROC** or **Resolver.BSD_PROCSTAT**
+ and the process_pid wasn't provided
+
+ * **IOError** if no connections are available or resolution fails
+ (generally they're indistinguishable). The common causes are the
+ command being unavailable or permissions.
+ """
+
+ def _log(msg):
+ if LOG_CONNECTION_RESOLUTION:
+ log.debug(msg)
+
+ _log('=' * 80)
+ _log('Querying connections for resolver: %s, pid: %s, name: %s' % (resolver, process_pid, process_name))
+
+ if isinstance(process_pid, str):
+ try:
+ process_pid = int(process_pid)
+ except ValueError:
+ raise ValueError('Process pid was non-numeric: %s' % process_pid)
+
+ if process_pid is None and process_name and resolver == Resolver.NETSTAT_WINDOWS:
+ process_pid = stem.util.system.pid_by_name(process_name)
+
+ if process_pid is None and resolver in (Resolver.NETSTAT_WINDOWS, Resolver.PROC, Resolver.BSD_PROCSTAT):
+ raise ValueError('%s resolution requires a pid' % resolver)
+
+ if resolver == Resolver.PROC:
+ return [Connection(*conn) for conn in stem.util.proc.connections(process_pid)]
+
+ resolver_command = RESOLVER_COMMAND[resolver].format(pid = process_pid)
+
+ try:
+ results = stem.util.system.call(resolver_command)
+ except OSError as exc:
+ raise IOError("Unable to query '%s': %s" % (resolver_command, exc))
+
+ resolver_regex_str = RESOLVER_FILTER[resolver].format(
+ protocol = '(?P<protocol>\S+)',
+ local_address = '(?P<local_address>[0-9.]+)',
+ local_port = '(?P<local_port>[0-9]+)',
+ remote_address = '(?P<remote_address>[0-9.]+)',
+ remote_port = '(?P<remote_port>[0-9]+)',
+ pid = process_pid if process_pid else '[0-9]*',
+ name = process_name if process_name else '\S*',
+ )
+
+ _log('Resolver regex: %s' % resolver_regex_str)
+ _log('Resolver results:\n%s' % '\n'.join(results))
+
+ connections = []
+ resolver_regex = re.compile(resolver_regex_str)
+
+ for line in results:
+ match = resolver_regex.match(line)
+
+ if match:
+ attr = match.groupdict()
+ local_addr = attr['local_address']
+ local_port = int(attr['local_port'])
+ remote_addr = attr['remote_address']
+ remote_port = int(attr['remote_port'])
+ protocol = attr['protocol'].lower()
+
+ if remote_addr == '0.0.0.0':
+ continue # procstat response for unestablished connections
+
+ if not (is_valid_ipv4_address(local_addr) and is_valid_ipv4_address(remote_addr)):
+ _log('Invalid address (%s or %s): %s' % (local_addr, remote_addr, line))
+ elif not (is_valid_port(local_port) and is_valid_port(remote_port)):
+ _log('Invalid port (%s or %s): %s' % (local_port, remote_port, line))
+ elif protocol not in ('tcp', 'udp'):
+ _log('Unrecognized protocol (%s): %s' % (protocol, line))
+
+ conn = Connection(local_addr, local_port, remote_addr, remote_port, protocol)
+ connections.append(conn)
+ _log(str(conn))
+
+ _log('%i connections found' % len(connections))
+
+ if not connections:
+ raise IOError('No results found using: %s' % resolver_command)
+
+ return connections
+
+
+def system_resolvers(system = None):
+ """
+ Provides the types of connection resolvers likely to be available on this platform.
+
+ .. versionadded:: 1.1.0
+
+ .. versionchanged:: 1.3.0
+ Renamed from get_system_resolvers() to system_resolvers(). The old name
+ still works as an alias, but will be dropped in Stem version 2.0.0.
+
+ :param str system: system to get resolvers for, this is determined by
+ platform.system() if not provided
+
+ :returns: **list** of :data:`~stem.util.connection.Resolver` instances available on this platform
+ """
+ if system is None:
+ if stem.util.system.is_gentoo():
+ system = 'Gentoo'
+ else:
+ system = platform.system()
+
+ if system == 'Windows':
+ resolvers = [Resolver.NETSTAT_WINDOWS]
+ elif system in ('Darwin', 'OpenBSD'):
+ resolvers = [Resolver.LSOF]
+ elif system == 'FreeBSD':
+ # Netstat is available, but lacks a '-p' equivalent so we can't associate
+ # the results to processes. The platform also has a ss command, but it
+ # belongs to a spreadsheet application.
+
+ resolvers = [Resolver.BSD_SOCKSTAT, Resolver.BSD_PROCSTAT, Resolver.LSOF]
+ else:
+ # Sockstat isn't available by default on ubuntu.
+
+ resolvers = [Resolver.NETSTAT, Resolver.SOCKSTAT, Resolver.LSOF, Resolver.SS]
+
+ # remove any that aren't in the user's PATH
+
+ resolvers = [r for r in resolvers if stem.util.system.is_available(RESOLVER_COMMAND[r])]
+
+ # proc resolution, by far, outperforms the others so defaults to this is able
+
+ if stem.util.proc.is_available() and os.access('/proc/net/tcp', os.R_OK) and os.access('/proc/net/udp', os.R_OK):
+ resolvers = [Resolver.PROC] + resolvers
+
+ return resolvers
+
+
+def port_usage(port):
+ """
+ Provides the common use of a given port. For example, 'HTTP' for port 80 or
+ 'SSH' for 22.
+
+ .. versionadded:: 1.2.0
+
+ :param int port: port number to look up
+
+ :returns: **str** with a description for the port, **None** if none is known
+ """
+
+ global PORT_USES
+
+ if PORT_USES is None:
+ config = conf.Config()
+ config_path = os.path.join(os.path.dirname(__file__), 'ports.cfg')
+
+ try:
+ config.load(config_path)
+ port_uses = {}
+
+ for key, value in config.get('port', {}).items():
+ if key.isdigit():
+ port_uses[int(key)] = value
+ elif '-' in key:
+ min_port, max_port = key.split('-', 1)
+
+ for port_entry in range(int(min_port), int(max_port) + 1):
+ port_uses[port_entry] = value
+ else:
+ raise ValueError("'%s' is an invalid key" % key)
+
+ PORT_USES = port_uses
+ except Exception as exc:
+ log.warn("BUG: stem failed to load its internal port descriptions from '%s': %s" % (config_path, exc))
+
+ if not PORT_USES:
+ return None
+
+ if isinstance(port, str) and port.isdigit():
+ port = int(port)
+
+ return PORT_USES.get(port)
+
+
+def is_valid_ipv4_address(address):
+ """
+ Checks if a string is a valid IPv4 address.
+
+ :param str address: string to be checked
+
+ :returns: **True** if input is a valid IPv4 address, **False** otherwise
+ """
+
+ if not isinstance(address, (bytes, str_type)):
+ return False
+
+ # checks if theres four period separated values
+
+ if address.count('.') != 3:
+ return False
+
+ # checks that each value in the octet are decimal values between 0-255
+ for entry in address.split('.'):
+ if not entry.isdigit() or int(entry) < 0 or int(entry) > 255:
+ return False
+ elif entry[0] == '0' and len(entry) > 1:
+ return False # leading zeros, for instance in '1.2.3.001'
+
+ return True
+
+
+def is_valid_ipv6_address(address, allow_brackets = False):
+ """
+ Checks if a string is a valid IPv6 address.
+
+ :param str address: string to be checked
+ :param bool allow_brackets: ignore brackets which form '[address]'
+
+ :returns: **True** if input is a valid IPv6 address, **False** otherwise
+ """
+
+ if allow_brackets:
+ if address.startswith('[') and address.endswith(']'):
+ address = address[1:-1]
+
+ # addresses are made up of eight colon separated groups of four hex digits
+ # with leading zeros being optional
+ # https://en.wikipedia.org/wiki/IPv6#Address_format
+
+ colon_count = address.count(':')
+
+ if colon_count > 7:
+ return False # too many groups
+ elif colon_count != 7 and '::' not in address:
+ return False # not enough groups and none are collapsed
+ elif address.count('::') > 1 or ':::' in address:
+ return False # multiple groupings of zeros can't be collapsed
+
+ for entry in address.split(':'):
+ if not re.match('^[0-9a-fA-f]{0,4}$', entry):
+ return False
+
+ return True
+
+
+def is_valid_port(entry, allow_zero = False):
+ """
+ Checks if a string or int is a valid port number.
+
+ :param list,str,int entry: string, integer or list to be checked
+ :param bool allow_zero: accept port number of zero (reserved by definition)
+
+ :returns: **True** if input is an integer and within the valid port range, **False** otherwise
+ """
+
+ try:
+ value = int(entry)
+
+ if str(value) != str(entry):
+ return False # invalid leading char, e.g. space or zero
+ elif allow_zero and value == 0:
+ return True
+ else:
+ return value > 0 and value < 65536
+ except TypeError:
+ if isinstance(entry, (tuple, list)):
+ for port in entry:
+ if not is_valid_port(port, allow_zero):
+ return False
+
+ return True
+ else:
+ return False
+ except ValueError:
+ return False
+
+
+def is_private_address(address):
+ """
+ Checks if the IPv4 address is in a range belonging to the local network or
+ loopback. These include:
+
+ * Private ranges: 10.*, 172.16.* - 172.31.*, 192.168.*
+ * Loopback: 127.*
+
+ .. versionadded:: 1.1.0
+
+ :param str address: string to be checked
+
+ :returns: **True** if input is in a private range, **False** otherwise
+
+ :raises: **ValueError** if the address isn't a valid IPv4 address
+ """
+
+ if not is_valid_ipv4_address(address):
+ raise ValueError("'%s' isn't a valid IPv4 address" % address)
+
+ # checks for any of the simple wildcard ranges
+
+ if address.startswith('10.') or address.startswith('192.168.') or address.startswith('127.'):
+ return True
+
+ # checks for the 172.16.* - 172.31.* range
+
+ if address.startswith('172.'):
+ second_octet = int(address.split('.')[1])
+
+ if second_octet >= 16 and second_octet <= 31:
+ return True
+
+ return False
+
+
+def expand_ipv6_address(address):
+ """
+ Expands abbreviated IPv6 addresses to their full colon separated hex format.
+ For instance...
+
+ ::
+
+ >>> expand_ipv6_address('2001:db8::ff00:42:8329')
+ '2001:0db8:0000:0000:0000:ff00:0042:8329'
+
+ >>> expand_ipv6_address('::')
+ '0000:0000:0000:0000:0000:0000:0000:0000'
+
+ :param str address: IPv6 address to be expanded
+
+ :raises: **ValueError** if the address can't be expanded due to being malformed
+ """
+
+ if not is_valid_ipv6_address(address):
+ raise ValueError("'%s' isn't a valid IPv6 address" % address)
+
+ # expands collapsed groupings, there can only be a single '::' in a valid
+ # address
+ if '::' in address:
+ missing_groups = 7 - address.count(':')
+ address = address.replace('::', '::' + ':' * missing_groups)
+
+ # inserts missing zeros
+ for index in range(8):
+ start = index * 5
+ end = address.index(':', start) if index != 7 else len(address)
+ missing_zeros = 4 - (end - start)
+
+ if missing_zeros > 0:
+ address = address[:start] + '0' * missing_zeros + address[start:]
+
+ return address
+
+
+def get_mask_ipv4(bits):
+ """
+ Provides the IPv4 mask for a given number of bits, in the dotted-quad format.
+
+ :param int bits: number of bits to be converted
+
+ :returns: **str** with the subnet mask representation for this many bits
+
+ :raises: **ValueError** if given a number of bits outside the range of 0-32
+ """
+
+ if bits > 32 or bits < 0:
+ raise ValueError('A mask can only be 0-32 bits, got %i' % bits)
+ elif bits == 32:
+ return FULL_IPv4_MASK
+
+ # get the binary representation of the mask
+ mask_bin = _get_binary(2 ** bits - 1, 32)[::-1]
+
+ # breaks it into eight character groupings
+ octets = [mask_bin[8 * i:8 * (i + 1)] for i in range(4)]
+
+ # converts each octet into its integer value
+ return '.'.join([str(int(octet, 2)) for octet in octets])
+
+
+def get_mask_ipv6(bits):
+ """
+ Provides the IPv6 mask for a given number of bits, in the hex colon-delimited
+ format.
+
+ :param int bits: number of bits to be converted
+
+ :returns: **str** with the subnet mask representation for this many bits
+
+ :raises: **ValueError** if given a number of bits outside the range of 0-128
+ """
+
+ if bits > 128 or bits < 0:
+ raise ValueError('A mask can only be 0-128 bits, got %i' % bits)
+ elif bits == 128:
+ return FULL_IPv6_MASK
+
+ # get the binary representation of the mask
+ mask_bin = _get_binary(2 ** bits - 1, 128)[::-1]
+
+ # breaks it into sixteen character groupings
+ groupings = [mask_bin[16 * i:16 * (i + 1)] for i in range(8)]
+
+ # converts each group into its hex value
+ return ':'.join(['%04x' % int(group, 2) for group in groupings]).upper()
+
+
+def _get_masked_bits(mask):
+ """
+ Provides the number of bits that an IPv4 subnet mask represents. Note that
+ not all masks can be represented by a bit count.
+
+ :param str mask: mask to be converted
+
+ :returns: **int** with the number of bits represented by the mask
+
+ :raises: **ValueError** if the mask is invalid or can't be converted
+ """
+
+ if not is_valid_ipv4_address(mask):
+ raise ValueError("'%s' is an invalid subnet mask" % mask)
+
+ # converts octets to binary representation
+ mask_bin = _get_address_binary(mask)
+ mask_match = re.match('^(1*)(0*)$', mask_bin)
+
+ if mask_match:
+ return 32 - len(mask_match.groups()[1])
+ else:
+ raise ValueError('Unable to convert mask to a bit count: %s' % mask)
+
+
+def _get_binary(value, bits):
+ """
+ Provides the given value as a binary string, padded with zeros to the given
+ number of bits.
+
+ :param int value: value to be converted
+ :param int bits: number of bits to pad to
+ """
+
+ # http://www.daniweb.com/code/snippet216539.html
+ return ''.join([str((value >> y) & 1) for y in range(bits - 1, -1, -1)])
+
+
+def _get_address_binary(address):
+ """
+ Provides the binary value for an IPv4 or IPv6 address.
+
+ :returns: **str** with the binary representation of this address
+
+ :raises: **ValueError** if address is neither an IPv4 nor IPv6 address
+ """
+
+ if is_valid_ipv4_address(address):
+ return ''.join([_get_binary(int(octet), 8) for octet in address.split('.')])
+ elif is_valid_ipv6_address(address):
+ address = expand_ipv6_address(address)
+ return ''.join([_get_binary(int(grouping, 16), 16) for grouping in address.split(':')])
+ else:
+ raise ValueError("'%s' is neither an IPv4 or IPv6 address" % address)
+
+
+def _hmac_sha256(key, msg):
+ """
+ Generates a sha256 digest using the given key and message.
+
+ :param str key: starting key for the hash
+ :param str msg: message to be hashed
+
+ :returns: sha256 digest of msg as bytes, hashed using the given key
+ """
+
+ return hmac.new(key, msg, hashlib.sha256).digest()
+
+
+def _cryptovariables_equal(x, y):
+ """
+ Compares two strings for equality securely.
+
+ :param str x: string to be compared.
+ :param str y: the other string to be compared.
+
+ :returns: **True** if both strings are equal, **False** otherwise.
+ """
+
+ return (
+ _hmac_sha256(CRYPTOVARIABLE_EQUALITY_COMPARISON_NONCE, x) ==
+ _hmac_sha256(CRYPTOVARIABLE_EQUALITY_COMPARISON_NONCE, y))
+
+# TODO: drop with stem 2.x
+# We renamed our methods to drop a redundant 'get_*' prefix, so alias the old
+# names for backward compatability.
+
+get_system_resolvers = system_resolvers
diff --git a/ext/stem/util/enum.py b/ext/stem/util/enum.py
new file mode 100644
index 00000000..5cf81b8b
--- /dev/null
+++ b/ext/stem/util/enum.py
@@ -0,0 +1,172 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Basic enumeration, providing ordered types for collections. These can be
+constructed as simple type listings...
+
+::
+
+ >>> from stem.util import enum
+ >>> insects = enum.Enum('ANT', 'WASP', 'LADYBUG', 'FIREFLY')
+ >>> insects.ANT
+ 'Ant'
+ >>> tuple(insects)
+ ('Ant', 'Wasp', 'Ladybug', 'Firefly')
+
+... or with overwritten string counterparts...
+
+::
+
+ >>> from stem.util import enum
+ >>> pets = enum.Enum(('DOG', 'Skippy'), 'CAT', ('FISH', 'Nemo'))
+ >>> pets.DOG
+ 'Skippy'
+ >>> pets.CAT
+ 'Cat'
+
+**Module Overview:**
+
+::
+
+ UppercaseEnum - Provides an enum instance with capitalized values
+
+ Enum - Provides a basic, ordered enumeration
+ |- keys - string representation of our enum keys
+ |- index_of - index of an enum value
+ |- next - provides the enum after a given enum value
+ |- previous - provides the enum before a given value
+ |- __getitem__ - provides the value for an enum key
+ +- __iter__ - iterator over our enum keys
+"""
+
+from stem import str_type
+
+
+def UppercaseEnum(*args):
+ """
+ Provides an :class:`~stem.util.enum.Enum` instance where the values are
+ identical to the keys. Since the keys are uppercase by convention this means
+ the values are too. For instance...
+
+ ::
+
+ >>> from stem.util import enum
+ >>> runlevels = enum.UppercaseEnum('DEBUG', 'INFO', 'NOTICE', 'WARN', 'ERROR')
+ >>> runlevels.DEBUG
+ 'DEBUG'
+
+ :param list args: enum keys to initialize with
+
+ :returns: :class:`~stem.util.enum.Enum` instance with the given keys
+ """
+
+ return Enum(*[(v, v) for v in args])
+
+
+class Enum(object):
+ """
+ Basic enumeration.
+ """
+
+ def __init__(self, *args):
+ from stem.util.str_tools import _to_camel_case
+
+ # ordered listings of our keys and values
+ keys, values = [], []
+
+ for entry in args:
+ if isinstance(entry, (bytes, str_type)):
+ key, val = entry, _to_camel_case(entry)
+ elif isinstance(entry, tuple) and len(entry) == 2:
+ key, val = entry
+ else:
+ raise ValueError('Unrecognized input: %s' % args)
+
+ keys.append(key)
+ values.append(val)
+ setattr(self, key, val)
+
+ self._keys = tuple(keys)
+ self._values = tuple(values)
+
+ def keys(self):
+ """
+ Provides an ordered listing of the enumeration keys in this set.
+
+ :returns: **list** with our enum keys
+ """
+
+ return list(self._keys)
+
+ def index_of(self, value):
+ """
+ Provides the index of the given value in the collection.
+
+ :param str value: entry to be looked up
+
+ :returns: **int** index of the given entry
+
+ :raises: **ValueError** if no such element exists
+ """
+
+ return self._values.index(value)
+
+ def next(self, value):
+ """
+ Provides the next enumeration after the given value.
+
+ :param str value: enumeration for which to get the next entry
+
+ :returns: enum value following the given entry
+
+ :raises: **ValueError** if no such element exists
+ """
+
+ if value not in self._values:
+ raise ValueError('No such enumeration exists: %s (options: %s)' % (value, ', '.join(self._values)))
+
+ next_index = (self._values.index(value) + 1) % len(self._values)
+ return self._values[next_index]
+
+ def previous(self, value):
+ """
+ Provides the previous enumeration before the given value.
+
+ :param str value: enumeration for which to get the previous entry
+
+ :returns: enum value proceeding the given entry
+
+ :raises: **ValueError** if no such element exists
+ """
+
+ if value not in self._values:
+ raise ValueError('No such enumeration exists: %s (options: %s)' % (value, ', '.join(self._values)))
+
+ prev_index = (self._values.index(value) - 1) % len(self._values)
+ return self._values[prev_index]
+
+ def __getitem__(self, item):
+ """
+ Provides the values for the given key.
+
+ :param str item: key to be looked up
+
+ :returns: **str** with the value for the given key
+
+ :raises: **ValueError** if the key doesn't exist
+ """
+
+ if item in vars(self):
+ return getattr(self, item)
+ else:
+ keys = ', '.join(self.keys())
+ raise ValueError("'%s' isn't among our enumeration keys, which includes: %s" % (item, keys))
+
+ def __iter__(self):
+ """
+ Provides an ordered listing of the enums in this set.
+ """
+
+ for entry in self._values:
+ yield entry
diff --git a/ext/stem/util/log.py b/ext/stem/util/log.py
new file mode 100644
index 00000000..4154706f
--- /dev/null
+++ b/ext/stem/util/log.py
@@ -0,0 +1,253 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Functions to aid library logging. The default logging
+:data:`~stem.util.log.Runlevel` is usually NOTICE and above.
+
+**Stem users are more than welcome to listen for stem events, but these
+functions are not being vended to our users. They may change in the future, use
+them at your own risk.**
+
+**Module Overview:**
+
+::
+
+ get_logger - provides the stem's Logger instance
+ logging_level - converts a runlevel to its logging number
+ escape - escapes special characters in a message in preparation for logging
+
+ log - logs a message at the given runlevel
+ log_once - logs a message, deduplicating if it has already been logged
+ trace - logs a message at the TRACE runlevel
+ debug - logs a message at the DEBUG runlevel
+ info - logs a message at the INFO runlevel
+ notice - logs a message at the NOTICE runlevel
+ warn - logs a message at the WARN runlevel
+ error - logs a message at the ERROR runlevel
+
+ LogBuffer - Buffers logged events so they can be iterated over.
+ |- is_empty - checks if there's events in our buffer
+ +- __iter__ - iterates over and removes the buffered events
+
+ log_to_stdout - reports further logged events to stdout
+
+.. data:: Runlevel (enum)
+
+ Enumeration for logging runlevels.
+
+ ========== ===========
+ Runlevel Description
+ ========== ===========
+ **ERROR** critical issue occurred, the user needs to be notified
+ **WARN** non-critical issue occurred that the user should be aware of
+ **NOTICE** information that is helpful to the user
+ **INFO** high level library activity
+ **DEBUG** low level library activity
+ **TRACE** request/reply logging
+ ========== ===========
+"""
+
+import logging
+
+import stem.prereq
+import stem.util.enum
+import stem.util.str_tools
+
+# Logging runlevels. These are *very* commonly used so including shorter
+# aliases (so they can be referenced as log.DEBUG, log.WARN, etc).
+
+Runlevel = stem.util.enum.UppercaseEnum('TRACE', 'DEBUG', 'INFO', 'NOTICE', 'WARN', 'ERROR')
+TRACE, DEBUG, INFO, NOTICE, WARN, ERR = list(Runlevel)
+
+# mapping of runlevels to the logger module's values, TRACE and DEBUG aren't
+# built into the module
+
+LOG_VALUES = {
+ Runlevel.TRACE: logging.DEBUG - 5,
+ Runlevel.DEBUG: logging.DEBUG,
+ Runlevel.INFO: logging.INFO,
+ Runlevel.NOTICE: logging.INFO + 5,
+ Runlevel.WARN: logging.WARN,
+ Runlevel.ERROR: logging.ERROR,
+}
+
+logging.addLevelName(LOG_VALUES[TRACE], 'TRACE')
+logging.addLevelName(LOG_VALUES[NOTICE], 'NOTICE')
+
+LOGGER = logging.getLogger('stem')
+LOGGER.setLevel(LOG_VALUES[TRACE])
+
+# There's some messages that we don't want to log more than once. This set has
+# the messages IDs that we've logged which fall into this category.
+DEDUPLICATION_MESSAGE_IDS = set()
+
+# Adds a default nullhandler for the stem logger, suppressing the 'No handlers
+# could be found for logger "stem"' warning as per...
+# http://docs.python.org/release/3.1.3/library/logging.html#configuring-logging-for-a-library
+
+
+class _NullHandler(logging.Handler):
+ def emit(self, record):
+ pass
+
+if not LOGGER.handlers:
+ LOGGER.addHandler(_NullHandler())
+
+
+def get_logger():
+ """
+ Provides the stem logger.
+
+ :return: **logging.Logger** for stem
+ """
+
+ return LOGGER
+
+
+def logging_level(runlevel):
+ """
+ Translates a runlevel into the value expected by the logging module.
+
+ :param stem.util.log.Runlevel runlevel: runlevel to be returned, no logging if **None**
+ """
+
+ if runlevel:
+ return LOG_VALUES[runlevel]
+ else:
+ return logging.FATAL + 5
+
+
+def escape(message):
+ """
+ Escapes specific sequences for logging (newlines, tabs, carriage returns). If
+ the input is **bytes** then this converts it to **unicode** under python 3.x.
+
+ :param str message: string to be escaped
+
+ :returns: str that is escaped
+ """
+
+ if stem.prereq.is_python_3():
+ message = stem.util.str_tools._to_unicode(message)
+
+ for pattern, replacement in (('\n', '\\n'), ('\r', '\\r'), ('\t', '\\t')):
+ message = message.replace(pattern, replacement)
+
+ return message
+
+
+def log(runlevel, message):
+ """
+ Logs a message at the given runlevel.
+
+ :param stem.util.log.Runlevel runlevel: runlevel to log the message at, logging is skipped if **None**
+ :param str message: message to be logged
+ """
+
+ if runlevel:
+ LOGGER.log(LOG_VALUES[runlevel], message)
+
+
+def log_once(message_id, runlevel, message):
+ """
+ Logs a message at the given runlevel. If a message with this ID has already
+ been logged then this is a no-op.
+
+ :param str message_id: unique message identifier to deduplicate on
+ :param stem.util.log.Runlevel runlevel: runlevel to log the message at, logging is skipped if **None**
+ :param str message: message to be logged
+
+ :returns: **True** if we log the message, **False** otherwise
+ """
+
+ if not runlevel or message_id in DEDUPLICATION_MESSAGE_IDS:
+ return False
+ else:
+ DEDUPLICATION_MESSAGE_IDS.add(message_id)
+ log(runlevel, message)
+
+# shorter aliases for logging at a runlevel
+
+
+def trace(message):
+ log(Runlevel.TRACE, message)
+
+
+def debug(message):
+ log(Runlevel.DEBUG, message)
+
+
+def info(message):
+ log(Runlevel.INFO, message)
+
+
+def notice(message):
+ log(Runlevel.NOTICE, message)
+
+
+def warn(message):
+ log(Runlevel.WARN, message)
+
+
+def error(message):
+ log(Runlevel.ERROR, message)
+
+
+class LogBuffer(logging.Handler):
+ """
+ Basic log handler that listens for stem events and stores them so they can be
+ read later. Log entries are cleared as they are read.
+
+ .. versionchanged:: 1.4.0
+ Added the yield_records argument.
+ """
+
+ def __init__(self, runlevel, yield_records = False):
+ # TODO: At least in python 2.6 logging.Handler has a bug in that it doesn't
+ # extend object, causing our super() call to fail. When we drop python 2.6
+ # support we should switch back to using super() instead.
+ #
+ # super(LogBuffer, self).__init__(level = logging_level(runlevel))
+
+ logging.Handler.__init__(self, level = logging_level(runlevel))
+
+ self.formatter = logging.Formatter(
+ fmt = '%(asctime)s [%(levelname)s] %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S')
+
+ self._buffer = []
+ self._yield_records = yield_records
+
+ def is_empty(self):
+ return not bool(self._buffer)
+
+ def __iter__(self):
+ while self._buffer:
+ record = self._buffer.pop(0)
+ yield record if self._yield_records else self.formatter.format(record)
+
+ def emit(self, record):
+ self._buffer.append(record)
+
+
+class _StdoutLogger(logging.Handler):
+ def __init__(self, runlevel):
+ logging.Handler.__init__(self, level = logging_level(runlevel))
+
+ self.formatter = logging.Formatter(
+ fmt = '%(asctime)s [%(levelname)s] %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S')
+
+ def emit(self, record):
+ print(self.formatter.format(record))
+
+
+def log_to_stdout(runlevel):
+ """
+ Logs further events to stdout.
+
+ :param stem.util.log.Runlevel runlevel: minimum runlevel a message needs to be to be logged
+ """
+
+ get_logger().addHandler(_StdoutLogger(runlevel))
diff --git a/ext/stem/util/lru_cache.py b/ext/stem/util/lru_cache.py
new file mode 100644
index 00000000..011d4456
--- /dev/null
+++ b/ext/stem/util/lru_cache.py
@@ -0,0 +1,182 @@
+# Drop in replace for python 3.2's collections.lru_cache, from...
+# http://code.activestate.com/recipes/578078-py26-and-py30-backport-of-python-33s-lru-cache/
+#
+# ... which is under the MIT license. Stem users should *not* rely upon this
+# module. It will be removed when we drop support for python 3.2 and below.
+
+"""
+Memoization decorator that caches a function's return value. If later called
+with the same arguments then the cached value is returned rather than
+reevaluated.
+
+This is a a python 2.x port of `functools.lru_cache
+<http://docs.python.org/3/library/functools.html#functools.lru_cache>`_. If
+using python 3.2 or later you should use that instead.
+"""
+
+from collections import namedtuple
+from functools import update_wrapper
+from threading import RLock
+
+_CacheInfo = namedtuple('CacheInfo', ['hits', 'misses', 'maxsize', 'currsize'])
+
+
+class _HashedSeq(list):
+ __slots__ = 'hashvalue'
+
+ def __init__(self, tup, hash=hash):
+ self[:] = tup
+ self.hashvalue = hash(tup)
+
+ def __hash__(self):
+ return self.hashvalue
+
+
+def _make_key(args, kwds, typed,
+ kwd_mark = (object(),),
+ fasttypes = set([int, str, frozenset, type(None)]),
+ sorted=sorted, tuple=tuple, type=type, len=len):
+ 'Make a cache key from optionally typed positional and keyword arguments'
+ key = args
+ if kwds:
+ sorted_items = sorted(kwds.items())
+ key += kwd_mark
+ for item in sorted_items:
+ key += item
+ if typed:
+ key += tuple(type(v) for v in args)
+ if kwds:
+ key += tuple(type(v) for k, v in sorted_items)
+ elif len(key) == 1 and type(key[0]) in fasttypes:
+ return key[0]
+ return _HashedSeq(key)
+
+
+def lru_cache(maxsize=100, typed=False):
+ """Least-recently-used cache decorator.
+
+ If *maxsize* is set to None, the LRU features are disabled and the cache
+ can grow without bound.
+
+ If *typed* is True, arguments of different types will be cached separately.
+ For example, f(3.0) and f(3) will be treated as distinct calls with
+ distinct results.
+
+ Arguments to the cached function must be hashable.
+
+ View the cache statistics named tuple (hits, misses, maxsize, currsize) with
+ f.cache_info(). Clear the cache and statistics with f.cache_clear().
+ Access the underlying function with f.__wrapped__.
+
+ See: http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used
+
+ """
+
+ # Users should only access the lru_cache through its public API:
+ # cache_info, cache_clear, and f.__wrapped__
+ # The internals of the lru_cache are encapsulated for thread safety and
+ # to allow the implementation to change (including a possible C version).
+
+ def decorating_function(user_function):
+
+ cache = dict()
+ stats = [0, 0] # make statistics updateable non-locally
+ HITS, MISSES = 0, 1 # names for the stats fields
+ make_key = _make_key
+ cache_get = cache.get # bound method to lookup key or return None
+ _len = len # localize the global len() function
+ lock = RLock() # because linkedlist updates aren't threadsafe
+ root = [] # root of the circular doubly linked list
+ root[:] = [root, root, None, None] # initialize by pointing to self
+ nonlocal_root = [root] # make updateable non-locally
+ PREV, NEXT, KEY, RESULT = 0, 1, 2, 3 # names for the link fields
+
+ if maxsize == 0:
+
+ def wrapper(*args, **kwds):
+ # no caching, just do a statistics update after a successful call
+ result = user_function(*args, **kwds)
+ stats[MISSES] += 1
+ return result
+
+ elif maxsize is None:
+
+ def wrapper(*args, **kwds):
+ # simple caching without ordering or size limit
+ key = make_key(args, kwds, typed)
+ result = cache_get(key, root) # root used here as a unique not-found sentinel
+ if result is not root:
+ stats[HITS] += 1
+ return result
+ result = user_function(*args, **kwds)
+ cache[key] = result
+ stats[MISSES] += 1
+ return result
+
+ else:
+
+ def wrapper(*args, **kwds):
+ # size limited caching that tracks accesses by recency
+ key = make_key(args, kwds, typed) if kwds or typed else args
+ with lock:
+ link = cache_get(key)
+ if link is not None:
+ # record recent use of the key by moving it to the front of the list
+ root, = nonlocal_root
+ link_prev, link_next, key, result = link
+ link_prev[NEXT] = link_next
+ link_next[PREV] = link_prev
+ last = root[PREV]
+ last[NEXT] = root[PREV] = link
+ link[PREV] = last
+ link[NEXT] = root
+ stats[HITS] += 1
+ return result
+ result = user_function(*args, **kwds)
+ with lock:
+ root, = nonlocal_root
+ if key in cache:
+ # getting here means that this same key was added to the
+ # cache while the lock was released. since the link
+ # update is already done, we need only return the
+ # computed result and update the count of misses.
+ pass
+ elif _len(cache) >= maxsize:
+ # use the old root to store the new key and result
+ oldroot = root
+ oldroot[KEY] = key
+ oldroot[RESULT] = result
+ # empty the oldest link and make it the new root
+ root = nonlocal_root[0] = oldroot[NEXT]
+ oldkey = root[KEY]
+ root[KEY] = root[RESULT] = None
+ # now update the cache dictionary for the new links
+ del cache[oldkey]
+ cache[key] = oldroot
+ else:
+ # put result in a new link at the front of the list
+ last = root[PREV]
+ link = [last, root, key, result]
+ last[NEXT] = root[PREV] = cache[key] = link
+ stats[MISSES] += 1
+ return result
+
+ def cache_info():
+ """Report cache statistics"""
+ with lock:
+ return _CacheInfo(stats[HITS], stats[MISSES], maxsize, len(cache))
+
+ def cache_clear():
+ """Clear the cache and cache statistics"""
+ with lock:
+ cache.clear()
+ root = nonlocal_root[0]
+ root[:] = [root, root, None, None]
+ stats[:] = [0, 0]
+
+ wrapper.__wrapped__ = user_function
+ wrapper.cache_info = cache_info
+ wrapper.cache_clear = cache_clear
+ return update_wrapper(wrapper, user_function)
+
+ return decorating_function
diff --git a/ext/stem/util/ordereddict.py b/ext/stem/util/ordereddict.py
new file mode 100644
index 00000000..07c7d4ea
--- /dev/null
+++ b/ext/stem/util/ordereddict.py
@@ -0,0 +1,133 @@
+# Drop in replacement for python 2.7's OrderedDict, from...
+# http://pypi.python.org/pypi/ordereddict
+#
+# Stem users should *not* rely upon this module. It will be removed when we
+# drop support for python 2.6 and below.
+
+# Copyright (c) 2009 Raymond Hettinger
+#
+# Permission is hereby granted, free of charge, to any person
+# obtaining a copy of this software and associated documentation files
+# (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge,
+# publish, distribute, sublicense, and/or sell copies of the Software,
+# and to permit persons to whom the Software is furnished to do so,
+# subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+
+from UserDict import DictMixin
+
+
+class OrderedDict(dict, DictMixin):
+ def __init__(self, *args, **kwds):
+ if len(args) > 1:
+ raise TypeError('expected at most 1 arguments, got %d' % len(args))
+ try:
+ self.__end
+ except AttributeError:
+ self.clear()
+ self.update(*args, **kwds)
+
+ def clear(self):
+ self.__end = end = []
+ end += [None, end, end] # sentinel node for doubly linked list
+ self.__map = {} # key --> [key, prev, next]
+ dict.clear(self)
+
+ def __setitem__(self, key, value):
+ if key not in self:
+ end = self.__end
+ curr = end[1]
+ curr[2] = end[1] = self.__map[key] = [key, curr, end]
+ dict.__setitem__(self, key, value)
+
+ def __delitem__(self, key):
+ dict.__delitem__(self, key)
+ key, prev, next = self.__map.pop(key)
+ prev[2] = next
+ next[1] = prev
+
+ def __iter__(self):
+ end = self.__end
+ curr = end[2]
+ while curr is not end:
+ yield curr[0]
+ curr = curr[2]
+
+ def __reversed__(self):
+ end = self.__end
+ curr = end[1]
+ while curr is not end:
+ yield curr[0]
+ curr = curr[1]
+
+ def popitem(self, last=True):
+ if not self:
+ raise KeyError('dictionary is empty')
+ if last:
+ key = reversed(self).next()
+ else:
+ key = iter(self).next()
+ value = self.pop(key)
+ return key, value
+
+ def __reduce__(self):
+ items = [[k, self[k]] for k in self]
+ tmp = self.__map, self.__end
+ del self.__map, self.__end
+ inst_dict = vars(self).copy()
+ self.__map, self.__end = tmp
+ if inst_dict:
+ return (self.__class__, (items,), inst_dict)
+ return self.__class__, (items,)
+
+ def keys(self):
+ return list(self)
+
+ setdefault = DictMixin.setdefault
+ update = DictMixin.update
+ pop = DictMixin.pop
+ values = DictMixin.values
+ items = DictMixin.items
+ iterkeys = DictMixin.iterkeys
+ itervalues = DictMixin.itervalues
+ iteritems = DictMixin.iteritems
+
+ def __repr__(self):
+ if not self:
+ return '%s()' % (self.__class__.__name__,)
+ return '%s(%r)' % (self.__class__.__name__, self.items())
+
+ def copy(self):
+ return self.__class__(self)
+
+ @classmethod
+ def fromkeys(cls, iterable, value=None):
+ d = cls()
+ for key in iterable:
+ d[key] = value
+ return d
+
+ def __eq__(self, other):
+ if isinstance(other, OrderedDict):
+ if len(self) != len(other):
+ return False
+ for p, q in zip(self.items(), other.items()):
+ if p != q:
+ return False
+ return True
+ return dict.__eq__(self, other)
+
+ def __ne__(self, other):
+ return not self == other
diff --git a/ext/stem/util/ports.cfg b/ext/stem/util/ports.cfg
new file mode 100644
index 00000000..8b7829ed
--- /dev/null
+++ b/ext/stem/util/ports.cfg
@@ -0,0 +1,313 @@
+################################################################################
+#
+# Common usages for port . This is based on...
+#
+# https://secure.wikimedia.org/wikipedia/en/wiki/List_of_TCP_and_UDP_port numbers
+# http://isc.sans.edu/services.html
+#
+################################################################################
+
+port 1 => TCPMUX
+port 2 => CompressNET
+port 3 => CompressNET
+port 5 => RJE
+port 7 => Echo
+port 9 => Discard
+port 11 => SYSTAT
+port 13 => Daytime
+port 15 => netstat
+port 17 => QOTD
+port 18 => MSP
+port 19 => CHARGEN
+port 20 => FTP
+port 21 => FTP
+port 22 => SSH
+port 23 => Telnet
+port 24 => Priv-mail
+port 25 => SMTP
+port 34 => RF
+port 35 => Printer
+port 37 => TIME
+port 39 => RLP
+port 41 => Graphics
+port 42 => WINS
+port 43 => WHOIS
+port 47 => NI FTP
+port 49 => TACACS
+port 50 => Remote Mail
+port 51 => IMP
+port 52 => XNS
+port 53 => DNS
+port 54 => XNS
+port 55 => ISI-GL
+port 56 => RAP
+port 57 => MTP
+port 58 => XNS
+port 67 => BOOTP
+port 68 => BOOTP
+port 69 => TFTP
+port 70 => Gopher
+port 79 => Finger
+port 80 => HTTP
+port 81 => HTTP Alternate
+port 82 => Torpark
+port 83 => MIT ML
+port 88 => Kerberos
+port 90 => dnsix
+port 99 => WIP
+port 101 => NIC
+port 102 => ISO-TSAP
+port 104 => ACR/NEMA
+port 105 => CCSO
+port 107 => Telnet
+port 108 => SNA
+port 109 => POP2
+port 110 => POP3
+port 111 => ONC RPC
+port 113 => ident
+port 115 => SFTP
+port 117 => UUCP
+port 118 => SQL
+port 119 => NNTP
+port 123 => NTP
+port 135 => DCE
+port 137 => NetBIOS
+port 138 => NetBIOS
+port 139 => NetBIOS
+port 143 => IMAP
+port 152 => BFTP
+port 153 => SGMP
+port 156 => SQL
+port 158 => DMSP
+port 161 => SNMP
+port 162 => SNMPTRAP
+port 170 => Print-srv
+port 177 => XDMCP
+port 179 => BGP
+port 194 => IRC
+port 199 => SMUX
+port 201 => AppleTalk
+port 209 => QMTP
+port 210 => ANSI
+port 213 => IPX
+port 218 => MPP
+port 220 => IMAP
+port 256 => 2DEV
+port 259 => ESRO
+port 264 => BGMP
+port 308 => Novastor
+port 311 => OSX Admin
+port 318 => PKIX TSP
+port 319 => PTP
+port 320 => PTP
+port 323 => IMMP
+port 350 => MATIP
+port 351 => MATIP
+port 366 => ODMR
+port 369 => Rpc2port ap
+port 370 => codaauth2
+port 371 => ClearCase
+port 383 => HP Alarm Mgr
+port 384 => ARNS
+port 387 => AURP
+port 389 => LDAP
+port 401 => UPS
+port 402 => Altiris
+port 427 => SLP
+port 443 => HTTPS
+port 444 => SNPP
+port 445 => SMB
+port 464 => Kerberos (kpasswd)
+port 465 => SMTP
+port 475 => tcpnethaspsrv
+port 497 => Retrospect
+port 500 => ISAKMP
+port 501 => STMF
+port 502 => Modbus
+port 504 => Citadel
+port 510 => FirstClass
+port 512 => Rexec
+port 513 => rlogin
+port 514 => rsh
+port 515 => LPD
+port 517 => Talk
+port 518 => NTalk
+port 520 => efs
+port 524 => NCP
+port 530 => RPC
+port 531 => AIM/IRC
+port 532 => netnews
+port 533 => netwall
+port 540 => UUCP
+port 542 => commerce
+port 543 => Kerberos (klogin)
+port 544 => Kerberos (kshell)
+port 545 => OSISoft PI
+port 546 => DHCPv6
+port 547 => DHCPv6
+port 548 => AFP
+port 550 => new-who
+port 554 => RTSP
+port 556 => RFS
+port 560 => rmonitor
+port 561 => monitor
+port 563 => NNTPS
+port 587 => SMTP
+port 591 => FileMaker
+port 593 => HTTP RPC
+port 604 => TUNNEL
+port 623 => ASF-RMCP
+port 631 => CUPS
+port 635 => RLZ DBase
+port 636 => LDAPS
+port 639 => MSDP
+port 641 => Support oft
+port 646 => LDP
+port 647 => DHCP
+port 648 => RRP
+port 651 => IEEE-MMS
+port 652 => DTCP
+port 653 => Support oft
+port 654 => MMS/MMP
+port 657 => RMC
+port 660 => OSX Admin
+port 665 => sun-dr
+port 666 => Doom
+port 674 => ACAP
+port 691 => MS Exchange
+port 692 => Hyperwave-ISP
+port 694 => Linux-HA
+port 695 => IEEE-MMS-SSL
+port 698 => OLSR
+port 699 => Access Network
+port 700 => EPP
+port 701 => LMP
+port 702 => IRIS
+port 706 => SILC
+port 711 => MPLS
+port 712 => TBRPF
+port 720 => SMQP
+port 749 => Kerberos (admin)
+port 750 => rfile
+port 751 => pump
+port 752 => qrh
+port 753 => rrh
+port 754 => tell send
+port 760 => ns
+port 782 => Conserver
+port 783 => spamd
+port 829 => CMP
+port 843 => Flash
+port 847 => DHCP
+port 860 => iSCSI
+port 873 => rsync
+port 888 => CDDB
+port 901 => SWAT
+port 902-904 => VMware
+port 911 => NCA
+port 953 => DNS RNDC
+port 981 => SofaWare Firewall
+port 989 => FTPS
+port 990 => FTPS
+port 991 => NAS
+port 992 => Telnets
+port 993 => IMAPS
+port 994 => IRCS
+port 995 => POP3S
+port 999 => ScimoreDB
+port 1001 => JtoMB
+port 1002 => cogbot
+
+port 1080 => SOCKS
+port 1085 => WebObjects
+port 1109 => KPOP
+port 1169 => Tripwire
+port 1194 => OpenVPN
+port 1214 => Kazaa
+port 1220 => QuickTime
+port 1234 => VLC
+port 1241 => Nessus
+port 1270 => SCOM
+port 1293 => IPSec
+port 1433 => MSSQL
+port 1434 => MSSQL
+port 1500 => NetGuard
+port 1503 => MSN
+port 1512 => WINS
+port 1521 => Oracle
+port 1526 => Oracle
+port 1533 => Sametime
+port 1666 => Perforce
+port 1677 => GroupWise
+port 1723 => PPTP
+port 1725 => Steam
+port 1863 => MSNP
+port 2049 => NFS
+port 2082 => Infowave
+port 2083 => radsec
+port 2086 => GNUnet
+port 2087 => ELI
+port 2095 => NBX SER
+port 2096 => NBX DIR
+port 2102-2104 => Zephyr
+port 2401 => CVS
+port 2525 => SMTP
+port 2710 => BitTorrent
+port 3074 => XBox LIVE
+port 3101 => BlackBerry
+port 3128 => SQUID
+port 3306 => MySQL
+port 3389 => WBT
+port 3690 => SVN
+port 3723 => Battle.net
+port 3724 => WoW
+port 4321 => RWHOIS
+port 4643 => Virtuozzo
+port 4662 => eMule
+port 5003 => FileMaker
+port 5050 => Yahoo IM
+port 5060 => SIP
+port 5061 => SIP
+port 5190 => AIM/ICQ
+port 5222 => Jabber
+port 5223 => Jabber
+port 5228 => Android Market
+port 5269 => Jabber
+port 5298 => Jabber
+port 5432 => PostgreSQL
+port 5500 => VNC
+port 5556 => Freeciv
+port 5666 => NRPE
+port 5667 => NSCA
+port 5800 => VNC
+port 5900 => VNC
+port 6346 => gnutella
+port 6347 => gnutella
+port 6660-6669 => IRC
+port 6679 => IRC
+port 6697 => IRC
+port 6881-6999 => BitTorrent
+port 8000 => iRDMI
+port 8008 => HTTP Alternate
+port 8010 => XMPP
+port 8074 => Gadu-Gadu
+port 8080 => HTTP Proxy
+port 8087 => SPP
+port 8088 => Radan HTTP
+port 8118 => Privoxy
+port 8123 => Polipo
+port 8332-8333 => Bitcoin
+port 8443 => PCsync HTTPS
+port 8888 => NewsEDGE
+port 9030 => Tor
+port 9050 => Tor
+port 9051 => Tor
+port 9418 => Git
+port 9999 => distinct
+port 10000 => Webmin
+port 19294 => Google Voice
+port 19638 => Ensim
+port 23399 => Skype
+port 30301 => BitTorrent
+port 33434 => traceroute
+
diff --git a/ext/stem/util/proc.py b/ext/stem/util/proc.py
new file mode 100644
index 00000000..e4a826ed
--- /dev/null
+++ b/ext/stem/util/proc.py
@@ -0,0 +1,547 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Helper functions for querying process and system information from the /proc
+contents. Fetching information this way provides huge performance benefits
+over lookups via system utilities (ps, netstat, etc). For instance, resolving
+connections this way cuts the runtime by around 90% verses the alternatives.
+These functions may not work on all platforms (only Linux?).
+
+The method for reading these files (and a little code) are borrowed from
+`psutil <https://code.google.com/p/psutil/>`_, which was written by Jay Loden,
+Dave Daeschler, Giampaolo Rodola' and is under the BSD license.
+
+**These functions are not being vended to stem users. They may change in the
+future, use them at your own risk.**
+
+.. versionchanged:: 1.3.0
+ Dropped the get_* prefix from several function names. The old names still
+ work, but are deprecated aliases.
+
+**Module Overview:**
+
+::
+
+ is_available - checks if proc utilities can be used on this system
+ system_start_time - unix timestamp for when the system started
+ physical_memory - memory available on this system
+ cwd - provides the current working directory for a process
+ uid - provides the user id a process is running under
+ memory_usage - provides the memory usage of a process
+ stats - queries statistics about a process
+ file_descriptors_used - number of file descriptors used by a process
+ connections - provides the connections made by a process
+
+.. data:: Stat (enum)
+
+ Types of data available via the :func:`~stem.util.proc.stats` function.
+
+ ============== ===========
+ Stat Description
+ ============== ===========
+ **COMMAND** command name under which the process is running
+ **CPU_UTIME** total user time spent on the process
+ **CPU_STIME** total system time spent on the process
+ **START_TIME** when this process began, in unix time
+ ============== ===========
+"""
+
+import base64
+import os
+import platform
+import socket
+import sys
+import time
+
+import stem.util.enum
+
+from stem.util import log
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+# os.sysconf is only defined on unix
+try:
+ CLOCK_TICKS = os.sysconf(os.sysconf_names['SC_CLK_TCK'])
+except AttributeError:
+ CLOCK_TICKS = None
+
+Stat = stem.util.enum.Enum(
+ ('COMMAND', 'command'), ('CPU_UTIME', 'utime'),
+ ('CPU_STIME', 'stime'), ('START_TIME', 'start time')
+)
+
+
+@lru_cache()
+def is_available():
+ """
+ Checks if proc information is available on this platform.
+
+ :returns: **True** if proc contents exist on this platform, **False** otherwise
+ """
+
+ if platform.system() != 'Linux':
+ return False
+ else:
+ # list of process independent proc paths we use
+ proc_paths = ('/proc/stat', '/proc/meminfo', '/proc/net/tcp', '/proc/net/udp')
+
+ for path in proc_paths:
+ if not os.path.exists(path):
+ return False
+
+ return True
+
+
+@lru_cache()
+def system_start_time():
+ """
+ Provides the unix time (seconds since epoch) when the system started.
+
+ :returns: **float** for the unix time of when the system started
+
+ :raises: **IOError** if it can't be determined
+ """
+
+ start_time, parameter = time.time(), 'system start time'
+ btime_line = _get_line('/proc/stat', 'btime', parameter)
+
+ try:
+ result = float(btime_line.strip().split()[1])
+ _log_runtime(parameter, '/proc/stat[btime]', start_time)
+ return result
+ except:
+ exc = IOError('unable to parse the /proc/stat btime entry: %s' % btime_line)
+ _log_failure(parameter, exc)
+ raise exc
+
+
+@lru_cache()
+def physical_memory():
+ """
+ Provides the total physical memory on the system in bytes.
+
+ :returns: **int** for the bytes of physical memory this system has
+
+ :raises: **IOError** if it can't be determined
+ """
+
+ start_time, parameter = time.time(), 'system physical memory'
+ mem_total_line = _get_line('/proc/meminfo', 'MemTotal:', parameter)
+
+ try:
+ result = int(mem_total_line.split()[1]) * 1024
+ _log_runtime(parameter, '/proc/meminfo[MemTotal]', start_time)
+ return result
+ except:
+ exc = IOError('unable to parse the /proc/meminfo MemTotal entry: %s' % mem_total_line)
+ _log_failure(parameter, exc)
+ raise exc
+
+
+def cwd(pid):
+ """
+ Provides the current working directory for the given process.
+
+ :param int pid: process id of the process to be queried
+
+ :returns: **str** with the path of the working directory for the process
+
+ :raises: **IOError** if it can't be determined
+ """
+
+ start_time, parameter = time.time(), 'cwd'
+ proc_cwd_link = '/proc/%s/cwd' % pid
+
+ if pid == 0:
+ cwd = ''
+ else:
+ try:
+ cwd = os.readlink(proc_cwd_link)
+ except OSError:
+ exc = IOError('unable to read %s' % proc_cwd_link)
+ _log_failure(parameter, exc)
+ raise exc
+
+ _log_runtime(parameter, proc_cwd_link, start_time)
+ return cwd
+
+
+def uid(pid):
+ """
+ Provides the user ID the given process is running under.
+
+ :param int pid: process id of the process to be queried
+
+ :returns: **int** with the user id for the owner of the process
+
+ :raises: **IOError** if it can't be determined
+ """
+
+ start_time, parameter = time.time(), 'uid'
+ status_path = '/proc/%s/status' % pid
+ uid_line = _get_line(status_path, 'Uid:', parameter)
+
+ try:
+ result = int(uid_line.split()[1])
+ _log_runtime(parameter, '%s[Uid]' % status_path, start_time)
+ return result
+ except:
+ exc = IOError('unable to parse the %s Uid entry: %s' % (status_path, uid_line))
+ _log_failure(parameter, exc)
+ raise exc
+
+
+def memory_usage(pid):
+ """
+ Provides the memory usage in bytes for the given process.
+
+ :param int pid: process id of the process to be queried
+
+ :returns: **tuple** of two ints with the memory usage of the process, of the
+ form **(resident_size, virtual_size)**
+
+ :raises: **IOError** if it can't be determined
+ """
+
+ # checks if this is the kernel process
+
+ if pid == 0:
+ return (0, 0)
+
+ start_time, parameter = time.time(), 'memory usage'
+ status_path = '/proc/%s/status' % pid
+ mem_lines = _get_lines(status_path, ('VmRSS:', 'VmSize:'), parameter)
+
+ try:
+ residentSize = int(mem_lines['VmRSS:'].split()[1]) * 1024
+ virtualSize = int(mem_lines['VmSize:'].split()[1]) * 1024
+
+ _log_runtime(parameter, '%s[VmRSS|VmSize]' % status_path, start_time)
+ return (residentSize, virtualSize)
+ except:
+ exc = IOError('unable to parse the %s VmRSS and VmSize entries: %s' % (status_path, ', '.join(mem_lines)))
+ _log_failure(parameter, exc)
+ raise exc
+
+
+def stats(pid, *stat_types):
+ """
+ Provides process specific information. See the :data:`~stem.util.proc.Stat`
+ enum for valid options.
+
+ :param int pid: process id of the process to be queried
+ :param Stat stat_types: information to be provided back
+
+ :returns: **tuple** with all of the requested statistics as strings
+
+ :raises: **IOError** if it can't be determined
+ """
+
+ if CLOCK_TICKS is None:
+ raise IOError('Unable to look up SC_CLK_TCK')
+
+ start_time, parameter = time.time(), 'process %s' % ', '.join(stat_types)
+
+ # the stat file contains a single line, of the form...
+ # 8438 (tor) S 8407 8438 8407 34818 8438 4202496...
+ stat_path = '/proc/%s/stat' % pid
+ stat_line = _get_line(stat_path, str(pid), parameter)
+
+ # breaks line into component values
+ stat_comp = []
+ cmd_start, cmd_end = stat_line.find('('), stat_line.find(')')
+
+ if cmd_start != -1 and cmd_end != -1:
+ stat_comp.append(stat_line[:cmd_start])
+ stat_comp.append(stat_line[cmd_start + 1:cmd_end])
+ stat_comp += stat_line[cmd_end + 1:].split()
+
+ if len(stat_comp) < 44 and _is_float(stat_comp[13], stat_comp[14], stat_comp[21]):
+ exc = IOError('stat file had an unexpected format: %s' % stat_path)
+ _log_failure(parameter, exc)
+ raise exc
+
+ results = []
+ for stat_type in stat_types:
+ if stat_type == Stat.COMMAND:
+ if pid == 0:
+ results.append('sched')
+ else:
+ results.append(stat_comp[1])
+ elif stat_type == Stat.CPU_UTIME:
+ if pid == 0:
+ results.append('0')
+ else:
+ results.append(str(float(stat_comp[13]) / CLOCK_TICKS))
+ elif stat_type == Stat.CPU_STIME:
+ if pid == 0:
+ results.append('0')
+ else:
+ results.append(str(float(stat_comp[14]) / CLOCK_TICKS))
+ elif stat_type == Stat.START_TIME:
+ if pid == 0:
+ return system_start_time()
+ else:
+ # According to documentation, starttime is in field 21 and the unit is
+ # jiffies (clock ticks). We divide it for clock ticks, then add the
+ # uptime to get the seconds since the epoch.
+ p_start_time = float(stat_comp[21]) / CLOCK_TICKS
+ results.append(str(p_start_time + system_start_time()))
+
+ _log_runtime(parameter, stat_path, start_time)
+ return tuple(results)
+
+
+def file_descriptors_used(pid):
+ """
+ Provides the number of file descriptors currently being used by a process.
+
+ .. versionadded:: 1.3.0
+
+ :param int pid: process id of the process to be queried
+
+ :returns: **int** of the number of file descriptors used
+
+ :raises: **IOError** if it can't be determined
+ """
+
+ try:
+ pid = int(pid)
+
+ if pid < 0:
+ raise IOError("Process pids can't be negative: %s" % pid)
+ except (ValueError, TypeError):
+ raise IOError('Process pid was non-numeric: %s' % pid)
+
+ try:
+ return len(os.listdir('/proc/%i/fd' % pid))
+ except Exception as exc:
+ raise IOError('Unable to check number of file descriptors used: %s' % exc)
+
+
+def connections(pid):
+ """
+ Queries connection related information from the proc contents. This provides
+ similar results to netstat, lsof, sockstat, and other connection resolution
+ utilities (though the lookup is far quicker).
+
+ :param int pid: process id of the process to be queried
+
+ :returns: A listing of connection tuples of the form **[(local_ipAddr1,
+ local_port1, foreign_ipAddr1, foreign_port1, protocol), ...]** (addresses
+ and protocols are strings and ports are ints)
+
+ :raises: **IOError** if it can't be determined
+ """
+
+ try:
+ pid = int(pid)
+
+ if pid < 0:
+ raise IOError("Process pids can't be negative: %s" % pid)
+ except (ValueError, TypeError):
+ raise IOError('Process pid was non-numeric: %s' % pid)
+
+ if pid == 0:
+ return []
+
+ # fetches the inode numbers for socket file descriptors
+
+ start_time, parameter = time.time(), 'process connections'
+ inodes = []
+
+ for fd in os.listdir('/proc/%s/fd' % pid):
+ fd_path = '/proc/%s/fd/%s' % (pid, fd)
+
+ try:
+ # File descriptor link, such as 'socket:[30899]'
+
+ fd_name = os.readlink(fd_path)
+
+ if fd_name.startswith('socket:['):
+ inodes.append(fd_name[8:-1])
+ except OSError as exc:
+ if not os.path.exists(fd_path):
+ continue # descriptors may shift while we're in the middle of iterating over them
+
+ # most likely couldn't be read due to permissions
+ exc = IOError('unable to determine file descriptor destination (%s): %s' % (exc, fd_path))
+ _log_failure(parameter, exc)
+ raise exc
+
+ if not inodes:
+ # unable to fetch any connections for this process
+ return []
+
+ # check for the connection information from the /proc/net contents
+
+ conn = []
+
+ for proc_file_path in ('/proc/net/tcp', '/proc/net/udp'):
+ try:
+ proc_file = open(proc_file_path)
+ proc_file.readline() # skip the first line
+
+ for line in proc_file:
+ _, l_addr, f_addr, status, _, _, _, _, _, inode = line.split()[:10]
+
+ if inode in inodes:
+ # if a tcp connection, skip if it isn't yet established
+ if proc_file_path.endswith('/tcp') and status != '01':
+ continue
+
+ local_ip, local_port = _decode_proc_address_encoding(l_addr)
+ foreign_ip, foreign_port = _decode_proc_address_encoding(f_addr)
+ protocol = proc_file_path[10:]
+ conn.append((local_ip, local_port, foreign_ip, foreign_port, protocol))
+
+ proc_file.close()
+ except IOError as exc:
+ exc = IOError("unable to read '%s': %s" % (proc_file_path, exc))
+ _log_failure(parameter, exc)
+ raise exc
+ except Exception as exc:
+ exc = IOError("unable to parse '%s': %s" % (proc_file_path, exc))
+ _log_failure(parameter, exc)
+ raise exc
+
+ _log_runtime(parameter, '/proc/net/[tcp|udp]', start_time)
+ return conn
+
+
+def _decode_proc_address_encoding(addr):
+ """
+ Translates an address entry in the /proc/net/* contents to a human readable
+ form (`reference <http://linuxdevcenter.com/pub/a/linux/2000/11/16/LinuxAdmin.html>`_,
+ for instance:
+
+ ::
+
+ "0500000A:0016" -> ("10.0.0.5", 22)
+
+ :param str addr: proc address entry to be decoded
+
+ :returns: **tuple** of the form **(addr, port)**, with addr as a string and port an int
+ """
+
+ ip, port = addr.split(':')
+
+ # the port is represented as a two-byte hexadecimal number
+ port = int(port, 16)
+
+ if sys.version_info >= (3,):
+ ip = ip.encode('ascii')
+
+ # The IPv4 address portion is a little-endian four-byte hexadecimal number.
+ # That is, the least significant byte is listed first, so we need to reverse
+ # the order of the bytes to convert it to an IP address.
+ #
+ # This needs to account for the endian ordering as per...
+ # http://code.google.com/p/psutil/issues/detail?id=201
+ # https://trac.torproject.org/projects/tor/ticket/4777
+
+ if sys.byteorder == 'little':
+ ip = socket.inet_ntop(socket.AF_INET, base64.b16decode(ip)[::-1])
+ else:
+ ip = socket.inet_ntop(socket.AF_INET, base64.b16decode(ip))
+
+ return (ip, port)
+
+
+def _is_float(*value):
+ try:
+ for v in value:
+ float(v)
+
+ return True
+ except ValueError:
+ return False
+
+
+def _get_line(file_path, line_prefix, parameter):
+ return _get_lines(file_path, (line_prefix, ), parameter)[line_prefix]
+
+
+def _get_lines(file_path, line_prefixes, parameter):
+ """
+ Fetches lines with the given prefixes from a file. This only provides back
+ the first instance of each prefix.
+
+ :param str file_path: path of the file to read
+ :param tuple line_prefixes: string prefixes of the lines to return
+ :param str parameter: description of the proc attribute being fetch
+
+ :returns: mapping of prefixes to the matching line
+
+ :raises: **IOError** if unable to read the file or can't find all of the prefixes
+ """
+
+ try:
+ remaining_prefixes = list(line_prefixes)
+ proc_file, results = open(file_path), {}
+
+ for line in proc_file:
+ if not remaining_prefixes:
+ break # found everything we're looking for
+
+ for prefix in remaining_prefixes:
+ if line.startswith(prefix):
+ results[prefix] = line
+ remaining_prefixes.remove(prefix)
+ break
+
+ proc_file.close()
+
+ if remaining_prefixes:
+ if len(remaining_prefixes) == 1:
+ msg = '%s did not contain a %s entry' % (file_path, remaining_prefixes[0])
+ else:
+ msg = '%s did not contain %s entries' % (file_path, ', '.join(remaining_prefixes))
+
+ raise IOError(msg)
+ else:
+ return results
+ except IOError as exc:
+ _log_failure(parameter, exc)
+ raise exc
+
+
+def _log_runtime(parameter, proc_location, start_time):
+ """
+ Logs a message indicating a successful proc query.
+
+ :param str parameter: description of the proc attribute being fetch
+ :param str proc_location: proc files we were querying
+ :param int start_time: unix time for when this query was started
+ """
+
+ runtime = time.time() - start_time
+ log.debug('proc call (%s): %s (runtime: %0.4f)' % (parameter, proc_location, runtime))
+
+
+def _log_failure(parameter, exc):
+ """
+ Logs a message indicating that the proc query failed.
+
+ :param str parameter: description of the proc attribute being fetch
+ :param Exception exc: exception that we're raising
+ """
+
+ log.debug('proc call failed (%s): %s' % (parameter, exc))
+
+# TODO: drop with stem 2.x
+# We renamed our methods to drop a redundant 'get_*' prefix, so alias the old
+# names for backward compatability.
+
+get_system_start_time = system_start_time
+get_physical_memory = physical_memory
+get_cwd = cwd
+get_uid = uid
+get_memory_usage = memory_usage
+get_stats = stats
+get_connections = connections
diff --git a/ext/stem/util/str_tools.py b/ext/stem/util/str_tools.py
new file mode 100644
index 00000000..497564e3
--- /dev/null
+++ b/ext/stem/util/str_tools.py
@@ -0,0 +1,558 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Toolkit for various string activity.
+
+.. versionchanged:: 1.3.0
+ Dropped the get_* prefix from several function names. The old names still
+ work, but are deprecated aliases.
+
+**Module Overview:**
+
+::
+
+ crop - shortens string to a given length
+
+ size_label - human readable label for a number of bytes
+ time_label - human readable label for a number of seconds
+ time_labels - human readable labels for each time unit
+ short_time_label - condensed time label output
+ parse_short_time_label - seconds represented by a short time label
+"""
+
+import codecs
+import datetime
+import re
+import sys
+
+import stem.prereq
+import stem.util.enum
+
+from stem import str_type
+
+# label conversion tuples of the form...
+# (bits / bytes / seconds, short label, long label)
+
+SIZE_UNITS_BITS = (
+ (140737488355328.0, ' Pb', ' Petabit'),
+ (137438953472.0, ' Tb', ' Terabit'),
+ (134217728.0, ' Gb', ' Gigabit'),
+ (131072.0, ' Mb', ' Megabit'),
+ (128.0, ' Kb', ' Kilobit'),
+ (0.125, ' b', ' Bit'),
+)
+
+SIZE_UNITS_BYTES = (
+ (1125899906842624.0, ' PB', ' Petabyte'),
+ (1099511627776.0, ' TB', ' Terabyte'),
+ (1073741824.0, ' GB', ' Gigabyte'),
+ (1048576.0, ' MB', ' Megabyte'),
+ (1024.0, ' KB', ' Kilobyte'),
+ (1.0, ' B', ' Byte'),
+)
+
+TIME_UNITS = (
+ (86400.0, 'd', ' day'),
+ (3600.0, 'h', ' hour'),
+ (60.0, 'm', ' minute'),
+ (1.0, 's', ' second'),
+)
+
+_timestamp_re = re.compile(r'(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})')
+
+if stem.prereq.is_python_3():
+ def _to_bytes_impl(msg):
+ if isinstance(msg, str):
+ return codecs.latin_1_encode(msg, 'replace')[0]
+ else:
+ return msg
+
+ def _to_unicode_impl(msg):
+ if msg is not None and not isinstance(msg, str):
+ return msg.decode('utf-8', 'replace')
+ else:
+ return msg
+else:
+ def _to_bytes_impl(msg):
+ if msg is not None and isinstance(msg, str_type):
+ return codecs.latin_1_encode(msg, 'replace')[0]
+ else:
+ return msg
+
+ def _to_unicode_impl(msg):
+ if msg is not None and not isinstance(msg, str_type):
+ return msg.decode('utf-8', 'replace')
+ else:
+ return msg
+
+
+def _to_bytes(msg):
+ """
+ Provides the ASCII bytes for the given string. This is purely to provide
+ python 3 compatability, normalizing the unicode/ASCII change in the version
+ bump. For an explanation of this see...
+
+ http://python3porting.com/problems.html#nicer-solutions
+
+ :param str,unicode msg: string to be converted
+
+ :returns: ASCII bytes for string
+ """
+
+ return _to_bytes_impl(msg)
+
+
+def _to_unicode(msg):
+ """
+ Provides the unicode string for the given ASCII bytes. This is purely to
+ provide python 3 compatability, normalizing the unicode/ASCII change in the
+ version bump.
+
+ :param str,unicode msg: string to be converted
+
+ :returns: unicode conversion
+ """
+
+ return _to_unicode_impl(msg)
+
+
+def _to_camel_case(label, divider = '_', joiner = ' '):
+ """
+ Converts the given string to camel case, ie:
+
+ ::
+
+ >>> _to_camel_case('I_LIKE_PEPPERJACK!')
+ 'I Like Pepperjack!'
+
+ :param str label: input string to be converted
+ :param str divider: word boundary
+ :param str joiner: replacement for word boundaries
+
+ :returns: camel cased string
+ """
+
+ words = []
+ for entry in label.split(divider):
+ if len(entry) == 0:
+ words.append('')
+ elif len(entry) == 1:
+ words.append(entry.upper())
+ else:
+ words.append(entry[0].upper() + entry[1:].lower())
+
+ return joiner.join(words)
+
+
+# This needs to be defined after _to_camel_case() to avoid a circular
+# dependency with the enum module.
+
+Ending = stem.util.enum.Enum('ELLIPSE', 'HYPHEN')
+
+
+def crop(msg, size, min_word_length = 4, min_crop = 0, ending = Ending.ELLIPSE, get_remainder = False):
+ """
+ Shortens a string to a given length.
+
+ If we crop content then a given ending is included (counting itself toward
+ the size limitation). This crops on word breaks so we only include a word if
+ we can display at least **min_word_length** characters of it.
+
+ If there isn't room for even a truncated single word (or one word plus the
+ ellipse if including those) then this provides an empty string.
+
+ If a cropped string ends with a comma or period then it's stripped (unless
+ we're providing the remainder back). For example...
+
+ >>> crop('This is a looooong message', 17)
+ 'This is a looo...'
+
+ >>> crop('This is a looooong message', 12)
+ 'This is a...'
+
+ >>> crop('This is a looooong message', 3)
+ ''
+
+ The whole point of this method is to provide human friendly croppings, and as
+ such details of how this works might change in the future. Callers should not
+ rely on the details of how this crops.
+
+ .. versionadded:: 1.3.0
+
+ :param str msg: text to be processed
+ :param int size: space available for text
+ :param int min_word_length: minimum characters before which a word is
+ dropped, requires whole word if **None**
+ :param int min_crop: minimum characters that must be dropped if a word is
+ cropped
+ :param Ending ending: type of ending used when truncating, no special
+ truncation is used if **None**
+ :param bool get_remainder: returns a tuple with the second part being the
+ cropped portion of the message
+
+ :returns: **str** of the text truncated to the given length
+ """
+
+ # checks if there's room for the whole message
+
+ if len(msg) <= size:
+ return (msg, '') if get_remainder else msg
+
+ if size < 0:
+ raise ValueError("Crop size can't be negative (received %i)" % size)
+ elif min_word_length and min_word_length < 0:
+ raise ValueError("Crop's min_word_length can't be negative (received %i)" % min_word_length)
+ elif min_crop < 0:
+ raise ValueError("Crop's min_crop can't be negative (received %i)" % min_crop)
+
+ # since we're cropping, the effective space available is less with an
+ # ellipse, and cropping words requires an extra space for hyphens
+
+ if ending == Ending.ELLIPSE:
+ size -= 3
+ elif min_word_length and ending == Ending.HYPHEN:
+ min_word_length += 1
+
+ if min_word_length is None:
+ min_word_length = sys.maxsize
+
+ # checks if there isn't the minimum space needed to include anything
+
+ last_wordbreak = msg.rfind(' ', 0, size + 1)
+
+ if last_wordbreak == -1:
+ # we're splitting the first word
+
+ if size < min_word_length:
+ return ('', msg) if get_remainder else ''
+
+ include_crop = True
+ else:
+ last_wordbreak = len(msg[:last_wordbreak].rstrip()) # drops extra ending whitespaces
+ include_crop = size - last_wordbreak - 1 >= min_word_length
+
+ # if there's a max crop size then make sure we're cropping at least that many characters
+
+ if include_crop and min_crop:
+ next_wordbreak = msg.find(' ', size)
+
+ if next_wordbreak == -1:
+ next_wordbreak = len(msg)
+
+ include_crop = next_wordbreak - size + 1 >= min_crop
+
+ if include_crop:
+ return_msg, remainder = msg[:size], msg[size:]
+
+ if ending == Ending.HYPHEN:
+ remainder = return_msg[-1] + remainder
+ return_msg = return_msg[:-1].rstrip() + '-'
+ else:
+ return_msg, remainder = msg[:last_wordbreak], msg[last_wordbreak:]
+
+ # if this is ending with a comma or period then strip it off
+
+ if not get_remainder and return_msg and return_msg[-1] in (',', '.'):
+ return_msg = return_msg[:-1]
+
+ if ending == Ending.ELLIPSE:
+ return_msg = return_msg.rstrip() + '...'
+
+ return (return_msg, remainder) if get_remainder else return_msg
+
+
+def size_label(byte_count, decimal = 0, is_long = False, is_bytes = True):
+ """
+ Converts a number of bytes into a human readable label in its most
+ significant units. For instance, 7500 bytes would return "7 KB". If the
+ is_long option is used this expands unit labels to be the properly pluralized
+ full word (for instance 'Kilobytes' rather than 'KB'). Units go up through
+ petabytes.
+
+ ::
+
+ >>> size_label(2000000)
+ '1 MB'
+
+ >>> size_label(1050, 2)
+ '1.02 KB'
+
+ >>> size_label(1050, 3, True)
+ '1.025 Kilobytes'
+
+ :param int byte_count: number of bytes to be converted
+ :param int decimal: number of decimal digits to be included
+ :param bool is_long: expands units label
+ :param bool is_bytes: provides units in bytes if **True**, bits otherwise
+
+ :returns: **str** with human readable representation of the size
+ """
+
+ if is_bytes:
+ return _get_label(SIZE_UNITS_BYTES, byte_count, decimal, is_long)
+ else:
+ return _get_label(SIZE_UNITS_BITS, byte_count, decimal, is_long)
+
+
+def time_label(seconds, decimal = 0, is_long = False):
+ """
+ Converts seconds into a time label truncated to its most significant units.
+ For instance, 7500 seconds would return "2h". Units go up through days.
+
+ This defaults to presenting single character labels, but if the is_long
+ option is used this expands labels to be the full word (space included and
+ properly pluralized). For instance, "4h" would be "4 hours" and "1m" would
+ become "1 minute".
+
+ ::
+
+ >>> time_label(10000)
+ '2h'
+
+ >>> time_label(61, 1, True)
+ '1.0 minute'
+
+ >>> time_label(61, 2, True)
+ '1.01 minutes'
+
+ :param int seconds: number of seconds to be converted
+ :param int decimal: number of decimal digits to be included
+ :param bool is_long: expands units label
+
+ :returns: **str** with human readable representation of the time
+ """
+
+ return _get_label(TIME_UNITS, seconds, decimal, is_long)
+
+
+def time_labels(seconds, is_long = False):
+ """
+ Provides a list of label conversions for each time unit, starting with its
+ most significant units on down. Any counts that evaluate to zero are omitted.
+ For example...
+
+ ::
+
+ >>> time_labels(400)
+ ['6m', '40s']
+
+ >>> time_labels(3640, True)
+ ['1 hour', '40 seconds']
+
+ :param int seconds: number of seconds to be converted
+ :param bool is_long: expands units label
+
+ :returns: **list** of strings with human readable representations of the time
+ """
+
+ time_labels = []
+
+ for count_per_unit, _, _ in TIME_UNITS:
+ if abs(seconds) >= count_per_unit:
+ time_labels.append(_get_label(TIME_UNITS, seconds, 0, is_long))
+ seconds %= count_per_unit
+
+ return time_labels
+
+
+def short_time_label(seconds):
+ """
+ Provides a time in the following format:
+ [[dd-]hh:]mm:ss
+
+ ::
+
+ >>> short_time_label(111)
+ '01:51'
+
+ >>> short_time_label(544100)
+ '6-07:08:20'
+
+ :param int seconds: number of seconds to be converted
+
+ :returns: **str** with the short representation for the time
+
+ :raises: **ValueError** if the input is negative
+ """
+
+ if seconds < 0:
+ raise ValueError("Input needs to be a non-negative integer, got '%i'" % seconds)
+
+ time_comp = {}
+
+ for amount, _, label in TIME_UNITS:
+ count = int(seconds / amount)
+ seconds %= amount
+ time_comp[label.strip()] = count
+
+ label = '%02i:%02i' % (time_comp['minute'], time_comp['second'])
+
+ if time_comp['day']:
+ label = '%i-%02i:%s' % (time_comp['day'], time_comp['hour'], label)
+ elif time_comp['hour']:
+ label = '%02i:%s' % (time_comp['hour'], label)
+
+ return label
+
+
+def parse_short_time_label(label):
+ """
+ Provides the number of seconds corresponding to the formatting used for the
+ cputime and etime fields of ps:
+ [[dd-]hh:]mm:ss or mm:ss.ss
+
+ ::
+
+ >>> parse_short_time_label('01:51')
+ 111
+
+ >>> parse_short_time_label('6-07:08:20')
+ 544100
+
+ :param str label: time entry to be parsed
+
+ :returns: **int** with the number of seconds represented by the label
+
+ :raises: **ValueError** if input is malformed
+ """
+
+ days, hours, minutes, seconds = '0', '0', '0', '0'
+
+ if '-' in label:
+ days, label = label.split('-', 1)
+
+ time_comp = label.split(':')
+
+ if len(time_comp) == 3:
+ hours, minutes, seconds = time_comp
+ elif len(time_comp) == 2:
+ minutes, seconds = time_comp
+ else:
+ raise ValueError("Invalid time format, we expected '[[dd-]hh:]mm:ss' or 'mm:ss.ss': %s" % label)
+
+ try:
+ time_sum = int(float(seconds))
+ time_sum += int(minutes) * 60
+ time_sum += int(hours) * 3600
+ time_sum += int(days) * 86400
+ return time_sum
+ except ValueError:
+ raise ValueError('Non-numeric value in time entry: %s' % label)
+
+
+def _parse_timestamp(entry):
+ """
+ Parses the date and time that in format like like...
+
+ ::
+
+ 2012-11-08 16:48:41
+
+ :param str entry: timestamp to be parsed
+
+ :returns: **datetime** for the time represented by the timestamp
+
+ :raises: **ValueError** if the timestamp is malformed
+ """
+
+ if not isinstance(entry, (str, str_type)):
+ raise ValueError('parse_timestamp() input must be a str, got a %s' % type(entry))
+
+ try:
+ time = [int(x) for x in _timestamp_re.match(entry).groups()]
+ except AttributeError:
+ raise ValueError('Expected timestamp in format YYYY-MM-DD HH:MM:ss but got ' + entry)
+
+ return datetime.datetime(time[0], time[1], time[2], time[3], time[4], time[5])
+
+
+def _parse_iso_timestamp(entry):
+ """
+ Parses the ISO 8601 standard that provides for timestamps like...
+
+ ::
+
+ 2012-11-08T16:48:41.420251
+
+ :param str entry: timestamp to be parsed
+
+ :returns: **datetime** for the time represented by the timestamp
+
+ :raises: **ValueError** if the timestamp is malformed
+ """
+
+ if not isinstance(entry, (str, str_type)):
+ raise ValueError('parse_iso_timestamp() input must be a str, got a %s' % type(entry))
+
+ # based after suggestions from...
+ # http://stackoverflow.com/questions/127803/how-to-parse-iso-formatted-date-in-python
+
+ if '.' in entry:
+ timestamp_str, microseconds = entry.split('.')
+ else:
+ timestamp_str, microseconds = entry, '000000'
+
+ if len(microseconds) != 6 or not microseconds.isdigit():
+ raise ValueError("timestamp's microseconds should be six digits")
+
+ if timestamp_str[10] == 'T':
+ timestamp_str = timestamp_str[:10] + ' ' + timestamp_str[11:]
+ else:
+ raise ValueError("timestamp didn't contain delimeter 'T' between date and time")
+
+ timestamp = _parse_timestamp(timestamp_str)
+ return timestamp + datetime.timedelta(microseconds = int(microseconds))
+
+
+def _get_label(units, count, decimal, is_long):
+ """
+ Provides label corresponding to units of the highest significance in the
+ provided set. This rounds down (ie, integer truncation after visible units).
+
+ :param tuple units: type of units to be used for conversion, containing
+ (count_per_unit, short_label, long_label)
+ :param int count: number of base units being converted
+ :param int decimal: decimal precision of label
+ :param bool is_long: uses the long label if **True**, short label otherwise
+ """
+
+ # formatted string for the requested number of digits
+ label_format = '%%.%if' % decimal
+
+ if count < 0:
+ label_format = '-' + label_format
+ count = abs(count)
+ elif count == 0:
+ units_label = units[-1][2] + 's' if is_long else units[-1][1]
+ return '%s%s' % (label_format % count, units_label)
+
+ for count_per_unit, short_label, long_label in units:
+ if count >= count_per_unit:
+ # Rounding down with a '%f' is a little clunky. Reducing the count so
+ # it'll divide evenly as the rounded down value.
+
+ count -= count % (count_per_unit / (10 ** decimal))
+ count_label = label_format % (count / count_per_unit)
+
+ if is_long:
+ # Pluralize if any of the visible units make it greater than one. For
+ # instance 1.0003 is plural but 1.000 isn't.
+
+ if decimal > 0:
+ is_plural = count > count_per_unit
+ else:
+ is_plural = count >= count_per_unit * 2
+
+ return count_label + long_label + ('s' if is_plural else '')
+ else:
+ return count_label + short_label
+
+# TODO: drop with stem 2.x
+# We renamed our methods to drop a redundant 'get_*' prefix, so alias the old
+# names for backward compatability.
+
+get_size_label = size_label
+get_time_label = time_label
+get_time_labels = time_labels
+get_short_time_label = short_time_label
diff --git a/ext/stem/util/system.py b/ext/stem/util/system.py
new file mode 100644
index 00000000..aa13cb98
--- /dev/null
+++ b/ext/stem/util/system.py
@@ -0,0 +1,1176 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Helper functions for working with the underlying system. These are mostly os
+dependent, only working on linux, osx, and bsd. In almost all cases they're
+best-effort, providing **None** if the lookup fails.
+
+.. versionchanged:: 1.3.0
+ Dropped the get_* prefix from several function names. The old names still
+ work, but are deprecated aliases.
+
+**Module Overview:**
+
+::
+
+ is_windows - checks if we're running on windows
+ is_mac - checks if we're running on a mac
+ is_gentoo - checks if we're running on gentoo
+ is_bsd - checks if we're running on the bsd family of operating systems
+
+ is_available - determines if a command is available on this system
+ is_running - determines if a given process is running
+ call - runs the given system command and provides back the results
+
+ name_by_pid - gets the name for a process by the given pid
+ pid_by_name - gets the pid for a process by the given name
+ pid_by_port - gets the pid for a process listening to a given port
+ pid_by_open_file - gets the pid for the process with an open file
+ cwd - provides the current working directory for a given process
+ user - provides the user a process is running under
+ start_time - provides the unix timestamp when the process started
+ tail - provides lines from the end of a file
+ bsd_jail_id - provides the BSD jail id a given process is running within
+ bsd_jail_path - provides the path of the given BSD jail
+
+ is_tarfile - checks if the given path is a tarball
+ expand_path - expands relative paths and ~ entries
+ files_with_suffix - provides files with the given suffix
+
+ get_process_name - provides our process' name
+ set_process_name - changes our process' name
+"""
+
+import ctypes
+import ctypes.util
+import distutils.spawn
+import mimetypes
+import os
+import platform
+import re
+import subprocess
+import tarfile
+import time
+
+import stem.util.proc
+import stem.util.str_tools
+
+from stem import UNDEFINED, str_type
+from stem.util import log
+
+# Mapping of commands to if they're available or not.
+
+CMD_AVAILABLE_CACHE = {}
+
+# An incomplete listing of commands provided by the shell. Expand this as
+# needed. Some noteworthy things about shell commands...
+#
+# * They're not in the path so is_available() will fail.
+# * subprocess.Popen() without the 'shell = True' argument will fail with...
+# OSError: [Errno 2] No such file or directory
+
+SHELL_COMMANDS = ['ulimit']
+
+IS_RUNNING_PS_LINUX = 'ps -A co command'
+IS_RUNNING_PS_BSD = 'ps -ao ucomm='
+GET_NAME_BY_PID_PS = 'ps -p %s -o comm'
+GET_PID_BY_NAME_PGREP = 'pgrep -x %s'
+GET_PID_BY_NAME_PIDOF = 'pidof %s'
+GET_PID_BY_NAME_PS_LINUX = 'ps -o pid -C %s'
+GET_PID_BY_NAME_PS_BSD = 'ps axc'
+GET_PID_BY_NAME_LSOF = 'lsof -tc %s'
+GET_PID_BY_PORT_NETSTAT = 'netstat -npltu'
+GET_PID_BY_PORT_SOCKSTAT = 'sockstat -4l -P tcp -p %s'
+GET_PID_BY_PORT_LSOF = 'lsof -wnP -iTCP -sTCP:LISTEN'
+GET_PID_BY_FILE_LSOF = 'lsof -tw %s'
+GET_CWD_PWDX = 'pwdx %s'
+GET_CWD_LSOF = 'lsof -a -p %s -d cwd -Fn'
+GET_BSD_JAIL_ID_PS = 'ps -p %s -o jid'
+GET_BSD_JAIL_PATH = 'jls -j %s'
+
+BLOCK_SIZE = 1024
+
+# flag for setting the process name, found in '/usr/include/linux/prctl.h'
+
+PR_SET_NAME = 15
+
+argc_t = ctypes.POINTER(ctypes.c_char_p)
+
+# The following can fail with pypy...
+# AttributeError: No symbol Py_GetArgcArgv found in library <None>
+
+try:
+ Py_GetArgcArgv = ctypes.pythonapi.Py_GetArgcArgv
+ Py_GetArgcArgv.restype = None
+ Py_GetArgcArgv.argtypes = [
+ ctypes.POINTER(ctypes.c_int),
+ ctypes.POINTER(argc_t),
+ ]
+except:
+ Py_GetArgcArgv = None
+
+# This is both a cache for get_process_name() and tracks what we've changed our
+# process name to.
+
+_PROCESS_NAME = None
+
+# Length of our original process name.
+#
+# The original author our process renaming is based on did a memset for 256,
+# while Jake did it for the original process name length (capped at 1608). I'm
+# not sure of the reasons for either of these limits, but setting it to
+# anything higher than our original name length should be pointless, so opting
+# for Jake's limit.
+
+_MAX_NAME_LENGTH = -1
+
+
+def is_windows():
+ """
+ Checks if we are running on Windows.
+
+ :returns: **bool** to indicate if we're on Windows
+ """
+
+ return platform.system() == 'Windows'
+
+
+def is_mac():
+ """
+ Checks if we are running on Mac OSX.
+
+ :returns: **bool** to indicate if we're on a Mac
+ """
+
+ return platform.system() == 'Darwin'
+
+
+def is_gentoo():
+ """
+ Checks if we're running on Gentoo.
+
+ :returns: **bool** to indicate if we're on Gentoo
+ """
+
+ return os.path.exists('/etc/gentoo-release')
+
+
+def is_bsd():
+ """
+ Checks if we are within the BSD family of operating systems. This currently
+ recognizes Macs, FreeBSD, and OpenBSD but may be expanded later.
+
+ :returns: **bool** to indicate if we're on a BSD OS
+ """
+
+ return platform.system() in ('Darwin', 'FreeBSD', 'OpenBSD')
+
+
+def is_available(command, cached=True):
+ """
+ Checks the current PATH to see if a command is available or not. If more
+ than one command is present (for instance "ls -a | grep foo") then this
+ just checks the first.
+
+ Note that shell (like cd and ulimit) aren't in the PATH so this lookup will
+ try to assume that it's available. This only happends for recognized shell
+ commands (those in SHELL_COMMANDS).
+
+ :param str command: command to search for
+ :param bool cached: makes use of available cached results if **True**
+
+ :returns: **True** if an executable we can use by that name exists in the
+ PATH, **False** otherwise
+ """
+
+ if ' ' in command:
+ command = command.split(' ')[0]
+
+ if command in SHELL_COMMANDS:
+ # we can't actually look it up, so hope the shell really provides it...
+
+ return True
+ elif cached and command in CMD_AVAILABLE_CACHE:
+ return CMD_AVAILABLE_CACHE[command]
+ else:
+ cmd_exists = distutils.spawn.find_executable(command) is not None
+ CMD_AVAILABLE_CACHE[command] = cmd_exists
+ return cmd_exists
+
+
+def is_running(command):
+ """
+ Checks for if a process with a given name is running or not.
+
+ :param str command: process name to be checked
+
+ :returns: **True** if the process is running, **False** if it's not among ps
+ results, and **None** if ps can't be queried
+ """
+
+ # Linux and the BSD families have different variants of ps. Guess based on
+ # the is_bsd() check which to try first, then fall back to the other.
+ #
+ # Linux
+ # -A - Select all processes.
+ # -co command - Shows just the base command.
+ #
+ # Mac / BSD
+ # -a - Display information about other users' processes as well as
+ # our own.
+ # -o ucomm= - Shows just the ucomm attribute ("name to be used for
+ # accounting")
+
+ if is_available('ps'):
+ if is_bsd():
+ primary_resolver = IS_RUNNING_PS_BSD
+ secondary_resolver = IS_RUNNING_PS_LINUX
+ else:
+ primary_resolver = IS_RUNNING_PS_LINUX
+ secondary_resolver = IS_RUNNING_PS_BSD
+
+ command_listing = call(primary_resolver, None)
+
+ if not command_listing:
+ command_listing = call(secondary_resolver, None)
+
+ if command_listing:
+ command_listing = map(str_type.strip, command_listing)
+ return command in command_listing
+
+ return None
+
+
+def name_by_pid(pid):
+ """
+ Attempts to determine the name a given process is running under (not
+ including arguments). This uses...
+
+ ::
+
+ 1. Information from /proc
+ 2. ps -p <pid> -o command
+
+ :param int pid: process id of the process to be queried
+
+ :returns: **str** with the process name, **None** if it can't be determined
+ """
+
+ process_name = None
+
+ if stem.util.proc.is_available():
+ try:
+ process_name = stem.util.proc.stats(pid, stem.util.proc.Stat.COMMAND)[0]
+ except IOError:
+ pass
+
+ # attempts to resolve using ps, failing if:
+ # - system's ps variant doesn't handle these flags (none known at the moment)
+ #
+ # example output:
+ # atagar@morrigan:~$ ps -p 5767 -o comm
+ # COMMAND
+ # vim
+
+ if not process_name:
+ try:
+ results = call(GET_NAME_BY_PID_PS % pid)
+ except OSError:
+ results = None
+
+ if results and len(results) == 2 and results[0] == 'COMMAND':
+ process_name = results[1].strip()
+
+ return process_name
+
+
+def pid_by_name(process_name, multiple = False):
+ """
+ Attempts to determine the process id for a running process, using...
+
+ ::
+
+ 1. pgrep -x <name>
+ 2. pidof <name>
+ 3. ps -o pid -C <name> (linux)
+ ps axc | egrep " <name>$" (bsd)
+ 4. lsof -tc <name>
+ 5. tasklist | str <name>.exe
+
+ :param str process_name: process name for which to fetch the pid
+ :param bool multiple: provides a list of all pids if **True**, otherwise
+ results with multiple processes are discarded
+
+ :returns:
+ Response depends upon the 'multiple' argument as follows...
+
+ * if **False** then this provides an **int** with the process id or **None** if it can't be determined
+ * if **True** then this provides a **list** of all **int** process ids, and an empty list if it can't be determined
+ """
+
+ # attempts to resolve using pgrep, failing if:
+ # - we're running on bsd (command unavailable)
+ #
+ # example output:
+ # atagar@morrigan:~$ pgrep -x vim
+ # 3283
+ # 3392
+
+ if is_available('pgrep'):
+ results = call(GET_PID_BY_NAME_PGREP % process_name, None)
+
+ if results:
+ try:
+ pids = list(map(int, results))
+
+ if multiple:
+ return pids
+ elif len(pids) == 1:
+ return pids[0]
+ except ValueError:
+ pass
+
+ # attempts to resolve using pidof, failing if:
+ # - we're running on bsd (command unavailable)
+ #
+ # example output:
+ # atagar@morrigan:~$ pidof vim
+ # 3392 3283
+
+ if is_available('pidof'):
+ results = call(GET_PID_BY_NAME_PIDOF % process_name, None)
+
+ if results and len(results) == 1:
+ try:
+ pids = list(map(int, results[0].split()))
+
+ if multiple:
+ return pids
+ elif len(pids) == 1:
+ return pids[0]
+ except ValueError:
+ pass
+
+ # attempts to resolve using ps, failing if:
+ # - system's ps variant doesn't handle these flags (none known at the moment)
+ #
+ # example output:
+ # atagar@morrigan:~/Desktop/stem$ ps -o pid -C vim
+ # PID
+ # 3283
+ # 3392
+ #
+ # atagar$ ps axc
+ # PID TT STAT TIME COMMAND
+ # 1 ?? Ss 9:00.22 launchd
+ # 10 ?? Ss 0:09.97 kextd
+ # 11 ?? Ss 5:47.36 DirectoryService
+ # 12 ?? Ss 3:01.44 notifyd
+
+ if is_available('ps'):
+ if not is_bsd():
+ # linux variant of ps
+ results = call(GET_PID_BY_NAME_PS_LINUX % process_name, None)
+
+ if results:
+ try:
+ pids = list(map(int, results[1:]))
+
+ if multiple:
+ return pids
+ elif len(pids) == 1:
+ return pids[0]
+ except ValueError:
+ pass
+
+ if is_bsd():
+ # bsd variant of ps
+ results = call(GET_PID_BY_NAME_PS_BSD, None)
+
+ if results:
+ # filters results to those with our process name
+ results = [r.split()[0] for r in results if r.endswith(' %s' % process_name)]
+
+ try:
+ pids = list(map(int, results))
+
+ if multiple:
+ return pids
+ elif len(pids) == 1:
+ return pids[0]
+ except ValueError:
+ pass
+
+ # resolves using lsof which works on both Linux and BSD, only failing if:
+ # - lsof is unavailable (not included by default on OpenBSD)
+ # - the process being run as a different user due to permissions
+ # - the process doesn't have any open files to be reported by lsof?
+ #
+ # flags:
+ # t - only show pids
+ # c - restrict results to that command
+ #
+ # example output:
+ # atagar@morrigan:~$ lsof -t -c vim
+ # 2470
+ # 2561
+
+ if is_available('lsof'):
+ results = call(GET_PID_BY_NAME_LSOF % process_name, None)
+
+ if results:
+ try:
+ pids = list(map(int, results))
+
+ if multiple:
+ return pids
+ elif len(pids) == 1:
+ return pids[0]
+ except ValueError:
+ pass
+
+ if is_available('tasklist') and is_windows():
+ if not process_name.endswith('.exe'):
+ process_name = process_name + '.exe'
+
+ process_ids = []
+
+ results = stem.util.system.call('tasklist', None)
+
+ if results:
+ tasklist_regex = re.compile('^\s*%s\s+(?P<pid>[0-9]*)' % process_name)
+
+ for line in results:
+ match = tasklist_regex.search(line)
+
+ if match:
+ process_ids.append(int(match.group('pid')))
+
+ if multiple:
+ return process_ids
+ elif len(process_ids) > 0:
+ return process_ids[0]
+
+ log.debug("failed to resolve a pid for '%s'" % process_name)
+ return [] if multiple else None
+
+
+def pid_by_port(port):
+ """
+ Attempts to determine the process id for a process with the given port,
+ using...
+
+ ::
+
+ 1. netstat -npltu | grep 127.0.0.1:<port>
+ 2. sockstat -4l -P tcp -p <port>
+ 3. lsof -wnP -iTCP -sTCP:LISTEN | grep ":<port>"
+
+ Most queries limit results to listening TCP connections. This function likely
+ won't work on Mac OSX.
+
+ :param int port: port where the process we're looking for is listening
+
+ :returns: **int** with the process id, **None** if it can't be determined
+ """
+
+ # attempts to resolve using netstat, failing if:
+ # - netstat doesn't accept these flags (Linux only)
+ # - the process being run as a different user due to permissions
+ #
+ # flags:
+ # n - numeric (disables hostname lookups)
+ # p - program (include pids)
+ # l - listening (include listening sockets)
+ # tu - show tcp and udp sockets, and nothing else
+ #
+ # example output:
+ # atagar@morrigan:~$ netstat -npltu
+ # Active Internet connections (only servers)
+ # Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
+ # tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN -
+ # tcp 0 0 127.0.0.1:9051 0.0.0.0:* LISTEN 1641/tor
+ # tcp6 0 0 ::1:631 :::* LISTEN -
+ # udp 0 0 0.0.0.0:5353 0.0.0.0:* -
+ # udp6 0 0 fe80::7ae4:ff:fe2f::123 :::* -
+
+ if is_available('netstat'):
+ results = call(GET_PID_BY_PORT_NETSTAT, None)
+
+ if results:
+ # filters to results with our port
+ results = [r for r in results if '127.0.0.1:%s' % port in r]
+
+ if len(results) == 1 and len(results[0].split()) == 7:
+ results = results[0].split()[6] # process field (ex. "7184/tor")
+ pid = results[:results.find('/')]
+
+ if pid.isdigit():
+ return int(pid)
+
+ # attempts to resolve using sockstat, failing if:
+ # - sockstat doesn't accept the -4 flag (BSD only)
+ # - sockstat isn't available (encountered with OSX 10.5.8)
+ # - there are multiple instances using the same port on different addresses
+ #
+ # flags:
+ # 4 - only show IPv4 sockets
+ # l - listening sockets
+ # P tcp - only show tcp connections
+ # p - only includes results if the local or foreign port match this
+ #
+ # example output:
+ # # sockstat -4 | grep tor
+ # _tor tor 4397 7 tcp4 51.64.7.84:9050 *:*
+ # _tor tor 4397 8 udp4 51.64.7.84:53 *:*
+ # _tor tor 4397 12 tcp4 51.64.7.84:54011 80.3.121.7:9001
+ # _tor tor 4397 15 tcp4 51.64.7.84:59374 7.42.1.102:9001
+ # _tor tor 4397 20 tcp4 51.64.7.84:51946 32.83.7.104:443
+
+ if is_available('sockstat'):
+ results = call(GET_PID_BY_PORT_SOCKSTAT % port, None)
+
+ if results:
+ # filters to results where this is the local port
+ results = [r for r in results if (len(r.split()) == 7 and (':%s' % port) in r.split()[5])]
+
+ if len(results) == 1:
+ pid = results[0].split()[2]
+
+ if pid.isdigit():
+ return int(pid)
+
+ # resolves using lsof which works on both Linux and BSD, only failing if:
+ # - lsof is unavailable (not included by default on OpenBSD)
+ # - lsof doesn't provide the port ip/port, nor accept the -i and -s args
+ # (encountered with OSX 10.5.8)
+ # - the process being run as a different user due to permissions
+ # - there are multiple instances using the same port on different addresses
+ #
+ # flags:
+ # w - disables warning messages
+ # n - numeric addresses (disables hostname lookups)
+ # P - numeric ports (disables replacement of ports with their protocol)
+ # iTCP - only show tcp connections
+ # sTCP:LISTEN - listening sockets
+ #
+ # example output:
+ # atagar@morrigan:~$ lsof -wnP -iTCP -sTCP:LISTEN
+ # COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
+ # tor 1745 atagar 6u IPv4 14229 0t0 TCP 127.0.0.1:9051 (LISTEN)
+
+ if is_available('lsof'):
+ results = call(GET_PID_BY_PORT_LSOF, None)
+
+ if results:
+ # filters to results with our port
+ results = [r for r in results if (len(r.split()) == 10 and (':%s' % port) in r.split()[8])]
+
+ if len(results) == 1:
+ pid = results[0].split()[1]
+
+ if pid.isdigit():
+ return int(pid)
+
+ return None # all queries failed
+
+
+def pid_by_open_file(path):
+ """
+ Attempts to determine the process id for a process with the given open file,
+ using...
+
+ ::
+
+ lsof -w <path>
+
+ :param str path: location of the socket file to query against
+
+ :returns: **int** with the process id, **None** if it can't be determined
+ """
+
+ # resolves using lsof which works on both Linux and BSD, only failing if:
+ # - lsof is unavailable (not included by default on OpenBSD)
+ # - the file can't be read due to permissions
+ #
+ # flags:
+ # t - only show pids
+ # w - disables warning messages
+ #
+ # example output:
+ # atagar@morrigan:~$ lsof -tw /tmp/foo
+ # 4762
+
+ if is_available('lsof'):
+ results = call(GET_PID_BY_FILE_LSOF % path, [])
+
+ if len(results) == 1:
+ pid = results[0].strip()
+
+ if pid.isdigit():
+ return int(pid)
+
+ return None # all queries failed
+
+
+def cwd(pid):
+ """
+ Provides the working directory of the given process.
+
+ :param int pid: process id of the process to be queried
+
+ :returns: **str** with the absolute path for the process' present working
+ directory, **None** if it can't be determined
+ """
+
+ # try fetching via the proc contents if it's available
+ if stem.util.proc.is_available():
+ try:
+ return stem.util.proc.cwd(pid)
+ except IOError:
+ pass
+
+ # Fall back to a pwdx query. This isn't available on BSD.
+ logging_prefix = 'cwd(%s):' % pid
+
+ if is_available('pwdx'):
+ # pwdx results are of the form:
+ # 3799: /home/atagar
+ # 5839: No such process
+
+ results = call(GET_CWD_PWDX % pid, None)
+
+ if not results:
+ log.debug("%s pwdx didn't return any results" % logging_prefix)
+ elif results[0].endswith('No such process'):
+ log.debug('%s pwdx processes reported for this pid' % logging_prefix)
+ elif len(results) != 1 or results[0].count(' ') != 1 or not results[0].startswith('%s: ' % pid):
+ log.debug('%s we got unexpected output from pwdx: %s' % (logging_prefix, results))
+ else:
+ return results[0].split(' ', 1)[1].strip()
+
+ # Use lsof as the final fallback. This is available on both Linux and is the
+ # only lookup method here that works for BSD...
+ # https://trac.torproject.org/projects/tor/ticket/4236
+ #
+ # flags:
+ # a - presents the intersection of the following arguments
+ # p - limits results to this pid
+ # d cwd - limits results to just the cwd rather than all open files
+ # Fn - short listing in a single column, with just the pid and cwd
+ #
+ # example output:
+ # ~$ lsof -a -p 75717 -d cwd -Fn
+ # p75717
+ # n/Users/atagar/tor/src/or
+
+ if is_available('lsof'):
+ results = call(GET_CWD_LSOF % pid, [])
+
+ if len(results) == 2 and results[1].startswith('n/'):
+ lsof_result = results[1][1:].strip()
+
+ # If we lack read permissions for the cwd then it returns...
+ # p2683
+ # n/proc/2683/cwd (readlink: Permission denied)
+
+ if ' ' not in lsof_result:
+ return lsof_result
+ else:
+ log.debug('%s we got unexpected output from lsof: %s' % (logging_prefix, results))
+
+ return None # all queries failed
+
+
+def user(pid):
+ """
+ Provides the user a process is running under.
+
+ :param int pid: process id of the process to be queried
+
+ :returns: **str** with the username a process is running under, **None** if
+ it can't be determined
+ """
+
+ if not isinstance(pid, int) or pid < 0:
+ return None
+
+ if stem.util.proc.is_available():
+ try:
+ import pwd # only available on unix platforms
+
+ uid = stem.util.proc.uid(pid)
+
+ if uid and uid.isdigit():
+ return pwd.getpwuid(int(uid)).pw_name
+ except:
+ pass
+
+ if is_available('ps'):
+ results = call('ps -o user %s' % pid, [])
+
+ if len(results) >= 2:
+ return results[1].strip()
+
+ return None
+
+
+def start_time(pid):
+ """
+ Provides the unix timestamp when the given process started.
+
+ :param int pid: process id of the process to be queried
+
+ :returns: **float** for the unix timestamp when the process began, **None**
+ if it can't be determined
+ """
+
+ if not isinstance(pid, int) or pid < 0:
+ return None
+
+ if stem.util.proc.is_available():
+ try:
+ return float(stem.util.proc.stats(pid, stem.util.proc.Stat.START_TIME)[0])
+ except IOError:
+ pass
+
+ try:
+ ps_results = call('ps -p %s -o etime' % pid, [])
+
+ if len(ps_results) >= 2:
+ etime = ps_results[1].strip()
+ return time.time() - stem.util.str_tools.parse_short_time_label(etime)
+ except:
+ pass
+
+ return None
+
+
+def tail(target, lines = None):
+ """
+ Provides lines of a file starting with the end. For instance,
+ 'tail -n 50 /tmp/my_log' could be done with...
+
+ ::
+
+ reversed(list(tail('/tmp/my_log', 50)))
+
+ :param str,file target: path or file object to read from
+ :param int lines: number of lines to read
+
+ :returns: **generator** that reads lines, starting with the end
+
+ :raises: **IOError** if unable to read the file
+ """
+
+ if isinstance(target, str):
+ with open(target) as target_file:
+ for line in tail(target_file, lines):
+ yield line
+
+ return
+
+ # based on snippet from...
+ # https://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-python-similar-to-tail
+
+ target.seek(0, 2) # go to the end of the file
+ block_end_byte = target.tell()
+ block_number = -1
+ content = ''
+
+ while (lines is None or lines > 0) and block_end_byte > 0:
+ if (block_end_byte - BLOCK_SIZE > 0):
+ # read the last block we haven't yet read
+ target.seek(block_number * BLOCK_SIZE, 2)
+ content, completed_lines = (target.read(BLOCK_SIZE) + content).split('\n', 1)
+ else:
+ # reached the start of the file, just read what's left
+ target.seek(0, 0)
+ completed_lines = target.read(block_end_byte) + content
+
+ for line in reversed(completed_lines.splitlines()):
+ if lines is None or lines > 0:
+ if lines is not None:
+ lines -= 1
+
+ yield line
+
+ block_end_byte -= BLOCK_SIZE
+ block_number -= 1
+
+
+def bsd_jail_id(pid):
+ """
+ Gets the jail id for a process. These seem to only exist for FreeBSD (this
+ style for jails does not exist on Linux, OSX, or OpenBSD).
+
+ :param int pid: process id of the jail id to be queried
+
+ :returns: **int** for the jail id, zero if this can't be determined
+ """
+
+ # Output when called from a FreeBSD jail or when Tor isn't jailed:
+ # JID
+ # 0
+ #
+ # Otherwise it's something like:
+ # JID
+ # 1
+
+ ps_output = call(GET_BSD_JAIL_ID_PS % pid, [])
+
+ if len(ps_output) == 2 and len(ps_output[1].split()) == 1:
+ jid = ps_output[1].strip()
+
+ if jid.isdigit():
+ return int(jid)
+
+ os_name = platform.system()
+ if os_name == 'FreeBSD':
+ log.warn('Unable to get the jail id for process %s.' % pid)
+ else:
+ log.debug('bsd_jail_id(%s): jail ids do not exist on %s' % (pid, os_name))
+
+ return 0
+
+
+def bsd_jail_path(jid):
+ """
+ Provides the path of the given FreeBSD jail.
+
+ :param int jid: jail id to be queried
+
+ :returns: **str** of the path prefix, **None** if this can't be determined
+ """
+
+ if jid != 0:
+ # Output should be something like:
+ # JID IP Address Hostname Path
+ # 1 10.0.0.2 tor-jail /usr/jails/tor-jail
+
+ jls_output = call(GET_BSD_JAIL_PATH % jid, [])
+
+ if len(jls_output) == 2 and len(jls_output[1].split()) == 4:
+ return jls_output[1].split()[3]
+
+ return None
+
+
+def is_tarfile(path):
+ """
+ Returns if the path belongs to a tarfile or not.
+
+ .. versionadded:: 1.2.0
+
+ :param str path: path to be checked
+
+ :returns: **True** if the path belongs to a tarball, **False** otherwise
+ """
+
+ # Checking if it's a tar file may fail due to permissions so failing back
+ # to the mime type...
+ #
+ # IOError: [Errno 13] Permission denied: '/vmlinuz.old'
+ #
+ # With python 3 insuffient permissions raises an AttributeError instead...
+ #
+ # http://bugs.python.org/issue17059
+
+ try:
+ return tarfile.is_tarfile(path)
+ except (IOError, AttributeError):
+ return mimetypes.guess_type(path)[0] == 'application/x-tar'
+
+
+def expand_path(path, cwd = None):
+ """
+ Provides an absolute path, expanding tildes with the user's home and
+ appending a current working directory if the path was relative.
+
+ :param str path: path to be expanded
+ :param str cwd: current working directory to expand relative paths with, our
+ process' if this is **None**
+
+ :returns: **str** of the path expanded to be an absolute path, never with an
+ ending slash
+ """
+
+ if is_windows():
+ relative_path = path.replace('/', '\\').rstrip('\\')
+ else:
+ relative_path = path.rstrip('/')
+
+ if not relative_path or os.path.isabs(relative_path):
+ # empty or already absolute - nothing to do
+ pass
+ elif relative_path.startswith('~'):
+ # prefixed with a ~ or ~user entry
+ relative_path = os.path.expanduser(relative_path)
+ else:
+ # relative path, expand with the cwd
+
+ if not cwd:
+ cwd = os.getcwd()
+
+ # we'll be dealing with both "my/path/" and "./my/path" entries, so
+ # cropping the later
+ if relative_path.startswith('./') or relative_path.startswith('.\\'):
+ relative_path = relative_path[2:]
+ elif relative_path == '.':
+ relative_path = ''
+
+ if relative_path == '':
+ relative_path = cwd
+ else:
+ relative_path = os.path.join(cwd, relative_path)
+
+ return relative_path
+
+
+def files_with_suffix(base_path, suffix):
+ """
+ Iterates over files in a given directory, providing filenames with a certain
+ suffix.
+
+ .. versionadded:: 1.2.0
+
+ :param str base_path: directory to be iterated over
+ :param str suffix: filename suffix to look for
+
+ :returns: iterator that yields the absolute path for files with the given suffix
+ """
+
+ if os.path.isfile(base_path):
+ if base_path.endswith(suffix):
+ yield base_path
+ else:
+ for root, _, files in os.walk(base_path):
+ for filename in files:
+ if filename.endswith(suffix):
+ yield os.path.join(root, filename)
+
+
+def call(command, default = UNDEFINED, ignore_exit_status = False):
+ """
+ Issues a command in a subprocess, blocking until completion and returning the
+ results. This is not actually ran in a shell so pipes and other shell syntax
+ are not permitted.
+
+ :param str,list command: command to be issued
+ :param object default: response if the query fails
+ :param bool ignore_exit_status: reports failure if our command's exit status
+ was non-zero
+
+ :returns: **list** with the lines of output from the command
+
+ :raises: **OSError** if this fails and no default was provided
+ """
+
+ if isinstance(command, str):
+ command_list = command.split(' ')
+ else:
+ command_list = command
+
+ try:
+ is_shell_command = command_list[0] in SHELL_COMMANDS
+
+ start_time = time.time()
+ process = subprocess.Popen(command_list, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = is_shell_command)
+
+ stdout, stderr = process.communicate()
+ stdout, stderr = stdout.strip(), stderr.strip()
+ runtime = time.time() - start_time
+
+ log.debug('System call: %s (runtime: %0.2f)' % (command, runtime))
+ trace_prefix = 'Received from system (%s)' % command
+
+ if stdout and stderr:
+ log.trace(trace_prefix + ', stdout:\n%s\nstderr:\n%s' % (stdout, stderr))
+ elif stdout:
+ log.trace(trace_prefix + ', stdout:\n%s' % stdout)
+ elif stderr:
+ log.trace(trace_prefix + ', stderr:\n%s' % stderr)
+
+ exit_code = process.poll()
+
+ if not ignore_exit_status and exit_code != 0:
+ raise OSError('%s returned exit status %i' % (command, exit_code))
+
+ if stdout:
+ return stdout.decode('utf-8', 'replace').splitlines()
+ else:
+ return []
+ except OSError as exc:
+ log.debug('System call (failed): %s (error: %s)' % (command, exc))
+
+ if default != UNDEFINED:
+ return default
+ else:
+ raise exc
+
+
+def get_process_name():
+ """
+ Provides the present name of our process.
+
+ :returns: **str** with the present name of our process
+ """
+
+ global _PROCESS_NAME, _MAX_NAME_LENGTH
+
+ if _PROCESS_NAME is None:
+ # Example output...
+ #
+ # COMMAND
+ # python run_tests.py --unit
+
+ ps_output = call('ps -p %i -o args' % os.getpid(), [])
+
+ if len(ps_output) == 2 and ps_output[0] in ('COMMAND', 'ARGS'):
+ _PROCESS_NAME = ps_output[1]
+ else:
+ # Falling back on using ctypes to get our argv. Unfortunately the simple
+ # method for getting this...
+ #
+ # ' '.join(['python'] + sys.argv)
+ #
+ # ... doesn't do the trick since this will miss interpreter arguments.
+ #
+ # python -W ignore::DeprecationWarning my_script.py
+
+ args, argc = [], argc_t()
+
+ for i in range(100):
+ # The ending index can be either None or raise a ValueError when
+ # accessed...
+ #
+ # ValueError: NULL pointer access
+
+ try:
+ if argc[i] is None:
+ break
+ except ValueError:
+ break
+
+ args.append(str(argc[i]))
+
+ _PROCESS_NAME = ' '.join(args)
+
+ _MAX_NAME_LENGTH = len(_PROCESS_NAME)
+
+ return _PROCESS_NAME
+
+
+def set_process_name(process_name):
+ """
+ Renames our current process from "python <args>" to a custom name. This is
+ best-effort, not necessarily working on all platforms.
+
+ **Note:** This might have issues on FreeBSD (:trac:`9804`).
+
+ :param str process_name: new name for our process
+ """
+
+ # This is mostly based on...
+ #
+ # http://www.rhinocerus.net/forum/lang-python/569677-setting-program-name-like-0-perl.html#post2272369
+ #
+ # ... and an adaptation by Jake...
+ #
+ # https://github.com/ioerror/chameleon
+ #
+ # A cleaner implementation is available at...
+ #
+ # https://github.com/cream/libs/blob/b38970e2a6f6d2620724c828808235be0445b799/cream/util/procname.py
+ #
+ # but I'm not quite clear on their implementation, and it only does targeted
+ # argument replacement (ie, replace argv[0], argv[1], etc but with a string
+ # the same size).
+
+ _set_argv(process_name)
+
+ if platform.system() == 'Linux':
+ _set_prctl_name(process_name)
+ elif platform.system() in ('Darwin', 'FreeBSD', 'OpenBSD'):
+ _set_proc_title(process_name)
+
+
+def _set_argv(process_name):
+ """
+ Overwrites our argv in a similar fashion to how it's done in C with:
+ strcpy(argv[0], 'new_name');
+ """
+
+ if Py_GetArgcArgv is None:
+ return
+
+ global _PROCESS_NAME
+
+ # both gets the current process name and initializes _MAX_NAME_LENGTH
+
+ current_name = get_process_name()
+
+ argv, argc = ctypes.c_int(0), argc_t()
+ Py_GetArgcArgv(argv, ctypes.pointer(argc))
+
+ if len(process_name) > _MAX_NAME_LENGTH:
+ raise IOError("Can't rename process to something longer than our initial name (this would overwrite memory used for the env)")
+
+ # space we need to clear
+ zero_size = max(len(current_name), len(process_name))
+
+ ctypes.memset(argc.contents, 0, zero_size + 1) # null terminate the string's end
+ process_name_encoded = process_name.encode('utf8')
+ ctypes.memmove(argc.contents, process_name_encoded, len(process_name))
+ _PROCESS_NAME = process_name
+
+
+def _set_prctl_name(process_name):
+ """
+ Sets the prctl name, which is used by top and killall. This appears to be
+ Linux specific and has the max of 15 characters.
+
+ This is from...
+ http://stackoverflow.com/questions/564695/is-there-a-way-to-change-effective-process-name-in-python/923034#923034
+ """
+
+ libc = ctypes.CDLL(ctypes.util.find_library('c'))
+ name_buffer = ctypes.create_string_buffer(len(process_name) + 1)
+ name_buffer.value = stem.util.str_tools._to_bytes(process_name)
+ libc.prctl(PR_SET_NAME, ctypes.byref(name_buffer), 0, 0, 0)
+
+
+def _set_proc_title(process_name):
+ """
+ BSD specific calls (should be compataible with both FreeBSD and OpenBSD:
+ http://fxr.watson.org/fxr/source/gen/setproctitle.c?v=FREEBSD-LIBC
+ http://www.rootr.net/man/man/setproctitle/3
+ """
+
+ libc = ctypes.CDLL(ctypes.util.find_library('c'))
+ name_buffer = ctypes.create_string_buffer(len(process_name) + 1)
+ name_buffer.value = process_name
+
+ try:
+ libc.setproctitle(ctypes.byref(name_buffer))
+ except AttributeError:
+ # Possible issue (seen on OSX):
+ # AttributeError: dlsym(0x7fff6a41d1e0, setproctitle): symbol not found
+
+ pass
+
+
+# TODO: drop with stem 2.x
+# We renamed our methods to drop a redundant 'get_*' prefix, so alias the old
+# names for backward compatability.
+
+get_name_by_pid = name_by_pid
+get_pid_by_name = pid_by_name
+get_pid_by_port = pid_by_port
+get_pid_by_open_file = pid_by_open_file
+get_cwd = cwd
+get_user = user
+get_start_time = start_time
+get_bsd_jail_id = bsd_jail_id
+get_bsd_jail_path = bsd_jail_path
diff --git a/ext/stem/util/term.py b/ext/stem/util/term.py
new file mode 100644
index 00000000..bb110ea1
--- /dev/null
+++ b/ext/stem/util/term.py
@@ -0,0 +1,116 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Utilities for working with the terminal.
+
+**Module Overview:**
+
+::
+
+ format - wrap text with ANSI for the given colors or attributes
+
+.. data:: Color (enum)
+.. data:: BgColor (enum)
+
+ Enumerations for foreground or background terminal color.
+
+ =========== ===========
+ Color Description
+ =========== ===========
+ **BLACK** black color
+ **BLUE** blue color
+ **CYAN** cyan color
+ **GREEN** green color
+ **MAGENTA** magenta color
+ **RED** red color
+ **WHITE** white color
+ **YELLOW** yellow color
+ =========== ===========
+
+.. data:: Attr (enum)
+
+ Enumerations of terminal text attributes.
+
+ =================== ===========
+ Attr Description
+ =================== ===========
+ **BOLD** heavy typeface
+ **HILIGHT** inverted foreground and background
+ **UNDERLINE** underlined text
+ **READLINE_ESCAPE** wrap encodings in `RL_PROMPT_START_IGNORE and RL_PROMPT_END_IGNORE sequences <https://stackoverflow.com/questions/9468435/look-how-to-fix-column-calculation-in-python-readline-if-use-color-prompt>`_
+ =================== ===========
+"""
+
+import stem.util.enum
+import stem.util.str_tools
+
+TERM_COLORS = ('BLACK', 'RED', 'GREEN', 'YELLOW', 'BLUE', 'MAGENTA', 'CYAN', 'WHITE')
+
+# DISABLE_COLOR_SUPPORT is *not* being vended to Stem users. This is likely to
+# go away if I can think of a more graceful method for color toggling.
+
+DISABLE_COLOR_SUPPORT = False
+
+Color = stem.util.enum.Enum(*TERM_COLORS)
+BgColor = stem.util.enum.Enum(*['BG_' + color for color in TERM_COLORS])
+Attr = stem.util.enum.Enum('BOLD', 'UNDERLINE', 'HILIGHT', 'READLINE_ESCAPE')
+
+# mappings of terminal attribute enums to their ANSI escape encoding
+FG_ENCODING = dict([(list(Color)[i], str(30 + i)) for i in range(8)])
+BG_ENCODING = dict([(list(BgColor)[i], str(40 + i)) for i in range(8)])
+ATTR_ENCODING = {Attr.BOLD: '1', Attr.UNDERLINE: '4', Attr.HILIGHT: '7'}
+
+CSI = '\x1B[%sm'
+RESET = CSI % '0'
+
+
+def format(msg, *attr):
+ """
+ Simple terminal text formatting using `ANSI escape sequences
+ <https://en.wikipedia.org/wiki/ANSI_escape_code#CSI_codes>`_.
+ The following are some toolkits providing similar capabilities:
+
+ * `django.utils.termcolors <https://github.com/django/django/blob/master/django/utils/termcolors.py>`_
+ * `termcolor <https://pypi.python.org/pypi/termcolor>`_
+ * `colorama <https://pypi.python.org/pypi/colorama>`_
+
+ :param str msg: string to be formatted
+ :param str attr: text attributes, this can be :data:`~stem.util.term.Color`,
+ :data:`~stem.util.term.BgColor`, or :data:`~stem.util.term.Attr` enums
+ and are case insensitive (so strings like 'red' are fine)
+
+ :returns: **str** wrapped with ANSI escape encodings, starting with the given
+ attributes and ending with a reset
+ """
+
+ if DISABLE_COLOR_SUPPORT:
+ return msg
+
+ # if we have reset sequences in the message then apply our attributes
+ # after each of them
+
+ if RESET in msg:
+ return ''.join([format(comp, *attr) for comp in msg.split(RESET)])
+
+ encodings = []
+
+ for text_attr in attr:
+ text_attr, encoding = stem.util.str_tools._to_camel_case(text_attr), None
+ encoding = FG_ENCODING.get(text_attr, encoding)
+ encoding = BG_ENCODING.get(text_attr, encoding)
+ encoding = ATTR_ENCODING.get(text_attr, encoding)
+
+ if encoding:
+ encodings.append(encoding)
+
+ if encodings:
+ prefix, suffix = CSI % ';'.join(encodings), RESET
+
+ if Attr.READLINE_ESCAPE in attr:
+ prefix = '\001%s\002' % prefix
+ suffix = '\001%s\002' % suffix
+
+ return prefix + msg + suffix
+ else:
+ return msg
diff --git a/ext/stem/util/test_tools.py b/ext/stem/util/test_tools.py
new file mode 100644
index 00000000..d6a81c45
--- /dev/null
+++ b/ext/stem/util/test_tools.py
@@ -0,0 +1,341 @@
+# Copyright 2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Helper functions for testing.
+
+.. versionadded:: 1.2.0
+
+::
+
+ clean_orphaned_pyc - delete *.pyc files without corresponding *.py
+
+ is_pyflakes_available - checks if pyflakes is available
+ is_pep8_available - checks if pep8 is available
+
+ stylistic_issues - checks for PEP8 and other stylistic issues
+ pyflakes_issues - static checks for problems via pyflakes
+"""
+
+import collections
+import linecache
+import os
+import re
+
+import stem.util.conf
+import stem.util.system
+
+CONFIG = stem.util.conf.config_dict('test', {
+ 'pep8.ignore': [],
+ 'pyflakes.ignore': [],
+ 'exclude_paths': [],
+})
+
+Issue = collections.namedtuple('Issue', [
+ 'line_number',
+ 'message',
+ 'line',
+])
+
+
+def clean_orphaned_pyc(paths):
+ """
+ Deletes any file with a *.pyc extention without a corresponding *.py. This
+ helps to address a common gotcha when deleting python files...
+
+ * You delete module 'foo.py' and run the tests to ensure that you haven't
+ broken anything. They pass, however there *are* still some 'import foo'
+ statements that still work because the bytecode (foo.pyc) is still around.
+
+ * You push your change.
+
+ * Another developer clones our repository and is confused because we have a
+ bunch of ImportErrors.
+
+ :param list paths: paths to search for orphaned pyc files
+
+ :returns: list of absolute paths that were deleted
+ """
+
+ orphaned_pyc = []
+
+ for path in paths:
+ for pyc_path in stem.util.system.files_with_suffix(path, '.pyc'):
+ py_path = pyc_path[:-1]
+
+ # If we're running python 3 then the *.pyc files are no longer bundled
+ # with the *.py. Rather, they're in a __pycache__ directory.
+
+ pycache = '%s__pycache__%s' % (os.path.sep, os.path.sep)
+
+ if pycache in pyc_path:
+ directory, pycache_filename = pyc_path.split(pycache, 1)
+
+ if not pycache_filename.endswith('.pyc'):
+ continue # should look like 'test_tools.cpython-32.pyc'
+
+ py_path = os.path.join(directory, pycache_filename.split('.')[0] + '.py')
+
+ if not os.path.exists(py_path):
+ orphaned_pyc.append(pyc_path)
+ os.remove(pyc_path)
+
+ return orphaned_pyc
+
+
+def is_pyflakes_available():
+ """
+ Checks if pyflakes is availalbe.
+
+ :returns: **True** if we can use pyflakes and **False** otherwise
+ """
+
+ try:
+ import pyflakes.api
+ import pyflakes.reporter
+ return True
+ except ImportError:
+ return False
+
+
+def is_pep8_available():
+ """
+ Checks if pep8 is availalbe.
+
+ :returns: **True** if we can use pep8 and **False** otherwise
+ """
+
+ try:
+ import pep8
+
+ if not hasattr(pep8, 'BaseReport'):
+ raise ImportError()
+
+ return True
+ except ImportError:
+ return False
+
+
+def stylistic_issues(paths, check_two_space_indents = False, check_newlines = False, check_trailing_whitespace = False, check_exception_keyword = False, prefer_single_quotes = False):
+ """
+ Checks for stylistic issues that are an issue according to the parts of PEP8
+ we conform to. You can suppress PEP8 issues by making a 'test' configuration
+ that sets 'pep8.ignore'.
+
+ For example, with a 'test/settings.cfg' of...
+
+ ::
+
+ # PEP8 compliance issues that we're ignoreing...
+ #
+ # * E111 and E121 four space indentations
+ # * E501 line is over 79 characters
+
+ pep8.ignore E111
+ pep8.ignore E121
+ pep8.ignore E501
+
+ ... you can then run tests with...
+
+ ::
+
+ import stem.util.conf
+
+ test_config = stem.util.conf.get_config('test')
+ test_config.load('test/settings.cfg')
+
+ issues = stylistic_issues('my_project')
+
+ If a 'exclude_paths' was set in our test config then we exclude any absolute
+ paths matching those regexes.
+
+ .. versionchanged:: 1.3.0
+ Renamed from get_stylistic_issues() to stylistic_issues(). The old name
+ still works as an alias, but will be dropped in Stem version 2.0.0.
+
+ .. versionchanged:: 1.4.0
+ Changing tuples in return value to be namedtuple instances, and adding the
+ line that had the issue.
+
+ .. versionchanged:: 1.4.0
+ Added the prefer_single_quotes option.
+
+ :param list paths: paths to search for stylistic issues
+ :param bool check_two_space_indents: check for two space indentations and
+ that no tabs snuck in
+ :param bool check_newlines: check that we have standard newlines (\\n), not
+ windows (\\r\\n) nor classic mac (\\r)
+ :param bool check_trailing_whitespace: check that our lines don't end with
+ trailing whitespace
+ :param bool check_exception_keyword: checks that we're using 'as' for
+ exceptions rather than a comma
+ :param bool prefer_single_quotes: standardize on using single rather than
+ double quotes for strings, when reasonable
+
+ :returns: **dict** of the form ``path => [(line_number, message)...]``
+ """
+
+ issues = {}
+
+ if is_pep8_available():
+ import pep8
+
+ class StyleReport(pep8.BaseReport):
+ def __init__(self, options):
+ super(StyleReport, self).__init__(options)
+
+ def error(self, line_number, offset, text, check):
+ code = super(StyleReport, self).error(line_number, offset, text, check)
+
+ if code:
+ issues.setdefault(self.filename, []).append(Issue(line_number, '%s %s' % (code, text), text))
+
+ style_checker = pep8.StyleGuide(ignore = CONFIG['pep8.ignore'], reporter = StyleReport)
+ style_checker.check_files(list(_python_files(paths)))
+
+ if check_two_space_indents or check_newlines or check_trailing_whitespace or check_exception_keyword:
+ for path in _python_files(paths):
+ with open(path) as f:
+ file_contents = f.read()
+
+ lines = file_contents.split('\n')
+ is_block_comment = False
+
+ for index, line in enumerate(lines):
+ whitespace, content = re.match('^(\s*)(.*)$', line).groups()
+
+ # TODO: This does not check that block indentations are two spaces
+ # because differentiating source from string blocks ("""foo""") is more
+ # of a pita than I want to deal with right now.
+
+ if '"""' in content:
+ is_block_comment = not is_block_comment
+
+ if check_two_space_indents and '\t' in whitespace:
+ issues.setdefault(path, []).append(Issue(index + 1, 'indentation has a tab', line))
+ elif check_newlines and '\r' in content:
+ issues.setdefault(path, []).append(Issue(index + 1, 'contains a windows newline', line))
+ elif check_trailing_whitespace and content != content.rstrip():
+ issues.setdefault(path, []).append(Issue(index + 1, 'line has trailing whitespace', line))
+ elif check_exception_keyword and content.lstrip().startswith('except') and content.endswith(', exc:'):
+ # Python 2.6 - 2.7 supports two forms for exceptions...
+ #
+ # except ValueError, exc:
+ # except ValueError as exc:
+ #
+ # The former is the old method and no longer supported in python 3
+ # going forward.
+
+ # TODO: This check only works if the exception variable is called
+ # 'exc'. We should generalize this via a regex so other names work
+ # too.
+
+ issues.setdefault(path, []).append(Issue(index + 1, "except clause should use 'as', not comma", line))
+
+ if prefer_single_quotes and line and not is_block_comment:
+ content = line.strip().split('#', 1)[0]
+
+ if '"' in content and "'" not in content and '"""' not in content and not content.endswith('\\'):
+ # Checking if the line already has any single quotes since that
+ # usually means double quotes are preferable for the content (for
+ # instance "I'm hungry"). Also checking for '\' at the end since
+ # that can indicate a multi-line string.
+
+ issues.setdefault(path, []).append(Issue(index + 1, "use single rather than double quotes", line))
+
+ return issues
+
+
+def pyflakes_issues(paths):
+ """
+ Performs static checks via pyflakes. False positives can be ignored via
+ 'pyflakes.ignore' entries in our 'test' config. For instance...
+
+ ::
+
+ pyflakes.ignore stem/util/test_tools.py => 'pyflakes' imported but unused
+ pyflakes.ignore stem/util/test_tools.py => 'pep8' imported but unused
+
+ If a 'exclude_paths' was set in our test config then we exclude any absolute
+ paths matching those regexes.
+
+ .. versionchanged:: 1.3.0
+ Renamed from get_pyflakes_issues() to pyflakes_issues(). The old name
+ still works as an alias, but will be dropped in Stem version 2.0.0.
+
+ .. versionchanged:: 1.4.0
+ Changing tuples in return value to be namedtuple instances, and adding the
+ line that had the issue.
+
+ :param list paths: paths to search for problems
+
+ :returns: dict of the form ``path => [(line_number, message)...]``
+ """
+
+ issues = {}
+
+ if is_pyflakes_available():
+ import pyflakes.api
+ import pyflakes.reporter
+
+ class Reporter(pyflakes.reporter.Reporter):
+ def __init__(self):
+ self._ignored_issues = {}
+
+ for line in CONFIG['pyflakes.ignore']:
+ path, issue = line.split('=>')
+ self._ignored_issues.setdefault(path.strip(), []).append(issue.strip())
+
+ def unexpectedError(self, filename, msg):
+ self._register_issue(filename, None, msg, None)
+
+ def syntaxError(self, filename, msg, lineno, offset, text):
+ self._register_issue(filename, lineno, msg, text)
+
+ def flake(self, msg):
+ self._register_issue(msg.filename, msg.lineno, msg.message % msg.message_args, None)
+
+ def _is_ignored(self, path, issue):
+ # Paths in pyflakes_ignore are relative, so we need to check to see if our
+ # path ends with any of them.
+
+ for ignored_path, ignored_issues in self._ignored_issues.items():
+ if path.endswith(ignored_path) and issue in ignored_issues:
+ return True
+
+ return False
+
+ def _register_issue(self, path, line_number, issue, line):
+ if not self._is_ignored(path, issue):
+ if path and line_number and not line:
+ line = linecache.getline(path, line_number)
+
+ issues.setdefault(path, []).append(Issue(line_number, issue, line))
+
+ reporter = Reporter()
+
+ for path in _python_files(paths):
+ pyflakes.api.checkPath(path, reporter)
+
+ return issues
+
+
+def _python_files(paths):
+ for path in paths:
+ for file_path in stem.util.system.files_with_suffix(path, '.py'):
+ skip = False
+
+ for exclude_path in CONFIG['exclude_paths']:
+ if re.match(exclude_path, file_path):
+ skip = True
+ break
+
+ if not skip:
+ yield file_path
+
+# TODO: drop with stem 2.x
+# We renamed our methods to drop a redundant 'get_*' prefix, so alias the old
+# names for backward compatability.
+
+get_stylistic_issues = stylistic_issues
+get_pyflakes_issues = pyflakes_issues
diff --git a/ext/stem/util/tor_tools.py b/ext/stem/util/tor_tools.py
new file mode 100644
index 00000000..01c29eee
--- /dev/null
+++ b/ext/stem/util/tor_tools.py
@@ -0,0 +1,151 @@
+# Copyright 2012-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Miscellaneous utility functions for working with tor.
+
+.. versionadded:: 1.2.0
+
+**Module Overview:**
+
+::
+
+ is_valid_fingerprint - checks if a string is a valid tor relay fingerprint
+ is_valid_nickname - checks if a string is a valid tor relay nickname
+ is_valid_circuit_id - checks if a string is a valid tor circuit id
+ is_valid_stream_id - checks if a string is a valid tor stream id
+ is_valid_connection_id - checks if a string is a valid tor connection id
+ is_valid_hidden_service_address - checks if a string is a valid hidden service address
+ is_hex_digits - checks if a string is only made up of hex digits
+"""
+
+import re
+
+# The control-spec defines the following as...
+#
+# Fingerprint = "$" 40*HEXDIG
+# NicknameChar = "a"-"z" / "A"-"Z" / "0" - "9"
+# Nickname = 1*19 NicknameChar
+#
+# CircuitID = 1*16 IDChar
+# IDChar = ALPHA / DIGIT
+#
+# HEXDIG is defined in RFC 5234 as being uppercase and used in RFC 5987 as
+# case insensitive. Tor doesn't define this in the spec so flipping a coin
+# and going with case insensitive.
+
+NICKNAME_PATTERN = re.compile('^[a-zA-Z0-9]{1,19}$')
+CIRC_ID_PATTERN = re.compile('^[a-zA-Z0-9]{1,16}$')
+
+# Hidden service addresses are sixteen base32 characters.
+
+HS_ADDRESS_PATTERN = re.compile('^[a-z2-7]{16}$')
+
+
+def is_valid_fingerprint(entry, check_prefix = False):
+ """
+ Checks if a string is a properly formatted relay fingerprint. This checks for
+ a '$' prefix if check_prefix is true, otherwise this only validates the hex
+ digits.
+
+ :param str entry: string to be checked
+ :param bool check_prefix: checks for a '$' prefix
+
+ :returns: **True** if the string could be a relay fingerprint, **False** otherwise
+ """
+
+ try:
+ if check_prefix:
+ if not entry or entry[0] != '$':
+ return False
+
+ entry = entry[1:]
+
+ return is_hex_digits(entry, 40)
+ except TypeError:
+ return False
+
+
+def is_valid_nickname(entry):
+ """
+ Checks if a string is a valid format for being a nickname.
+
+ :param str entry: string to be checked
+
+ :returns: **True** if the string could be a nickname, **False** otherwise
+ """
+
+ try:
+ return bool(NICKNAME_PATTERN.match(entry))
+ except TypeError:
+ return False
+
+
+def is_valid_circuit_id(entry):
+ """
+ Checks if a string is a valid format for being a circuit identifier.
+
+ :returns: **True** if the string could be a circuit id, **False** otherwise
+ """
+
+ try:
+ return bool(CIRC_ID_PATTERN.match(entry))
+ except TypeError:
+ return False
+
+
+def is_valid_stream_id(entry):
+ """
+ Checks if a string is a valid format for being a stream identifier.
+ Currently, this is just an alias to :func:`~stem.util.tor_tools.is_valid_circuit_id`.
+
+ :returns: **True** if the string could be a stream id, **False** otherwise
+ """
+
+ return is_valid_circuit_id(entry)
+
+
+def is_valid_connection_id(entry):
+ """
+ Checks if a string is a valid format for being a connection identifier.
+ Currently, this is just an alias to :func:`~stem.util.tor_tools.is_valid_circuit_id`.
+
+ :returns: **True** if the string could be a connection id, **False** otherwise
+ """
+
+ return is_valid_circuit_id(entry)
+
+
+def is_valid_hidden_service_address(entry):
+ """
+ Checks if a string is a valid format for being a hidden service address (not
+ including the '.onion' suffix).
+
+ :returns: **True** if the string could be a hidden service address, **False** otherwise
+ """
+
+ try:
+ return bool(HS_ADDRESS_PATTERN.match(entry))
+ except TypeError:
+ return False
+
+
+def is_hex_digits(entry, count):
+ """
+ Checks if a string is the given number of hex digits. Digits represented by
+ letters are case insensitive.
+
+ :param str entry: string to be checked
+ :param int count: number of hex digits to be checked for
+
+ :returns: **True** if the given number of hex digits, **False** otherwise
+ """
+
+ try:
+ if len(entry) != count:
+ return False
+
+ int(entry, 16) # attempt to convert it as hex
+ return True
+ except (ValueError, TypeError):
+ return False
diff --git a/ext/stem/version.py b/ext/stem/version.py
new file mode 100644
index 00000000..1182bfca
--- /dev/null
+++ b/ext/stem/version.py
@@ -0,0 +1,376 @@
+# Copyright 2011-2015, Damian Johnson and The Tor Project
+# See LICENSE for licensing information
+
+"""
+Tor versioning information and requirements for its features. These can be
+easily parsed and compared, for instance...
+
+::
+
+ >>> from stem.version import get_system_tor_version, Requirement
+ >>> my_version = get_system_tor_version()
+ >>> print(my_version)
+ 0.2.1.30
+ >>> my_version >= Requirement.TORRC_CONTROL_SOCKET
+ True
+
+**Module Overview:**
+
+::
+
+ get_system_tor_version - gets the version of our system's tor installation
+
+ Version - Tor versioning information
+
+.. data:: Requirement (enum)
+
+ Enumerations for the version requirements of features.
+
+ ===================================== ===========
+ Requirement Description
+ ===================================== ===========
+ **AUTH_SAFECOOKIE** SAFECOOKIE authentication method
+ **DROPGUARDS** DROPGUARDS requests
+ **EVENT_AUTHDIR_NEWDESCS** AUTHDIR_NEWDESC events
+ **EVENT_BUILDTIMEOUT_SET** BUILDTIMEOUT_SET events
+ **EVENT_CIRC_MINOR** CIRC_MINOR events
+ **EVENT_CLIENTS_SEEN** CLIENTS_SEEN events
+ **EVENT_CONF_CHANGED** CONF_CHANGED events
+ **EVENT_DESCCHANGED** DESCCHANGED events
+ **EVENT_GUARD** GUARD events
+ **EVENT_HS_DESC_CONTENT** HS_DESC_CONTENT events
+ **EVENT_NEWCONSENSUS** NEWCONSENSUS events
+ **EVENT_NS** NS events
+ **EVENT_SIGNAL** SIGNAL events
+ **EVENT_STATUS** STATUS_GENERAL, STATUS_CLIENT, and STATUS_SERVER events
+ **EVENT_STREAM_BW** STREAM_BW events
+ **EVENT_TRANSPORT_LAUNCHED** TRANSPORT_LAUNCHED events
+ **EVENT_CONN_BW** CONN_BW events
+ **EVENT_CIRC_BW** CIRC_BW events
+ **EVENT_CELL_STATS** CELL_STATS events
+ **EVENT_TB_EMPTY** TB_EMPTY events
+ **EVENT_HS_DESC** HS_DESC events
+ **EXTENDCIRCUIT_PATH_OPTIONAL** EXTENDCIRCUIT queries can omit the path if the circuit is zero
+ **FEATURE_EXTENDED_EVENTS** 'EXTENDED_EVENTS' optional feature
+ **FEATURE_VERBOSE_NAMES** 'VERBOSE_NAMES' optional feature
+ **GETINFO_CONFIG_TEXT** 'GETINFO config-text' query
+ **HSFETCH** HSFETCH requests
+ **HSPOST** HSPOST requests
+ **ADD_ONION** ADD_ONION and DEL_ONION requests
+ **LOADCONF** LOADCONF requests
+ **MICRODESCRIPTOR_IS_DEFAULT** Tor gets microdescriptors by default rather than server descriptors
+ **TAKEOWNERSHIP** TAKEOWNERSHIP requests
+ **TORRC_CONTROL_SOCKET** 'ControlSocket <path>' config option
+ **TORRC_PORT_FORWARDING** 'PortForwarding' config option
+ **TORRC_DISABLE_DEBUGGER_ATTACHMENT** 'DisableDebuggerAttachment' config option
+ **TORRC_VIA_STDIN** Allow torrc options via 'tor -f -' (:trac:`13865`)
+ ===================================== ===========
+"""
+
+import os
+import re
+
+import stem.util.enum
+import stem.util.system
+
+try:
+ # added in python 3.2
+ from functools import lru_cache
+except ImportError:
+ from stem.util.lru_cache import lru_cache
+
+# cache for the get_system_tor_version function
+VERSION_CACHE = {}
+
+
+def get_system_tor_version(tor_cmd = 'tor'):
+ """
+ Queries tor for its version. This is os dependent, only working on linux,
+ osx, and bsd.
+
+ :param str tor_cmd: command used to run tor
+
+ :returns: :class:`~stem.version.Version` provided by the tor command
+
+ :raises: **IOError** if unable to query or parse the version
+ """
+
+ if tor_cmd not in VERSION_CACHE:
+ version_cmd = '%s --version' % tor_cmd
+
+ try:
+ version_output = stem.util.system.call(version_cmd)
+ except OSError as exc:
+ # make the error message nicer if this is due to tor being unavialable
+
+ if 'No such file or directory' in str(exc):
+ if os.path.isabs(tor_cmd):
+ exc = "Unable to check tor's version. '%s' doesn't exist." % tor_cmd
+ else:
+ exc = "Unable to run '%s'. Mabye tor isn't in your PATH?" % version_cmd
+
+ raise IOError(exc)
+
+ if version_output:
+ # output example:
+ # Oct 21 07:19:27.438 [notice] Tor v0.2.1.30. This is experimental software. Do not rely on it for strong anonymity. (Running on Linux i686)
+ # Tor version 0.2.1.30.
+
+ last_line = version_output[-1]
+
+ if last_line.startswith('Tor version ') and last_line.endswith('.'):
+ try:
+ version_str = last_line[12:-1]
+ VERSION_CACHE[tor_cmd] = Version(version_str)
+ except ValueError as exc:
+ raise IOError(exc)
+ else:
+ raise IOError("Unexpected response from '%s': %s" % (version_cmd, last_line))
+ else:
+ raise IOError("'%s' didn't have any output" % version_cmd)
+
+ return VERSION_CACHE[tor_cmd]
+
+
+@lru_cache()
+def _get_version(version_str):
+ return Version(version_str)
+
+
+class Version(object):
+ """
+ Comparable tor version. These are constructed from strings that conform to
+ the 'new' style in the `tor version-spec
+ <https://gitweb.torproject.org/torspec.git/tree/version-spec.txt>`_,
+ such as "0.1.4" or "0.2.2.23-alpha (git-7dcd105be34a4f44)".
+
+ :var int major: major version
+ :var int minor: minor version
+ :var int micro: micro version
+ :var int patch: patch level (**None** if undefined)
+ :var str status: status tag such as 'alpha' or 'beta-dev' (**None** if undefined)
+ :var str extra: extra information without its parentheses such as
+ 'git-8be6058d8f31e578' (**None** if undefined)
+ :var str git_commit: git commit id (**None** if it wasn't provided)
+
+ :param str version_str: version to be parsed
+
+ :raises: **ValueError** if input isn't a valid tor version
+ """
+
+ def __init__(self, version_str):
+ self.version_str = version_str
+ version_parts = re.match(r'^([0-9]+)\.([0-9]+)\.([0-9]+)(\.[0-9]+)?(-\S*)?( \(\S*\))?$', version_str)
+ self._hash = None
+
+ if version_parts:
+ major, minor, micro, patch, status, extra = version_parts.groups()
+
+ # The patch and status matches are optional (may be None) and have an extra
+ # proceeding period or dash if they exist. Stripping those off.
+
+ if patch:
+ patch = int(patch[1:])
+
+ if status:
+ status = status[1:]
+
+ if extra:
+ extra = extra[2:-1]
+
+ self.major = int(major)
+ self.minor = int(minor)
+ self.micro = int(micro)
+ self.patch = patch
+ self.status = status
+ self.extra = extra
+
+ if extra and re.match('^git-[0-9a-f]{16}$', extra):
+ self.git_commit = extra[4:]
+ else:
+ self.git_commit = None
+ else:
+ raise ValueError("'%s' isn't a properly formatted tor version" % version_str)
+
+ def __str__(self):
+ """
+ Provides the string used to construct the version.
+ """
+
+ return self.version_str
+
+ def _compare(self, other, method):
+ """
+ Compares version ordering according to the spec.
+ """
+
+ if not isinstance(other, Version):
+ return False
+
+ for attr in ('major', 'minor', 'micro', 'patch'):
+ my_version = getattr(self, attr)
+ other_version = getattr(other, attr)
+
+ if my_version is None:
+ my_version = 0
+
+ if other_version is None:
+ other_version = 0
+
+ if my_version != other_version:
+ return method(my_version, other_version)
+
+ # According to the version spec...
+ #
+ # If we *do* encounter two versions that differ only by status tag, we
+ # compare them lexically as ASCII byte strings.
+
+ my_status = self.status if self.status else ''
+ other_status = other.status if other.status else ''
+
+ return method(my_status, other_status)
+
+ def __eq__(self, other):
+ return self._compare(other, lambda s, o: s == o)
+
+ def __gt__(self, other):
+ """
+ Checks if this version meets the requirements for a given feature. We can
+ be compared to either a :class:`~stem.version.Version` or
+ :class:`~stem.version._VersionRequirements`.
+ """
+
+ if isinstance(other, _VersionRequirements):
+ for rule in other.rules:
+ if rule(self):
+ return True
+
+ return False
+
+ return self._compare(other, lambda s, o: s > o)
+
+ def __ge__(self, other):
+ if isinstance(other, _VersionRequirements):
+ for rule in other.rules:
+ if rule(self):
+ return True
+
+ return False
+
+ return self._compare(other, lambda s, o: s >= o)
+
+ def __hash__(self):
+ if self._hash is None:
+ my_hash = 0
+
+ for attr in ('major', 'minor', 'micro', 'patch', 'status'):
+ my_hash *= 1024
+
+ attr_value = getattr(self, attr)
+
+ if attr_value is not None:
+ my_hash += hash(attr_value)
+
+ self._hash = my_hash
+
+ return self._hash
+
+
+class _VersionRequirements(object):
+ """
+ Series of version constraints that can be compared to. For instance, this
+ allows for comparisons like 'if I'm greater than version X in the 0.2.2
+ series, or greater than version Y in the 0.2.3 series'.
+
+ This is a logical 'or' of the series of rules.
+ """
+
+ def __init__(self):
+ self.rules = []
+
+ def greater_than(self, version, inclusive = True):
+ """
+ Adds a constraint that we're greater than the given version.
+
+ :param stem.version.Version version: version we're checking against
+ :param bool inclusive: if comparison is inclusive or not
+ """
+
+ if inclusive:
+ self.rules.append(lambda v: version <= v)
+ else:
+ self.rules.append(lambda v: version < v)
+
+ def less_than(self, version, inclusive = True):
+ """
+ Adds a constraint that we're less than the given version.
+
+ :param stem.version.Version version: version we're checking against
+ :param bool inclusive: if comparison is inclusive or not
+ """
+
+ if inclusive:
+ self.rules.append(lambda v: version >= v)
+ else:
+ self.rules.append(lambda v: version > v)
+
+ def in_range(self, from_version, to_version, from_inclusive = True, to_inclusive = False):
+ """
+ Adds constraint that we're within the range from one version to another.
+
+ :param stem.version.Version from_version: beginning of the comparison range
+ :param stem.version.Version to_version: end of the comparison range
+ :param bool from_inclusive: if comparison is inclusive with the starting version
+ :param bool to_inclusive: if comparison is inclusive with the ending version
+ """
+
+ if from_inclusive and to_inclusive:
+ new_rule = lambda v: from_version <= v <= to_version
+ elif from_inclusive:
+ new_rule = lambda v: from_version <= v < to_version
+ else:
+ new_rule = lambda v: from_version < v < to_version
+
+ self.rules.append(new_rule)
+
+safecookie_req = _VersionRequirements()
+safecookie_req.in_range(Version('0.2.2.36'), Version('0.2.3.0'))
+safecookie_req.greater_than(Version('0.2.3.13'))
+
+Requirement = stem.util.enum.Enum(
+ ('AUTH_SAFECOOKIE', safecookie_req),
+ ('DROPGUARDS', Version('0.2.5.1-alpha')),
+ ('EVENT_AUTHDIR_NEWDESCS', Version('0.1.1.10-alpha')),
+ ('EVENT_BUILDTIMEOUT_SET', Version('0.2.2.7-alpha')),
+ ('EVENT_CIRC_MINOR', Version('0.2.3.11-alpha')),
+ ('EVENT_CLIENTS_SEEN', Version('0.2.1.10-alpha')),
+ ('EVENT_CONF_CHANGED', Version('0.2.3.3-alpha')),
+ ('EVENT_DESCCHANGED', Version('0.1.2.2-alpha')),
+ ('EVENT_GUARD', Version('0.1.2.5-alpha')),
+ ('EVENT_HS_DESC_CONTENT', Version('0.2.7.1-alpha')),
+ ('EVENT_NS', Version('0.1.2.3-alpha')),
+ ('EVENT_NEWCONSENSUS', Version('0.2.1.13-alpha')),
+ ('EVENT_SIGNAL', Version('0.2.3.1-alpha')),
+ ('EVENT_STATUS', Version('0.1.2.3-alpha')),
+ ('EVENT_STREAM_BW', Version('0.1.2.8-beta')),
+ ('EVENT_TRANSPORT_LAUNCHED', Version('0.2.5.0-alpha')),
+ ('EVENT_CONN_BW', Version('0.2.5.2-alpha')),
+ ('EVENT_CIRC_BW', Version('0.2.5.2-alpha')),
+ ('EVENT_CELL_STATS', Version('0.2.5.2-alpha')),
+ ('EVENT_TB_EMPTY', Version('0.2.5.2-alpha')),
+ ('EVENT_HS_DESC', Version('0.2.5.2-alpha')),
+ ('EXTENDCIRCUIT_PATH_OPTIONAL', Version('0.2.2.9')),
+ ('FEATURE_EXTENDED_EVENTS', Version('0.2.2.1-alpha')),
+ ('FEATURE_VERBOSE_NAMES', Version('0.2.2.1-alpha')),
+ ('GETINFO_CONFIG_TEXT', Version('0.2.2.7-alpha')),
+ ('HSFETCH', Version('0.2.7.1-alpha')),
+ ('HSPOST', Version('0.2.7.1-alpha')),
+ ('ADD_ONION', Version('0.2.7.1-alpha')),
+ ('LOADCONF', Version('0.2.1.1')),
+ ('MICRODESCRIPTOR_IS_DEFAULT', Version('0.2.3.3')),
+ ('TAKEOWNERSHIP', Version('0.2.2.28-beta')),
+ ('TORRC_CONTROL_SOCKET', Version('0.2.0.30')),
+ ('TORRC_PORT_FORWARDING', Version('0.2.3.1-alpha')),
+ ('TORRC_DISABLE_DEBUGGER_ATTACHMENT', Version('0.2.3.9')),
+ ('TORRC_VIA_STDIN', Version('0.2.6.3-alpha')),
+)