3 h2@sddZddlmZddlZejeeeedddlmZddlm Z m Z m Z eZ Gdd d eZ dS) z3 Python Lexical Analyser Scanning an input stream )absolute_importN)BOLEOLEOF NOT_FOUND)Errors)rrrc@seZdZdZdddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ dddZddZeddZdS) Scannera A Scanner is used to read tokens from a stream of characters using the token set specified by a Plex.Lexicon. Constructor: Scanner(lexicon, stream, name = '') See the docstring of the __init__ method for details. Methods: See the docstrings of the individual methods for more information. read() --> (value, text) Reads the next lexical token from the stream. position() --> (name, line, col) Returns the position of the last token read using the read() method. begin(state_name) Causes scanner to change state. produce(value [, text]) Causes return of a token value to the caller of the Scanner. NcCsd|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_ ||_ ||_ ||_ g|_d|_|jdd|_d|_d|_t|_d|_|dk r|d|d |_|_dS)a Scanner(lexicon, stream, name = '') |lexicon| is a Plex.Lexicon instance specifying the lexical tokens to be recognised. |stream| can be a file object or anything which implements a compatible read() method. |name| is optional, and may be the name of the file being scanned or any other identifying string. rr rN)r rr)r rr)tracebuffer buf_start_posnext_poscur_poscur_line start_poscurrent_scanner_position_tuplelast_token_position_tupletext state_namelexiconstreamnamequeue initial_statebegincur_line_startrcur_char input_state)selfrrrZ initial_posr!f/var/www/html/CrowdFlow/Picklecon/ble_analysis_env/lib/python3.6/site-packages/Cython/Plex/Scanners.py__init__Ms0  zScanner.__init__cCsr|j}xR|sX|j\|_}|dkr6|jd|jq|j||j}|dk r|j|qW|d\}|_|d=|S)a" Read the next lexical token from the stream and return a tuple (value, text), where |value| is the value associated with the token as specified by the Lexicon, and |text| is the actual string read from the stream. Returns (None, '') on end of file. Nr)r scan_a_tokenrproduceeofZperformr)r ractionvalueresultr!r!r"readus  z Scanner.readcCs|jjd||f|fdS)Nr)rinsert)r tokenr(positionr!r!r"unreadszScanner.unreadcCs|jS)N)r)r r!r!r"get_current_scan_posszScanner.get_current_scan_poscCs|j|_|j|j|j|jf|_|j}|dk rr|jrLtd||j|jf|j |j|j |j|j }||fS|j|jkr|j t kr|j |j dks|j tkrdStj||jdS)z Read the next input sequence recognised by the machine and return (text, action). Returns ('', None) on end of file. Nz"Scanner: read: Performing %s %d:%dr )r N)rrrrrrrun_machine_inlinedr printr rrr next_charrrZUnrecognizedInputr)r r'rr!r!r"r$s"   zScanner.scan_a_tokencCs|j}|j}|j}|j}|j}|j}|j}|j}|j} t |} d\} } } }}}}|j }x|rzt d|d||t |f|d}|dk r|||||||f\} } } }}}}|}|j |t}|tkr|o|j d}|r|rt d|d|}|d kr|}|| }|| kr ||}|d 7}nh|j| }|jjd }|j|d|}||_| |7} | |_t |} ||8}|r||}|d 7}nd}|d krt}d }n|st}d }n|}nP|d krd }d}n<|dkr|d 7}|}}t}d }n|d krt}d}nd}qZ|rt d| dk r>| | | ||||f\}}}}}}}nd}PqZW||_||_||_||_||_||_|r|dk rt d||S)z; Inlined version of run_machine for speed. Nrr zState %d, %d/%d:%s -->numberr'elsezState %dri r ZblockedzDoing %s)Nrrrr rr)rrrrrrrr rlenr r1reprgetrrrr*rrr)r staterrrrrrr rZbuf_lenZb_actionZ b_cur_posZ b_cur_lineZb_cur_line_startZ b_cur_charZ b_input_stateZ b_next_posr r'cZ new_stateZ buf_indexdiscarddatar!r!r"r0s               zScanner.run_machine_inlinedcCs|j}|jr tdd ||jf|dkrh|j|_|j}|dkrNt|_d|_q|s`t|_d|_q||_nd|dkr~d|_d|_nN|dkr|jd7_|j|_ |_t |_d|_n|dkrt |_d |_nd |_|jrtd ||j|jfdS) NzScanner: next: %s [%d] %d rr5r r6r7r8r z--> [%d] %d %rz ) rr r1rrZ read_charrrrrrr)r rr=r!r!r"r2s6zScanner.next_charcCs|jS)am Return a tuple (name, line, col) representing the location of the last token read using the read() method. |name| is the name that was provided to the Scanner constructor; |line| is the line number in the stream (1-based); |col| is the position within the line of the first character of the token (0-based). )r)r r!r!r"r-8s zScanner.positioncCs|jS)zX Python accessible wrapper around position(), only for error reporting. )r-)r r!r!r" get_positionCszScanner.get_positioncCs|jj||_||_dS)z8Set the current state of the scanner to the named state.N)rZget_initial_staterr)r rr!r!r"rIsz Scanner.begincCs(|dkr|j}|jj||f|jfdS)a Called from an action procedure, causes |value| to be returned as the token value from read(). If |text| is supplied, it is returned in place of the scanned text. produce() can be called more than once during a single call to an action procedure, in which case the tokens are queued up and returned one at a time by subsequent calls to read(), until the queue is empty, whereupon scanning resumes. N)rrappendr)r r(rr!r!r"r%Os zScanner.producecCsdS)z_ Override this method if you want something to be done at end of file. Nr!)r r!r!r"r&^sz Scanner.eofcCs |jdS)Nr)r)r r!r!r" start_lineeszScanner.start_line)r N)N)__name__ __module__ __qualname____doc__r#r*r.r/r$r0r2r-rBrr%r&propertyrDr!r!r!r"r s (m  r )rH __future__rZcythonZdeclareobjectr rZRegexpsrrrrr r!r!r!r"s