֦i$ d Z ddlZddlZddlZddlZdgZ ej dd Z G d d Z G d d Z
G d d
Z y) a% robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://www.robotstxt.org/norobots-rfc.txt
NRobotFileParser RequestRatezrequests secondsc Z e Zd ZdZddZd Zd Zd Zd Zd Z d Z
d Z d
Z d Z
d Zd
Zy)r zs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
c z g | _ g | _ d | _ d| _ d| _ | j | d| _ y )NFr )entriessitemaps
default_entry disallow_all allow_allset_url last_checkedselfurls +/usr/lib64/python3.12/urllib/robotparser.py__init__zRobotFileParser.__init__ s;
! ! S c | j S )zReturns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r
r s r mtimezRobotFileParser.mtime& s r c 6 ddl }|j | _ y)zYSets the time the robots.txt file was last fetched to the
current time.
r N)timer
)r r s r modifiedzRobotFileParser.modified/ s
IIKr c p || _ t j j | dd \ | _ | _ y)z,Sets the URL referring to a robots.txt file. N)r urllibparseurlparsehostpathr s r r zRobotFileParser.set_url7 s- %|| 4 4S 9!A > 49r c t j j | j }|j }| j |j
d j y# t j j $ rU}|j dv rd| _ n%|j dk\ r|j dk rd| _ |j Y d}~yd}~ww xY w)z4Reads the robots.txt URL and feeds it to the parser.zutf-8)i i Ti i N)r requesturlopenr readr decode
splitlineserror HTTPErrorcoder
r close)r frawerrs r r% zRobotFileParser.read< s
9&&txx0A &&(C JJszz'*557 8 ||%% xx:%$(!S SXX^!% IIKK s )A* * C A CC c d|j v r| j || _ y y | j j | y N*)
useragentsr r append)r entrys r
_add_entryzRobotFileParser._add_entryJ s= %"" "!!)%*" *
LL &r c d}t }| j |D ] }|s4|dk( r
t }d}n"|dk( r | j | t }d}|j d }|dk\ r|d| }|j }|sh|j dd }t
| dk( s|d j j |d< t j j |d j |d< |d dk( rB|dk( r| j | t }|j j |d d}*|d dk( r3|dk7 s9|j j t |d d d}e|d d
k( r3|dk7 st|j j t |d d d}|d d k( r?|dk7 s|d j j rt! |d |_ d}|d d
k( r|dk7 s|d j d }t
| dk( rk|d j j rJ|d j j r)t% t! |d t! |d |_ d}|d dk( s| j( j |d |dk( r| j | yy)zParse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r #N:z
user-agentdisallowFallowTz crawl-delayz request-rate/sitemap)Entryr r5 findstripsplitlenlowerr r unquoter2 r3 rulelinesRuleLineisdigitintdelayr req_rater )r linesstater4 lineinumberss r r zRobotFileParser.parseS s
DA: !GE EaZOOE* !GE E #AAvBQx::>
\\**6<<+?+?+DE
ll%%r"Z__ j . .
0C0C'E Fll %C\\E * s++ " %%//4 4r c | j sy | j D ]! }|j | s|j c S | j r| j j S y N)r r rW rI r r rY r4 s r crawl_delayzRobotFileParser.crawl_delay sT zz|\\E * {{" " %%++ +r c | j sy | j D ]! }|j | s|j c S | j r| j j S y r] )r r rW rJ r r^ s r request_ratez RobotFileParser.request_rate sT zz|\\E * ~~% " %%.. .r c 4 | j sy | j S r] )r r s r site_mapszRobotFileParser.site_maps s }}}} r c | j }| j || j gz }dj t t | S )Nz
)r r joinmapstr)r r s r __str__zRobotFileParser.__str__ s@ ,, ) !3!3 44G{{3sG,--r N)rQ )__name__
__module__ __qualname____doc__r r r r r% r5 r r[ r_ ra rc rh r r r r sE
!(?
9 'G#R:
.r c " e Zd ZdZd Zd Zd Zy)rF zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c |dk( r|sd}t j j t j j | }t j j | | _ || _ y )NrQ T)r r rR r rV r! rX )r r! rX s r r zRuleLine.__init__ sP 2:i I||&&v||'<'>zTADIIMMr N)ri rj rk rl r rW rh rm r r rF rF s 1#BNr rF c ( e Zd ZdZd Zd Zd Zd Zy)r> z?An entry has one or more user-agents and zero or more rulelinesc < g | _ g | _ d | _ d | _ y r] )r2 rE rI rJ r s r r zEntry.__init__ s
r c g }| j D ] }|j d| | j |j d| j | j 7| j }|j d|j d|j
|j
t t | j dj | S )Nz User-agent: z
Crawl-delay: zRequest-rate: r<
) r2 r3 rI rJ requestssecondsextendrf rg rE re )r retagentrates r rh z
Entry.__str__ s __E JJ eW- . % :: ! JJ tzzl3 4 == $==D JJ
a ~F G
3sDNN+,yy~ r c |j d d j }| j D ] }|dk( r y|j }||v s y y)z2check if this entry applies to the specified agentr< r r1 TF)rA rC r2 )r rY r~ s r rW zEntry.applies_to sS OOC(+113 __E|KKME !
% r c d | j D ]! }|j | s|j c S y)zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)rE rW rX )r rr rM s r rX zEntry.allowance s- NNDx(~~% # r N)ri rj rk rl r rh rW rX rm r r r> r> s I
r r> ) rl collections urllib.errorr urllib.parseurllib.request__all__
namedtupler r rF r> rm r r r sX
$k$$]4FG . .DN N$( (r