hR^dZgdZddlZddlZddlZddlZddlZddl Zddl Zddl m Z GddejjZdZdZdZd Zd Zd ZGd d Zddej.dddfdZddZddZddZddZddZddZddZddZ e!dk(rddl"Z"e"jFe!yy)aencutils - encoding detection collection for Python A collection of helper functions to detect encodings of text files (like HTML, XHTML, XML, CSS, etc.) retrieved via HTTP, file or string. :func:`getEncodingInfo` is probably the main function of interest which uses other supplied functions itself and gathers all information together and supplies an :class:`EncodingInfo` object. example:: >>> import encutils >>> info = encutils.getEncodingInfo(url='http://jaraco.com') >>> print(info) utf-8 >>> info # doctest:+ELLIPSIS >>> print(info.logtext) HTTP media_type: text/html HTTP encoding: utf-8 Encoding (probably): utf-8 (Mismatch: False) references XML RFC 3023 (http://www.ietf.org/rfc/rfc3023.txt) easier explained in - http://feedparser.org/docs/advanced.html - http://www.xml.com/pub/a/2004/07/21/dive.html HTML http://www.w3.org/TR/REC-html40/charset.html#h-5.2.2 TODO - parse @charset of HTML elements? - check for more texttypes if only text given )buildlogencodingByMediaType getHTTPInfo getMetaInfodetectXMLEncodinggetEncodingInfo tryEncodings EncodingInfoN)MessageceZdZdZdZdZy)_MetaHTMLParserz6Parse given data for .Nc |dk(ry|jsl|Dcic]$\}}|j|j&}}}|jddjdk(r|jd|_yyyycc}}w)Nmetaz http-equiv content-typecontent) content_typelowergetstrip)selftagattrsavattss Q/var/www/api/v1/venv_getwork_v1/lib/python3.12/site-packages/encutils/__init__.pyhandle_starttagz_MetaHTMLParser.handle_starttagEsw &=!2!25:;UTQAGGIqwwy(UD;xx b)//1^C$(HHY$7!D"3=;s)B)__name__ __module__ __qualname____doc__rrrr r @s@L8r$r c"eZdZdZdZdZdZy)r a All encoding related information, returned by :func:`getEncodingInfo`. Attributes filled: - ``encoding``: The guessed encoding Encoding is the explicit or implicit encoding or None and always lowercase. - from HTTP response * ``http_encoding`` * ``http_media_type`` - from HTML element * ``meta_encoding`` * ``meta_media_type`` - from XML declaration * ``xml_encoding`` - ``mismatch``: True if mismatch between XML declaration and HTTP header. Mismatch is True if any mismatches between HTTP header, XML declaration or textcontent (meta) are found. More detailed mismatch reports are written to the optional log or ``logtext`` Mismatches are not necessarily errors as preferences are defined. For details see the specifications. - ``logtext``: if no log was given log reports are given here ctdx|_x|_x|_x|_x|_x|_x|_|_y)zVInitialize all possible properties to ``None``, see class description N)encodingmismatchlogtext http_encodinghttp_media_type meta_encodingmeta_media_type xml_encodingrs r__init__zEncodingInfo.__init__s[ MQ Q  Q  Q  Qt7I Q   Q   Q!%!58Ir$c4|jr |jSy)z7Output the guessed encoding itself or the empty string.r)r,r4s r__str__zEncodingInfo.__str__s ==== r$c d|jjd|jjd|jd|jdt |dd S)N<.z object encoding=z mismatch=z at 0xx>) __class__r rr,r-idr4s r__repr__zEncodingInfo.__repr__s4>>,,-Qt~~/F/F.GGXY]YfYfXiistxuBuBtCCIJLMQJRSTIUUVW Wr$N)rr r!r"r5r7r?r#r$rr r as>QWr$r encutilsINFOwz%(levelname)s %(message)scLddl}|j|}|r|j||}n|j|}|j |} |j | |j ||j|jj||j|S)aeHelper to build a basic log - if `filename` is given returns a log logging to `filename` with mode `filemode` - else uses a log streaming to `stream` which defaults to `sys.stderr` - `level` defines the level of the log - `format` defines the formatter format of the log :returns: a log with the name `logname` r N) logging getLogger FileHandler StreamHandler Formatter setFormatter addHandlersetLevel__dict__rrA) lognamelevelstreamfilenamefilemodeformatrDloghdlr formatters rrrs(   G $C""8X6$$V,!!&)Ii NN4LL!!%%eW\\:; Jr$c|stSgd}gd}|jj}||vsJtj|d|tj tj ztjzrtS||vsJtj|d|tj tj ztjzrtS|dk(rtS|dk(rtS|jdrtStS)zF :returns: type as defined by constants in this class )zapplication/.*?\+xmlzapplication/xmlzapplication/xml-dtdz&application/xml-external-parsed-entity)ztext\/.*?\+xmlztext/xmlztext/xml-external-parsed-entityr z text/htmlztext/cssztext/) _OTHER_TYPErrrematchISX_XML_APPLICATION_TYPE_XML_TEXT_TYPE_HTML_TEXT_TYPE _TEXT_UTF8 startswith _TEXT_TYPE) media_typerSxml_application_typesxml_text_typess r_getTextTypeByMediaTyperfs  XN!!#))+J**bhha *bddRTTkBDD.@/%$ ~ %q:rttbdd{RTT'9* { " z !   w 'r$cH|ddjddk7rtStS)zSCheck if given text is XML (**naive test!**) used if no content-type given Nz>#  & 3 $h/ x r$ct} |j||j rvt }|j |d<|j}|jd}|r|j}|r$|jd||jd|||fSdx}}||fS#tjj$rYwxYw)a= :param text: a byte string :returns: ``(media_type, encoding)`` information from (first) X/HTML Content-Type ```` element if available in `text`. XHTML format:: rcharsetzHTML META media_type: %szHTML META encoding: %sN) r feedhtmlparserHTMLParseErrorrr rv get_paramrru)rkrSpmrcr,s rrrs A  t  ~~ INN.'') ;;y) ~~'H  HH/ < HH-x 8 x !%$ X x # ;; % %   sB''CCcHt|trtj|}dddddd}|j }|j dt tt|jd\}}}}|j||||f} | s,|j|||d f} | s|j||d d f} | r)|r|jd | z|j || S|j d|jd } d } tj| tj} | j| } |j || r7| j!d j#}|r|jd|z|S|r|r|jdyy )apAttempt to detect the character encoding of the xml file given by a file object `fp`. `fp` must not be a codec wrapped file object! `fp` may be a string or unicode string though. Based on a recipe by Lars Tiede: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/363841 which itself is based on Paul Prescotts recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52257 :returns: - if detection of the BOM succeeds, the codec name of the corresponding unicode charset is returned - if BOM detection fails, the xml declaration is searched for the encoding attribute and its value returned. the "<" character has to be the very first in the file then (it's xml standard after all). - if BOM and xml declaration fail, utf-8 is returned according to XML 1.0. utf_32_be utf_32_le utf_16_be utf_16_lern))r r )rrr r )rrNN)rrNN)Nr r)NzXML BOM encoding: %sia ^<\?xml # w/o BOM, xmldecl starts with # what's matched in the brackets will be named encstr [^"']+ # every character not delimiter (not overly exact!) ) # closes the brackets pair for the named group ["'] # attribute end delimiter .*? # some chars optionally (standalone decl or whitespace) \?> # xmldecl end encstrzXML encoding="%s"zXML encoding default utf-8) isinstancestrioStringIOtellseektuplemapordreadrrurXcompileVERBOSEsearchgroupr)fprSincludeDefaultbomDictoldFPbyte1byte2byte3byte4 bomDetectionbufferxmlDeclPattern xmlDeclRErYencs rrrCs,"c [[_ #."-"-"-") G GGIEGGAJ#(S"''!*)=#> UE5%;;ueU;? ";;udD'ABL  HH+l: ; GGAJ WWT]F N >2::6I   V $EGGEN kk(#))+  HH(3. / 56r$cT ddl}|j|d}|S#t$rd}|r|j|n t |d}d}|D]T} |j |d|k(r' d|j dvrYyn#t $rYnwxYw|ccYS#t $rYRwxYwY|SwxYw) a7If installed uses chardet http://chardet.feedparser.org/ to detect encoding, else tries different encodings on `text` and returns the one that does not raise an exception which is not very advanced or may be totally wrong. The tried encoding are in order 'ascii', 'iso-8859-1', 'windows-1252' (which probably will never happen as 'iso-8859-1' can decode these strings too) and lastly 'utf-8'. :param text: a byte string :returns: Working encoding or ``None`` if no encoding does work at all. The returned encoding might nevertheless be not the one intended by the author as it is only checked if the text might be encoded in that encoding. Some texts might be working in "iso-8859-1" *and* "windows-1252" *and* "ascii" *and* "utf-8" and ... r Nr,zGUsing simplified encoding detection, you might want to install chardet.)rorprnrpu€z windows-1252)chardetdetect ImportErrorwarnprintdecodeUnicodeDecodeError)rkrSrr,msg encodingses rrrs$ >>$' 3> O; W  HHSM #J A  A 1$ DKK$??#1@-&   O;s]3B'B!B''A>:B'=B'> B B' B  B' BB'BB'&B'c|rtjj|}| |j}|d}t }t j}|s t|d}|r0t||\|_ |_ t|j|}n t||}|tk(r t|||_|t&k(r t||d|_|t&k(s |t(k(rt+||\|_|_|j|_d|_|tk(r|j0s|j |_n|t&k(r`|j0s|j.|_|j0st5|j|_|j0syt7||_nh|t8k(s |t(k(r'|j0sJt5|j|_n/|t:k(r&|j0st5|j|_|jrX|j rL|j|j k7r3d|_|j=d|jd |j d |jrX|j.rL|j|j.k7r3d|_|j=d|jd |j.d |j rX|j.rL|j |j.k7r3d|_|j=d|j d |j.d |j?d |j0|j2|jA|_!|S#t$rYXwxYw#t"t$f$r d|_YwxYw#t"t$f$r d|_YwxYw)a Find all encoding related information in given `text`. Information in headers of supplied HTTPResponse, possible XML declaration and X/HTML ```` elements are used. :param response: HTTP response object, e.g. via ``urllib.urlopen('url')`` :param text: a byte string to guess encoding for. XML prolog with encoding pseudo attribute or HTML meta element will be used to detect the encoding :param url: When given fetches document at `url` and all needed information. No `reponse` or `text` parameters are needed in this case. :param log: an optional logging logger to which messages may go, if no log given all log messages are available from resulting ``EncodingInfo`` :returns: instance of :class:`EncodingInfo`. How the resulting encoding is retrieved: XML RFC 3023 states if media type given in the Content-Type HTTP header is application/xml, application/xml-dtd, application/xml-external-parsed-entity, or any one of the subtypes of application/xml such as application/atom+xml or application/rss+xml etc then the character encoding is determined in this order: 1. the encoding given in the charset parameter of the Content-Type HTTP header, or 2. the encoding given in the encoding attribute of the XML declaration within the document, or 3. utf-8. Mismatch possibilities: - HTTP + XMLdecla - HTTP + HTMLmeta application/xhtml+xml ? XMLdecla + HTMLmeta If the media type given in the Content-Type HTTP header is text/xml, text/xml-external-parsed-entity, or a subtype like text/Anything+xml, the encoding attribute of the XML declaration is ignored completely and the character encoding is determined in the order: 1. the encoding given in the charset parameter of the Content-Type HTTP header, or 2. ascii. No mismatch possible. If no media type is given the XML encoding pseuso attribute is used if present. No mismatch possible. HTML For HTML served as text/html: http://www.w3.org/TR/REC-html40/charset.html#h-5.2.2 1. An HTTP "charset" parameter in a "Content-Type" field. (maybe defaults to ISO-8859-1, but should not assume this) 2. A META declaration with "http-equiv" set to "Content-Type" and a value set for "charset". 3. The charset attribute set on an element that designates an external resource. (NOT IMPLEMENTED HERE YET) Mismatch possibilities: - HTTP + HTMLmeta TEXT For most text/* types the encoding will be reported as iso-8859-1. Exceptions are XML formats send as text/* mime type (see above) and text/css which has a default encoding of UTF-8. Nrz %(message)s)rOrRF)rT"z " (HTTP) != "z" (XML) encoding mismatchz!" (HTML ) encoding mismatchz " (XML) != "z&Encoding (probably): %s (Mismatch: %s))"urllibrequesturlopenrOSErrorr rrrrr0r/rfrlr]rr3AttributeError ValueErrorr_rbrr2r1r,r-rrr^r`warningrugetvaluer.)rxrkrSurlencinfo logstreamrss rrrs\b >>))#. | ==?D |nG I i >9DXs9S6!6*7+B+BCH c*(( (#4T3#?G ?" (#4T3u#UG ?"h*&<9DT39O6!6,,GG((&33G  _ $&44G 273J3JKG +D1G  ^ #x:'=273J3JKG  Z 273J3JKG     ! !W%9%9 9 $$g&:&: <   ! !  ! !W%:%: : $$g&;&; =   ! !  G$9$9 9 ##W%:%: < HH0'2B2BGDTDT ((*GO N[   2 + (#'G  ( + (#'G  (s5N.N N. N NN+*N+.OO__main__)N)NT)NrNN)$r"__all__ html.parserr|rrXsys urllib.errorr urllib.parseurllib.request email.messager r} HTMLParserr r]r^r_rbr`rWr stderrrrfrlrrrrrrrpydochelpr#r$rrs(T  ! 8dkk,, 8   0W0Wh   ::   ' #L F@ ." J^B4nFR zEJJxr$