BfiS=HdZdZdZdZdZdZdZddlZdd lm Z ddl Z ddl Z e jdd kDr=ddl mZddlmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddl m!Z"ddl#m$Z%n6ddl&Zddl'Zdd l(mZdd l(mZdd l(mZddl(mZddl(mZddl(mZddl(m!Z"ddl)Z%ddl*m+Z+ddl,m-Z-ddl.m/Z/ej`e1Z2GddeZ3Gdde4Z5d#dZ6d#dZ7d#dZ8d#dZ9d$dZ:dZ;Gdd e<Z=d!Z>e1d"k(rej~e>yy)%z`Utilities using NDG HTTPS Client, including a main module that can be used to fetch from a URL. z R B Wilkinsonz09/12/11z2(C) 2011 Science and Technology Facilities Councilz-BSD - see LICENSE file in top-level directoryzPhilip.Kershaw@stfc.ac.ukz$Id$N) OptionParser)Request) HTTPHandler)HTTPCookieProcessor)HTTPBasicAuthHandler)HTTPPasswordMgrWithDefaultRealm) ProxyHandler) HTTPError) build_opener)HTTPSContextHandler)ssl_context_utilceZdZdZdZeZy)AccumulatingHTTPCookieProcessorztCookie processor that adds new cookies (instead of replacing the existing ones as HTTPCookieProcessor does) cd}t|j|ji|j|j}|j j ||j|}|r[|j|r8|j|}dj||g}|j|||S|j|||S)zProcesses cookies for a HTTP request. @param request: request to process @type request: urllib2.Request @return: request @rtype: urllib2.Request Cookiez; ) Request_ get_full_urldataorigin_req_host unverifiable cookiejaradd_cookie_header get_header has_headerjoinadd_unredirected_header)selfrequestCOOKIE_HEADER_NAME tmp_request new_cookies old_cookiesmerged_cookiess g/var/www/html/BankruptcyAI-uat/bankruptcy-ai/venv/lib/python3.12/site-packages/ndg/httpsclient/utils.py http_requestz,AccumulatingHTTPCookieProcessor.http_request3s&w335w||R&-&=&=&-&:&:<  ((5!,,-?@ !!"45%001CD !%K+E!F//0B0>@ //0BKPN)__name__ __module__ __qualname____doc__r& https_requestr'r%rr/s6!Mr'rceZdZdZy) URLFetchErrorzError fetching content from URLN)r(r)r*r+r-r'r%r/r/Qs)r'r/ct||||\}}}|r5|tjk(r"|j}|j |St |)zReturns data retrieved from a URL. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @return data retrieved from URL or None rhandlers)open_url http_client_OKreadcloser/)urlconfigrr2 return_codereturn_messageresponse return_datas r%fetch_from_urlr>UsS-5S&t>F-H)K{loo5mmo N++r'ct||||\}}}|tjk(rM|j}|j t |d} | j || j |||tjk(fS)aWrites data retrieved from a URL to a file. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @param output_file: output file @type output_file: basestring @return: tuple ( returned HTTP status code or 0 if an error occurred returned message boolean indicating whether access was successful) r1w)r3r4r5r6r7openwrite) r8r9 output_filerr2r:r;r<r=outfiles r%fetch_from_url_to_filerEfsw-5S&t>F-H)Kloo%mmo {C( k"   |(F FFr'clt||||\}}}|r|tjk(r|St|)aReturns data retrieved from a URL. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @param data: HTTP POST data @type data: str @param handlers: list of custom urllib2 handlers to add to the request @type handlers: iterable @return: data retrieved from URL or None @rtype: file derived type r1)r3r4r5r/)r8r9rr2r:r;r<s r%fetch_stream_from_urlrGs=-5S&t>F-H)K{loo5N++r'cp|jrdnd}|jr |j}ntj}t |}|sg}|j ||jr6t |}t|j|}|j||g|jrStt} | jd||jd|jd|j | t||js0|j t!it"jdnR|j$rF|j t!|j$t"jd|j$zt'|d|ji} |j(} | i} t+||| } d} d }d} | j-| }|j.}|j0} t"j3t4j6r*t9|D]\}}t"jd ||| ||fS#t:$rs}|j0} d |j.z}t"j3t4j6r+t"jd |j0|j.Yd}~|d}~wt<$rg}d |j?z}t"j3t4j6r'ddl }t"j|jCYd}~d}~wwxYw) aAttempts to open a connection to a specified URL. @param url: URL to attempt to open @param config: SSL context configuration @type config: Configuration @param data: HTTP POST data @type data: str @param handlers: list of custom urllib2 handlers to add to the request @type handlers: iterable @return: tuple ( returned HTTP status code or 0 if an error occurred returned message or error description response object) r) debuglevelN)realmuriuserpasswdzNot using proxyzConfiguring proxies: %s ssl_contextz %s : %sz Error: %sz%s %s)"debugcookie cookiejar_ CookieJarrappend HTTPHandler_r rOextendhttp_basicauthHTTPBasicAuthHandler_ HTTPPasswordMgrWithDefaultRealm_ add_password_should_use_proxyno_proxy ProxyHandler_logproxiesr headersrrAmsgcode isEnabledForloggingDEBUG enumerate HTTPError_ Exception__str__ traceback format_exc)r8r9rr2rJcjcookie_handler http_handler https_handler auth_handleropenerrarr:r;r<indexrRexcrks r%r3r3sllJ}} ]]  ! ! # 5R8N  OON# ||#z: +F,>,>7AC }56 ,-M-OP !!#'-'<')>q)A " C  % S&// 2 b)* #$  fnn56 +fnn<= 8 D1C1C DFnnGsD'*GKNH.;;w'!mm   GMM *!*2 6 v +uf5 6  22 2hh $sww.   GMM * IIgsxx 1 .$s{{}4   GMM *  IIi**, - .s'A6I L5A)K L5AL00L5c|!tjjdd}n|}tjt |}|j dDcgc]}|jc}D]}|j|k(syycc}w)zDetermines whether a proxy should be used to open a connection to the specified URL, based on the value of the no_proxy environment variable. @param url: URL @type url: basestring or urllib2.Request r]rP,FT) osenvironget urlparse_urlparse_url_as_stringsplitstriphostname)r8r]no_proxy_effectiveurlObjhnps r%r\r\s ZZ^^J;%   s 3 4F"4":":3"?@Qqwwy@ ??b   AsBct|tr|jSt|tr|St dtdt)zReturns the URL string from a URL value that is either a string or urllib2.Request.. @param url: URL @type url: basestring or urllib2.Request @return: URL string @rtype: basestring zExpected type z or ) isinstancerrstr TypeError)r8s r%r|r|sC#x !! C  h() )r'ceZdZdZ ddZy) ConfigurationzConnection configuration. Ncf||_||_||_||_||_||_||_y)a @param ssl_context: SSL context to use with this configuration @type ssl_context: OpenSSL.SSL.Context @param debug: if True, output debugging information @type debug: bool @param proxies: proxies to use for @type proxies: dict with basestring keys and values @param no_proxy: hosts for which a proxy should not be used @type no_proxy: basestring @param cookie: cookies to set for request @type cookie: cookielib.CookieJar (python 3 - http.cookiejar) @param http_basicauth: http authentication, or None @type http_basicauth: tuple of (username,password) @param headers: http headers @type headers: dict N)rOrQr`r]rRrXra)rrOrQr`r]rRrXras r%__init__zConfiguration.__init__s7$'     , r')FNNNNN)r(r)r*r+rr-r'r%rrsHL;?r'rc ftd}|jddddtjj dd |jd d d dd d |jddddd d |jdddddd|jddddd d |jdd d!dd d" |jd#d$dd%dd&|jd'd(d)d*d d+ |jd,d-d.d/d01|j \}}t |d2k7r|jd3|d4}|jr1tjjtj|jr6tjj|jr |j}nd }|jr6tjj|jr |j}nd }|j r6tjj|j r |j }nd }|j" }|j$r_tjj|j$r6t'|j$}|j)} |j+nd } |j,r|j,j/d5d2} nd } i} |j0rG|j0D]8} | j/d5d2\} }|j3| | j5<:t7j8||d |||}t;||j| | 6}|j<r*t?|||j<| d d7\}}tA||tC||} tE| y )8zHUtility to fetch data using HTTP or HTTPS GET from a specified URL. z%prog [options] url)usagez-cz --certificate cert_fileFILEz~/credentials.pemz4Certificate file - defaults to $HOME/credentials.pem)destmetavardefaulthelpz-kz --private-keykey_fileNz3Private key file - defaults to the certificate filez-tz--ca-certificate-dirca_dirPATHz%Trusted CA certificate file directoryz-dz--debug store_truerQFzPrint debug information.)actionrrrz-pz--post-data-file data_filezPOST data filez-fz--fetchrCz Output filez-nz--no-verify-peerno_verify_peerz&Skip verification of peer certificate.z-az --basicauth basicauthz USER:PASSWDzHTTP authentication credentialsz--headerrUraz HEADER: VALUEzAdd HTTP header to request)rrrrrIzIncorrect number of argumentsr:)rXrar)#r add_optionrwpath expanduser parse_argslenerrorrQre getLoggersetLevelrfrexistsrrrrrAr6r7rr}ralstripr~rmake_ssl_contextrrCrE SystemExitr>print)parseroptionsargsr8rrr verify_peerrrrXrarkeyvalrOr9r:r;s r%mainr4s 5 6F dO+v gg001DEQS dO*f"PR d2$"BD dIl#57 d.[$d+- dIM6"8 d.|+UCE dM +"<> j -79'')OWd 4yA~ 45 q'C}}$$W]]3BGGNN7+;+;<##RWW^^G,=,=>%%  ~~"''..8,,,KRWW^^G,=,=>**+ ~~ **00a8G 0AwwsAHC#&::rErGr3r\r|objectrr basicConfigr-r'r%rs D = )  ! A'&2:JLM<4$""+3CEM5/ =5,g!!&:!D*I*,"G2,*\3~$)"F