Monday, November 08, 2010 at 12:06 AM.
system.verbs.builtins.tcp.httpClient
on httpClient (method="GET", server="", port=nil, path="", proxy=user.webBrowser.proxy.domain, proxyPort=user.webBrowser.proxy.port, proxyUserName=user.webBrowser.proxy.userName, proxyPassword=user.webBrowser.proxy.password, data="", datatype="", username="", password="", adrHdrTable=nil, cookiesOn=false, debug=false, timeOutTicks=60*30, flMessages=true, ctFollowRedirects=0, flJustHeaders=false, flAcceptOpml=false, flUseProxy=nil, adrRedirectInfo=nil) { <<Changes <<1/22/06; 7:33:35 PM by DW <<The code that checked to see if we're running on the "samemachine" wasn't checking if server was 127.0.0.1, and was only checking the port on user.inetd.config.http, when there may be more listeners running on different ports, as there are in the OPML Editor. It would only shortcut of it was localhost and if it was running on one of the daemons. <<2/7/03; 3:49:12 PM by JES <<When testing for proxy-exempt servers, don't inclue the port number if specified in the server parameter. <<10/20/02; 2:32:23 AM by JES <<Respect user.webBrowser.proxy.exceptions. <<10/16/02; 6:55:56 PM by JES <<Removed debugging code. <<10/14/02; 11:10:16 AM by JES <<If server is 127.0.0.1 or "localhost", set flUseProxy to false. <<10/13/02; 6:23:28 PM by DW <<New optional param -- adrRedirectInfo. If it's non-nil (the default) and we're redirecting, we fill the table with info about the redirect. The RSS aggregator needs this information, and it may prove useful in other places. See xml.rss.readService for an example of its usage. <<12/19/01; 9:58:10 PM by JES <<Added a call to webBrowser.init, which initializes prefs at user.webBrowser.proxy. Added optional parameter, flUseProxy, which defaults to nil. If nil, then the value at user.webBrowser.proxy.enabled is used. Pass the value of flUseProxy when following redirects. <<8/14/01; 1:49:11 PM by JES <<When recursing to following redirects, pass in the flJustHeaders and flAcceptOpml parameters. <<8/9/01; 6:10:17 PM by JES <<If the stream is unexpectedly closed by the remote host, call tcp.abortStream, to avoid consuming the available connections. <<8/8/01; 3:21:27 PM by JES <<If no Accept header has been passed in, add an Accept header with the value: text/x-opml, */*, so that we get the response in opml instead of html. <<3/1/01; 1:57:33 PM by PBS <<Trim white space around the content-length header before coercing it to a number. If a server adds white space here, the coercion would fail, resulting in a connection counter leak. (History: this was discovered in Radio when reading the Red Herring channel.) <<1/10/01; 10:36:30 AM by DW <<Add an optional parameter, flJustHeaders, if true, we don't read the body, we just return the header portion of the response. This is useful if you just want to know how big a download is, and want to defer downloading big things until the middle of the night when it's quiet on the Internet. <<1/7/01; 9:44:18 PM by DW <<In Radio, set User-Agent to Radio UserLand. <<Changed the header so it doesn't use backslashes. The outliner now supports wrapping headlines. <<9/22/00; 11:27:50 AM by PBS <<Respect the user's offline settings. <<11/5/99; 3:00:39 AM by AR <<If the server doesn't send a Content-Length header, read until connection closes. <<10/21/99; 3:57:29 PM by AR <<Adapted to use new TCP verbs in 6.1. <<6/14/99; 5:23:37 AM by AR <<If flMessages is true, we now clean up the message area before we return. <<6/13/99; 9:20:04 AM by AR <<Both the read and write loop now yield time to other Frontier threads and to the system. <<The Host header is now generated correctly if the request goes thru a proxy server. <<Fixed a few inconsistencies in the cookie handling code. Thinks should now work correctly when connecting via a proxy server. <<Added an optional parameter, ctFollowRedirects, determining the number of times that HTTP redirects will be followed. In order to preserve previous behaviour, the default value is 0. <<3/4/99; 11:54:30 AM by PBS <<If adrHdrTable^ contains a "Host" sub-item, don't add a Host header item. <<3/2/99; 10:43:18 AM by DW <<if adrHdrTable^ contains a "User-Agent" sub-item, don't add a User-Agent header item <<10/25/98; 6:33:59 AM by DW <<allow the use of a colon in the server name to specify the port <<example: www.scripting.com:80 <<if present, we use that port, not the one specified in the call <<the assumption is that most callers either specify 80 or don't specify the port at all <<for crawlers, for example, they definitely want to use the port specified in the server string <<some commands, esp in the webBrowser class, get the url from another piece of software <<9/28/98; 4:42:12 AM by DW <<if we're sending to the same machine, set the stream ID to 0 <<user.webserver.responders.CGI.methods.any is copying it, hope no CGIs are using it! <<9/17/98; 7:11:56 AM by DW <<msgs now say that they're from tcp.httpClient. <<changed Connection to xxx timed out message to... <<Can't process the request because the connection to xxx timed out. <<allow server to be "localhost", if so, we totally optimize the functionality <<2/13/98; 8:57:10 AM by DW <<Initial code assembled from various sample scripts by Dave Winer. <<The goal is to have a single documented way to talk to HTTP servers. <<It's got a long parameter list, but requires no configuration, doesn't run off a user.xxx table. <<The target user of this script is someone who needs to understand how the code works. <<It will be distributed with Frontier on both platforms as soon as we agree that this is complete and works. <<When you call it, use parameter labels, don't depend on the order of these params. <<For examples, see tcp.examples. if tcp.isOffline () { //PBS 09/22/00: check offline status scriptError ("Can't send HTTP request because TCP is offline.")}; webBrowser.init (); try { //set flUseProxy if (flUseProxy == nil) and (typeOf (flUseProxy) == unknownType) { flUseProxy = user.webBrowser.proxy.enabled}; if (server == "127.0.0.1" or string.lower (server) == "localhost") { //JES 10/14/02: don't call the loopback address through the proxy flUseProxy = false}; if flUseProxy { //check the exceptions table -- if we find a match then set flUseProxy to false on wildcardIpMatch (s, oneip) { local (i); if s == oneip { //optimization return (true)}; for i = 1 to 4 { try { local (part = string.nthField (oneip, ".", i)); local (pattern = string.nthField (s, ".", i)); if pattern == "*" { continue}; if pattern contains "-" { local (min = number (string.nthField (pattern, "-", 1)) ); local (max = number (string.nthField (pattern, "-", 2)) ); local (n = number (part)); if n >= min and n <= max { continue} else { return (false)}}; if pattern != part { return (false)}} else { return (false)}}; return (true)}; local (flIpAddress = false, hostToCheck = string.lower (string.nthField (server, ":", 1)) ); try { //see if flIpAddress should be true local (serverWithoutDots = string.replaceAll (server, ".", "")); double (serverWithoutDots); //will error if non-numeric characters are present flIpAddress = true} else { //check domains, and then try a DNS lookup local (adr); local (lowerserver = string.lower (server)); for adr in @user.webBrowser.proxy.exceptions { if string.wildcardMatch (hostToCheck, string.lower (nameOf (adr^)) ) { if adr^ { flUseProxy = false; break}}}; if flUseProxy { //try a reverse lookup so we can check against IP address try { hostToCheck = tcp.dns.getDottedId (server); flIpAddress = true}}}; if flUseProxy { //check to see if this specific IP is excepted if defined (user.webBrowser.proxy.exceptions.[hostToCheck]) { if user.webBrowser.proxy.exceptions.[hostToCheck] { flUseProxy = false}}}; if flUseProxy { //check for IP-address exceptions which contain wildcard characters and ranges local (i); for i = 1 to sizeOf (user.webBrowser.proxy.exceptions) { if user.webBrowser.proxy.exceptions[i] { local (oneexception = nameOf (user.webBrowser.proxy.exceptions[i])); if wildcardIpMatch (oneexception, hostToCheck) { flUseProxy = false; break}}}}}}; on displayMsg (s) { msg ("tcp.httpClient: " + s)}; local (myAddress = tcp.myDottedID (), flSameMachine = false); bundle { //look for the port specified in the server string if server contains ":" { try {port = number (string.nthField (server, ":", 2))}; //it might not be a number server = string.nthField (server, ":", 1)}; if port == nil { //no port was specified, either in the call or in the server string port = 80}}; <<bundle //see if it's the same machine <<local (lowerserver = string.lower (server)) <<if (lowerserver == "localhost") or (lowerserver == "127.0.0.1") <<flSameMachine = true <<else <<try {flSameMachine = tcp.equalNames (server, myAddress)} //DW 3/9/98 << <<bundle //check that we're running on the requested port, 1/22/06 by DW <<if flsamemachine //may turn false <<local (adrd, flfound = false) <<for adrd in @user.inetd.config <<if port == adrd^.port <<if inetd.isDaemonRunning (adrd) <<flfound = true <<break <<if not flfound <<flsamemachine = false <<bundle //old code <<PBS 9/18/98: check that the server on the requested port is Frontier. <<if port != user.inetd.config.http.port <<flSameMachine = false //it's the same machine, but it's not Frontier as a server <<else <<if not (inetd.isDaemonRunning (@user.inetd.config.http)) <<flSameMachine = false //Frontier isn't running as a webserver, there must be another server on this machine if not (path beginsWith "/") { path = "/" + path}; local (httpCommand = "", adrCookiesTable = @user.webBrowser.cookies); bundle { //build the HTTP request in "httpCommand" on add (s) { httpCommand = httpCommand + s + "\r\n"}; local (uri = path); if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server: set uri accordingly uri = "http://" + server; if port != 80 { uri = uri + ":" + port}; uri = uri + path}; add (method + " " + uri + " HTTP/1.0"); if (adrHdrTable == nil) or (not defined (adrHdrTable^.["User-Agent"])) { //3/2/99; 10:43:10 AM by DW if system.environment.isPike { add ("User-Agent: Radio UserLand/" + Frontier.version () + " (" + sys.os () + ")")} else { add ("User-Agent: Frontier/" + Frontier.version () + " (" + sys.os () + ")")}}; local (host = "Host: " + server); if port != 80 { host = host + ":" + port}; if (adrHdrTable == nil) or (not defined (adrHdrTable^.["Host"])) { //3/4/99; 11:54:30 AM by PBS add (host)}; if flAcceptOpml { //8/8/01; 3:19:50 PM by JES if (adrHdrTable == nil) { add ("Accept: text/x-opml, */*")} else { //headers were passed in if defined (adrHdrTable^.Accept) { local (acceptHeader = adrHdrTable^.Accept); if not (acceptHeader contains "text/x-opml") { if sizeOf (string.trimWhiteSpace (acceptHeader)) > 0 { acceptHeader = "text/x-opml, " + acceptHeader} else { acceptHeader = "text/x-opml"}}; if not (acceptHeader contains "*/*") { acceptHeader = acceptHeader + ", */*"}; add ("Accept: " + acceptHeader)} else { add ("Accept: text/x-opml, */*")}}}; if sizeOf (data) > 0 { add ("Content-Type: " + datatype); add ("Content-length: " + sizeOf (data))}; if (sizeOf (username) > 0) or (sizeOf (password) > 0) { add ("Authorization: Basic " + base64.encode (username + ":" + password, 0))}; if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server if (sizeOf (proxyUsername) > 0) or (sizeOf (proxyPassword) > 0) { //the proxy requires authentication add ("Proxy-Authorization: Basic " + base64.encode (proxyUsername + ":" + proxyPassword, 0))}}; if adrHdrTable != nil { //added 3.27.98 by ASG to include header items local (i); for i = 1 to sizeOf (adrHdrTable^) { local (headerName = nameOf (adrHdrTable^[i])); if flAcceptOpml { //8/8/01; 3:19:50 PM by JES: Only add the Acept header once if string.lower (headerName) == "accept" { continue}}; add (headerName + ": " + adrHdrTable^[i])}}; if cookiesOn and defined (adrCookiesTable^) { //add cookies to the request header: ASG 4/9/98 local (i, j, k, cookiePath = "/", cookiestring); if path != "/" { cookiePath = string.popSuffix (path, "/")}; for i = 1 to sizeOf (adrCookiesTable^) { //loop through cookies table local (adrdomain = @adrCookiesTable^ [i]); if host endsWith nameOf (adrdomain^) { for j = 1 to sizeOf (adrdomain^) { local (adrcookie = @adrdomain^ [j]); if cookiePath contains nameOf (adrcookie^) { for k = 1 to sizeOf (adrcookie^) { if adrcookie^ [k].expires > clock.now () { //add this cookie cookiestring = cookiestring + nameOf (adrcookie^ [k]) + "=" + adrcookie^ [k].value + "; "} else { delete (@adrcookie^ [k])}}}}}}; //this cookie has expired so delete it if cookiestring != nil { cookiestring = string.delete (cookiestring, sizeof (cookiestring) - 1, 2); //delete trailing "; " add ("Cookie: " + cookiestring)}}; //add the cookies to the header add (""); //blank line if sizeOf (data) > 0 { httpCommand = httpCommand + data}}; if debug { //save a copy of the HTTP request for debugging wp.newTextObject (httpCommand, @scratchpad.httpCommand)}; local (httpResult = ""); if flSameMachine { //shortcut the call, turn our worst case into our best case: 9/17/98; 7:21:13 AM by DW local (params); new (tabletype, @params); params.client = myAddress; params.port = port; params.stream = 0; //9/28/98; 4:42:45 AM by DW params.request = httpCommand; httpResult = webserver.server (@params, httpCommand)} else { local (serverAddress = server, serverPort = port); if flUseProxy and (sizeOf (proxy) > 0) { //the request is going thru a proxy server: switch server and port serverAddress = proxy; serverPort = proxyPort}; if flMessages { //PBS 5/5/98 displayMsg ("Connecting to " + serverAddress + ".")}; local (stream = tcp.openStream (serverAddress, serverPort)); local (chunksize = 5 * 1024); if flMessages { displayMsg ("Sending request to " + serverAddress + ".")}; try { tcp.writeStringToStream (stream, httpCommand, chunkSize, timeOutTicks/60)} //write the command to the server else { tcp.abortStream (stream); scriptError (tryError)}; if flMessages { displayMsg ("Waiting for data from " + serverAddress + ".")}; local (pat = "\r\n\r\n"); try { tcp.readStreamUntil (stream, pat, timeOutTicks/60, @httpResult)} else { tcp.abortStream (stream); scriptError (tryerror)}; local (headerTable); webserver.util.parseHeaders (httpResult, @headerTable); if flJustHeaders { //the caller only wants the headers, not the body 1/10/01 by DW tcp.closeStream (stream)} else { if (string.upper (method) != "HEAD") and defined (headerTable.["Content-Length"]) { //we know there's content left to be read local (lenHeader = string.patternMatch (pat, httpResult) - 1); local (lenContent = number (string.trimWhiteSpace (headerTable.["Content-Length"]))); //PBS 03/01/01: trim white space before coercing local (lenTotal = lenHeader + string.length (pat) + lenContent); if flMessages { displayMsg ("Receiving response body (" + string.kBytes (lenContent) + ") from " + serverAddress + ".")}; try { tcp.readStreamBytes (stream, lenTotal, timeOutTicks/60, @httpResult)} else { tcp.abortStream (stream); scriptError (tryerror)}} else { //just read until connection closes if flMessages { displayMsg ("Receiving response from " + serverAddress + ".")}; try { tcp.readStreamUntilClosed (stream, timeOutTicks/60, @httpResult)} else { tcp.abortStream (stream); scriptError (tryerror)}}; tcp.closeStream (stream); if (httpResult != "") and flMessages { displayMsg ("Finished receiving " + string.kBytes (string.length (httpResult)) + " from " + serverAddress + ".")}}}; if (httpResult != "") and cookiesOn { // find, parse, and store any cookies: ASG 4/9/98 local (ixEndHeader, responseHeader, i, j); ixEndHeader = string.patternMatch ("\r\n\r\n", httpResult); //try to find the end of the response header if ixEndHeader > 0 { //there is a valid header, look for cookies responseHeader = string.mid (httpResult, 1, ixEndHeader - 1); //get the header if string.lower (responseHeader) contains "set-cookie:" { //the response header contains at least one cookie if not defined (adrCookiesTable^) { new (tableType, adrCookiesTable)}; for i = 1 to string.countFields (responseHeader, "\r") { //find all the cookies local (onecookie = string.nthField (responseHeader, "\r", i)); //get the cookie onecookie = string.popLeading (onecookie, '\n'); if string.lower (onecookie) beginsWith "set-cookie:" { local (flsecure = false); //not secure by default local (domain = server); //default domain to this server local (cookiePath = "/"); local (expires = clock.now () + (24 * 60 * 60)); //default expires at "end of session" local (cookieName, cookieValue); if path != "/" { cookiePath = string.popSuffix (path, "/")}; onecookie = string.popLeading (string.mid (onecookie, 12, infinity), ' '); for j = 1 to string.countFields (onecookie, ';') { local (cookiepart = string.nthField (onecookie, ';', j)); cookiepart = string.popLeading (cookiepart, ' '); case string.lower (string.nthField (cookiepart, '=', 1)) { //take care of standard parts "domain" { domain = string.nthField (cookiepart, '=', 2)}; "path" { cookiePath = string.nthField (cookiepart, '=', 2)}; "expires" { on cookieDateToSystemDate (cookieDate) { try { //some cookie dates can be converted to dates directly local (systemDate = date (cookieDate)); if systemDate < date ("Monday, 6-Feb-2040") { return (systemDate)}}; try { //it may be a date in the 21st century local (lastpart = string.nthField (cookieDate, "-", 3)); local (ixstart = string.patternMatch (lastpart, cookieDate)); return (date (string.insert ("20", cookieDate, ixstart)))}; return (false)}; expires = cookieDateToSystemDate (string.nthField (cookiepart, '=', 2)); if expires == false { //invalid cookie, ignore it break}}; "secure" { flsecure = true}} else { //this is the actual cookie name/value cookieName = string.nthField (cookiepart, '=', 1); cookieValue = string.nthField (cookiepart, '=', 2)}}; if clock.now () < expires { //store non-expired cookies only if not defined (adrCookiesTable^.[domain]) { //have we cookies already for this domain? new (tableType, @adrCookiesTable^.[domain])}; if not defined (adrCookiesTable^.[domain].[cookiePath]) { new (tableType, @adrCookiesTable^.[domain].[cookiePath])}; if not defined (adrCookiesTable^.[domain].[cookiePath].[cookieName]) { new (tableType, @adrCookiesTable^.[domain].[cookiePath].[cookieName])}; adrCookiesTable^.[domain].[cookiePath].[cookieName].value = cookieValue; adrCookiesTable^.[domain].[cookiePath].[cookieName].expires = expires; adrCookiesTable^.[domain].[cookiePath].[cookieName].secure = flsecure}}}}}}; if debug { //save a copy of the server's response for debugging wp.newTextObject (httpResult, @scratchpad.httpResult)}; if (ctFollowRedirects > 0) { //check whether we are being redirected: AR 6/13/1999 local (statusCode = string.nthField (httpResult, ' ', 2)); if ({"301", "302", "303"} contains statusCode) { //response codes for Moved Permanently, Moved Temporarily, See Other local (flRedirect = false, redirectServer, redirectPath); local (t); new (tableType, @t); webserver.util.parseHeaders (httpResult, @t); if defined (t.location) { //the response has a Location: header field on isAbsoluteURL (s) { if not (s contains ":") { return (false)}; local (scheme = string.lower (string.nthField (s, ":", 1))); if scheme == "" { return (false)}; local (ix); for ix = 1 to sizeof (scheme) { if not string.isAlpha (scheme[ix]) { return (false)}}; return (true)}; on isAbsolutePath (s) { return (s beginsWith "/")}; case true { isAbsoluteURL (t.location) { if string.lower (t.location) beginsWith "http://" { //it's an absolute HTTP URL local (s = string.delete (t.location, 1, string.length ("http://"))); redirectServer = string.nthField (s, "/", 1); redirectPath = string.delete (s, 1, string.length (redirectServer) + 1); flRedirect = true}}; isAbsolutePath (t.location) { redirectServer = server; redirectPath = t.location; flRedirect = true}} else { //assume it is a relative path redirectServer = server; redirectPath = string.popSuffix (path, "/") + "/" + t.location; flRedirect = true}}; if flRedirect { if flMessages { displayMsg ("Redirected to " + t.Location + ".")}; if statusCode == "303" { //use the GET method to retrieve the specified entity method = "GET"}; //see section 10.3.4 of RFC2068 if adrRedirectInfo != nil { //10/13/02 by DW, accumulate info about the redirect local (adrsubtable = @adrRedirectInfo^.[string.padWithZeros (sizeof (adrRedirectInfo^) + 1, 5)]); new (tableType, adrsubtable); adrsubtable^.code = statusCode; adrsubtable^.server = redirectServer; adrsubtable^.port = port; adrsubtable^.path = redirectPath}; httpResult = tcp.httpClient (method, redirectServer, port, redirectPath, proxy, proxyPort, proxyUserName, proxyPassword, data, datatype, username, password, adrHdrTable, cookiesOn, debug, timeOutTicks, flMessages, --ctFollowRedirects, flJustHeaders, flAcceptOpml, flUseProxy, adrRedirectInfo)}}}; if flMessages { //clean up the message area: AR 6/14/1999 msg ("")}; return (httpResult)}
This listing is for code that runs in the OPML Editor environment. I created these listings because I wanted the search engines to index it, so that when I want to look up something in my codebase I don't have to use the much slower search functionality in my object database. Dave Winer.