Click here to show or hide the menubar.
Top > System > system.verbs.builtins > tcp

on httpClient (method="GET", server="", port=nil, path="", proxy=user.webBrowser.proxy.domain, proxyPort=user.webBrowser.proxy.port, proxyUserName=user.webBrowser.proxy.userName, proxyPassword=user.webBrowser.proxy.password, data="", datatype="", username="", password="", adrHdrTable=nil, cookiesOn=false, debug=false, timeOutTicks=60*30, flMessages=true, ctFollowRedirects=0, flJustHeaders=false, flAcceptOpml=false, flUseProxy=nil, adrRedirectInfo=nil)
Changes
11/10/11; 4:12:08 PM by DW
Allow data to be an address, if so we don't make a copy of it.
Before November 2011
1/22/06; 7:33:35 PM by DW
The code that checked to see if we're running on the "samemachine" wasn't checking if server was 127.0.0.1, and was only checking the port on user.inetd.config.http, when there may be more listeners running on different ports, as there are in the OPML Editor. It would only shortcut of it was localhost and if it was running on one of the daemons.
2/7/03; 3:49:12 PM by JES
When testing for proxy-exempt servers, don't inclue the port number if specified in the server parameter.
10/20/02; 2:32:23 AM by JES
Respect user.webBrowser.proxy.exceptions.
10/16/02; 6:55:56 PM by JES
Removed debugging code.
10/14/02; 11:10:16 AM by JES
If server is 127.0.0.1 or "localhost", set flUseProxy to false.
10/13/02; 6:23:28 PM by DW
New optional param -- adrRedirectInfo. If it's non-nil (the default) and we're redirecting, we fill the table with info about the redirect. The RSS aggregator needs this information, and it may prove useful in other places. See xml.rss.readService for an example of its usage.
12/19/01; 9:58:10 PM by JES
Added a call to webBrowser.init, which initializes prefs at user.webBrowser.proxy. Added optional parameter, flUseProxy, which defaults to nil. If nil, then the value at user.webBrowser.proxy.enabled is used. Pass the value of flUseProxy when following redirects.
8/14/01; 1:49:11 PM by JES
When recursing to following redirects, pass in the flJustHeaders and flAcceptOpml parameters.
8/9/01; 6:10:17 PM by JES
If the stream is unexpectedly closed by the remote host, call tcp.abortStream, to avoid consuming the available connections.
8/8/01; 3:21:27 PM by JES
If no Accept header has been passed in, add an Accept header with the value: text/x-opml, */*, so that we get the response in opml instead of html.
3/1/01; 1:57:33 PM by PBS
Trim white space around the content-length header before coercing it to a number. If a server adds white space here, the coercion would fail, resulting in a connection counter leak. (History: this was discovered in Radio when reading the Red Herring channel.)
1/10/01; 10:36:30 AM by DW
Add an optional parameter, flJustHeaders, if true, we don't read the body, we just return the header portion of the response. This is useful if you just want to know how big a download is, and want to defer downloading big things until the middle of the night when it's quiet on the Internet.
1/7/01; 9:44:18 PM by DW
In Radio, set User-Agent to Radio UserLand.
Changed the header so it doesn't use backslashes. The outliner now supports wrapping headlines.
9/22/00; 11:27:50 AM by PBS
Respect the user's offline settings.
11/5/99; 3:00:39 AM by AR
If the server doesn't send a Content-Length header, read until connection closes.
10/21/99; 3:57:29 PM by AR
Adapted to use new TCP verbs in 6.1.
6/14/99; 5:23:37 AM by AR
If flMessages is true, we now clean up the message area before we return.
6/13/99; 9:20:04 AM by AR
Both the read and write loop now yield time to other Frontier threads and to the system.
The Host header is now generated correctly if the request goes thru a proxy server.
Fixed a few inconsistencies in the cookie handling code. Thinks should now work correctly when connecting via a proxy server.
Added an optional parameter, ctFollowRedirects, determining the number of times that HTTP redirects will be followed. In order to preserve previous behaviour, the default value is 0.
3/4/99; 11:54:30 AM by PBS
If adrHdrTable^ contains a "Host" sub-item, don't add a Host header item.
3/2/99; 10:43:18 AM by DW
if adrHdrTable^ contains a "User-Agent" sub-item, don't add a User-Agent header item
10/25/98; 6:33:59 AM by DW
allow the use of a colon in the server name to specify the port
example: www.scripting.com:80
if present, we use that port, not the one specified in the call
the assumption is that most callers either specify 80 or don't specify the port at all
for crawlers, for example, they definitely want to use the port specified in the server string
some commands, esp in the webBrowser class, get the url from another piece of software
9/28/98; 4:42:12 AM by DW
if we're sending to the same machine, set the stream ID to 0
user.webserver.responders.CGI.methods.any is copying it, hope no CGIs are using it!
9/17/98; 7:11:56 AM by DW
msgs now say that they're from tcp.httpClient.
changed Connection to xxx timed out message to...
Can't process the request because the connection to xxx timed out.
allow server to be "localhost", if so, we totally optimize the functionality
2/13/98; 8:57:10 AM by DW
Initial code assembled from various sample scripts by Dave Winer.
The goal is to have a single documented way to talk to HTTP servers.
It's got a long parameter list, but requires no configuration, doesn't run off a user.xxx table.
The target user of this script is someone who needs to understand how the code works.
It will be distributed with Frontier on both platforms as soon as we agree that this is complete and works.
When you call it, use parameter labels, don't depend on the order of these params.
For examples, see tcp.examples.
local (adrdata)
if tcp.isOffline () //PBS 09/22/00: check offline status
scriptError ("Can't send HTTP request because TCP is offline.")
bundle //set adrdata, 11/10/11 by DW
if sizeof (data) > 0
if typeof (data) == addresstype
adrdata = data
else
adrdata = @data
else
adrdata = @data
webBrowser.init ()
try //set flUseProxy
if (flUseProxy == nil) and (typeOf (flUseProxy) == unknownType)
flUseProxy = user.webBrowser.proxy.enabled
if (server == "127.0.0.1" or string.lower (server) == "localhost") //JES 10/14/02: don't call the loopback address through the proxy
flUseProxy = false
if flUseProxy //check the exceptions table -- if we find a match then set flUseProxy to false
on wildcardIpMatch (s, oneip)
local (i)
if s == oneip //optimization
return (true)
for i = 1 to 4
try
local (part = string.nthField (oneip, ".", i))
local (pattern = string.nthField (s, ".", i))
if pattern == "*"
continue
if pattern contains "-"
local (min = number (string.nthField (pattern, "-", 1)) )
local (max = number (string.nthField (pattern, "-", 2)) )
local (n = number (part))
if n >= min and n <= max
continue
else
return (false)
if pattern != part
return (false)
else
return (false)
return (true)
local (flIpAddress = false, hostToCheck = string.lower (string.nthField (server, ":", 1)) )
try //see if flIpAddress should be true
local (serverWithoutDots = string.replaceAll (server, ".", ""))
double (serverWithoutDots) //will error if non-numeric characters are present
flIpAddress = true
else //check domains, and then try a DNS lookup
local (adr)
local (lowerserver = string.lower (server))
for adr in @user.webBrowser.proxy.exceptions
if flUseProxy //try a reverse lookup so we can check against IP address
try
hostToCheck = tcp.dns.getDottedId (server)
flIpAddress = true
if flUseProxy //check to see if this specific IP is excepted
if flUseProxy //check for IP-address exceptions which contain wildcard characters and ranges
local (i)
for i = 1 to sizeOf (user.webBrowser.proxy.exceptions)
if user.webBrowser.proxy.exceptions[i]
local (oneexception = nameOf (user.webBrowser.proxy.exceptions[i]))
if wildcardIpMatch (oneexception, hostToCheck)
flUseProxy = false
break
on displayMsg (s)
msg ("tcp.httpClient: " + s)
local (myAddress = tcp.myDottedID (), flSameMachine = false)
bundle //look for the port specified in the server string
if server contains ":"
try {port = number (string.nthField (server, ":", 2))} //it might not be a number
server = string.nthField (server, ":", 1)
if port == nil //no port was specified, either in the call or in the server string
port = 80
bundle //see if it's the same machine
local (lowerserver = string.lower (server))
if (lowerserver == "localhost") or (lowerserver == "127.0.0.1")
flSameMachine = true
else
try {flSameMachine = tcp.equalNames (server, myAddress)} //DW 3/9/98
bundle //check that we're running on the requested port, 1/22/06 by DW
if flsamemachine //may turn false
local (adrd, flfound = false)
for adrd in @user.inetd.config
if not flfound
flsamemachine = false
bundle //old code
PBS 9/18/98: check that the server on the requested port is Frontier.
if port != user.inetd.config.http.port
flSameMachine = false //it's the same machine, but it's not Frontier as a server
else
if not (inetd.isDaemonRunning (@user.inetd.config.http))
flSameMachine = false //Frontier isn't running as a webserver, there must be another server on this machine
if not (path beginsWith "/")
path = "/" + path
local (httpCommand = "", adrCookiesTable = @user.webBrowser.cookies)
bundle //build the HTTP request in "httpCommand"
on add (s)
httpCommand = httpCommand + s + "\r\n"
local (uri = path)
if flUseProxy and (sizeOf (proxy) > 0) //the request is going thru a proxy server: set uri accordingly
uri = "http://" + server
if port != 80
uri = uri + ":" + port
uri = uri + path
add (method + " " + uri + " HTTP/1.0")
if (adrHdrTable == nil) or (not defined (adrHdrTable^.["User-Agent"])) //3/2/99; 10:43:10 AM by DW
if system.environment.isPike
add ("User-Agent: Radio UserLand/" + Frontier.version () + " (" + sys.os () + ")")
else
add ("User-Agent: Frontier/" + Frontier.version () + " (" + sys.os () + ")")
local (host = "Host: " + server)
if port != 80
host = host + ":" + port
if (adrHdrTable == nil) or (not defined (adrHdrTable^.["Host"])) //3/4/99; 11:54:30 AM by PBS
add (host)
if flAcceptOpml //8/8/01; 3:19:50 PM by JES
if (adrHdrTable == nil)
add ("Accept: text/x-opml, */*")
else //headers were passed in
if defined (adrHdrTable^.Accept)
local (acceptHeader = adrHdrTable^.Accept)
if not (acceptHeader contains "text/x-opml")
if sizeOf (string.trimWhiteSpace (acceptHeader)) > 0
acceptHeader = "text/x-opml, " + acceptHeader
else
acceptHeader = "text/x-opml"
if not (acceptHeader contains "*/*")
acceptHeader = acceptHeader + ", */*"
add ("Accept: " + acceptHeader)
else
add ("Accept: text/x-opml, */*")
if sizeOf (adrdata^) > 0
add ("Content-Type: " + datatype)
add ("Content-length: " + sizeOf (adrdata^))
if (sizeOf (username) > 0) or (sizeOf (password) > 0)
add ("Authorization: Basic " + base64.encode (username + ":" + password, 0))
if flUseProxy and (sizeOf (proxy) > 0) //the request is going thru a proxy server
if (sizeOf (proxyUsername) > 0) or (sizeOf (proxyPassword) > 0) //the proxy requires authentication
add ("Proxy-Authorization: Basic " + base64.encode (proxyUsername + ":" + proxyPassword, 0))
if adrHdrTable != nil //added 3.27.98 by ASG to include header items
local (i)
for i = 1 to sizeOf (adrHdrTable^)
local (headerName = nameOf (adrHdrTable^[i]))
if flAcceptOpml //8/8/01; 3:19:50 PM by JES: Only add the Acept header once
add (headerName + ": " + adrHdrTable^[i])
if cookiesOn and defined (adrCookiesTable^) //add cookies to the request header: ASG 4/9/98
local (i, j, k, cookiePath = "/", cookiestring)
if path != "/"
cookiePath = string.popSuffix (path, "/")
for i = 1 to sizeOf (adrCookiesTable^) //loop through cookies table
local (adrdomain = @adrCookiesTable^ [i])
if host endsWith nameOf (adrdomain^)
for j = 1 to sizeOf (adrdomain^)
local (adrcookie = @adrdomain^ [j])
if cookiePath contains nameOf (adrcookie^)
for k = 1 to sizeOf (adrcookie^)
if adrcookie^ [k].expires > clock.now () //add this cookie
cookiestring = cookiestring + nameOf (adrcookie^ [k]) + "=" + adrcookie^ [k].value + "; "
else
delete (@adrcookie^ [k]) //this cookie has expired so delete it
if cookiestring != nil
cookiestring = string.delete (cookiestring, sizeof (cookiestring) - 1, 2) //delete trailing "; "
add ("Cookie: " + cookiestring) //add the cookies to the header
add ("") //blank line
if sizeOf (adrdata^) > 0
httpCommand = httpCommand + adrdata^
if debug //save a copy of the HTTP request for debugging
wp.newTextObject (httpCommand, @scratchpad.httpCommand)
local (httpResult = "")
if flSameMachine //shortcut the call, turn our worst case into our best case: 9/17/98; 7:21:13 AM by DW
local (params)
new (tabletype, @params)
params.client = myAddress
params.port = port
params.stream = 0 //9/28/98; 4:42:45 AM by DW
params.request = httpCommand
httpResult = webserver.server (@params, httpCommand)
else
local (serverAddress = server, serverPort = port)
if flUseProxy and (sizeOf (proxy) > 0) //the request is going thru a proxy server: switch server and port
serverAddress = proxy
serverPort = proxyPort
if flMessages //PBS 5/5/98
displayMsg ("Connecting to " + serverAddress + ".")
local (stream = tcp.openStream (serverAddress, serverPort))
local (chunksize = 5 * 1024)
if flMessages
displayMsg ("Sending request to " + serverAddress + ".")
try
tcp.writeStringToStream (stream, httpCommand, chunkSize, timeOutTicks/60) //write the command to the server
else
tcp.abortStream (stream)
scriptError (tryError)
if flMessages
displayMsg ("Waiting for data from " + serverAddress + ".")
local (pat = "\r\n\r\n")
try
tcp.readStreamUntil (stream, pat, timeOutTicks/60, @httpResult)
else
tcp.abortStream (stream)
scriptError (tryerror)
local (headerTable)
webserver.util.parseHeaders (httpResult, @headerTable)
if flJustHeaders //the caller only wants the headers, not the body 1/10/01 by DW
tcp.closeStream (stream)
else
if (string.upper (method) != "HEAD") and defined (headerTable.["Content-Length"]) //we know there's content left to be read
local (lenHeader = string.patternMatch (pat, httpResult) - 1)
local (lenContent = number (string.trimWhiteSpace (headerTable.["Content-Length"]))) //PBS 03/01/01: trim white space before coercing
local (lenTotal = lenHeader + string.length (pat) + lenContent)
if flMessages
displayMsg ("Receiving response body (" + string.kBytes (lenContent) + ") from " + serverAddress + ".")
try
tcp.readStreamBytes (stream, lenTotal, timeOutTicks/60, @httpResult)
else
tcp.abortStream (stream)
scriptError (tryerror)
else //just read until connection closes
if flMessages
displayMsg ("Receiving response from " + serverAddress + ".")
try
tcp.readStreamUntilClosed (stream, timeOutTicks/60, @httpResult)
else
tcp.abortStream (stream)
scriptError (tryerror)
tcp.closeStream (stream)
if (httpResult != "") and flMessages
displayMsg ("Finished receiving " + string.kBytes (string.length (httpResult)) + " from " + serverAddress + ".")
if (httpResult != "") and cookiesOn // find, parse, and store any cookies: ASG 4/9/98
local (ixEndHeader, responseHeader, i, j)
ixEndHeader = string.patternMatch ("\r\n\r\n", httpResult) //try to find the end of the response header
if ixEndHeader > 0 //there is a valid header, look for cookies
responseHeader = string.mid (httpResult, 1, ixEndHeader - 1) //get the header
if string.lower (responseHeader) contains "set-cookie:" //the response header contains at least one cookie
if not defined (adrCookiesTable^)
new (tableType, adrCookiesTable)
for i = 1 to string.countFields (responseHeader, "\r") //find all the cookies
local (onecookie = string.nthField (responseHeader, "\r", i)) //get the cookie
onecookie = string.popLeading (onecookie, '\n')
if string.lower (onecookie) beginsWith "set-cookie:"
local (flsecure = false) //not secure by default
local (domain = server) //default domain to this server
local (cookiePath = "/")
local (expires = clock.now () + (24 * 60 * 60)) //default expires at "end of session"
local (cookieName, cookieValue)
if path != "/"
cookiePath = string.popSuffix (path, "/")
onecookie = string.popLeading (string.mid (onecookie, 12, infinity), ' ')
for j = 1 to string.countFields (onecookie, ';')
local (cookiepart = string.nthField (onecookie, ';', j))
cookiepart = string.popLeading (cookiepart, ' ')
case string.lower (string.nthField (cookiepart, '=', 1)) //take care of standard parts
"domain"
domain = string.nthField (cookiepart, '=', 2)
"path"
cookiePath = string.nthField (cookiepart, '=', 2)
"expires"
on cookieDateToSystemDate (cookieDate)
try //some cookie dates can be converted to dates directly
local (systemDate = date (cookieDate))
if systemDate < date ("Monday, 6-Feb-2040")
return (systemDate)
try //it may be a date in the 21st century
local (lastpart = string.nthField (cookieDate, "-", 3))
local (ixstart = string.patternMatch (lastpart, cookieDate))
return (date (string.insert ("20", cookieDate, ixstart)))
return (false)
expires = cookieDateToSystemDate (string.nthField (cookiepart, '=', 2))
if expires == false //invalid cookie, ignore it
break
"secure"
flsecure = true
else //this is the actual cookie name/value
cookieName = string.nthField (cookiepart, '=', 1)
cookieValue = string.nthField (cookiepart, '=', 2)
if clock.now () < expires //store non-expired cookies only
if not defined (adrCookiesTable^.[domain]) //have we cookies already for this domain?
new (tableType, @adrCookiesTable^.[domain])
if not defined (adrCookiesTable^.[domain].[cookiePath])
new (tableType, @adrCookiesTable^.[domain].[cookiePath])
if not defined (adrCookiesTable^.[domain].[cookiePath].[cookieName])
new (tableType, @adrCookiesTable^.[domain].[cookiePath].[cookieName])
adrCookiesTable^.[domain].[cookiePath].[cookieName].value = cookieValue
adrCookiesTable^.[domain].[cookiePath].[cookieName].expires = expires
adrCookiesTable^.[domain].[cookiePath].[cookieName].secure = flsecure
if debug //save a copy of the server's response for debugging
wp.newTextObject (httpResult, @scratchpad.httpResult)
if (ctFollowRedirects > 0) //check whether we are being redirected: AR 6/13/1999
local (statusCode = string.nthField (httpResult, ' ', 2))
if ({"301", "302", "303"} contains statusCode) //response codes for Moved Permanently, Moved Temporarily, See Other
local (flRedirect = false, redirectServer, redirectPath)
local (t); new (tableType, @t)
webserver.util.parseHeaders (httpResult, @t)
if defined (t.location) //the response has a Location: header field
on isAbsoluteURL (s)
if not (s contains ":")
return (false)
local (scheme = string.lower (string.nthField (s, ":", 1)))
if scheme == ""
return (false)
local (ix)
for ix = 1 to sizeof (scheme)
return (true)
on isAbsolutePath (s)
return (s beginsWith "/")
case true
isAbsoluteURL (t.location)
if string.lower (t.location) beginsWith "http://" //it's an absolute HTTP URL
local (s = string.delete (t.location, 1, string.length ("http://")))
redirectServer = string.nthField (s, "/", 1)
redirectPath = string.delete (s, 1, string.length (redirectServer) + 1)
flRedirect = true
isAbsolutePath (t.location)
redirectServer = server
redirectPath = t.location
flRedirect = true
else //assume it is a relative path
redirectServer = server
redirectPath = string.popSuffix (path, "/") + "/" + t.location
flRedirect = true
if flRedirect
if flMessages
displayMsg ("Redirected to " + t.Location + ".")
if statusCode == "303" //use the GET method to retrieve the specified entity
method = "GET" //see section 10.3.4 of RFC2068
if adrRedirectInfo != nil //10/13/02 by DW, accumulate info about the redirect
local (adrsubtable = @adrRedirectInfo^.[string.padWithZeros (sizeof (adrRedirectInfo^) + 1, 5)])
new (tableType, adrsubtable)
adrsubtable^.code = statusCode
adrsubtable^.server = redirectServer
adrsubtable^.port = port
adrsubtable^.path = redirectPath
httpResult = tcp.httpClient (method, redirectServer, port, redirectPath, proxy, proxyPort, proxyUserName, proxyPassword, data, datatype, username, password, adrHdrTable, cookiesOn, debug, timeOutTicks, flMessages, --ctFollowRedirects, flJustHeaders, flAcceptOpml, flUseProxy, adrRedirectInfo)
if flMessages //clean up the message area: AR 6/14/1999
msg ("")
return (httpResult)
XML Stats & Atts.

This aggression will not stand.

Note: This page is being rendered using a template I have in a root, named My Template. This feels very nice and natural and I'm excited that it will lead very easily to themes. For more info, read this post.