Monday, November 08, 2010 at 12:05 AM.
system.verbs.builtins.radio.upstream.uploadChangedFiles
on uploadChangedFiles () { <<Changes <<1/22/03; 3:21:33 PM by JES <<If the serial number is expired, log an error to the Events page if upstreaming to UserLand's server fails as a result. <<12/21/02; 4:13:17 PM by JES <<Added support for serial number renewals. <<6/23/02; 11:17:34 PM by JES <<If a file's modification date is in the future, set adrfile^.whenLastUpstream to the file's mod-date, instead of to now. Prevents files with future mod-dates from being repeatedly upstreamed. <<2/1/02; 1:59:54 PM by JES <<Upload a maximum of 25 files per pass. <<1/31/02; 3:05:23 AM by JES <<Skip busy files until the next time through the loop. (A busy file is most likely opened with write permissions by another application.) <<1/29/02; 3:29:30 AM by JES <<Break upstreaming passes up into 128KB blocks. (The last file that breaks the 128KB limit is included with the pass.) <<1/7/02; 11:41:41 AM by JES <<Don't upstream files whose name begins with '.' or is contained in a folder whose name begins with '.'. <<12/12/01; 8:38:29 PM by DW <<Don't upstream if the file is contained within a folder whose name begins with a #. <<12/11/01; 9:38:47 AM by DW --Tweaks and tuneups after rewrite. <<Trick the upstreamer into doing "index" first. <<The goal is to make it more likely that when you click on Home on the DWHP that it will already be updated. If index is moved to the front of the list, that becomes more likely. Why should it wait for the other two files? It shouldn't. <<Tell the thread not to go to sleep if directory.opml needs to be written <<See comment at head of radio.thread.script. <<12/10/01; 9:20:24 AM by DW -- Total rewrite. <<The main loop now fills a table with lists of files that need to be upstreamed. <<Each list is actually a table (for performance). All files in a given list are going to the same place. <<Then in a separate pass iterate over the lists, send all the files in a given list in one shot. <<This required a rearchitect of the upstream drivers. (Better now than later.) <<12/9/01; 7:43:57 AM by DW <<In the main loop, instead of checking against the modified file attribute, check against the upstream.whenLastUploaded attribute. <<12/8/01; 5:24:00 PM by DW <<If there was a change to a #upstream.xml, mark all the files under its control as needing upstreaming. <<12/7/01; 9:24:29 AM by DW <<Optimization -- after upstreaming, be sure the file isn't in recentlyWrittenWwwFiles, avoid upstreaming it twice. <<12/6/01; 10:34:54 AM by DW <<Re-coded for performance. <<bundle //old code <<on dofolder (folder) <<local (f, lastfileurl = "") <<fileloop (f in folder) <<thread.sleepTicks (0) //PBS 11/16/01: more background-friendly //PBS 11/20/01: interferes with debugging <<if file.isVisible (f) //skip invisible files -- icon files on the Mac, in particular <<local (fname = file.filefrompath (f)) <<if not (fname beginswith "#") <<if file.isfolder (f) <<lastfileurl = dofolder (f) <<else <<local (extension = string.nthfield (fname, ".", string.countfields (fname, "."))) <<radio.file.getFileAttributes (f, @adrfile) <<if file.modified (f) > adrfile^.upstream.whenLastUploaded <<upstreamfile (adrfile) <<adrfile^.upstream.whenLastUploaded = now //only try once <<lastfileurl = adrfile^.upstream.url <<local (folderurl = "") <<bundle //set the folder's url <<if lastfileurl != "" <<local (s) <<bundle //set s <<local (ix) <<s = lastfileurl <<if s endswith "/" <<s = string.delete (s, sizeof (s), 1) <<for ix = sizeof (s) downto 1 <<if s [ix] == "/" <<s = string.mid (s, 1, ix) <<break <<local (adrfolder) <<radio.file.getFileAttributes (folder, @adrfolder) <<adrfolder^.upstream.url = s <<folderurl = s <<return (folderurl) <<local (now = clock.now ()) <<dofolder (user.radio.prefs.upstream.folder) <<12/4/01; 2:41:59 PM by JES <<If there's an error uploading a file, record it in flError, in the upstream sub-table of the filetable, instead of at the top-level of the filetable. If there's no error, set adrfile^.upstream.flError to false. <<12/1/01; 10:55:59 PM by JES <<Fixed a bug where directory.opml would only be updated when files in the same folder as the upstream spec were modified. <<11/17/01; 3:11:23 AM by JES <<When making log entries, the text of the link is the upstreamed filename, instead of the on-disk filename. <<11/16/01; 8:24:21 PM by PBS <<To make this more background-friendly, call thread.sleepTicks (0) in the fileloop. <<11/13/01; 2:24:05 AM by JES <<Set an item in temp.radio.recentlyChangedFolders.[adrfile^.baseUpstreamFolder], when a file is upstreamed. This triggers the re-writing of the directory.opml file for the upstream folder. <<Note: This item is *not* set for files named directory.opml, which live in the baseUpstreamFolder. Don't change that, or you'll get an infinite loop of directory.opml's upstreaming into the cloud. <<9/28/01; 2:25:56 PM by JES <<Call callbacks when a file is upstreamed. First call callbacks at radio.upstream.callbacks.upstream, and then call user callbacks at user.radio.callbacks.upstream. <<9/5/01; 12:08:59 AM by JES <<Lock a semaphore on the file path while processing. This enables us to stay out of the way of weblog Theme application, and possibly others in the future. <<7/24/01; 1:20:23 AM by JES <<Don't attempt to upstream invisible files. This avoids upstreaming icon files on MacOS. <<6/19/01; 10:03:02 AM by DW <<Rewrite. Out with the two passes. Implement #upstream.xml. on logadd (htmltext) { if user.radio.prefs.upstream.logging { radio.log.add ("Upstream", htmltext, startticks)}}; on runCallbacks (filelist, adrcallbacktable) { local (nomad); try { for nomad in adrcallbacktable { while typeOf (nomad^) == addressType { nomad = nomad^}; try {nomad^ (filelist)}}}}; //site of an outage -- 12/6/01; 7:49:55 PM by DW local (maxBytesPerPass = 128 * 1024); //128KB local (maxFilesPerPass = 25); local (now = clock.now ()); local (streams); new (tabletype, @streams); bundle { //pass 1 -- fill streams with files that changed, sorted by stream local (f, pc = file.getPathChar (), ctbytes = 0, ctfiles = 0); fileloop (f in user.radio.prefs.upstream.folder, infinity) { local (whenModified = file.modified (f)); local (flupstream = true); try { if whenModified <= user.radio.settings.files.[f].upstream.whenLastUploaded { flupstream = false}}; if flupstream { ctbytes = ctbytes + file.size (f); local (adrfile); radio.file.getFileAttributes (f, @adrfile); bundle { //optimization <<12/9/01; 7:43:57 AM by DW <<If it's a new file make sure it has an entry in the cache, makes future scans faster. Inside the "try" above, it is looking for the cache element to exist for the file. Consider the case of an invisible file (doesn't happen often) or a #file (more often) -- they should be in the cache too, with a "whenLastUploaded" field set, so we can dispose of them in future scans without having to see if they're visible or have a name that begins with a #. if now < whenModified { adrfile^.upstream.whenLastUploaded = whenModified} else { adrfile^.upstream.whenLastUploaded = now}}; if file.isvisible (f) { if not file.isBusy (f) { //1/31/02 by JES: skip this file until it's not busy (opened with write permissions) if not (f contains (pc + ".")) { //01/07/02 by JES: files/folders that start with '.' are invisible if not radio.upstream.inPoundFolder (f) { //12/12/01 by DW local (name = file.filefrompath (f)); if name beginswith "#" { if string.lower (name) == radio.data.fileNames.upstreamFileName { radio.upstream.folderNeedsUpstream (file.folderfrompath (f))}} else { local (adrspec); if radio.upstream.getUpstreamSpec (adrfile, @adrspec) { local (adrstream = @streams.[nameof (adrspec^)]); if not defined (adrstream^) { new (tabletype, adrstream); adrstream^.adrspec = adrspec; adrstream^.files = {}}; adrstream^.files = adrstream^.files + {adrfile}; if ctbytes >= maxBytesPerPass { break}; if ++ctfiles >= maxFilesPerPass { break}}}}}}}}}}; <<scratchpad.streams = streams <<streams = scratchpad.streams //for debugging bundle { //pass 2 -- iterate over streams table, routing the changed files to their proper destinations local (startticks); local (adrstream, adrspec, type, adrdriver); for adrstream in @streams { startticks = clock.ticks (); adrspec = adrstream^.adrspec; bundle { //expiration check -- if expired, don't upstream to UserLand if defined (user.radio.settings.whenSNExpires) { local (whenexpires = date (user.radio.settings.whenSNExpires)); if whenexpires < now { case string.lower (adrspec^.server) { "rcs.salon.com"; "radio.xmlstoragesystem.com" { logadd ("Can't upstream to " + adrspec^.server + " because your serial number is expired."); continue}}}}}; type = adrspec^.type; if not radio.upstream.findDriver (type, @adrdriver) { if string.lower (type) != "none" { logadd ("Can't upstream because there is no driver for type \"" + type + "\".")}; continue}; bundle { //trick the upstreamer into doing "index" first. local (adrfile, ix = 1); for adrfile in adrstream^.files { if string.lower (file.filefrompath (nameof (adrfile^))) beginswith "index" { if ix != 1 { local (adrtmp = adrstream^.files [1]); adrstream^.files [1] = adrfile; adrstream^.files [ix] = adrtmp}; break}; ix++}}; try { local (response); bundle { //set relativePath for each file local (adrfile); for adrfile in adrstream^.files { local (s = string.delete (nameof (adrfile^), 1, sizeof (adrfile^.baseUpstreamFolder))); s = string.replaceAll (s, file.getpathchar (), "/"); adrfile^.relativePath = s}}; adrdriver^.upstreamMultipleFiles (adrstream^.files, adrspec, @response); bundle { //set an item in system.temp.radio.recentlyChangedFolders to trigger rebuild of directory.opml local (adrfile); for adrfile in adrstream^.files { if string.lower (file.filefrompath (nameof (adrfile^))) != string.lower (radio.data.upstream.directoryFileName) { system.temp.radio.recentlyChangedFolders.[adrfile^.baseUpstreamFolder] = clock.now (); system.temp.radio.misc.flThreadSleeps = false}}}; //12/11/01; 10:23:43 AM by DW bundle { //recording info in file tables, log the upstream local (url, ct = 1, adrfile, htmltext = ""); for url in response.urllist { adrfile = adrstream^.files [ct++]; if url == "" { //error on the server processing this file adrfile^.upstream.flError = true; htmltext = htmltext + "<font color=\"red\">" + string.nthfield (url, "/", string.countfields (url, "/")) + "</font>, "} else { //no error on this file adrfile^.upstream.ctUploads++; adrfile^.upstream.url = url; adrfile^.upstream.flError = false; htmltext = htmltext + "<a href=\"" + url + "\">" + string.nthfield (url, "/", string.countfields (url, "/")) + "</a>, "}}; htmltext = string.mid (htmltext, 1, sizeof (htmltext) - 2) + "."; //pop off extra comma and space, add period if response.flerror { htmltext = htmltext + " The server reported an error: " + response.message}; if sizeof (response.urllist) == 1 { htmltext = "1 file: " + htmltext} else { htmltext = sizeof (response.urllist) + " files: " + htmltext}; logadd (htmltext)}; bundle { //set the parent folder urls local (adrfile); for adrfile in adrstream^.files { local (s, adrfolder); radio.file.getFileAttributes (file.folderfrompath (nameof (adrfile^)), @adrfolder); bundle { //set s local (ix); s = adrfile^.upstream.url; if s endswith "/" { s = string.delete (s, sizeof (s), 1)}; for ix = sizeof (s) downto 1 { if s [ix] == "/" { s = string.mid (s, 1, ix); break}}}; adrfolder^.upstream.url = s}}; bundle { //run callbacks local (filepathlist = {}, adrfile); for adrfile in adrstream^.files { filepathlist = filepathlist + {nameof (adrfile^)}}; if defined (radio.upstream.callbacks.upstream) { runCallbacks (filepathlist, @radio.upstream.callbacks.upstream)}; if defined (user.radio.callbacks.upstream) { runCallbacks (filepathlist, @user.radio.callbacks.upstream)}}} else { logadd ("Can't upstream because \"" + tryerror + "\"")}}}} <<bundle //pre 12/10/01 code <<local (startticks = clock.ticks (), ctupstreams = 0, htmltext = "") <<on upstreamfile (f) <<local (adrfile, adrspec) <<radio.file.getFileAttributes (f, @adrfile) <<adrfile^.upstream.whenLastUploaded = clock.now () //mark as upstreamed, don't retry errant uploads until they change again <<if not radio.upstream.getUpstreamSpec (adrfile, @adrspec) <<return (false) <<bundle //upstream the file, according to the spec <<local (adrdriver) <<if not radio.upstream.findDriver (adrspec^.type, @adrdriver) <<return (false) <<bundle //set the relative path <<local (s = string.delete (f, 1, sizeof (adrfile^.baseUpstreamFolder))) <<s = string.replaceAll (s, file.getpathchar (), "/") <<adrfile^.relativePath = s <<try <<adrdriver^.upstream (adrfile, adrspec) <<adrfile^.upstream.flError = false <<adrfile^.upstream.ctUploads++ <<bundle //set an item in system.temp.radio.recentlyChangedFolders to trigger rebuild of directory.opml <<if string.lower (file.folderFromPath (f)) == string.lower (adrfile^.baseUpstreamFolder) <<if string.lower (file.filefrompath (f)) != string.lower (radio.data.upstream.directoryFileName) <<system.temp.radio.recentlyChangedFolders.[adrfile^.baseUpstreamFolder] = clock.now () <<if defined (radio.upstream.callbacks.upstream) <<runCallbacks (adrfile, @radio.upstream.callbacks.upstream) <<if defined (user.radio.callbacks.upstream) <<runCallbacks (adrfile, @user.radio.callbacks.upstream) <<local (upstreamFname = string.nthField (adrfile^.upstream.url, "/", string.countFields (adrfile^.upstream.url, "/"))) <<htmltext = htmltext + "<a href=\"" + adrfile^.upstream.url + "\">" + upstreamFname + "</a>, " <<ctupstreams++ <<else <<adrfile^.upstream.flError = true <<htmltext = htmltext + file.filefrompath (f) + "(Error = " + tryError + "), " <<bundle //set the parent folder url <<local (s, adrfolder) <<radio.file.getFileAttributes (file.folderfrompath (f), @adrfolder) <<bundle //set s <<local (ix) <<s = adrfile^.upstream.url <<if s endswith "/" <<s = string.delete (s, sizeof (s), 1) <<for ix = sizeof (s) downto 1 <<if s [ix] == "/" <<s = string.mid (s, 1, ix) <<break <<adrfolder^.upstream.url = s <<bundle //make sure file isn't in recentlyWrittenWwwFiles, don't upstream twice <<try {delete (@system.temp.radio.recentlyWrittenWwwFiles.[f])} <<return (true) <<local (f) <<fileloop (f in user.radio.prefs.upstream.folder, infinity) <<local (flupstream = true) <<try <<if file.modified (f) <= user.radio.settings.files.[f].upstream.whenLastUploaded <<flupstream = false <<if flupstream <<bundle //optimization <<If it's a new file make sure it has an entry in the cache, makes future scans faster. Inside the "try" above, it is looking for the cache element to exist for the file. Consider the case of an invisible file (doesn't happen often) or a #file (more often) -- they should be in the cache too, with a "whenLastUploaded" field set, so we can dispose of them in future scans without having to see if they're visible or have a name that begins with a #. <<local (adrfile) <<radio.file.getFileAttributes (f, @adrfile) <<adrfile^.upstream.whenLastUploaded = clock.now () <<if file.isvisible (f) <<local (name = file.filefrompath (f)) <<if name beginswith "#" <<if string.lower (name) == radio.data.fileNames.upstreamFileName <<radio.upstream.folderNeedsUpstream (file.folderfrompath (f)) <<else <<upstreamfile (f) <<if user.radio.prefs.upstream.logging <<if ctupstreams > 0 <<if ctupstreams == 1 <<htmltext = ctupstreams + " file: " + htmltext <<else <<htmltext = ctupstreams + " files: " + htmltext <<htmltext = string.delete (htmltext, sizeof (htmltext) - 1, 2) + "." <<radio.log.add ("Upstream", htmltext, startticks, nil) <<return (true) <<bundle //test code <<local (startticks = clock.ticks ()) <<uploadChangedFiles () <<dialog.alert (clock.ticks () - startticks)
This listing is for code that runs in the OPML Editor environment. I created these listings because I wanted the search engines to index it, so that when I want to look up something in my codebase I don't have to use the much slower search functionality in my object database. Dave Winer.