Monday, November 08, 2010 at 12:04 AM.
system.verbs.builtins.radio.aggregator.downloadEnclosures
on downloadEnclosures (fltesting = false) { <<Changes <<4/26/02; 10:29:46 AM by DW <<A few changes to make downloading enclosures more pleasant and reliable. <<1. Who wants %20's in file names? I sure don't. So we pass the names through string.urlDecode to make them more presentable. <<2. Do think it's crazy to download a file that you already have? We thought so too. So there's a new table that contains the URLs of all the files that have been downloaded and the time when they were downloaded. If you sort this table by name, you get a chronologic table of downloads, suitable for display on the desktop website home page someday. <<3. Save changes to aggregatorData.root after every file is downloaded, for good luck. <<12/20/01; 1:01:30 AM by JES <<Use file.writeTextFile instead of file.writeWholeFile to write the descriptions.opml to disk. <<10/19/01; 4:58:08 AM by JES <<Ported from myUserLandSuite.downloadEnclosures. <<Archive <<3/1/01; 11:42:46 PM by JES <<Use Frontier.id for the id, instead of hard-coding 'Radu'. <<2/21/01; 1:15:55 AM by JES <<White descriptions.opml using the correct type/creator for Macs. <<1/19/01; 11:42:23 AM by DW <<Generate an OPML document in each folder containing descriptions of each of the items in your download folder. Open the outline when we're done. This is going to be quite interesting! <<The file is called descriptions.opml. If the pref is set true, we create it every time we download at least one file. Each item is a link to the download, so there's no need to open the folder. <<1/19/01; 10:39:20 AM by DW <<Added callback at the request of Sam DeVore. The callback goes in myUserLandData.callbacks.downloadEnclosure. It takes two params, the address of a table in the stories table, adrstory, and the file path, f. adrstory^.url contains the URL. All the callbacks are called. They may check to see if the file exists, and not do the download if it does. The callbacks should check for errors, and run synchronously, in other words don't return until the file is completely downloaded. If after calling the callbacks, the file does not exist, we download it with tcp.httpClient. <<Sam said: "Dave do you think it would be possible to add a callback so that one (ok me) could use another application for downloading enclosures (like anarchie) that handles memory a little better for big downloads, i just can't use radio if it is going to have to have such big memory requirements for Adams downloads (which make me laugh so I would like to still get them). <<Added fltesting boolean param, defaults to false. This way I can test the script more easily by running it in the debugger, and not have it care that it's not the right time of day to do downloads. <<1/18/01; 10:03:02 AM by DW <<If the file already exists in today's folder, and it's the same size, don't download it. <<Some people re-route items with enclosures, ending up with files being downloaded 2 or more times. <<1/16/01; 5:28:03 PM by DW <<Fix bug in logging errors, move startticks initialization and declaration out one level. <<1/14/01; 9:31:03 AM by DW <<Log errors. <<1/10/01; 9:42:46 PM by DW <<Created. system.temp.radio.misc.flDownloadingEnclosures = true; local (day, month, year, hour, minute, second); date.get (clock.now (), @day, @month, @year, @hour, @minute, @second); local (todaysFolderName = year + "-" + string.padwithzeros (month, 2) + "-" + string.padwithzeros (day, 2)); local (folder = user.radio.prefs.enclosures.enclosureFolder + todaysFolderName + file.getpathchar ()); local (enclosureOutline, oldtarget, insertdir); if user.radio.prefs.enclosures.flOutlineEnclosures { //set up outline new (outlinetype, @enclosureOutline); local (oldtarget = target.set (@enclosureOutline)); op.setlinetext ("Enclosures downloaded on " + clock.now ()); insertdir = right; target.set (oldtarget)}; on statusmsg (s) { temp.radio.stats.status = s; msg (clock.now () + "; " + s)}; on docallbacks (adrstory, f) { if defined (user.radio.callbacks.downloadEnclosure) { local (adrcallback); for adrcallback in @user.radio.callbacks.downloadEnclosure { while typeOf (adrcallback^) == addressType { adrcallback = adrcallback^}; try {adrcallback^ (adrstory, f)}}}}; local (adrdata = xml.aggregator.init ()); local (adrenclosurestats = @adrdata^.stats.enclosureDownloads); if not defined (adrenclosurestats^) { new (tabletype, adrenclosurestats)}; local (adrstory); for adrstory in @adrdata^.stories { bundle { //look for reasons to stop downloading now if not fltesting { if not user.radio.prefs.enclosures.flDownloadsEnabled { //you can stop it by setting the pref break}; date.get (clock.now (), @day, @month, @year, @hour, @minute, @second); if hour >= user.radio.prefs.enclosures.hourToStopDownloads { break}}}; if defined (adrstory^.enclosure) { <<edit (adrstory) if not defined (adrstory^.enclosure.f) { //hasn't been downloaded yet local (url = string.urldecode (adrstory^.enclosure.url)); local (f = folder + string.nthfield (url, "/", string.countfields (url, "/"))); local (fldownload = true); if file.exists (f) { if file.size (f) == adrstory^.enclosure.length { //already have it fldownload = false}}; bundle { //if it's already in the enclosureDownloads table, don't download local (adr, urllookfor = adrstory^.enclosure.url); for adr in adrenclosurestats { if adr^.enclosure.url == urllookfor { fldownload = false; break}}}; if fldownload { local (startticks = clock.ticks ()); try { docallbacks (adrstory, f); if not file.exists (f) { //the callbacks didn't do the download for us, so we do it here local (urllist = string.urlSplit (url), server = urllist [2], path = urllist [3]); statusmsg ("Downloading " + url + "."); local (httpresult = string.httpResultSplit (tcp.httpClient (server:server, path:path, timeOutTicks:60*60, flMessages:false))); file.surefilepath (f); file.writewholefile (f, httpresult)}; adrstory^.enclosure.f = f; if user.radio.prefs.enclosures.flLogDownloads { local (s = "<a href=\"" + html.getfileurl (f) + "\">" + file.filefrompath (f) + "</a>."); radio.log.add ("Download enclosure", s, startticks)}; if user.radio.prefs.enclosures.flOutlineEnclosures { local (oldtarget = target.set (@enclosureOutline)); op.insert (adrstory^.channeltitle + ": " + adrstory^.storytext, insertdir); insertdir = down; local (atts); new (tabletype, @atts); atts.type = "link"; atts.url = html.getfileurl (f); op.attributes.addGroup (@atts); target.set (oldtarget)}; bundle { //add it to the table of already-downloaded enclosures adrenclosurestats^.[clock.now ()] = adrstory^}} else { //log the error if user.radio.prefs.enclosures.flLogDownloads { radio.log.add ("Download enclosure", "Error: " + tryError, startticks)}}; filemenu.savemyroot (adrdata)}}}}; temp.radio.stats.status = "Enclosure downloads complete at " + date.timestring () + "."; msg (""); system.temp.radio.misc.flDownloadingEnclosures = false; if user.radio.prefs.enclosures.flOutlineEnclosures { //save outline if there's anything there if insertdir == down { //at least one line was added local (f = folder + "descriptions.opml"); local (oldtarget = target.set (@enclosureOutline)); op.firstsummit (); target.set (oldtarget); file.writeTextFile (f, op.outlineToXml (@enclosureOutline)); //12/20/01 JES: changed to file.writeTextFile filemenu.open (f)}}} <<bundle //test code <<downloadEnclosures (true)
This listing is for code that runs in the OPML Editor environment. I created these listings because I wanted the search engines to index it, so that when I want to look up something in my codebase I don't have to use the much slower search functionality in my object database. Dave Winer.