| View previous topic :: View next topic |
| Author |
Message |
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
 Posted: Fri Aug 12, 2005 11:31 pm Post subject: RSS news by demond |
 |
|
since the other thread that people have been posting in about this wasn't actually relevant from the beginning, I'm opening this dedicated to the subject new thread
for those who don't know what the fuss is all about: I made a RSS news script for my own purposes (it still doesn't have an official name, I haven't released it officially yet) and I posted the beta version on several occasions here on these forums, after seeing that people tend to have problems with other RSS scripts; my version differs from all other similar scripts in the mechanism of parsing - it uses the object-oriented and XML facilities of TclLib and TclXML, also the RSS package from BDK, posted on Tcl'ers Wiki; and as it (unfortunately) turns out, most people are not used to installing third-party Tcl packages and therefore have been having problems with my script's requirements
I'm a big proponent of software component re-use, and I think we should encourage usage of good (and de facto standard) component packages like TclLib and TclXML
I'm also inclined to release the script officially, but that would depend on how many people actually manage to successfully install the required packages |
|
| Back to top |
|
 |
greenbear
Owner
Joined: 24 Sep 2001
Posts: 733
Location: Norway
|
| Posted: Sat Aug 13, 2005 11:05 am Post subject: |
|
|
| Installing aditional packages will probably be a big pita for everyone running bots from their cheap $5 a month shell accounts... |
|
| Back to top |
|
|
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
| Posted: Sat Aug 13, 2005 1:34 pm Post subject: |
|
|
well you certainly have a point here, although if you have at least 5mb quota you should be able to install TclLib and TclXML in your homedir:
| Code: |
[demond@whitepine demond]$ du -h -d0 lib/tcllib1.7/ lib/tclxml3.0/
3.1M lib/tcllib1.7/
156K lib/tclxml3.0/
|
|
|
| Back to top |
|
|
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
| Posted: Wed Aug 17, 2005 12:50 am Post subject: |
|
|
| I might rewrite this thing so that it would no longer depend on external packages, but it will continue to be rather minimalistic RSS solution, I'm not interested in adding bells & whistles |
|
| Back to top |
|
|
sKy
Op
Joined: 14 Apr 2005
Posts: 194
Location: Germany
|
| Posted: Fri Aug 19, 2005 7:43 pm Post subject: |
|
|
I think that`s an good idea.
Well, RSS. I know what RSS means but i think its inconnu as well. Please don`t get me wrong, but if you want some feedback you then should try to explain in some easy sentence what RSS is and what the purpose of your script is.
| Quote: | | Installing aditional packages will probably be a big pita for everyone running bots from their cheap $5 a month shell accounts... |
I think he is right :/. |
|
| Back to top |
|
|
tonyrayo
Voice
Joined: 31 Jul 2003
Posts: 20
Location: Waldorf, MD
|
| Posted: Sun Aug 21, 2005 4:42 pm Post subject: |
|
|
| I believe demond has been brief because there has been no offical release... or maybe even because no one should try to use the script at this stage that doesn't know what an rss parser is and how it would work based on looking at source... or maybe it's just because demond refuses to wear domed shape hats!!! PORQUE!!!! |
|
| Back to top |
|
|
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
| Posted: Sun Aug 21, 2005 4:46 pm Post subject: |
|
|
| an official release that doesn't require external packages is coming, stay tuned |
|
| Back to top |
|
|
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
| Posted: Mon Aug 22, 2005 10:57 pm Post subject: |
|
|
ok folks, here it is, feedback is much appreciated:
| Code: |
#######################################################################
# #
# rssnews.tcl - RSS news announcer for eggdrop by demond@demond.net #
# #
# this will announce the updated news from RSS feed(s), #
# periodically polling the feed(s); supports multiple #
# channels, one feed per channel; you only need to set #
# the feeds array, see below #
# #
# Usage: !rss <news#|*> (read news# or * for headlines list) #
# #
#######################################################################
package require Tcl 8.3
package require eggdrop 1.6
package require http 2.0
namespace eval rssnews {
# set your feed(s) sources here: channel, poll frequency in mins, feed URL
#
#set feeds(#chan1) {17 http://www.osnews.com/files/recent.rdf}
#set feeds(#chan2) {11 http://news.google.com/news?ned=us&topic=h&output=rss}
variable version "rssnews-1.0"
variable timeout 20 ;# seconds
bind pub - !rss [namespace current]::news
bind time - * [namespace current]::timer
putlog "$version by demond loaded"
proc timer {min hour day month year} {
variable feeds
if {[info exists feeds]} {
if {$min} {set min [string trimleft $min 0]}
foreach {chan feed} [array get feeds] {
if {$min && $min % [lindex $feed 0] == 0} {
fetch [lindex $feed 1] $chan
}
}}
}
proc fetch {url chan} {
variable timeout
variable version; variable token
set to [expr {$timeout * 1000}]
set cmd [namespace current]::callback
::http::config -useragent "$version by demond"
if {[catch {set t [::http::geturl $url -command $cmd -timeout $to]} err]} {
putlog "$version: ERROR($chan): $err"
} {
set token($t) $chan
}
}
proc callback {t} {
variable version; variable token
set chan $token($t)
switch -exact [::http::status $t] {
"timeout" {
putlog "$version: ERROR($chan): timeout"
}
"error" {
putlog "$version: ERROR($chan): [::http::error $t]"
}
"ok" {
if {[::http::ncode $t] != 200} {
putlog "$version: ERROR($chan): [::http::code $t]"
} {
process [::http::data $t] $chan
}
}
default {
putlog "$version: ERROR($chan): got EOF from socket"
}}
::http::cleanup $t
}
proc process {data chan} {
variable news; variable hash
set idx 1; set news($chan) {}
regsub -all {(?i)<items.*?>.*?</items>} $data {} data
foreach {foo item} [regexp -all -inline {(?i)<item.*?>(.*?)</item>} $data] {
regexp {(?i)<title>(.*?)</title>} $item -> title
regexp {(?i)<link>(.*?)</link} $item -> link
regexp {(?i)<description>(.*?)</description>} $item -> descr
strip title descr
if {[info exists hash($chan)]} {
if {[lsearch -exact $hash($chan) [md5 $title]] == -1} {
if {[botonchan $chan]} {
puthelp "privmsg $chan :($idx) $title"
}
}}
lappend news($chan) [list $title $link $descr]
lappend hashes [md5 $title]
incr idx
}
set hash($chan) $hashes
}
proc strip {args} {
variable html
foreach a $args {
upvar $a x
set amp {& &}
set x [string map $amp $x]
set x [string map $html $x]
regsub -all {<[^<]+?>} $x {} x
}
}
proc news {nick uhost hand chan text} {
variable news; variable feeds
set num [lindex [split $text] 0]
if {$num == ""} {
puthelp "notice $nick :Usage: $::lastbind <news#|*>"
return
}
if {$num != "*" && ![string is integer $num]} {
puthelp "notice $nick :argument must be number or *"
return
}
if {![info exists news($chan)]} {
puthelp "notice $nick :no news for this channel"
return
}
if {$num == "*"} {
set idx 1
puthelp "notice $nick :News source: [lindex $feeds($chan) 1]"
foreach item $news($chan) {
puthelp "notice $nick :($idx) [lindex $item 0]"
incr idx
}
return 1
} {
if {$num < 1 || $num > [llength $news($chan)]} {
puthelp "notice $nick :no such news index, try $::lastbind *"
} {
set idx [expr {$num-1}]
puthelp "notice $nick :......title($num): [lindex [lindex $news($chan) $idx] 0]"
puthelp "notice $nick :description($num): [lindex [lindex $news($chan) $idx] 2]"
puthelp "notice $nick :.......link($num): [lindex [lindex $news($chan) $idx] 1]"
return 1
}}
}
variable html {
" \x22 ' \x27 & \x26 < \x3C
> \x3E \x20 ¡ \xA1 ¤ \xA4
¢ \xA2 £ \xA3 ¥ \xA5 ¦ \xA6
§ \xA7 ¨ \xA8 © \xA9 ª \xAA
« \xAB ¬ \xAC ­ \xAD ® \xAE
¯ \xAF ° \xB0 ± \xB1 ² \xB2
³ \xB3 ´ \xB4 µ \xB5 ¶ \xB6
· \xB7 ¸ \xB8 ¹ \xB9 º \xBA
» \xBB ¼ \xBC ½ \xBD ¾ \xBE
¿ \xBF × \xD7 ÷ \xF7 À \xC0
Á \xC1 Â \xC2 Ã \xC3 Ä \xC4
Å \xC5 Æ \xC6 Ç \xC7 È \xC8
É \xC9 Ê \xCA Ë \xCB Ì \xCC
Í \xCD Î \xCE Ï \xCF Ð \xD0
Ñ \xD1 Ò \xD2 Ó \xD3 Ô \xD4
Õ \xD5 Ö \xD6 Ø \xD8 Ù \xD9
Ú \xDA Û \xDB Ü \xDC Ý \xDD
Þ \xDE ß \xDF à \xE0 á \xE1
â \xE2 ã \xE3 ä \xE4 å \xE5
æ \xE6 ç \xE7 è \xE8 é \xE9
ê \xEA ë \xEB ì \xEC í \xED
î \xEE ï \xEF ð \xF0 ñ \xF1
ò \xF2 ó \xF3 ô \xF4 õ \xF5
ö \xF6 ø \xF8 ù \xF9 ú \xFA
û \xFB ü \xFC ý \xFD þ \xFE
ÿ \xFF ' \x27 <p> \x20 <br> \x20 \n \x20
}
}
|
|
|
| Back to top |
|
|
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
| Posted: Mon Aug 22, 2005 11:01 pm Post subject: |
|
|
| ' on the last line actually needs to be & #39; (without space between & and #) |
|
| Back to top |
|
|
r00tw00t
Voice
Joined: 23 Aug 2005
Posts: 2
|
| Posted: Tue Aug 23, 2005 8:24 am Post subject: It works great!!! |
|
|
Wow, after two hours of trying to get your 'beta' version working, I found this. One single file does it all. kudos!
Now if I wanned the bot to say the headline + link in the channel, how do I do? Like: <eggdrop> [nameofnewssource]: [headline] - [link]
And everytime a new headline comes up it will say it in the #. (somekind of pseudo-live news feeder.)
Thanks!
EDIT: huh. The script already does that.  I was refreshing the RSS until a new headline was up, and checked if the bot had said that. The poll frequency is 3 min but it took 15-20 minutes for the bot to say the headline. Laggy?
 |
|
| Back to top |
|
|
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
| Posted: Tue Aug 23, 2005 11:40 am Post subject: |
|
|
| you set the poll frequency in the feeds array to 3mins but it took 15mins for an update headline(s) to show up? that means the RSS feed got updated 12mins (or more) after the script started polling |
|
| Back to top |
|
|
r00tw00t
Voice
Joined: 23 Aug 2005
Posts: 2
|
|
| Back to top |
|
|
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
| Posted: Wed Aug 24, 2005 5:48 pm Post subject: |
|
|
you don't have <description> tags in that feed
I'll fix that in 1.1 by providing default values for <title>, <link> and <description> tags |
|
| Back to top |
|
|
demond
Revered One
Joined: 12 Jun 2004
Posts: 3073
Location: San Francisco, CA
|
| Posted: Sat Aug 27, 2005 3:49 pm Post subject: |
|
|
okay 1.1 is out with that fixed and improved handling of HTML encoding
find rssnews.tcl here, I'll also submit it to Tcl script archive |
|
| Back to top |
|
|
sarius
Voice
Joined: 05 Aug 2005
Posts: 20
|
| Posted: Sun Aug 28, 2005 2:43 pm Post subject: |
|
|
Sweet!
It works for most the the news websites so far. But is it possible to include CDATA support too?
It can't seem to read from this site - http://soccernet.espn.go.com/rss/news
Really appreciate the work done. Cheers! |
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
