This is the new home of the egghelp.org community forum.
All data has been migrated (including user logins/passwords) to a new phpBB version.


For more information, see this announcement post. Click the X in the top right-corner of this box to dismiss this message.

RSS news by demond

Support & discussion of released scripts, and announcements of new releases.
Post Reply
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

RSS news by demond

Post by demond »

since the other thread that people have been posting in about this wasn't actually relevant from the beginning, I'm opening this dedicated to the subject new thread

for those who don't know what the fuss is all about: I made a RSS news script for my own purposes (it still doesn't have an official name, I haven't released it officially yet) and I posted the beta version on several occasions here on these forums, after seeing that people tend to have problems with other RSS scripts; my version differs from all other similar scripts in the mechanism of parsing - it uses the object-oriented and XML facilities of TclLib and TclXML, also the RSS package from BDK, posted on Tcl'ers Wiki; and as it (unfortunately) turns out, most people are not used to installing third-party Tcl packages and therefore have been having problems with my script's requirements

I'm a big proponent of software component re-use, and I think we should encourage usage of good (and de facto standard) component packages like TclLib and TclXML

I'm also inclined to release the script officially, but that would depend on how many people actually manage to successfully install the required packages
g
greenbear
Owner
Posts: 733
Joined: Mon Sep 24, 2001 8:00 pm
Location: Norway

Post by greenbear »

Installing aditional packages will probably be a big pita for everyone running bots from their cheap $5 a month shell accounts...
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

Post by demond »

well you certainly have a point here, although if you have at least 5mb quota you should be able to install TclLib and TclXML in your homedir:

Code: Select all

[demond@whitepine demond]$ du -h -d0 lib/tcllib1.7/ lib/tclxml3.0/
3.1M    lib/tcllib1.7/
156K    lib/tclxml3.0/
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

Post by demond »

I might rewrite this thing so that it would no longer depend on external packages, but it will continue to be rather minimalistic RSS solution, I'm not interested in adding bells & whistles
User avatar
sKy
Op
Posts: 194
Joined: Thu Apr 14, 2005 5:58 pm
Location: Germany

Post by sKy »

I think that`s an good idea.

Well, RSS. I know what RSS means but i think its inconnu as well. Please don`t get me wrong, but if you want some feedback you then should try to explain in some easy sentence what RSS is and what the purpose of your script is.
Installing aditional packages will probably be a big pita for everyone running bots from their cheap $5 a month shell accounts...
I think he is right :/.
t
tonyrayo
Voice
Posts: 20
Joined: Thu Jul 31, 2003 3:29 pm
Location: Waldorf, MD
Contact:

Post by tonyrayo »

I believe demond has been brief because there has been no offical release... or maybe even because no one should try to use the script at this stage that doesn't know what an rss parser is and how it would work based on looking at source... or maybe it's just because demond refuses to wear domed shape hats!!! PORQUE!!!!
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

Post by demond »

an official release that doesn't require external packages is coming, stay tuned
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

Post by demond »

ok folks, here it is, feedback is much appreciated:

Code: Select all

#######################################################################
#                                                                     #
# rssnews.tcl - RSS news announcer for eggdrop by demond@demond.net   #
#                                                                     #
#               this will announce the updated news from RSS feed(s), #
#               periodically polling the feed(s); supports multiple   #
#               channels, one feed per channel; you only need to set  #
#               the feeds array, see below                            #
#                                                                     #
#        Usage: !rss <news#|*> (read news# or * for headlines list)   #
#                                                                     #
#######################################################################

package require Tcl 8.3
package require eggdrop 1.6
package require http 2.0

namespace eval rssnews {

# set your feed(s) sources here: channel, poll frequency in mins, feed URL
#
#set feeds(#chan1) {17 http://www.osnews.com/files/recent.rdf}
#set feeds(#chan2) {11 http://news.google.com/news?ned=us&topic=h&output=rss}

variable version "rssnews-1.0"

variable timeout 20 ;# seconds

bind pub  - !rss [namespace current]::news
bind time - *    [namespace current]::timer

putlog "$version by demond loaded"

proc timer {min hour day month year} {
	variable feeds
	if {[info exists feeds]} {
	if {$min} {set min [string trimleft $min 0]}
	foreach {chan feed} [array get feeds] {
		if {$min && $min % [lindex $feed 0] == 0} {
			fetch [lindex $feed 1] $chan
		}
	}}
}

proc fetch {url chan} {
	variable timeout
	variable version; variable token
	set to [expr {$timeout * 1000}]
	set cmd [namespace current]::callback
	::http::config -useragent "$version by demond"
	if {[catch {set t [::http::geturl $url -command $cmd -timeout $to]} err]} {
		putlog "$version: ERROR($chan): $err"
	} {
		set token($t) $chan
	}
}

proc callback {t} {
	variable version; variable token
	set chan $token($t)
	switch -exact [::http::status $t] {
	"timeout" {
		putlog "$version: ERROR($chan): timeout"
	}
	"error" {
		putlog "$version: ERROR($chan): [::http::error $t]"
	}
	"ok" {
		if {[::http::ncode $t] != 200} {
			putlog "$version: ERROR($chan): [::http::code $t]"
		} {
			process [::http::data $t] $chan
		}
	}
	default {
		putlog "$version: ERROR($chan): got EOF from socket"
	}}
	::http::cleanup $t
}

proc process {data chan} {
	variable news; variable hash
	set idx 1; set news($chan) {}
	regsub -all {(?i)<items.*?>.*?</items>} $data {} data
	foreach {foo item} [regexp -all -inline {(?i)<item.*?>(.*?)</item>} $data] {
		regexp {(?i)<title>(.*?)</title>}             $item -> title
		regexp {(?i)<link>(.*?)</link}                $item -> link
		regexp {(?i)<description>(.*?)</description>} $item -> descr
		strip title descr
		if {[info exists hash($chan)]} {
		if {[lsearch -exact $hash($chan) [md5 $title]] == -1} {
			if {[botonchan $chan]} {
				puthelp "privmsg $chan :($idx) $title"
			}
		}}
		lappend news($chan) [list $title $link $descr]
		lappend hashes [md5 $title]
		incr idx
	}
	set hash($chan) $hashes
}

proc strip {args} {
	variable html
	foreach a $args {
		upvar $a x
		set amp {& &}
		set x [string map $amp $x]
		set x [string map $html $x]
		regsub -all {<[^<]+?>} $x {} x
	}
}

proc news {nick uhost hand chan text} {
	variable news; variable feeds
	set num [lindex [split $text] 0]
	if {$num == ""} {
		puthelp "notice $nick :Usage: $::lastbind <news#|*>"
		return
	}
	if {$num != "*" && ![string is integer $num]} {
		puthelp "notice $nick :argument must be number or *"
		return
	}
	if {![info exists news($chan)]} {
		puthelp "notice $nick :no news for this channel"
		return
	}
	if {$num == "*"} {
		set idx 1
		puthelp "notice $nick :News source: [lindex $feeds($chan) 1]"
		foreach item $news($chan) {
			puthelp "notice $nick :($idx) [lindex $item 0]"
			incr idx
		}
		return 1
	} {
	if {$num < 1 || $num > [llength $news($chan)]} {
		puthelp "notice $nick :no such news index, try $::lastbind *"
	} {
		set idx [expr {$num-1}]
		puthelp "notice $nick :......title($num): [lindex [lindex $news($chan) $idx] 0]"
		puthelp "notice $nick :description($num): [lindex [lindex $news($chan) $idx] 2]"
		puthelp "notice $nick :.......link($num): [lindex [lindex $news($chan) $idx] 1]"
		return 1
	}}
}

variable html {
	"     \x22  &apos;     \x27  &      \x26  <       \x3C
	>       \x3E        \x20  ¡    \xA1  ¤   \xA4
	¢     \xA2  £    \xA3  ¥      \xA5  ¦   \xA6
	§     \xA7  ¨      \xA8  ©     \xA9  ª     \xAA
	«    \xAB  ¬      \xAC  ­      \xAD  ®      \xAE
	¯     \xAF  °      \xB0  ±   \xB1  ²     \xB2
	³     \xB3  ´    \xB4  µ    \xB5  ¶     \xB6
	·   \xB7  ¸    \xB8  ¹     \xB9  º     \xBA
	»    \xBB  ¼   \xBC  ½   \xBD  ¾   \xBE
	¿   \xBF  ×    \xD7  ÷   \xF7  À   \xC0
	Á   \xC1  Â    \xC2  Ã   \xC3  Ä     \xC4
	Å    \xC5  Æ    \xC6  Ç   \xC7  È   \xC8
	É   \xC9  Ê    \xCA  Ë     \xCB  Ì   \xCC
	Í   \xCD  Î    \xCE  Ï     \xCF  Ð      \xD0
	Ñ   \xD1  Ò   \xD2  Ó   \xD3  Ô    \xD4
	Õ   \xD5  Ö     \xD6  Ø   \xD8  Ù   \xD9
	Ú   \xDA  Û    \xDB  Ü     \xDC  Ý   \xDD
	Þ    \xDE  ß    \xDF  à   \xE0  á   \xE1
	â    \xE2  ã   \xE3  ä     \xE4  å    \xE5
	æ    \xE6  ç   \xE7  è   \xE8  é   \xE9
	ê    \xEA  ë     \xEB  ì   \xEC  í   \xED
	î    \xEE  ï     \xEF  ð      \xF0  ñ   \xF1
	ò   \xF2  ó   \xF3  ô    \xF4  õ   \xF5
	ö     \xF6  ø   \xF8  ù   \xF9  ú   \xFA
	û    \xFB  ü     \xFC  ý   \xFD  þ    \xFE
	ÿ     \xFF  '      \x27  <p> \x20 <br> \x20 \n \x20
}

}
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

Post by demond »

' on the last line actually needs to be & #39; (without space between & and #)
User avatar
r00tw00t
Voice
Posts: 2
Joined: Tue Aug 23, 2005 7:02 am

It works great!!!

Post by r00tw00t »

Wow, after two hours of trying to get your 'beta' version working, I found this. One single file does it all. kudos!

Now if I wanned the bot to say the headline + link in the channel, how do I do? Like: <eggdrop> [nameofnewssource]: [headline] - [link]

And everytime a new headline comes up it will say it in the #. (somekind of pseudo-live news feeder.)

Thanks!

EDIT: huh. The script already does that. :) I was refreshing the RSS until a new headline was up, and checked if the bot had said that. The poll frequency is 3 min but it took 15-20 minutes for the bot to say the headline. Laggy?
:D
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

Post by demond »

you set the poll frequency in the feeds array to 3mins but it took 15mins for an update headline(s) to show up? that means the RSS feed got updated 12mins (or more) after the script started polling
User avatar
r00tw00t
Voice
Posts: 2
Joined: Tue Aug 23, 2005 7:02 am

rssnews cant read this rss?

Post by r00tw00t »

http://www.fragbite.com/xml/headlines_rss.php i use this rss but the script doesnt seem to handle it?
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

Post by demond »

you don't have <description> tags in that feed

I'll fix that in 1.1 by providing default values for <title>, <link> and <description> tags
User avatar
demond
Revered One
Posts: 3073
Joined: Sat Jun 12, 2004 9:58 am
Location: San Francisco, CA
Contact:

Post by demond »

okay 1.1 is out with that fixed and improved handling of HTML encoding

find rssnews.tcl here, I'll also submit it to Tcl script archive
s
sarius
Voice
Posts: 20
Joined: Fri Aug 05, 2005 10:30 am

Post by sarius »

Sweet!

It works for most the the news websites so far. But is it possible to include CDATA support too?

It can't seem to read from this site - http://soccernet.espn.go.com/rss/news

Really appreciate the work done. Cheers! :D
Post Reply