| View previous topic :: View next topic |
| Author |
Message |
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
 Posted: Sat Aug 08, 2009 12:57 pm Post subject: |
 |
|
Download: Incith:Google v1.9.9m
New version but... this one WON'T fix every issue people are having regarding inconsistent results with "weather" and "calculator" within !google itself. What it does do is fix literally a ton of things that have broken over the course of the last month. This is the _EXACT_ version I am using myself. When I have more time I'll try to fix the problem with "weather/calculator"results. Until then consider this something to hold everybody over until then.
If you have _ANY_ issues, broken parsers, missing output, etc please feel free to shout it out here. If it has to do with google's "weather/calculator" results keep in mind this is already on the table and going to be addressed shortly. 
_________________
speechles' eggdrop tcl archive |
|
| Back to top |
|
 |
Anahel
Halfop
Joined: 03 Jul 2009
Posts: 48
Location: Dom!
|
| Posted: Sun Aug 09, 2009 7:31 pm Post subject: |
|
|
i've other problem with your google script (im using the newest "m" version)
when i use !trans with japanese characters bot send [censored] to channel:
| Code: | <&maruda> !trans jp@en オタク
<~Lilith> Google says: (ja->en) Translation: Japanese » English >> ŞŻGerman German German circumstances
|
and heres google:
| Code: | <tomek> !g オタク
<~Lilith> 7,360,000 Results | ĄŞ? Ą?? §?????$ @ http://jahjahsphinx.blogspot.com/ | ĄŞ? Ą?? §?????$ @ http://jahjahsphinx.blogspot.com/2009/08/blog-post.html | Ordinal indicator - Wikipedia, the free @ http://en.wikipedia.org/wiki/Ordinal_indicator | [PDF] : Ş " " Ş ˙ . . : Ş : 1. . 2 @ http://www.bibalex.org/ARABIC/Egyptology/images/coptlanglect.pdf |
maybe i've something unset with charset (i havent changed anything about it - only changes i've made was set wiki to .pl language, even google is set to .com)
could you help me? |
|
| Back to top |
|
|
spithash
Master
Joined: 12 Jul 2007
Posts: 248
Location: Libera
|
| Posted: Mon Aug 10, 2009 7:33 pm Post subject: |
|
|
I don't think this is an issue of the script, it works just awesome,
I believe its a UTF fonts issue.
_________________
Libera ##rtlsdr & ##re - Nick: spithash
Click here for troll.tcl |
|
| Back to top |
|
|
gh0st
Voice
Joined: 20 Aug 2009
Posts: 2
|
| Posted: Thu Aug 20, 2009 3:48 pm Post subject: |
|
|
i got this error since 2 days with the weather script. im using incith-google-v1.99m
| Code: | | Tcl error [incith::google::public_message]: can't read "f2": no such variable |
|
|
| Back to top |
|
|
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
| Posted: Thu Aug 20, 2009 6:02 pm Post subject: |
|
|
| Anahel wrote: | i've other problem with your google script (im using the newest "m" version)
when i use !trans with japanese characters bot send [censored] to channel:
| Code: | <&maruda> !trans jp@en オタク
<~Lilith> Google says: (ja->en) Translation: Japanese » English >> ŞŻGerman German German circumstances
|
|
Trans has issues regarding eggdrop and how it improperly handles utf-8. Also, http-package has large flaws which this script presently only fixes slightly (See here for http-packages flaw).
| Anahel wrote: | and heres google:
| Code: | <tomek> !g オタク
<~Lilith> 7,360,000 Results | ĄŞ? Ą?? §?????$ @ http://jahjahsphinx.blogspot.com/ | ĄŞ? Ą?? §?????$ @ http://jahjahsphinx.blogspot.com/2009/08/blog-post.html | Ordinal indicator - Wikipedia, the free @ http://en.wikipedia.org/wiki/Ordinal_indicator | [PDF] : Ş " " Ş ˙ . . : Ş : 1. . 2 @ http://www.bibalex.org/ARABIC/Egyptology/images/coptlanglect.pdf |
maybe i've something unset with charset (i havent changed anything about it - only changes i've made was set wiki to .pl language, even google is set to .com)
could you help me? |
| Code: | # encoding conversion lookups
# here is where you can correct language encoding problems by pointing their
# abbreviation towards an encoding. if you want more, feel free to add more.
# this is a somewhat poor example below, there are possibly hundreds of additions
# that need to be added to this section, this is just something to see if this
# indeed is a work around, enjoy and merry christmas ;P
# ------
variable encode_strings {
zh:gb2312
de:iso8859-1
es:iso8859-1
it:iso8859-1
nl:iso8859-1
com:iso8859-1
co.uk:iso8859-1
en:iso8859-1
fr:iso8859-1
ro:cp1251
bg:cp1251
rs:cp1251
sr:cp1251
sr-el:cp1252
ru:cp1251
ar:cp1256
jp:shiftjis
ja:shiftjis
co.jp:shiftjis
tr:cp857
kr:iso2022-kr
co.kr:iso2022-kr
co.il:iso8859-8
com.ua:koi8-u
uk:koi8-u
hu:cp1250
pl:iso8859-2
} |
This is where you set the encodings for each TLD (or in the case of wikimedia, the entire website domain) listed. if this list has any incorrect encoding listed next to any region/site this is up to each user to fix individually. This list is by no means, complete, accurate, nor guaranteed. Together with using the list and reading the output from debug mode of the script you should be able to figure out most of it. You might also use webby to visit the same url the bot has and see if webby can parse the page correctly. Webby features advanced encoding detection, reporting, and conflict resolution. So it's display should always be accurate and when it isn't append --swap to your url and it will resolve conflicts inversely. This isn't just a shameless plug to promote webby, this is me admitting that it's so useful to me everyday that I find it indispensable.
If you need any of the config settings explained, you should ask how to use them or actually read the comments completely and I pretty much explain how everything operates. If you have any questions on any of this, it isn't dumb to ask a question. It implies you have read the comments and need clarfication.
_________________
speechles' eggdrop tcl archive
Last edited by speechles on Thu Aug 20, 2009 6:14 pm; edited 1 time in total |
|
| Back to top |
|
|
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
| Posted: Thu Aug 20, 2009 6:12 pm Post subject: |
|
|
| gh0st wrote: | i got this error since 2 days with the weather script. im using incith-google-v1.99m
| Code: | | Tcl error [incith::google::public_message]: can't read "f2": no such variable |
|
Doh! Why didn't you say something two days ago?! HAW 
but on a more serious note, will have this fixed shortly. Watch for this post to be editing soon with a new version fixing the issue with weather, as well as fixing a handful of other bugs experienced.
Edit:
Well, how does this look? I think this fixes everything... | Code: | # .9n: Corrected google weather and cruft html elements within#
# some google no-results-found messages. # |
| Quote: | <speechles> !g .de wetter berlin
<sp33chy> Wetter für Berlin: 24 °C, Aktuell: Klar, Wind: SO mit 11 km/h, Feuchtigkeit: 44 %; Forecast: Do., Sunny (31°C|18°C); Fr., Chance of storm (29°C|14°C); Sa., Mostly sunny (23°C|10°C)
<speechles> !g .nl weer amsterdam
<sp33chy> Weer voor Amsterdam: 19 °C, Actueel: Plaatselijk bewolkt, Wind: W - 16 km/h, Luchtvochtigheid: 63%; Forecast: do, Chance of storm (29°C|15°C); vr, Chance of rain (21°C|13°C); za, Chance of rain (21°C|12°C)
<speechles> !g weather san francisco, california
<sp33chy> Weather for San Francisco, California: 67°F, Current: Partly Cloudy, Wind: NW at 6 mph, Humidity: 73%; Forecast: Thu, Mostly sunny (63°F|54°F); Fri, Mostly sunny (67°F|56°F); Sat, Mostly sunny (67°F|54°F) |
Incith:Google v1.9.9n. Enjoy and as always, have a fun ;P
Will correct ebay and it's inconsistant shipping display for .com results shortly.. expect to see .o version at that time ;P
_________________
speechles' eggdrop tcl archive |
|
| Back to top |
|
|
gh0st
Voice
Joined: 20 Aug 2009
Posts: 2
|
| Posted: Fri Aug 21, 2009 9:08 am Post subject: |
|
|
| hey thank you! :thumps up: |
|
| Back to top |
|
|
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
| Posted: Fri Aug 21, 2009 9:26 pm Post subject: |
|
|
| Quote: | -- singapore
<speechles> !e .com.sg bubble gum
<sp33chy> 19 results | Vincci Bubblegum Pop Colorful Opentoe He, S$ 9.90(+S$ 2.50), 0 Bids, 8h 25m @ http://cgi.ebay.com.sg/_W0QQItemZ250483340761 | BN Bonne Bell Bubble Gum Lip Smacker Glo, S$ 4.50(Free), Buy It Now, 2h 45m @ http://cgi.ebay.com.sg/_W0QQItemZ250483289799 | Japanese Animation - Bubblegum Crisis To, S$ 3.00(+S$ 2.00), 0 Bids, 1d 3h 27m @ http://cgi.ebay.com.sg/_W0QQItemZ270444229706
-- malaysia
<speechles> !e .com.my bride
<sp33chy> 148 results | BRAND NEW The Petrakos Bride by Lynne Gr, RM 18.00(+RM 4.00), 0 Bids, 3h 9m @ http://cgi.ebay.com.my/_W0QQItemZ380149390697 | NEW The Italian's Future Bride by Michel, RM 18.00(+RM 4.00), 0 Bids, 3h 17m @ http://cgi.ebay.com.my/_W0QQItemZ380149391616 | NEW The Ranieri Bride by Michelle Reid, RM 18.00(+RM 4.00), 0 Bids, 3h 18m @ http://cgi.ebay.com.my/_W0QQItemZ380149391840
-- germany
<speechles> !e .de eier
<sp33chy> 45.236 Ergebnisse | WAHNSINNNNNN zwischen 2500 und 2700 Hpf, EUR 20,50(+EUR 15,00), 11 Gebote, 6T 16Std 38Min @ http://cgi.ebay.de/_W0QQItemZ230368738997 | O EI A Jubelausgabe 2003-04, EUR 1,00(+EUR 4,90), 0 Gebote, 25Min @ http://cgi.ebay.de/_W0QQItemZ350241866986 | Ü-Ei Figur Sonne, Mond und Sterne Stern, EUR 1,00(+EUR 2,00), 0 Gebote, 41Min @ http://cgi.ebay.de/_W0QQItemZ130324363220
-- united states
<speechles> !e .com eggs
<sp33chy> 18,628 results | 155.0ct RUBY AMETHYST GARNET BT STERLING, $202.50(), 10 Bids, 6d 17h 26m @ http://cgi.ebay.com/_W0QQItemZ310162811776 | Little Giant Egg Incubator w/ Forced Air, $142.95(Free shipping), Buy It Now, 26d 11h 41m @ http://cgi.ebay.com/_W0QQItemZ330353195089 | Ped Egg Professional Pedicure & 3 File B, $7.49(Free shipping), Buy It Now, 8d 17h 23m @ http://cgi.ebay.com/_W0QQItemZ260465595310
-- france
<speechles> !e .fr pink
<sp33chy> 24 022 résultats | 6 ÉPINGLES À CHEVEUX MARIÉE MARIAGE PART, 4,99 EUR(+3,99 EUR), 0 Enchère, 6m @ http://cgi.ebay.fr/_W0QQItemZ140339874791 | BONNET DE NAISSANCE "J'AIME PAPA" - ROSE, 3,50 EUR(+1,88 EUR), Achat immédiat, 8j 7h 17m @ http://cgi.ebay.fr/_W0QQItemZ200375358198 | GANTS SANS DOIGT MARIÉE MARIAGE PARTIE ~, 4,99 EUR(+3,99 EUR), 0 Enchère, 7m @ http://cgi.ebay.fr/_W0QQItemZ140339874835 |
Shipping and all parsing is now detected perfectly (an empty shipping field indicates just that this isn't a bug). This uses several different templates and schemes to extract information from almost every regional ebay server you feed it. The ones above are merely examples demonstrating this behavior. There are several more you can use as well somewhere around 20ish or so. This update has also corrected the bug within google video display causing things to run together.
Incith:Google v1.9.9o .. Enjoy everything now works as intended... 
_________________
speechles' eggdrop tcl archive |
|
| Back to top |
|
|
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
| Posted: Mon Aug 24, 2009 2:08 am Post subject: |
|
|
Had some time to investigate google translations and it's total lack of understanding encodings and seem to have stumbled upon an answer of sorts. | Code: | # Google translation input/output encodings
# set these below to either:
# 0 for automatic encoding
# 1 for utf-8
# You may use switches to inverse this behavior on-the-fly. Set them
# below as well.
# ------
variable trans_input 1
variable trans_output 0
variable trans_input_t "--swapi"
variable trans_output_t "--swapo" |
With these options you can now control how encodings for google translations are seen both as input and output. Using 0 will automatically detect, while using 1 will strictly assume utf-8. This is all fine and dandy until you discover the problem. It would be nice to inverse this, maybe on the fly for the few languages which don't work quite as expected and so people may experiment on their own. Well now this is possible, you can modify any of these attributes using the 2 switches you create within trans_input_t and trans_output_t.
| Quote: | Using 1 for input encoding seems to solves the accented character problem as shown below:
<speechles> !tr Google Traducciones parece odiar eggdrops por alguna razón
<sp33chy> Google says: (auto->en) Translation: Galician (automatically detected) » English >> Google seems to hate eggdrops translations for some reason
<speechles> !tr Google Traducciones parece odiar eggdrops por alguna razón --swapi
<sp33chy> Google says: (auto->en) Translation: Galician (automatically detected) » English >> Google translations seem to hate any race for eggdrops § Ã £ o
Now the problem with using 1 for output encoding is that it doesn't seem to work for all languages. This is probably a direct result of eggdrop not being able to handle utf-8 correctly which 1 means.. But, you do have the ability to inversely affect the output encodings as shown below:
<speechles> !tr Google Traducciones parece odiar eggdrops por alguna razón --swapo
<sp33chy> Google says: (auto->en) Translation: Galician (automatically detected) » English >> Google seems to hate eggdrops translations for some reason
Keep in mind you can place these "switches" anywhere in the string and they will be properly detected and removed. So you must be careful what you name them to keep them from being seen as of any text you may wish to translate. The debug output during these queries (if enabled and debugnick is set correctly) will clearly indicate which encodings are being used each time. |
The problem here is that choosing utf-8, which --swapo does in this case (using default options shown above), there is no notable difference between the automatic (iso8859-1, in this case) representation and the utf-8 one. Not until you start to deal with accented characters and eggdrop mishandles things. If your bot is patched with either the suzi or utf-8 patch this may work wonders now for you, as every query within google now embeds the encoding within the query line. This wasn't happening before and was the reason for alot of the problems. Utf-8 folks should now be alot happier. Even if you aren't using utf-8, this works much better than any previous version...
Incith:Google v1.9.9p .. Have a fun
_________________
speechles' eggdrop tcl archive |
|
| Back to top |
|
|
Anahel
Halfop
Joined: 03 Jul 2009
Posts: 48
Location: Dom!
|
| Posted: Mon Aug 24, 2009 7:00 am Post subject: |
|
|
hmm i've installeg the "p" version and still have issues with japanese language, i've tested polish too and there's a problem too:
xchat:
| Code: |
<tomek> !tr pl@en liść
<~Lilith> Google says: (pl->en) Translation: Polish » English >> If [?
<tomek> !tr ja@en 私
<~Lilith> Google says: (ja->en) Translation: Japanese » English >> Á
|
mirc:
| Code: |
<&anahel> !tr pl@en gość
<~Lilith> Google says: (pl->en) Translation: Polish » English >> it [?
<&anahel> !tr jp@en ポオランド
<~Lilith> Google says: (ja->en) Translation: Japanese » English >> ÝŞ éóÉ
|
so there's still an issue with iso-8859-2 (polish) and (shifts something and dont remember jap charset, later i've changed it to utf-8 which works much better than original)
and heres test of you example:
| Code: |
<tomek> !tr Google Traducciones parece odiar eggdrops por alguna razón
<~Lilith> Google says: (auto->en) Translation: Galician (automatically detected) » English >> Google seems to hate eggdrops translations for some reason
|
 |
|
| Back to top |
|
|
transacid
Voice
Joined: 08 Aug 2007
Posts: 12
Location: Hamburg / Germany
|
| Posted: Tue Sep 08, 2009 4:21 pm Post subject: |
|
|
apparently there is something wrong with the parser. (besides the wrong usage but this shouldn't happen at all IMHO)
the query:
| Quote: | | [21:36:14] ( transacid) !g hl:de define:potz |
the output:
another thing:
| Code: | [22:27:15] ( transacid) !w plenk
[22:27:16] ( NeXuS) Wikimedia Error: Unable to parse for: plenk @
http://en.wikipedia.org/wiki/index.php?title=Special%3ASearch&search=plenk&fulltext=Search
|
|
|
| Back to top |
|
|
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
| Posted: Tue Sep 08, 2009 10:38 pm Post subject: |
|
|
| transacid wrote: | | apparently there is something wrong with the parser. (besides the wrong usage but this shouldn't happen at all IMHO) |
It's bound to happen eventually. It's simply the nature of web scraping. Html changes with the wind. In a perfect (read this as free API) world we wouldn't need to screen/web scrape a thing. But this world we live in is far from that...
another thing:
| transacid wrote: | | Code: | [22:27:15] ( transacid) !w plenk
[22:27:16] ( NeXuS) Wikimedia Error: Unable to parse for: plenk @
http://en.wikipedia.org/wiki/index.php?title=Special%3ASearch&search=plenk&fulltext=Search
|
|
Note: This is actually a bug within wikipedia/wikimedia sites themselves, not anything lacking in the script. The problem has to do with problematic squid caches returning outdated incorrect replies. See here for more info.
This occurs because that page just so happens to be gzipped. This script expects text data to flow. Not binary compressed data. It's no longer an issue though because I've incorporated support for the zlib package within the script now. It will seamlessly decompress those pesky gzipped wikipedia/media pages easily.
http://wiki.tcl.tk/4610 - this url explains most of what is now required. And by required, I mean if you don't need gzip for your wiki pleasures then by all means it's not required. But.. If you experience any issue like above, your going to need it.
Windrop users without tcl8.6 handy (which includes myself ) can simply use the url below to download a precompiled binary package, copying the entire folder to their windrop/lib/tcl folder gets you going.
http://pascal.scheffers.net/software/zlib-1.1.1-win32.zip
Eggdrop users, on actual eggdrops running flavors of *nix can simply rely on trf package to work the gzip magic. This is to allow future versions of tcl (zlib) to work as well as the older (trf).
Now for the new additions to the script:
| Code: | # set this to the proxy you wish the main html fetching procedure to
# use. set both proxy_host and proxy_port. if you don't wish to use
# a proxy. set proxy_host ""
# --
variable proxy_host ""
variable proxy_port ""
# set this to the switch you want to use for youtube to filter content
# and display only high-definition videos.
# ------
variable youtube_highdef "@hd" |
Those previously getting 503 google sorry messages will now be much happier. There is now a proper proxy system present within the script as well. You can also now search youtube and filter high-definition videos only. This switch is customizeable as well and can be placed anywhere (except before the .region switch).
!y @hd .de whatever - Bad, the high-def switch is placed before the region.
!y .de whatever @hd / !y .fr @hd whatever - both 100% acceptable.
Edit: While on the subject of gzipped webpages. Why not incorporate this bandwidth saving feature into the script. This has now been done. If you have zlib or trf packages handy on your bot this script will now use them and save bandwidth using it for every query it makes. In my tests this seems to speed up page loading considerably. It is not a requirement to use zlib or trf packages, the script will still function without them. But for those with it, you will notice better response times.
Download: Incith:Google v1.9.9r
As you can see above using webby to illustrate. Simply gzipping the query arrives at 9576 bytes. While the entire html would've been 23390 bytes. This equates to quite a savings over time. Enjoy, and most important. Have a fun ;P
_________________
speechles' eggdrop tcl archive |
|
| Back to top |
|
|
glitchiker
Voice
Joined: 26 Aug 2009
Posts: 1
|
| Posted: Tue Sep 15, 2009 12:06 pm Post subject: |
|
|
zlib is now a part of tcl 8.6 for those of you / us who are eggdrop users.
If you are getting this error, it may be because zlib is not being called correctly in the script due to the version of tcl you may have. Versions 8.6b1.1 going forward all now have zlib packaged within tcl 8.6.
Error: | Quote: | | Incith:Google : Unable to find zlib package. Wikipedia and Wikimedia require gzip decompression. As a result, every article will not work for you. |
The shell provider and I worked on the issue and came up with this solution. Alter the code as needed.
*To find out what tcl packages are being used on your host use the following command:
| Quote: | | pkg_info | grep tcl |
For tcl 8.6b1.1 users and below. "DO NOT ALTER THIS CODE":
| Quote: | | if {[catch {package require zlib} error] !=0} { |
Anyone who is using tcl 8.6b1.1 and above, go ahead and modify the previous line to suit the following:
| Quote: | | if {[lsearch [info commands] zlib] == -1 && [catch {package require zlib} error] !=0} { |
Shell provider: "Just tell him that the zlib package is now a part of tcl and does not need to be loaded".
This fix may only be temporary until speechles does something new, and if none of this works, you may have to stay with the frontend script http 2.5. Make sure the script is loaded before incith-google*.
You can get the script from here: http://incith.com/eggdrop/http.tcl
Also you may have to comment the zlib section of the script out.
| Quote: | #if {[catch {package require zlib} error] !=0} {
#putlog "Incith:Google : Unable to find zlib package. Wikipedia and Wikimedia require gzip decompression. As a result, every article will not work for you."
#set incith_nozlib 1
#set incith_hdr ""
#} else {
set incith_hdr "Accept-Encoding gzip,deflate"
#} |
Another side note. If you are compiling eggdrop-1.6.19 on tcl 8.6* you may have some difficulty unless you hack into the compilation process first. Add the following code to config.h in between ./configure and make config.
| Quote: | | #define USE_INTERP_RESULT 1 |
Enjoy,
glitch
 |
|
| Back to top |
|
|
TehJibba
Voice
Joined: 19 Sep 2009
Posts: 1
|
| Posted: Sat Sep 19, 2009 5:14 pm Post subject: |
|
|
| Youtube changed its site code again and is the only part of the new version that doesnt work for me. Was hoping for maybe an updated snippet or workaround possibly |
|
| Back to top |
|
|
shadrach
Halfop
Joined: 14 Dec 2007
Posts: 74
|
| Posted: Mon Sep 21, 2009 2:28 pm Post subject: |
|
|
| 'locate' seems not to be working. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
