| View previous topic :: View next topic |
| Author |
Message |
w00f
Halfop
Joined: 04 Oct 2006
Posts: 49
|
 Posted: Fri Feb 26, 2010 7:24 am Post subject: HTTP/TLS Package |
 |
|
Hey guys,
i got this little problem within my wget tcl procedure when calling http::get to reach one ssl server. Works fine if i remove the ssl from the URL (-https +http).
here's is a bit of the script
| Code: |
package require http
package require tls
http::register https 443 [list ::tls::socket -require 0 -request 1]
proc s:wget { url } {
catch {set token [http::geturl $url -binary 1 -timeout 10000]} error
.....
if {[http::ncode $token] != 200} {
s:debug "[http::ncode $token] HTTP error"
return 0
}
}
|
It breaks in the http::ncode check ([http::ncode $token] = "", no code returned).
.errorInfo
can't read "state(sock)": no such element in array
while executing
"fileevent $state(sock) writable {}"
Thanks in advance. |
|
| Back to top |
|
 |
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
| Posted: Fri Feb 26, 2010 8:32 pm Post subject: |
|
|
Checking for a numeric code when one isn't available happens. Your method isn't a reliable way to check for the full range of errors which can occur during http transactions and parsing the response. This is the way I would handle it..... | Code: | package require http
package require tls
http::register https 443 [list ::tls::socket -require 0 -request 1]
proc s:wget { url } {
http::config -useragent "Mozilla/EggdropWget"
catch {set token [http::geturl $url -binary 1 -timeout 10000]} error
# error condition 1, invalid socket or other general error
if {![string match -nocase "::http::*" $error]} {
s:debug "Error: [string totitle [string map {"\n" " | "} $error]] \( $url \)"
return 0
}
# error condition 2, http error
if {![string equal -nocase [::http::status $token] "ok"]} {
s:debug "Http error: [string totitle [::http::status $token]] \( $url \)"
http::cleanup $token
return 0
}
# recursive redirect support, 300's
# this doesn't check for redirect to self nor does it check
# for nesting too deep, if you want these I can supply them
# but for using trusted urls, this shouldn't be necessary.
if {[string match "*[http::ncode $token]*" "303|302|301" ]} {
upvar #0 $token state
foreach {name value} $state(meta) {
if {[regexp -nocase ^location$ $name]} {
if {![string match "http*" $value]} {
# fix our locations if needed
if {![string match "/" [string index $value 0]]} {
set value "[join [lrange [split $url "/"] 0 2] "/"]/$value"
} else {
set value "[join [lrange [split $url "/"] 0 2] "/"]$value"
}
}
s:wget $value
# only the last iteration from our recursion is required
# we save time by using return on prior recurses
return
}
}
}
# waaay down here, we finally check the ncode for 400 or 500 codes
if {[string match 4* [http::ncode $token]] || [string match 5* [http::ncode $token]]} {
s:debug "Http resource is not evailable: [http::ncode $token] \( $url \)"
return 0
}
# uncomment the three lines below to return the http::data
# check for error by string equaling 0, if it isn't 0 it contains
# the data returned from the http transaction
# ---
# set data [http::data $token]
# http::cleanup $token
# return $data
} |
_________________
speechles' eggdrop tcl archive |
|
| Back to top |
|
|
w00f
Halfop
Joined: 04 Oct 2006
Posts: 49
|
| Posted: Mon Mar 01, 2010 7:20 pm Post subject: |
|
|
Thanks speechles, i've added the http status if (all the others are there already)
the status returned is 'eof', which is weird 
| Code: |
[11:10p] jackass • s:wget "https://xpto.com/index.php?action=dl&id=133"
[11:10p] -modtcl • [DEBUG] URL: https://xpto.com/index.php?action=dl&id=6437"
[11:10p] -modtcl • [DEBUG] Weirdo status returned, eof
[11:10p] -modtcl • 0
[11:10p] jackass • set errorInfo
[11:10p] -modtcl • can't read "state(sock)": no such element in array
[11:10p] -modtcl • while executing
[11:10p] -modtcl • "fileevent $state(sock) writable {}"
|
it worked just fine few days ago, dunno what changed since.
Something on the server maybe \: |
|
| Back to top |
|
|
nml375
Revered One
Joined: 04 Aug 2006
Posts: 2857
|
| Posted: Mon Mar 01, 2010 7:40 pm Post subject: |
|
|
w00f:
The EOF state simply means that the connection was dropped while the script was still retrieving the http headers (prematurely closed). Most likely the ssl handshake went wrong.
_________________
NML_375, idling at #eggdrop@IrcNET |
|
| Back to top |
|
|
w00f
Halfop
Joined: 04 Oct 2006
Posts: 49
|
| Posted: Mon Mar 01, 2010 8:02 pm Post subject: |
|
|
:/ That's what i though, just tried /usr/bin/wget with the SSL link and it worked like a charm.
So, it's most likely a SSL package problem ?
Using the non-ssl link isn't really a problem, i was just "curious"
Thanks :) |
|
| Back to top |
|
|
nml375
Revered One
Joined: 04 Aug 2006
Posts: 2857
|
| Posted: Tue Mar 02, 2010 12:07 am Post subject: |
|
|
Well, usually it's a matter of setting up a proper CA. I rare cases there might be some issues finding a common set of ciphers between the client and the server.
I'm not familiar with the tsl implementation, but in general when debugging OpenSSL issues, using the openssl s_client commandline client can be very helpful as it can use user-set CA's, verbose output, cerificate chain list, etc.
_________________
NML_375, idling at #eggdrop@IrcNET |
|
| Back to top |
|
|
w00f
Halfop
Joined: 04 Oct 2006
Posts: 49
|
| Posted: Tue Mar 02, 2010 7:25 pm Post subject: |
|
|
Well, i've updated the SSL certs (using openssl) then rehashed the certs, tested with curl/wget and it worked, everything seems to work but TCL =/
here's the output from curl
| Code: |
root@tehplanet:~# curl -I --capath /etc/ssl/certs https://xpto.com
HTTP/1.1 200 OK
Date: Tue, 02 Mar 2010 22:59:58 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Set-Cookie: PHPSESSID=29o1qivkuab8taakmfhhlg9ig5; path=/
Expires: Wed, 03 Mar 2010 04:59:58 GMT
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
Pragma:
Last-Modified: Tue, 02 Mar 2010 22:59:58 GMT
Connection: close
Content-Type: text/html; charset=UTF-8
root@tehplanet:~#
|
guess i'll have to exec wget if i want to use the SSL link. |
|
| Back to top |
|
|
nml375
Revered One
Joined: 04 Aug 2006
Posts: 2857
|
| Posted: Tue Mar 02, 2010 7:51 pm Post subject: |
|
|
Are you using the very same CA's with your tcl-script? I see that your xpto.com site uses a self-signed certificate. Unless this is listed as a trusted cert for your tls/tcl script, the connection would most likely be dropped.
Also, reading the docs for the tls tcl library, there are several additional parameters for tls connections, including -cadir and -cafile (specifying the root CA), and -command which may be used as a callback to track the status/failure of the tls/ssl handshake.
Further, depending on the version of tcl, you might have to patch the http-package. Supposedly, it's been submitted to the tcl developers to be included since v8.2.1
_________________
NML_375, idling at #eggdrop@IrcNET |
|
| Back to top |
|
|
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
| Posted: Tue Mar 02, 2010 9:42 pm Post subject: |
|
|
It's redirecting you. You've changed my messages to make it harder for me to tell, but this is obviously not the the url you chose. It's the one your being re-directed to.
| Code: | | PHPSESSID=29o1qivkuab8taakmfhhlg9ig5 |
It wants cookies too. This is why it's redirecting you. You aren't supplying the cookies during the redirect though. So all you'll get is an eof. Try the code below for a better outcome ... 
Note: What follows can work as "the textbook example" for anyone wishing to do the same or anything remotely similar. This covers every possible aspect and scenario that could arise from the http transaction (with the exception of security certificate authentications which nml375 discussed). This also includes redirect-to-self and nesting-too-deep when handling redirect traversals, even handles malicious url's safely. I've seen some poor web scripts that could benefit greatly just by reusing this code. Feel free to reuse it yourself for any non-malicious intents or purposes.
| Code: | package require http
package require tls
http::register https 443 [list ::tls::socket -require 0 -request 1]
# recursive wget with cookies and referer
proc s:wget { url {refer ""} {cookies ""} { re 0 } } {
http::config -useragent "Mozilla/EggdropWget"
# if we have cookies, let's use em ;)
if {![string length $cookies]} {
catch {set token [http::geturl $url -binary 1 -timeout 10000]} error
} else {
catch {set http [::http::geturl $url -binary 1 -headers [list "Referer" "$refer" "Cookie" "[string trim [join $cookies {;}] {;}]" ] -timeout 10000]} error
}
# error condition 1, invalid socket or other general error
if {![string match -nocase "::http::*" $error]} {
s:debug "Error: [string totitle [string map {"\n" " | "} $error]] \( $url \)"
return 0
}
# error condition 2, http error
if {![string equal -nocase [::http::status $token] "ok"]} {
s:debug "Http error: [string totitle [::http::status $token]] \( $url \)"
http::cleanup $token
return 0
}
upvar #0 $token state
# iterate through the meta array to grab cookies
foreach {name value} $state(meta) {
# do we have cookies?
if {[regexp -nocase ^Set-Cookie$ $name]} {
# yes, add them to cookie list
lappend ourCookies [lindex [split $value {;}] 0]
}
}
# if no cookies this iteration remember cookies from last
if {![info exists ourCookies] && [string length $cookies]} {
set ourCookies $cookies
}
# recursive redirect support, 300's
# the full gambit of browser support, hopefully ... ;)
if {[string match "*[http::ncode $token]*" "303|302|301" ]} {
foreach {name value} $state(meta) {
if {[regexp -nocase ^location$ $name]} {
if {![string match "http*" $value]} {
# fix our locations if needed
if {![string match "/" [string index $value 0]]} {
set value "[join [lrange [split $url "/"] 0 2] "/"]/$value"
} else {
set value "[join [lrange [split $url "/"] 0 2] "/"]$value"
}
}
# catch redirect to self's. There is one rule:
# A url can redirect to itself a few times to attempt to
# gain proper cookies, or referers. This is hard-coded at 2.
# We catch the 3rd time and poison our recursion with it.
# This will stop the madness ;)
if {[string match [string map {" " "%20"} $value] $url]} {
if {![info exists poison]} {
set poison 1
} else {
incr poison
if {$poison > 2} {
s:debug "HTTP Error: Redirect error self to self \(3rd instance poisoned\) \( $url \)""
return
}
}
}
# poison any nested recursion over 10 traversals deep. no legitimate
# site needs to do this. EVER!
if {[incr re] > 10} {
s:debug "HTTP Error: Redirect error (>10 too deep) \( $url \)"
return
}
# recursive redirect by passing cookies and referer
# this is what makes it now work! :)
s:wget [string map {" " "%20"} $value] $url $ourCookies $re
# only the last iteration from our recursion is required
# we save time by using return on prior recurses. this does
# not poison the recursion because we have invoked ourself
# before we poison the unneeded iteration.
return
}
}
}
# waaay down here, we finally check the ncode for 400 or 500 codes
if {[string match 4* [http::ncode $token]] || [string match 5* [http::ncode $token]]} {
s:debug "Http resource is not evailable: [http::ncode $token] \( $url \)"
return 0
}
# uncomment the three lines below to return the http::data
# check for error by string equaling 0, if it isn't 0 it contains
# the data returned from the http transaction. If it's an empty
# reply "", it means the site is either malicious or causing you
# to have endless redirects these are poison and no html is
# returned.
# ---
# set data [http::data $token]
# http::cleanup $token
# return $data
} |
_________________
speechles' eggdrop tcl archive |
|
| Back to top |
|
|
Elfriede
Halfop
Joined: 07 Aug 2007
Posts: 67
|
| Posted: Thu Mar 04, 2010 3:53 pm Post subject: |
|
|
| Code: | | set data [http::data $token] |
May i ask one question to that code.
What would be the easiest way to check $data, if theres eg html inside ? |
|
| Back to top |
|
|
speechles
Revered One
Joined: 26 Aug 2006
Posts: 1398
Location: emerald triangle, california (coastal redwoods)
|
| Posted: Thu Mar 04, 2010 7:50 pm Post subject: |
|
|
| Elfriede wrote: | | Code: | | set data [http::data $token] |
May i ask one question to that code.
What would be the easiest way to check $data, if theres eg html inside ? |
The easiest way is by knowing how html is constructed. There is a <head*> as well as a <body*> to most html pages. If these are present there is usually most always, a <title*> as well.
The http::geturl used above in my example meant for the original poster, w00f, includes "-binary 1". This omits the encoding as everything is bit-oriented. HTML obtained using this method probably won't appear correct for any languages that aren't composed of "ascii" characters (utf-8, for example, will lose its sequencing and render things incorrectly) as the HTML loses the encoding associated with the transaction when done in binary.
_________________
speechles' eggdrop tcl archive |
|
| Back to top |
|
|
w00f
Halfop
Joined: 04 Oct 2006
Posts: 49
|
| Posted: Fri Mar 05, 2010 4:49 pm Post subject: |
|
|
Yes nml375, i've already tried -cadir and -cafile options, same error.
The certificate it's not self-signed,
| Code: | (curl verbose output)
* successfully set certificate verify locations:
* CAfile: none
* CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* subject: /O=www.xpto.com/OU=Domain Control Validated/CN=www.xpto.com
* start date: 2009-11-11 05:55:55 GMT
* expire date: 2011-11-11 05:55:55 GMT
* subjectAltName: xpto.com matched
* issuer: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07
* SSL certificate verify ok.
|
I'm running Tcl 8.5.
Thanks for the code speechles, but the output is the same (status: eof)
ah! that "redirect" was a typo in the script, my bad.
Anyway, thanks guys =) |
|
| Back to top |
|
|
nml375
Revered One
Joined: 04 Aug 2006
Posts: 2857
|
| Posted: Fri Mar 05, 2010 6:29 pm Post subject: |
|
|
Ah well, was reported self-signed when I attempted to connect at that time.. nevertheless, try these settings:
| Code: | http::register https 443 [list ::tls::socket -require 0 -request 1 -command ::tls::callback]
set ::tls::debug 3
set ::tls::logcmd putlog |
That should dump quite a lot of data into the logs, hopefully revealing what is going on.
Edit: Sorry 'bout that, seems I was investigating the certificate of xpto.org, not xpto.com
Edit again: Should be -command, not -callback
_________________
NML_375, idling at #eggdrop@IrcNET |
|
| Back to top |
|
|
w00f
Halfop
Joined: 04 Oct 2006
Posts: 49
|
| Posted: Fri Mar 05, 2010 9:54 pm Post subject: |
|
|
here's the log/handshake
| Code: |
[01:40a] -modtcl • TLS/sock18: handshake/start: before/connect initialization
[01:40a] -modtcl • TLS/sock18: connect/loop: before/connect initialization
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv2/v3 write client hello A
[01:40a] -modtcl • TLS/sock18: connect/exit: SSLv2/v3 read server hello A
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 read server hello A
[01:40a] -modtcl • TLS/sock18: connect/exit: SSLv3 read server certificate A
[01:40a] -modtcl • TLS/sock18: connect/exit: SSLv3 read server certificate A
[01:40a] -modtcl • TLS/sock18: verify/3: Bad Cert: self signed certificate in certificate chain (rc = 0)
[01:40a] -modtcl • TLS/sock18: verify/3: /L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//emailAddress=info@valicert.com
[01:40a] -modtcl • TLS/sock18: verify/2: /C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
[01:40a] -modtcl • TLS/sock18: verify/1: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
[01:40a] -modtcl • TLS/sock18: verify/0: /O=www.xpto.com/OU=Domain Control Validated/CN=www.xpto.com
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 read server certificate A
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 read server key exchange A
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 read server done A
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 write client key exchange A
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 write change cipher spec A
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 write finished A
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 flush data
[01:40a] -modtcl • TLS/sock18: connect/exit: SSLv3 read finished A
[01:40a] -modtcl • TLS/sock18: connect/loop: SSLv3 read finished A
[01:40a] -modtcl • TLS/sock18: handshake/done: SSL negotiation finished successfully
[01:40a] -modtcl • TLS/sock18: connect/exit: SSL negotiation finished successfully
|
Then it hangs till times out
| Code: |
[01:41a] -modtcl • TLS/sock18: alert/write: close notify
[01:41a] -modtcl • [DEBUG] Request timeout (15s).
|
timeout only happens with the tls callback.
what does the "verify/3" log means? It's not a self-signed cert lol |
|
| Back to top |
|
|
nml375
Revered One
Joined: 04 Aug 2006
Posts: 2857
|
| Posted: Sat Mar 06, 2010 11:41 am Post subject: |
|
|
That was some interresting results.
The message regarding a self-signed cert means there was one certificate in the chain (not necessary yours) that was self-signed. Root CA certificates are generally self-signed, since there is no "higher" cert to sign it. Instead, we tell our clients (web browsers, etc) that we explicitly trust this cert, and any cert it has signed. An intermediate cert should however never be self-signed.
Setting the ::tls::debug variable to anything higher than 0, wil tell ::tl::callback to approve any cert, regardless of it's validity. The fact that your connection now doesn't close prematurely would suggest there might be some issue with the site-cert or your CA. However, the fact that it now freezes/times out suggests that the web server is not doing too well...
If you keep the ::tls::callback, but set ::tls::debug to 0, do you get the same behaviour?
_________________
NML_375, idling at #eggdrop@IrcNET |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
