The curl package needs updating/checking

[mad_ady]

I've been trying to repair the youtube-subscriptions plugin and I got my hands dirty with Google API.
Part of the authentication process is to connect via https to http://www.google.com and authenticate.

I'm using the following curl command line (replace $username and $password with your own if you want to try):

Code: Select all

curl -s -S --location https://www.google.com/accounts/ClientLogin --data 'Email=$username&Passwd=$password&service=youtube&source=wdtvext' --header 'Content-Type:application/x-www-form-urlencoded' 2>&1
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.


If I run the same command from my Ubuntu (10.04) box, the output is:

Code: Select all

adrianp@stingray:/tmp$ curl -s -S --location https://www.google.com/accounts/ClientLogin --data 'Email=$username&Passwd=$password&service=youtube&source=wdtvext' --header 'Content-Type:application/x-www-form-urlencoded' 2>&1
SID=DQAAAJsAAAAjyzkc74LJsEKbUMkxGz3t8Y6rglfqds26QYuJLfKz1k_CTvHXq3TMJAYsMC6KZ9kgsVKz73XErj6YcZswFI34bb9dZaJms0Ktqh-SrtnSHlabesTAwDCbFdDxBbYW897NYwm7e9R5tc94fbsFvJdKZaL3r6DCnbTFv2amjWOGt91Q8xX0zrFw0RxtcOyTBf25OJc0N7LTN2O6qvv5fBNa
LSID=DQAAAJ0AAADlzTtcmdZdOy6XXc_hafCt4TNvke8ShTaFGZpy5rQfg5qKTGmPMrfZIioytNofcI47pGt3Jbv88chsAKpBfuWzPsvCx-sgggbtLF4GUvSNjx2SsWLacq1XKGOWeUr_7Toru1Kdlqmfr-Y1qQDPcjYDzTlOuDpVwi3RySUWyWjF-iuP4DAG2ped0nVG0Ij3eAk8ltKqZUIGaCWNSlVbVix1
Auth=DQAAAJ0AAADlzTtcmdZdOy6XXc_hafCt4TNvke8ShTaFGZpy5rQfg5qKTGmPMrfZIioytNofcI6cQxV2HLP2WpHLEAJPRu_BuBwkXMlcrL5xH-WFvtYAnOsl8QILqjj6zcm4Cr676eBAfSub2iw1f32hekQlEtIbtHXJrs9bDvTR_nH27x9BQ-uQ9JBBLGCASjFVbwaM_DfOTbMgs8Cl8q1_nvdcBLYq


The WDTV curl version is:

Code: Select all

# curl -V
curl 7.20.0 (mipsel-unknown-linux-gnu) libcurl/7.20.1 OpenSSL/0.9.8k zlib/1.2.3.4 libidn/1.15 libssh2/1.2.5
Protocols: dict file ftp ftps http https imap imaps ldap pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz


The Ubuntu curl version is:

Code: Select all

adrianp@stingray:/tmp$ curl -V
curl 7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz


Even if the curl on the wdtv seems to be newer, there are some differences with the certificates shipped.

For now I will use --insecure, but that's just sweeping the problem under the rug.

[b-rad.cc]

figure out what certificate its using, forward it to me, and i'll put it in the right spot in the fw.

[mad_ady]

I did a bit of reading, and it seems that curl ships without a ca-cert list (http://curl.haxx.se/docs/sslcerts.html). This would mean it wouldn't be able to connect by using SSL without ignoring certificate warnings. Is this true? Did you manage to download anything through https with curl, or did I get this wrong?

I have downloaded the latest bundle of ca-certificates and tried it on with the current curl and it still didn't work (it read the certificates, but they still failed the check - maybe its format is not quite right...). It seems you can download weekly-updated ca-certs from here: http://curl.haxx.se/docs/caextract.html

Code: Select all

# strace curl -s -S --cacert /tmp/cacert.pem   --location https://www.google.com/accounts/ClientLogin --data 'Email=$username&Passwd=$password&service=youtube&source=wdtvext' --header 'Content-Type:application/x-www-form-urlencoded' 2>&1
.... output omitted ...
open("/tmp/cacert.pem", O_RDONLY|O_LARGEFILE) = 4
fstat64(0x4, 0x7f852de0)                = 0
old_mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b374000
read(4, "##\n## ca-bundle.crt -- Bundle of"..., 16384) = 16384
read(4, "ExFjAUBgNVBAMTDURTVCBS\nb290Q0EgW"..., 16384) = 16384
brk(0x478000)                           = 0x478000
read(4, "\n\nVerisign Class 1 Public Primar"..., 16384) = 16384
read(4, "BFZJU0ExLzAtBgNVBAsTJlZpc2EgSW50"..., 16384) = 16384
read(4, "gQUaJDkZ6SmU4DH\nhmak8fdLQ/uEvW0w"..., 16384) = 16384
read(4, "8wHQYDVR0OBBYEFNq7LqqwDLiIJlF0\nX"..., 16384) = 16384
brk(0x49c000)                           = 0x49c000
read(4, "DB/MQswCQYDVQQGEwJHQjEbMBkGA1UEC"..., 16384) = 16384
read(4, "l5vmwpMwci4YSM1gf/+rHhwLWjhOgeYl"..., 16384) = 16384
read(4, "U\nCbz0vGbMPVjQV0kK7iXiQe4T+Zs4NN"..., 16384) = 16384
brk(0x4c0000)                           = 0x4c0000
read(4, "WVuLiBJTVBPUlRBTlQhIFRoZSBpc3N1Y"..., 16384) = 16384
read(4, "uCp4Bx+ow0Syd3Tfom5h5VtP8c9/Qit5"..., 16384) = 16384
read(4, "oCggIBAMrfogLi2vj8Bxax3mCq3pZcZB"..., 16384) = 16384
read(4, "zerhChXDNjU1JlWbOOi/lmEtDHoM/hkl"..., 16384) = 16384
brk(0x4e4000)                           = 0x4e4000
read(4, "f131TN3ubY\n1gkIl2PlwS6wt0QmwCbAr"..., 16384) = 12836
read(4, "", 16384)                      = 0
close(4)                                = 0
... output omitted ...


I will try to troubleshoot this more and see if I can figure out why it's not working. I'll keep you posted.

[mad_ady]

Ok, I figured it out (did more man page reading).
* You would need to download the ca-cert bundle (either the official one, or one prepared by yourself)
* When running curl, you need to have an environment variable set (CURL_CA_BUNDLE) pointing to your cert file.
* MOST IMPORTANTLY (and I'm sorry I didn't remember this sooner!), you need to have the correct date and time set on your host. Certificates are valid in a specific time interval (e.g. between 2009 and 2011), and your client will refuse to accept the certificate if it's outside that time interval (e.g. living in 2000).

Default behavior with the correct time set:

Code: Select all

# ntpdate pool.ntp.org
14 Nov 11:41:12 ntpdate[6778]: step time server 91.207.120.6 offset 343042556.283647 sec
# date
Sun Nov 14 11:41:22 EET 2010
#
# curl -s -S --location https://www.google.com/accounts/ClientLogin --data 'Email=$username&Passwd=$password&service=youtube&source=wdtvext' --header 'Content-Type:application/x-www-form-urlencoded'
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.



Correct time set and a fresh cacert.pem file

Code: Select all

# wget http://curl.haxx.se/ca/cacert.pem -O /tmp/cacert.pem
--2010-11-14 11:43:35--  http://curl.haxx.se/ca/cacert.pem
Resolving curl.haxx.se... 80.67.6.50
Connecting to curl.haxx.se|80.67.6.50|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 225828 (221K) [text/plain]
Saving to: `/tmp/cacert.pem'

100%[========================================================================================================================================================================>] 225,828      556K/s   in 0.4s   

2010-11-14 11:43:35 (556 KB/s) - `/tmp/cacert.pem' saved [225828/225828]

# CURL_CA_BUNDLE=/tmp/cacert.pem curl -s -S --location https://www.google.com/accounts/ClientLogin --data 'Email=$username&Passwd=$password&service=youtube&source=wdtvext' --header 'Content-Type:application
/x-www-form-urlencoded'
SID=DQAAAJsAAAA1N41mVzzKtSoNORZLSgjazXpBeMNXDZtWrj-StV0C1owQAqlTdHNFplI-vfuY2f_MHkNEJld6gei7Xx2uHdiAUjfzP98hD3oT946UveAH_gj6_n0BmfcQ8BNNfXrIoPl_3L_BklbXuJ6spOT0o9pSkt8xO4mmf0mQ9qdCBhmcJhLkgdynTcb8E9ADAxwiBg7V2xTzM3JxiqyKN87ksnWb
LSID=DQAAAJ8AAADGUDh-gRynCNulUqfX1cs0RoWPAt8VEdpoJuJ6ecIy0yKbcj6Wt9Yaeq7q3kYwT2aT8zoCu_5TcGjMDxFdf_4jls2JYiE01CrJGJVlyvpttbnAK1dDFBmOd1ln1s9Jk8wl7mzeY5YAiT7zGgn4LNRNCs18QACgJUOakAPk9c1akj-NwAd8mMHEHLPb8tYbCyFUgpd3eOk87YA-LX7JRSBS
Auth=DQAAAJ8AAADGUDh-gRynCNulUqfX1cs0RoWPAt8VEdpoJuJ6ecIy0yKbcj6Wt9Yaeq7q3kYwT2ZHhswgyGQTJf9jM-zl3G8vrouqr2l3NAyyqygtjwF0N6rZcHlUUBaa_icwIzqF3Wb-hjM66XNfYJu8sfDZ6x6aepruOtP2AF49AByr_Id8Gh2gZBUr8var68wTFmhfzNTholsMSm_15ShsZyPfOTeC



Incorrect date set (the default date, without NTP adjustment) and a fresh cacert.pem file

Code: Select all

# date 010102002000
Sat Jan  1 02:00:00 EET 2000
# CURL_CA_BUNDLE=/tmp/cacert.pem curl -s -S --location https://www.google.com/accounts/ClientLogin --data 'Email=$username&Passwd=$password&service=youtube&source=wdtvext' --header 'Content-Type:application
/x-www-form-urlencoded'
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.



b-rad: you could include a fresh cacert.pem file in your next firmware release (burned in) and also, you can define the environment variable CURL_CA_BUNDLE to point to the correct file. I'm guessing the root Certificate Authorities won't change their keys very often, and even if they do, the user can append a script to download them and override the variable until the next firmware release. Now, the tricky part would be for the user to ensure to have a correct date set when using curl with https. I will update the error message in the youtube helper script to indicate this possible problem.

[mad_ady]

Sorry to revive this really old thread, but I think I fixed the issue (took only ~5 years)

I added the cacert.pem file and pointed curl to it (through .curlrc).

The issue should be fixed starting with firmwares larger than 0.5.2.0.

[workingman]

Hey mad_ady,

I just tried to add an https rss feed and am not getting any joy. I think it's curl related but don't know for sure yet.

[workingman]

Oh ya and running latest AFAIK...

Device: WDLXTV_LIVE
Current version: 0.5.2.2

[workingman]

Of course I should have mentioned this was via rutorrent front end - but you knew that right mad_ady

Think the problem has nothing to do with certs or https or anything tricky. Path to curl is hardcoded wrong.

# which curl
/bin/curl
# grep curl /var/www/plugins/rutorrent/conf/config.php
"curl" => '/usr/bin/curl', // Something like /usr/bin/curl. If empty, will be founded in PATH.

In the old app I built I just left it blank.

# grep curl /tmp/rt/work/app-build/src/rtorrent-0.9.2/var/www/rutorrent-3.4/conf
/config.php
"curl" => '', // Something like /usr/bin/curl. If empty, will be found in PATH.

In fact the only thing I defined was php-cgi as the php to use:

$pathToExternals = array(
"php" => '/usr/bin/php-cgi', // Something like /usr/bin/php. If empty, will be found in PATH.
"curl" => '', // Something like /usr/bin/curl. If empty, will be found in PATH.
"gzip" => '', // Something like /usr/bin/gzip. If empty, will be found in PATH.
"id" => '', // Something like /usr/bin/id. If empty, will be found in PATH.
"stat" => '', // Something like /usr/bin/stat. If empty, will be found in PATH.
);

I setup another box with rutorrent and created the rss feed then just copied over the file into /var/www/plugins/rutorrent/share/users/wdlxtv/settings/rss/cache/ and it worked fine.

[mad_ady]

Thanks for reporting the problem. I'll try to fix it but my development HDD is down

[mad_ady]

I'm having the same problem.