Plugin: YouTube Subscriptions + new subscription videos

[biomedica]

I still waiting for a solution for this question. I have tried different things but I was never able to increase the number of videos to be loaded by default. The maximum I get are 20 videos. Also, now my subscribed videos do not update unless I reboot the box each time. Why is that happening? That was not happening to me in the past.

[kharma]

Does anyone have an example of what the contents of the .cache file should look like? I'm getting a "no media in folder" error, and I'm looking to even manually create the file at first and then programatically reproduce this script. But I am not sure what the contents should even look like.

[mad_ady]

I have also enabled this plugin in UMSP. I can navigate to my account name -> New Subscription Videos but I get a "There is no media in the current folder".
I have run the subscription-helper script by hand previously:

Code: Select all

# /usr/bin/php5-cgi /tmp/umsp-plugins/youtube-subscriptions/youtube-subscriptions-helper.php
X-Powered-By: PHP/5.3.2-1
Content-type: text/html

0.1116480827331543#


The script finished in less than a second, so I suspect something went wrong inside. Haven't investigated yet, but I will try to debug it.
I have my account details typed in and I am logged in on the youtube page. I also have yt sharing enabled.

Is there a known fix for this? Or shall I start troubleshooting?

By the way, I'm running the latest firmware (1.02.21WDLXTV-LIVE-0.4.2.2)

[mad_ady]

Ok, I've done a little digging. I started to add some basic debug statements in the youtube-subscriptions-helper script to see what went wrong (username and password have been anonymized, of course ).

Here's what I got:

Code: Select all

# php5-cgi youtube-subscriptions-helper.php
X-Powered-By: PHP/5.3.2-1
Content-type: text/html

DBG: result of _getUser()
array(1) {
  [0]=>
  array(2) {
    ["youtube_username"]=>
    string(14) "my.youtube.username"
    ["youtube_password"]=>
    string(8) "12345678"
  }
}
DBG: _get_youtube_feed: about to send to 74.125.39.118:
GET /feeds/api/users/my.youtube.username/subscriptions?v=2&start-index=1&max-results=1 HTTP/1.0
Host: gdata.youtube.com
Connection: Close

DBG: _get_youtube_feed: received:

<errors xmlns='http://schemas.google.com/g/2005'>
<error>
<domain>GData</domain>
<code>InvalidRequestUriException</code>
<internalReason>Missing or invalid username.</internalReason>
</error>
</errors>

DBG: User subscriptions for my.youtube.username
array(0) {
}
0.1109580993652344



It seems that either the API has changed and it now requires other parameters, or the ip address (hardcoded in the script) is no longer a valid youtube server (and could be used for something else?)

The strange thing is the query doesn't send the password - so maybe the channels must be public. I will try to sniff some other client to see what the request should look like - but I need some advice -> where can I find documentation for this API?

If you want to run the changed (debug enabled) script, replace your /tmp/umsp-plugins/youtube-subscriptions/youtube-subscriptions-helper.php with this:
(disclamer - I am not a PHP programmer and the debugging is not complete -> I did it to discover this problem). Most likely after we manage to fix the problem I will release a patch which will be more complete. Also note, the debug will print your youtube password in clear - beware of snoops.

Code: Select all

#!/usr/bin/php5-cgi
<?php
// Youtube get subcriptions UMSP plugin by Dan
// http://forum.wdlxtv.com/viewtopic.php?f=49&t=713

//to enable debugging (goes to STDOUT) set $want_debug=1
//to disable debugging set $want_debug=0;
$want_debug=1;

define("LOG_FILE",'/tmp/umsp-log.txt');
set_time_limit(0);

$t = microtime(true);
main();
echo microtime(true) - $t;

function main()
{
   global $want_debug; //we want to access the global variable $want_debug
   $user = _getUser();
   if($want_debug){
   echo "DBG: result of _getUser()\n";
   var_dump($user);
   }
   $user_data = array();

   foreach ($user as $u) {
      $us = _getUserSubscriptions($u['youtube_username'], $u['youtube_password']);
      if($want_debug){
   echo "DBG: User subscriptions for ".$u['youtube_username']."\n";
   var_dump($us);
      }
      $t = _getUserVideos($us,20);

      $user_data[ $u['youtube_username'] ]['new_subscription_videos'] = $t['videos'];
      $user_data[ $u['youtube_username'] ]['subscriptions_videos'] = $t['user_videos'];
     
      //remove subscriptions without any content
      $user_with_videos = array_keys($t['user_videos']);
      foreach($us as $k => $v) {
         if (!in_array($v, $user_with_videos))
            unset($us[ $k ]);
      }
      $user_data[ $u['youtube_username'] ]['subscriptions'] = $us;
   }
   
   file_put_contents('/tmp/youtube-subscriptions.cache',serialize($user_data));
}

function _getUserVideos(array $users, $count)
{
   $videos = array();
   $user_videos = array();
   foreach($users as $user)
   {
      $t = _get_youtube_feed("74.125.39.118","gdata.youtube.com","/feeds/api/users/{$user}/uploads?orderby=published&max-results={$count}");
      $x = simplexml_load_string($t);
      if(!$x) {
         continue;
      }

      $x->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
      $entries = $x->xpath("//a:entry");

      foreach($entries as $t) {
         $ts = strtotime(substr($t->published, 0, 10).' '.substr($t->published, 11, 8));
         $vid = mb_substr($t->id,mb_strrpos($t->id,'/')+1,mb_strlen($t->id));
         $d = array(
            'user'=>$user,
            'title'=>(string)$t->title,
            'id'=>$vid,
             #'video_link'=>"http://www.youtube.com/watch?v={$vid}&t=&fmt=22",
            );
         $user_videos[ $user ][] = $d;
         $videos[ $ts ] = $d;
      }
   }
   
   foreach($user_videos as $k=>$v) {
      if (count($v) == 0) unset($user_videos[ $k ]);
   }

   krsort($videos);
   $videos = array_slice($videos,0,150,true);
   
   return array('videos'=>$videos, 'user_videos'=>$user_videos);
}


function _getUserSubscriptions($YouTubeUserName, $YouTubePassword)
{
   preg_match("/openSearch:totalResults>(\d+)</s",_get_youtube_feed("74.125.39.118","gdata.youtube.com","/feeds/api/users/{$YouTubeUserName}/subscriptions?v=2&start-index=1&max-results=1"),$m);

   $total_subscriptions = $m[1];
   $subscriptions = array();
   $pr = 20;

   //get channels
   for($i=0;$i<(int)ceil($total_subscriptions/$pr);$i++) {

      $start = ($i*$pr) + 1;
      $t = _get_youtube_feed("74.125.39.118","gdata.youtube.com","/feeds/api/users/{$YouTubeUserName}/subscriptions?v=2&start-index={$start}&max-results={$pr}");
      $x = simplexml_load_string($t);
      if(!$x) continue;

      $x->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
      $t = $x->xpath("//a:entry//a:title");
      foreach($t as $v) {
         preg_match("/:(.+)$/",$v,$s);
         $s = (string)trim($s[1]);
         if (!empty($s)) $subscriptions[] = $s;
         #break(2);
      }
   }
   natcasesort($subscriptions);
   return $subscriptions;   
}



function _getUser()
{
   $xml = simplexml_load_file('/conf/account_list.xml');
   $data = $xml->xpath('//service[@name="YOUTUBE"]/account');
   
   $accounts = array();
   
   foreach($data as $account)
   {
      $accounts[] = array('youtube_username' => (string)$account->username,
                     'youtube_password' => (string)$account->password);     
   }
   return $accounts;
}

/**
* Debug Logs
* stops if last param is 1
*/
function l()
{
   $t = debug_backtrace();
   $args = func_get_args();
   ob_start();
   echo basename($t[0]["file"]).":{$t[0]["line"]} > ";
   var_dump($args);
   $data = ob_get_contents();
   ob_end_clean();
   file_put_contents(LOG_FILE,$data,FILE_APPEND);
   if(end($args) === 1) die;
}

function _get_youtube_feed($ip,$host,$path)
{
   global $want_debug; //we want to access the global variable $want_debug
   $fp = fsockopen($ip, 80, $errno, $errstr, 30);
   if (!$fp) {
       echo "$errstr ($errno)<br />\n";
   } else {
       $out = "GET {$path} HTTP/1.0\r\n";
       $out .= "Host: {$host}\r\n";
       $out .= "Connection: Close\r\n\r\n";
       if($want_debug){
           echo "DBG: _get_youtube_feed: about to send to $ip:\n$out";
       }
       fwrite($fp, $out);
     
      $content = '';
      $headerPassed = false;
       while (!feof($fp)) {
           $l = fgets($fp);
         if($l == "\r\n") $headerPassed = true;
         if($headerPassed) $content .= $l;
   
       }
       fclose($fp);
   
       if($want_debug){
           echo "DBG: _get_youtube_feed: received:\n$content\n";
       }
      preg_match("/(<\?.*>)/s",$content,$m);
      return trim($m[1]);
   }
}

?>



[mad_ady]

I haven't had time to look into it any more, but I just wanted to know - can anyone get their subscriptions by using this plugin (meaning is it still working?), or is it me who has a problem?

This will help me to either check out the API or look into my YT account settings

[kharma]

Yeah, I don't know. My Boxee Box just shipped so I doubt I'll get anywhere on this unless I don't like the boxee.

Though I see it ALSO doesn't do subscription browsing...it must be ridiculously convoluted (even then the youtube app on my Epic and iPad seem to do it just fine) to do because I can't think of any other reason why something so simple would be missing in every single media streaming box on the market.

Plus, aside from line by line deconstructing the OP's (apparently abandoned) script to figure out how exactly to build the cache file so it can be read in a way that actually displays items in the menu (since we have no working version of the script and no example of a working .cache file) I don't see a simple way to just, recreate this plugin.

Which begs the question, why is a KNOWN bad plugin still being distributed via the SVN? Before my first post in this thread I deleted the script and used the webend to re-download all the plugins and this was part of what it downloaded.

Why keep including software that has been known broken since day 2 of the latest firmware release?

[biomedica]

Hi everybody,

I just want to let you know that my YouTube subscriptions plugin stills working and I am on the most updated software. The only difference is that in the past the videos would update automatically everytime I would open the plugin and now, to have them updated I need to reboot the box.

[mad_ady]

Thanks kharma, biomedia for your updates.

Since it works for somebody, I would guess the plugin works as designed, but maybe we're doing something wrong. From the debugging I did, it seems that youtube doesn't reply with the correct answers - which would probably mean we have a problem with youtube authentication (the script piggybacks on the WDTV's Youtube authentication - which might no longer work).
I will look again into the script when I have a bit of time on my hands and I'll see if I can add propper youtube authentication to it (without the need for shared sessions). But for this I would need to get dirty and look into the YT API.

biomedica - since it works for you, could you please post the firmware version you're using?
It would also be helpful (but rather difficult to obtain) a packet capture of the traffic your youtube-subscription-helper.php script does when downloading the subscription information. I am curious what it sends to youtube (unfortunately such a packet capture would expose your YT username and possibly password, and also we would see what channels you are viewing -- but you can temporarily change your YT password before doing the capture). If you are willing to do such a packet capture (you would need a usb stick or network share to store it) I can give you the details you need.

Or, instead of a packet capture you can run the script above (a few posts up) and paste back the output (with your username/password anonymized).

Also, updating the feeds more often can be easily fixed - either through cron, or running the script manually through the UI. We can discuss this also.

[kharma]

More interesting to me than a packet capture is actually a copy of the youtube-subscriptions.cache file.

And if I had to guess I'd say a) your cron entry got removed when you upgraded firmware b) the reason it works for you and not us is you said "still works", I never used it before the latest update, so maybe whatever needed to happen to set it up happened back when it worked, and now it doesn't.

Either way being able to see (without deconstructing, which is even harder when we just have to guess what the data from youtube would be) what it is SUPPOSED to create would be beneficial.

I'd still like the author or whomever maintains the SVN to explain why it's being pushed still. The author, if you read back, even notes there is a compatibility issue (which he didn't want to fix unless paid to do so) so there is definitely something wrong with the script.

[mad_ady]

Yes Kharma, your point is valid. It may have worked with a previous firmware and by upgrading the functionality was preserved. I've just tried it with one firmware version.
However - knowing the cache file format is only half of the problem - you would know how to read it (so you would be able to keep the UI scripts/proxy), but you'd still have to generate it somehow - meaning you would need access through the google API to your subscriptions.
I guess if somebody would attach a valid cache file we would at least know that half of the script is working fine...