Sign up here and you can log into the forum!

Plugin: YouTube Subscriptions + new subscription videos

Discussion, features, plugins--everything about zoster's UMSP UPnP Media Server software

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby kharma » Thu Nov 11, 2010 1:32 pm

Well my point with seeing the cache file is that at this point, theres little or nothing to work with in the original script, and I already traced back most of the calls it makes to youtube, so figuring out WHAT it's pulling down isn't hard, at this point, if I knew the format it is supposed to be in inside the .cache file it is only a matter of taking the feeds (which I know) and parsing them into a format (which I don't).

Redoing the whole script would be easier than fixing the original.

Especially since nobody either a) is willing to tell us what the format of a UMSP plugin "cache" is supposed to look like (i.e. what is it WDLX is looking for to populate a video list) and b) the author has abandoned the script.

At this point we know a source, we just need to know what kind of container to put it in to make it work Which is where a working .cache file comes in.

/edit

If anyone who has this working wants to provide a copy of their .cache file it is /tmp/youtube-subscriptions.cache

And I don't THINK (but again I don't know, this is just based on reading what the script says) it contains anything personal/account related other than a list of videos. So as long as no donkeys or asians are harmed in the course of your youtube viewing you are probably pretty safe posting it. ;)
Last edited by kharma on Thu Nov 11, 2010 1:36 pm, edited 1 time in total.
kharma
n00b
 
Posts: 5
Joined: Tue Nov 02, 2010 10:03 pm

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby kharma » Thu Nov 11, 2010 1:34 pm

That being said btw, I SINCERELY hope the boxee (which is out for delivery! woot!) isn't locked down by default and is hackable. I do like to tinker with my toys. :)
kharma
n00b
 
Posts: 5
Joined: Tue Nov 02, 2010 10:03 pm

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby b-rad.cc » Thu Nov 11, 2010 5:45 pm

None of the plugins I've looked at & modified had a cache file.
PM's are for private matters only, please post public matters on the forum to help others who might have the same issue.
:mrgreen:
User avatar
b-rad.cc
WDLXTV Team
 
Posts: 3003
Joined: Sat Apr 03, 2010 9:35 am
Location: New York

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby mad_ady » Sat Nov 13, 2010 11:11 am

Good news, folks!

I managed to get the helper script to work again after rewriting the integration with the Google API (it seems it has been updated in the mean time). Now, the helper script does proper authentication and retrieves your subscriptions. It works, it generates a valid cache file and the clips are totally watchable through the UMSP server.
But there are a few snags - it's not working properly yet - It can run just fine from my linux box (Ubuntu 10.04), but when I run it from the wdtv, I get two errors:
  • Certificate warning when connecting to google for authentication (can be bypassed)
  • The script gets your channels, but can't find any data inside them (e.g. - no videos posted)

When I run it from a computer in the same network, everything works... I will have to debug it on the wdtv and post back a fully working solution.
For now - for the impatient ones who want to try it out, do this:
  • Save the code below as youtube-subscription-helper.php in a folder on your computer
  • Make sure you have php5-cgi available[/i]
  • Copy your /conf/account_list.xml from your wdtv in the same folder as the script
  • Run the script (php5-cgi youtube-subscription-helper.php) by hand
  • You should get some output, and you should have a /tmp/youtube-subscriptions.cache file
  • scp /tmp/youtube-subscriptions.cache root@your_wdtv:/tmp
  • Navigate through UMSP to see your feeds

As I said these instructions are temporary, until I find a final fix (hope it won't take more than a few hours/days). All in all, after starting looking through the youtube API, it took me about 2-3 hours to rewrite the relevant bits of code and make it sort of work.

Code: Select all
#!/usr/bin/php5-cgi
<?php
// Youtube get subcriptions UMSP plugin by Dan
// http://forum.wdlxtv.com/viewtopic.php?f=49&t=713

// added proper youtube authentication and updated the API my mad_ady

/////////////////
// VERSION 0.2 //
/////////////////

//to enable debugging (goes to STDOUT) set $want_debug=1
//to disable debugging set $want_debug=0;
$want_debug=1;
$account_list_file='account_list.xml';
//$youtubeIP='74.125.39.118';
$youtubeIP='gdata.youtube.com';
$developerKey='AI39si77LEtwwQMWZlnJ-AeV_jQ24KbfVjRvgv7fpQMhsrd6XAkrop2FmjHzxbBozUaAD30aknjD1iwgnjogz-5S2lhGj_Ga-w'; //youtube developer key for mad_ady. Don't use it for other projects, please

define("LOG_FILE",'/tmp/umsp-log.txt');
set_time_limit(0);

$t = microtime(true);
main();
echo "Total execution time: ".(microtime(true) - $t);

function main()
{
   global $want_debug; //we want to access the global variable $want_debug
   $user = _getUser();
   if($want_debug){
   echo "DBG: result of _getUser()\n";
   var_dump($user);
   }
   $user_data = array();

   foreach ($user as $u) {
      //authenticate user
      echo "Authenticating ".$u['youtube_username']."\n";
      flush();
      $user_data[ $u['youtube_username'] ]['auth_token'] = _getAuthToken($u['youtube_username'], $u['youtube_password']);
      if($want_debug){
      echo "DBG: User data for ".$u['youtube_username']."\n";
      var_dump($user_data);
      }
      echo "Getting subscriptions for ".$u['youtube_username']."\n";
      flush();
      $us = _getUserSubscriptions($user_data[ $u['youtube_username'] ]['auth_token'], $u['youtube_username']);
      if($want_debug){
      echo "DBG: User subscriptions for ".$u['youtube_username']."\n";
      var_dump($us);
      }
     
      echo "Downloading first 20 channels from ". (implode(", ", $us))."\n";
      flush();
      $t = _getUserVideos($us,20, $user_data[ $u['youtube_username'] ]['auth_token']);

      $user_data[ $u['youtube_username'] ]['new_subscription_videos'] = $t['videos'];
      $user_data[ $u['youtube_username'] ]['subscriptions_videos'] = $t['user_videos'];
     
      //remove subscriptions without any content
      $user_with_videos = array_keys($t['user_videos']);
      foreach($us as $k => $v) {
         if (!in_array($v, $user_with_videos))
            unset($us[ $k ]);
      }
      $user_data[ $u['youtube_username'] ]['subscriptions'] = $us;
   }
   
   echo "Writing /tmp/youtube-subscriptions.cache\n";
   file_put_contents('/tmp/youtube-subscriptions.cache',serialize($user_data));
}

function _getUserVideos(array $users, $count, $authToken)
{
   //TODO: Change prototype
   global $youtubeIP;
   $videos = array();
   $user_videos = array();
   foreach($users as $user)
   {
      $t = _get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/{$user}/uploads?orderby=published&max-results={$count}", $authToken);
      $x = simplexml_load_string($t);
      if(!$x) {
         continue;
      }

      $x->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
      $entries = $x->xpath("//a:entry");

      foreach($entries as $t) {
         $ts = strtotime(substr($t->published, 0, 10).' '.substr($t->published, 11, 8));
         $vid = mb_substr($t->id,mb_strrpos($t->id,'/')+1,mb_strlen($t->id));
         $d = array(
            'user'=>$user,
            'title'=>(string)$t->title,
            'id'=>$vid,
             #'video_link'=>"http://www.youtube.com/watch?v={$vid}&t=&fmt=22",
            );
         $user_videos[ $user ][] = $d;
         $videos[ $ts ] = $d;
      }
   }
   
   foreach($user_videos as $k=>$v) {
      if (count($v) == 0) unset($user_videos[ $k ]);
   }

   krsort($videos);
   $videos = array_slice($videos,0,150,true);
   
   return array('videos'=>$videos, 'user_videos'=>$user_videos);
}


function _getUserSubscriptions($authToken, $YouTubeUserName)
{
   global $youtubeIP;
   preg_match("/openSearch:totalResults>(\d+)</s",_get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/default/subscriptions?v=2&start-index=1&max-results=1", $authToken),$m);

   $total_subscriptions = $m[1];
   $subscriptions = array();
   $pr = 20;

   //get channels
   for($i=0;$i<(int)ceil($total_subscriptions/$pr);$i++) {

      $start = ($i*$pr) + 1;
      $t = _get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/default/subscriptions?v=2&start-index={$start}&max-results={$pr}", $authToken);
      $x = simplexml_load_string($t);
      if(!$x) continue;

      $x->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
      $t = $x->xpath("//a:entry//a:title");
      foreach($t as $v) {
         preg_match("/:(.+)$/",$v,$s);
         $s = (string)trim($s[1]);
         if (!empty($s)) $subscriptions[] = $s;
         #break(2);
      }
   }
   natcasesort($subscriptions);
   return $subscriptions;   
}



function _getUser()
{
   global $account_list_file;
   $xml = simplexml_load_file($account_list_file);
   $data = $xml->xpath('//service[@name="YOUTUBE"]/account');
   
   $accounts = array();
   
   foreach($data as $account)
   {
      $accounts[] = array('youtube_username' => (string)$account->username,
                     'youtube_password' => (string)$account->password);     
   }
   return $accounts;
}

/**
* Debug Logs
* stops if last param is 1
*/
function l()
{
   $t = debug_backtrace();
   $args = func_get_args();
   ob_start();
   echo basename($t[0]["file"]).":{$t[0]["line"]} > ";
   var_dump($args);
   $data = ob_get_contents();
   ob_end_clean();
   file_put_contents(LOG_FILE,$data,FILE_APPEND);
   if(end($args) === 1) die;
}

function _get_youtube_feed($ip,$host,$path,$authToken)
{
   global $want_debug; //we want to access the global variable $want_debug
   global $developerKey;
   $fp = fsockopen($ip, 80, $errno, $errstr, 30);
   if (!$fp) {
       echo "$errstr ($errno)<br />\n";
   } else {
       $out = "GET {$path} HTTP/1.0\r\n";
       $out .= "Host: {$host}\r\n";
       $out .= "Authorization: GoogleLogin auth=$authToken\r\n";
       $out .= "X-GData-Key: key=$developerKey\r\n";
       $out .= "Connection: Close\r\n\r\n";
       if($want_debug){
           echo "DBG: _get_youtube_feed: about to send to $ip:\n$out";
       }
       fwrite($fp, $out);
     
      $content = '';
      $headerPassed = false;
       while (!feof($fp)) {
           $l = fgets($fp);
         if($l == "\r\n") $headerPassed = true;
         if($headerPassed) $content .= $l;
   
       }
       fclose($fp);
   
       if($want_debug){
           echo "DBG: _get_youtube_feed: received:\n$content\n";
       }
      preg_match("/(<\?.*>)/s",$content,$m);
      return trim($m[1]);
   }
}

function _getAuthToken ($username, $password){
   global $want_debug;
   global $youtubeIP;
   //url_encode the user/password
   $ytUser = urlencode($username);
   $ytPass = urlencode($password);
   //use curl since it uses https   
   $out = array();
   $command = "curl -s -S --location https://www.google.com/accounts/ClientLogin --data 'Email=$ytUser&Passwd=$ytPass&service=youtube&source=wdtvext' --header 'Content-Type:application/x-www-form-urlencoded' 2>&1";
   if($want_debug){
      echo "DBG: Authenticating with command: $command";
   }
   exec($command, $out);
   if($want_debug){
      print_r($out);
   }
   foreach ($out as $line){
      //we're looking for this line:
      // Auth=DQAAAJ8AAADBW15VA3cV9OmZeBF0maTTyCfHxdtEj62tyHCO696_CFYe6mUpwrF1DXHnpepVXKMFj3vgtoVHLXYv0qX9Hk1fawpea5XuZbEZZBNdfaW9-hNWbpDmE-_Rl_0g3RPoBNZLkIxJaeDz4d-M2WLmyiTcmVlcvbDhI1xc7mxfkDPLKdBLdJkC1fAFUbZHqq5mKfbWB0se4hLObnxWBEj-Rzr-
      preg_match("/^Auth=(.*)$/", $line, $result);
      if(isset ($result[1])){
         //there's a match, this is the line we were after
         return $result[1];
      }
   }
   
   //if we got here, there was an error during the authentication process
   die ("Unable to authenticate $username with Youtube. Please check username and password or run this script in debug mode");
}
?>

User avatar
mad_ady
Developer
 
Posts: 4568
Joined: Fri Nov 05, 2010 9:08 am
Location: Bucharest, Romania

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby b-rad.cc » Sat Nov 13, 2010 11:20 am

when you want svn rights to commit this (or anything else) lemme know. ;)
PM's are for private matters only, please post public matters on the forum to help others who might have the same issue.
:mrgreen:
User avatar
b-rad.cc
WDLXTV Team
 
Posts: 3003
Joined: Sat Apr 03, 2010 9:35 am
Location: New York

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby mad_ady » Sat Nov 13, 2010 11:27 am

Update - managed to get it working also on the wdtv.
There's something strange that I don't get - If I ask gdata.google.com for videos uploaded by some youtube user, it work from my PC. If I ask from the wdtv, it replies that that user doesn't exist. If I query the IP 74.125.39.118, then I get the correct answers from youtube from the wdtv. I'd like to know what's so special about that IP? Where did the original developer get it from?

Oops - should have looked:
Code: Select all
# cat /etc/hosts | grep gdata.youtube.com
127.0.0.1 gdata.youtube.com


May I ask why (I'm suspecting the wdtv acts like a proxy to youtube - but it doesn't seem to be complete)? Relying on a static IP address is a recipe for disaster!

Anyway - here's the updated code. Copy it to your wdtv (in /conf for instance) and run it by hand or through cron (as specified in the original thread).
Let me know if you run into problems. Also, success stories are welcome (to know that it's working for others as well).

Code: Select all
#!/usr/bin/php5-cgi
<?php
// Youtube get subcriptions UMSP plugin by Dan
// http://forum.wdlxtv.com/viewtopic.php?f=49&t=713

// added proper youtube authentication and updated the API my mad_ady

/////////////////
// VERSION 0.3 //
/////////////////

//to enable debugging (goes to STDOUT) set $want_debug=1
//to disable debugging set $want_debug=0;
$want_debug=0;
$account_list_file='/conf/account_list.xml';
$youtubeIP='74.125.39.118';
//$youtubeIP='gdata.youtube.com';
$developerKey='AI39si77LEtwwQMWZlnJ-AeV_jQ24KbfVjRvgv7fpQMhsrd6XAkrop2FmjHzxbBozUaAD30aknjD1iwgnjogz-5S2lhGj_Ga-w'; //youtube developer key for mad_ady. Don't use it for other projects, please

define("LOG_FILE",'/tmp/umsp-log.txt');
set_time_limit(0);

$t = microtime(true);
main();
echo "Total execution time: ".(microtime(true) - $t);

function main()
{
   global $want_debug; //we want to access the global variable $want_debug
   $user = _getUser();
   if($want_debug){
   echo "DBG: result of _getUser()\n";
   var_dump($user);
   }
   $user_data = array();

   foreach ($user as $u) {
      //authenticate user
      echo "Authenticating ".$u['youtube_username']."\n";
      flush();
      $user_data[ $u['youtube_username'] ]['auth_token'] = _getAuthToken($u['youtube_username'], $u['youtube_password']);
      if($want_debug){
      echo "DBG: User data for ".$u['youtube_username']."\n";
      var_dump($user_data);
      }
      echo "Getting subscriptions for ".$u['youtube_username']."\n";
      flush();
      $us = _getUserSubscriptions($user_data[ $u['youtube_username'] ]['auth_token'], $u['youtube_username']);
      if($want_debug){
      echo "DBG: User subscriptions for ".$u['youtube_username']."\n";
      var_dump($us);
      }
     
      echo "Downloading first 20 channels from ". (implode(", ", $us))."\n";
      flush();
      $t = _getUserVideos($us,20, $user_data[ $u['youtube_username'] ]['auth_token']);

      $user_data[ $u['youtube_username'] ]['new_subscription_videos'] = $t['videos'];
      $user_data[ $u['youtube_username'] ]['subscriptions_videos'] = $t['user_videos'];
     
      //remove subscriptions without any content
      $user_with_videos = array_keys($t['user_videos']);
      foreach($us as $k => $v) {
         if (!in_array($v, $user_with_videos))
            unset($us[ $k ]);
      }
      $user_data[ $u['youtube_username'] ]['subscriptions'] = $us;
   }
   
   echo "Writing /tmp/youtube-subscriptions.cache\n";
   file_put_contents('/tmp/youtube-subscriptions.cache',serialize($user_data));
}

function _getUserVideos(array $users, $count, $authToken)
{
   //TODO: Change prototype
   global $youtubeIP;
   $videos = array();
   $user_videos = array();
   foreach($users as $user)
   {
      $t = _get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/{$user}/uploads?orderby=published&max-results={$count}", $authToken);
      $x = simplexml_load_string($t);
      if(!$x) {
         continue;
      }

      $x->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
      $entries = $x->xpath("//a:entry");

      foreach($entries as $t) {
         $ts = strtotime(substr($t->published, 0, 10).' '.substr($t->published, 11, 8));
         $vid = mb_substr($t->id,mb_strrpos($t->id,'/')+1,mb_strlen($t->id));
         $d = array(
            'user'=>$user,
            'title'=>(string)$t->title,
            'id'=>$vid,
             #'video_link'=>"http://www.youtube.com/watch?v={$vid}&t=&fmt=22",
            );
         $user_videos[ $user ][] = $d;
         $videos[ $ts ] = $d;
      }
   }
   
   foreach($user_videos as $k=>$v) {
      if (count($v) == 0) unset($user_videos[ $k ]);
   }

   krsort($videos);
   $videos = array_slice($videos,0,150,true);
   
   return array('videos'=>$videos, 'user_videos'=>$user_videos);
}


function _getUserSubscriptions($authToken, $YouTubeUserName)
{
   global $youtubeIP;
   preg_match("/openSearch:totalResults>(\d+)</s",_get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/default/subscriptions?v=2&start-index=1&max-results=1", $authToken),$m);

   $total_subscriptions = $m[1];
   $subscriptions = array();
   $pr = 20;

   //get channels
   for($i=0;$i<(int)ceil($total_subscriptions/$pr);$i++) {

      $start = ($i*$pr) + 1;
      $t = _get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/default/subscriptions?v=2&start-index={$start}&max-results={$pr}", $authToken);
      $x = simplexml_load_string($t);
      if(!$x) continue;

      $x->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
      $t = $x->xpath("//a:entry//a:title");
      foreach($t as $v) {
         preg_match("/:(.+)$/",$v,$s);
         $s = (string)trim($s[1]);
         if (!empty($s)) $subscriptions[] = $s;
         #break(2);
      }
   }
   natcasesort($subscriptions);
   return $subscriptions;   
}



function _getUser()
{
   global $account_list_file;
   $xml = simplexml_load_file($account_list_file);
   $data = $xml->xpath('//service[@name="YOUTUBE"]/account');
   
   $accounts = array();
   
   foreach($data as $account)
   {
      $accounts[] = array('youtube_username' => (string)$account->username,
                     'youtube_password' => (string)$account->password);     
   }
   return $accounts;
}

/**
* Debug Logs
* stops if last param is 1
*/
function l()
{
   $t = debug_backtrace();
   $args = func_get_args();
   ob_start();
   echo basename($t[0]["file"]).":{$t[0]["line"]} > ";
   var_dump($args);
   $data = ob_get_contents();
   ob_end_clean();
   file_put_contents(LOG_FILE,$data,FILE_APPEND);
   if(end($args) === 1) die;
}

function _get_youtube_feed($ip,$host,$path,$authToken)
{
   global $want_debug; //we want to access the global variable $want_debug
   global $developerKey;
   $fp = fsockopen($ip, 80, $errno, $errstr, 30);
   if (!$fp) {
       echo "$errstr ($errno)<br />\n";
   } else {
       $out = "GET {$path} HTTP/1.0\r\n";
       $out .= "Host: {$host}\r\n";
       $out .= "Authorization: GoogleLogin auth=$authToken\r\n";
       $out .= "X-GData-Key: key=$developerKey\r\n";
       $out .= "Connection: Close\r\n\r\n";
       if($want_debug){
           echo "DBG: _get_youtube_feed: about to send to $ip:\n$out";
       }
       fwrite($fp, $out);
     
      $content = '';
      $headerPassed = false;
       while (!feof($fp)) {
           $l = fgets($fp);
         if($l == "\r\n") $headerPassed = true;
         if($headerPassed) $content .= $l;
   
       }
       fclose($fp);
   
       if($want_debug){
           echo "DBG: _get_youtube_feed: received:\n$content\n";
       }
      preg_match("/(<\?.*>)/s",$content,$m);
      return trim($m[1]);
   }
}

function _getAuthToken ($username, $password){
   global $want_debug;
   global $youtubeIP;
   //url_encode the user/password
   $ytUser = urlencode($username);
   $ytPass = urlencode($password);
   //use curl since it uses https   
   $out = array();
   $command = "/usr/bin/curl -s -S --insecure --location https://www.google.com/accounts/ClientLogin --data 'Email=$ytUser&Passwd=$ytPass&service=youtube&source=wdtvext' --header 'Content-Type:application/x-www-form-urlencoded' 2>&1";
   if($want_debug){
      echo "DBG: Authenticating with command: $command";
   }
   exec($command, $out);
   if($want_debug){
      print_r($out);
   }
   foreach ($out as $line){
      //we're looking for this line:
      // Auth=DQAAAJ8AAADBW15VA3cV9OmZeBF0maTTyCfHxdtEj62tyHCO696_CFYe6mUpwrF1DXHnpepVXKMFj3vgtoVHLXYv0qX9Hk1fawpea5XuZbEZZBNdfaW9-hNWbpDmE-_Rl_0g3RPoBNZLkIxJaeDz4d-M2WLmyiTcmVlcvbDhI1xc7mxfkDPLKdBLdJkC1fAFUbZHqq5mKfbWB0se4hLObnxWBEj-Rzr-
      preg_match("/^Auth=(.*)$/", $line, $result);
      if(isset ($result[1])){
         //there's a match, this is the line we were after
         return $result[1];
      }
   }
   
   //if we got here, there was an error during the authentication process
   die ("Unable to authenticate $username with Youtube. Please check username and password or run this script in debug mode");
}
?>



Running output:
Code: Select all
# php5-cgi youtube-subscriptions-helper.php
X-Powered-By: PHP/5.3.2-1
Content-type: text/html

Authenticating adrian.popa.gh
Getting subscriptions for adrian.popa.gh
Downloading first 20 channels from collegehumor, DiscoveryNetworks, froyoca, FunnyorDie, geekbeattv, Hak5Darren, HBO, HDstarcraft, HuskyStarcraft, PlanetGreenTV, revision3, ScienceChannel, sixtysymbols, TEDtalksDirector, TEKHD, TheOnion, thisweekinlinux, TLC, twit, ubunite
Writing /tmp/youtube-subscriptions.cache
Total execution time: 13.59104299545288


If it works ok for everyone, I'll try to convince the svn maintainers to include it there.
Enjoy!
User avatar
mad_ady
Developer
 
Posts: 4568
Joined: Fri Nov 05, 2010 9:08 am
Location: Bucharest, Romania

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby mad_ady » Mon Nov 15, 2010 8:38 am

I've made some changes to the youtube subscriptions helper script in order to make it easier to understand and adapt in the future. I'd like to hear some feedback from others - if it works, if it doesn't, if multiple accounts are supported correctly before I push this change on the svn. So, please, try it out and let me know how it works.

Code: Select all
#!/usr/bin/php5-cgi
<?php
// Youtube get subcriptions UMSP plugin by Dan
// http://forum.wdlxtv.com/viewtopic.php?f=49&t=713

// added proper youtube authentication and updated the API by mad_ady

/////////////////
// VERSION 0.4 //
/////////////////

//to enable debugging (goes to STDOUT) set $want_debug=1
//to disable debugging set $want_debug=0;
$want_debug=0;
$account_list_file='/conf/account_list.xml';
$youtubeIP='74.125.39.118';
$videoCountPerChannel=20; //change this if you want to see more than x new clips per channel
$maxChannels=20; //change this if you want more channels to be scanned
$maxTotalVideos=150; //change this if you want to keep more than x results for each account
//$youtubeIP='gdata.youtube.com';
$developerKey='AI39si77LEtwwQMWZlnJ-AeV_jQ24KbfVjRvgv7fpQMhsrd6XAkrop2FmjHzxbBozUaAD30aknjD1iwgnjogz-5S2lhGj_Ga-w'; //youtube developer key for mad_ady. Don't use it for other projects, please

$insecure='';
$options  = getopt('i', array('insecure'));
if(isset($options['i']) || isset($options['insecure'])){
   $insecure = '--insecure'; //pass to curl
}


define("LOG_FILE",'/tmp/umsp-log.txt');
set_time_limit(0);

$t = microtime(true);
main();
echo "Total execution time: ".(microtime(true) - $t);

function main()
{
   global $want_debug; //we want to access the global variable $want_debug
   global $videoCountPerChannel;
   $user = _getUser();
   if($want_debug){
      echo "DBG: result of _getUser()\n";
      var_dump($user);
   }
   $user_data = array();

   foreach ($user as $u) {
      //authenticate user
      echo "Authenticating ".$u['youtube_username']."\n";
      flush();
      $user_data[ $u['youtube_username'] ]['auth_token'] = _getAuthToken($u['youtube_username'], $u['youtube_password']);
      if($want_debug){
      echo "DBG: User data for ".$u['youtube_username']."\n";
      var_dump($user_data);
      }
      echo "Getting subscriptions for ".$u['youtube_username']."\n";
      flush();
      $us = _getUserSubscriptions($user_data[ $u['youtube_username'] ]['auth_token'], $u['youtube_username']);
      if($want_debug){
      echo "DBG: User subscriptions for ".$u['youtube_username']."\n";
      var_dump($us);
      }
     
      echo "Downloading information for first $videoCountPerChannel clips from each of these channels: ". (implode(", ", $us))."\n";
      flush();
      $t = _getUserVideos($us,$videoCountPerChannel, $user_data[ $u['youtube_username'] ]['auth_token']);

      $user_data[ $u['youtube_username'] ]['new_subscription_videos'] = $t['videos'];
      $user_data[ $u['youtube_username'] ]['subscriptions_videos'] = $t['user_videos'];
     
      //remove subscriptions without any content
      $user_with_videos = array_keys($t['user_videos']);
      foreach($us as $k => $v) {
         if (!in_array($v, $user_with_videos))
            unset($us[ $k ]);
      }
      $user_data[ $u['youtube_username'] ]['subscriptions'] = $us;
   }
   
   echo "Writing /tmp/youtube-subscriptions.cache\n";
   file_put_contents('/tmp/youtube-subscriptions.cache',serialize($user_data));
}

function _getUserVideos(array $users, $count, $authToken)
{
   global $youtubeIP;
   $videos = array();
   $user_videos = array();
   foreach($users as $user)
   {
      $t = _get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/{$user}/uploads?orderby=published&max-results={$count}", $authToken);
      $x = simplexml_load_string($t);
      if(!$x) {
         continue;
      }

      $x->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
      $entries = $x->xpath("//a:entry");

      foreach($entries as $t) {
         $ts = strtotime(substr($t->published, 0, 10).' '.substr($t->published, 11, 8));
         $vid = mb_substr($t->id,mb_strrpos($t->id,'/')+1,mb_strlen($t->id));
         $d = array(
            'user'=>$user,
            'title'=>(string)$t->title,
            'id'=>$vid,
             #'video_link'=>"http://www.youtube.com/watch?v={$vid}&t=&fmt=22",
            );
         $user_videos[ $user ][] = $d;
         $videos[ $ts ] = $d;
      }
   }
   
   foreach($user_videos as $k=>$v) {
      if (count($v) == 0) unset($user_videos[ $k ]);
      if (isset($user_videos[ $k ])) echo "$k: ".(count ($v))." videos\n";
   }

   krsort($videos);
   $videos = array_slice($videos,0,$maxTotalVideos,true);
   
   return array('videos'=>$videos, 'user_videos'=>$user_videos);
}


function _getUserSubscriptions($authToken, $YouTubeUserName)
{
   global $youtubeIP;
   global $maxChannels;
   preg_match("/openSearch:totalResults>(\d+)</s",_get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/default/subscriptions?v=2&start-index=1&max-results=1", $authToken),$m);

   $total_subscriptions = $m[1];
   $subscriptions = array();
   $pr = $maxChannels;

   //get channels
   for($i=0;$i<(int)ceil($total_subscriptions/$pr);$i++) {

      $start = ($i*$pr) + 1;
      $t = _get_youtube_feed($youtubeIP,"gdata.youtube.com","/feeds/api/users/default/subscriptions?v=2&start-index={$start}&max-results={$pr}", $authToken);
      $x = simplexml_load_string($t);
      if(!$x) continue;

      $x->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
      $t = $x->xpath("//a:entry//a:title");
      foreach($t as $v) {
         preg_match("/:(.+)$/",$v,$s);
         $s = (string)trim($s[1]);
         if (!empty($s)) $subscriptions[] = $s;
         #break(2);
      }
   }
   natcasesort($subscriptions);
   return $subscriptions;   
}



function _getUser()
{
   global $account_list_file;
   $xml = simplexml_load_file($account_list_file);
   $data = $xml->xpath('//service[@name="YOUTUBE"]/account');
   
   $accounts = array();
   
   foreach($data as $account)
   {
      $accounts[] = array('youtube_username' => (string)$account->username,
                     'youtube_password' => (string)$account->password);     
   }
   return $accounts;
}

/**
* Debug Logs
* stops if last param is 1
*/
/*function l()
{
   $t = debug_backtrace();
   $args = func_get_args();
   ob_start();
   echo basename($t[0]["file"]).":{$t[0]["line"]} > ";
   var_dump($args);
   $data = ob_get_contents();
   ob_end_clean();
   file_put_contents(LOG_FILE,$data,FILE_APPEND);
   if(end($args) === 1) die;
}
*/

function _get_youtube_feed($ip,$host,$path,$authToken)
{
   global $want_debug; //we want to access the global variable $want_debug
   global $developerKey;
   $fp = fsockopen($ip, 80, $errno, $errstr, 30);
   if (!$fp) {
       echo "$errstr ($errno)<br />\n";
   } else {
       $out = "GET {$path} HTTP/1.0\r\n";
       $out .= "Host: {$host}\r\n";
       $out .= "Authorization: GoogleLogin auth=$authToken\r\n";
       $out .= "X-GData-Key: key=$developerKey\r\n";
       $out .= "Connection: Close\r\n\r\n";
       if($want_debug){
           echo "DBG: _get_youtube_feed: about to send to $ip:\n$out";
       }
       fwrite($fp, $out);
     
      $content = '';
      $headerPassed = false;
       while (!feof($fp)) {
           $l = fgets($fp);
         if($l == "\r\n") $headerPassed = true;
         if($headerPassed) $content .= $l;
   
       }
       fclose($fp);
   
       if($want_debug){
           echo "DBG: _get_youtube_feed: received:\n$content\n";
       }
      preg_match("/(<\?.*>)/s",$content,$m);
      return trim($m[1]);
   }
}

function _getAuthToken ($username, $password){
   global $want_debug;
   global $youtubeIP;
   global $account_list_file;
   global $insecure;
   
   //url_encode the user/password
   $ytUser = urlencode($username);
   $ytPass = urlencode($password);
   //use curl since it uses https   
   $out = array();
   $command = "/usr/bin/curl -s -S $insecure --location https://www.google.com/accounts/ClientLogin --data 'Email=$ytUser&Passwd=$ytPass&service=youtube&source=wdtvext' --header 'Content-Type:application/x-www-form-urlencoded' 2>&1";
   if($want_debug){
      echo "DBG: Authenticating with command: $command";
   }
   exec($command, $out);
   if($want_debug){
      print_r($out);
   }
   foreach ($out as $line){
      //we're looking for this line:
      // Auth=DQAAAJ8AAADBW15VA3cV9OmZeBF0maTTyCfHxdtEj62tyHCO696_CFYe6mUpwrF1DXHnpepVXKMFj3vgtoVHLXYv0qX9Hk1fawpea5XuZbEZZBNdfaW9-hNWbpDmE-_Rl_0g3RPoBNZLkIxJaeDz4d-M2WLmyiTcmVlcvbDhI1xc7mxfkDPLKdBLdJkC1fAFUbZHqq5mKfbWB0se4hLObnxWBEj-Rzr-
      preg_match("/^Auth=(.*)$/", $line, $result);
      if(isset ($result[1])){
         //there's a match, this is the line we were after
         return $result[1];
      }
      preg_match("/SSL certificate problem, verify that the CA cert is OK./", $line, $result);
      if(isset ($result[0])){
         //curl certificate problem. Check if the time is reasonable
         if(date("Y") == 2000){
            echo "Your system date is not set. The script is unable to check the Google's/Youtube's server identity unless the date is correct. Set the date by hand, or run ntpdate pool.ntp.org and try again.\n";
         }
         else{
            echo "Unable to check Google's/Youtube's certificate (curl issue - see http://forum.wdlxtv.com/viewtopic.php?f=38&t=2469&start=0).\n";
         }
         echo "You can still connect without doing this check (but you will be vulnerable to Man-In-The-Middle attacks) by calling this script with --insecure parameter. Please note that your authentication is still encrypted, but you can't be 100% sure you are talking to a genuine Google/Youtube server.\n";
         exit;
      }
   }
   
   //if we got here, there was an error during the authentication process
   die ("Unable to authenticate $username with Youtube. Please check that the username and password are corectly set in $account_list_file or run this script in debug mode");
}
?>



With the current wdtv firmware version you'll have to run it with the --insecure parameter. This is because curl doesn't have the correct certificates in order to authenticate the Google/Youtube server, so you will still authenticate with encryption, but you will not verify the identity of the Google/Youtube server. This is a temporary fix - should be fixed in a future firmware release (or there is a manual way to do it, if you are afraid of the word insecure) :P See http://forum.wdlxtv.com/viewtopic.php?f=38&t=2469&start=0 for details.

Code: Select all
# php5-cgi /conf/youtube-subscriptions-helper.php
X-Powered-By: PHP/5.3.2-1
Content-type: text/html

Authenticating adrian.popa.gh
Your system date is not set. The script is unable to check the Google's/Youtube's server identity unless the date is correct. Set the date by hand, or run ntpdate pool.ntp.org and try again.
You can still connect without doing this check (but you will be vulnerable to Man-In-The-Middle attacks) by calling this script with --insecure parameter. Please note that your authentication is still encrypted, but you can't be 100% sure you are talking to a genuine Google/Youtube server.
# ntpdate pool.ntp.org
15 Nov 18:26:04 ntpdate[4820]: step time server 80.96.120.252 offset 343149032.899119 sec
# php5-cgi /conf/youtube-subscriptions-helper.php
X-Powered-By: PHP/5.3.2-1
Content-type: text/html

Authenticating adrian.popa.gh
Unable to check Google's/Youtube's certificate (curl issue - see http://forum.wdlxtv.com/viewtopic.php?f=38&t=2469&start=0).
You can still connect without doing this check (but you will be vulnerable to Man-In-The-Middle attacks) by calling this script with --insecure parameter. Please note that your authentication is still encrypted, but you can't be 100% sure you are talking to a genuine Google/Youtube server.
# php5-cgi /conf/youtube-subscriptions-helper.php --insecure
X-Powered-By: PHP/5.3.2-1
Content-type: text/html

Authenticating adrian.popa.gh
Getting subscriptions for adrian.popa.gh
Downloading information for first 20 clips from each of these channels: collegehumor, DiscoveryNetworks, froyoca, FunnyorDie, geekbeattv, Hak5Darren, HBO, HDstarcraft, HuskyStarcraft, PlanetGreenTV, revision3, ScienceChannel, sixtysymbols, TEDtalksDirector, TEKHD, TheOnion, thisweekinlinux, TLC, twit, ubunite
collegehumor: 20 videos
DiscoveryNetworks: 20 videos
froyoca: 11 videos
FunnyorDie: 20 videos
geekbeattv: 20 videos
Hak5Darren: 20 videos
HBO: 20 videos
HDstarcraft: 20 videos
HuskyStarcraft: 20 videos
PlanetGreenTV: 20 videos
revision3: 20 videos
ScienceChannel: 20 videos
sixtysymbols: 20 videos
TEDtalksDirector: 20 videos
TEKHD: 20 videos
TheOnion: 20 videos
thisweekinlinux: 20 videos
TLC: 20 videos
twit: 20 videos
ubunite: 20 videos
Writing /tmp/youtube-subscriptions.cache
Total execution time: 13.83636593818665
#


To get feeds more often, add the script (with the relevant parameter(s) in a cron/startup script). For now, I've copied the php file in my /conf directory and I can run it from there:
Code: Select all
# cat /conf/S99user-script
#!/bin/sh
# update youtube subscriptions - delay with 30s, because /conf is not available right now!
(sleep 30; /usr/bin/php5-cgi /conf/youtube-subscriptions-helper.php --insecure > /tmp/youtube-subscriptions.txt 2>&1)&


Edit: aparently the UMSP server goes haywire (plays just the first clip and then freezes) if you change the date while it's running (meaning, don't do ntpdate without restarting it). I have to test this more thoroughly though...
User avatar
mad_ady
Developer
 
Posts: 4568
Joined: Fri Nov 05, 2010 9:08 am
Location: Bucharest, Romania

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby mad_ady » Mon Nov 15, 2010 9:23 am

Ok - I've played with the videos and everything is nice and dandy, but all the clips are high quality (720p) and my network connection is chocking under the strain. I will look into the API to see how to request variable resolution clips. I'm very happy with 480p on my old-school TV :) Stay tuned...
User avatar
mad_ady
Developer
 
Posts: 4568
Joined: Fri Nov 05, 2010 9:08 am
Location: Bucharest, Romania

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby mad_ady » Tue Nov 16, 2010 8:37 am

I've managed to allow for a selectable video resolution. If you have the YOUTUBE_QUALITY=480P configuration variable set, then the plugin will download a lower quality video for you. Unfortunately I couldn't find the correct mapping, but I'm using this:

Code: Select all
//decide what video quality to request
   /*
   22: 720p, HD, ~3-5Mbps
   18: 360p, good quality, stereo sound, ~1Mbps
   6:  360p, good quality, mono sound, ~1Mbps
   0:  360p, lowest quality -- or original resolution?
   */
//set a default quality setting -> HD by default
  $fmt = 22; 

...

switch($resolution[0]){
      case '360':
        $fmt=6;
        break;
      case '480':
        $fmt=18;
        break;
      case '720':
      case '1080':
        $fmt=22; 
    }



For now, replace your /tmp/umsp-plugins/youtube-subscriptions/youtube-subscriptions-proxy.php with this code:
Code: Select all
<?php
// edited by mad_ady - honors user selected quality level in YOUTUBE_QUALITY

define("LOG_FILE",'/tmp/umsp-log.txt');

function parse_header($content)
{
    $newline = "\r\n";
    $parts = preg_split("/$newline . $newline/", $content);

    $header = array_shift($parts);
    $content = implode($parts, $newline . $newline);

    $parts = preg_split("/$newline/", $header);
    foreach ($parts as $part)
    {
        if (preg_match("/(.*)\: (.*)/", $part, $matches))
        {
            $headers[$matches[1]] = $matches[2];
        }
    }
    return $headers;
}

if (strlen($_GET['video_id']) < 50) {
   $url = _getYTVideo($_GET['video_id']);
} else {
   $url = $_GET['video_id'];
}
$url = _getYTVideo($_GET['video_id']);

_DownloadThru($url);

function _DownloadThru($url)
{
  foreach (array (' ',"\t","\n") as $char)
    $url = preg_replace("/$char/",urlencode($char),$url);
    $parsedURL = parse_url($url);

    $itemHost = $parsedURL['host'];
    $itemPath  = array_key_exists('path', $parsedURL) ? $parsedURL['path'] : "/";
    $itemPort  = array_key_exists('port', $parsedURL) ? (int)$parsedURL['port'] : 80;
    $itemPath  .= array_key_exists('query', $parsedURL) ? "?" . $parsedURL['query'] : "";

    $itemPath = urldecode($itemPath);
    _GetFile($itemHost, $itemPath, $itemPort);
}
function _GetFile($prmHost, $prmPath, $prmPort) {
   $fp = fsockopen($prmHost, $prmPort, $errno, $errstr, 30);
   if (!$fp) {
      echo "$errstr ($errno)<br />\n";
   } else {
      $out  = 'GET '. $prmPath .' HTTP/1.1' ."\r\n";
      $out .= 'Host: ' . $prmHost . "\r\n";
      $out .= "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\r\n";
      $out .= "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
      $out .= "Accept-Language: ru,en-us;q=0.7,en;q=0.3\r\n";
      $out .= "Accept-Encoding: gzip,deflate\r\n";
      $out .= "Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7\r\n";
      $out .= "Keep-Alive: 115\r\n";
      $out .= "Connection: keep-alive\r\n";
      //      $out .= 'Cache-Control: no-cache' ."\r\n";
      $out .= "\r\n";
        //echo "<pre>$out</pre>";
      fwrite($fp, $out);
      $headerpassed = false;
        $response_text = "";
        //HTTP/1.1 200 OK
        //HTTP/1.1 302 Found
        $http_code = "";
      while ($headerpassed == false) {
         $line = fgets( $fp);
         if( $line == "\r\n" )
            $headerpassed = true;
            else
            if ($http_code == "")
                $http_code = $line;
         else
                $response_text .= $line;
      }
        $response = parse_header($response_text);

        if ($response['Content-Type'] == 'video/x-flv')
            $response['Content-Disposition'] = 'attachment; filename="video.flv"';   
        if ($response['Content-Type'] == 'video/mp4')
            $response['Content-Disposition'] = 'attachment; filename="video.mp4"';   
        if ($http_code == "HTTP/1.1 302 Found\r\n" || $http_code == "HTTP/1.1 303 See Other\r\n")
        {                                         
            fclose($fp);
            _DownloadThru($response['Location']);
        }
        else
        if ($http_code == "HTTP/1.1 200 OK\r\n")
        {
            foreach (array_keys($response) as $header)
                header("$header: " . $response[$header]);
            fpassthru($fp);
            exit;
        }
        else
        {
            fclose($fp);
            echo "<p>$out</p>";
            echo "<p>$http_code</p>";
            echo "<p>$response_text</p>";
        }

   }
}

function _getYTVideo($id)
{
   //decide what video quality to request
   /*
   22: 720p, HD, ~3-5Mbps
   18: 360p, good quality, stereo sound, ~1Mbps
   6:  360p, good quality, mono sound, ~1Mbps
   0:  360p, lowest quality -- or original resolution?
   */

   //set a default quality setting -> HD by default
   $fmt = 22; 
   //get the current setting
   exec('grep YOUTUBE_QUALITY /conf/config | cut -d "=" -f 2 | sed "s/\'//g" | sed "s/\"//g" | sed "s/P//"', $resolution);
   if(isset($resolution[0])){
      switch($resolution[0]){
        case '360':
         $fmt=6;
         break;
        case '480':
         $fmt=18;
         break;
        case '720':
        case '1080':
         $fmt=22; 
      }
   }
   
   $c = file_get_contents("http://www.youtube.com/watch?v={$id}&fmt={$fmt}");

   preg_match("/\&t=([^(\&|$)]*)/", $c, $m);
   $ticket = $m[1];

   preg_match("/\&fmt_url_map=([^(\&|$)]*)/", $c, $m);

   $formats = explode('%2C',$m[1]);

   $data = array();
   foreach($formats as $v) {
      $t = explode('%7C',$v);
      if ($t[0] == 37) continue;
      $data[ $t[0] ] = urldecode($t[1]);
   }
   
   return !empty($data)?reset($data):array();
}

function l()
{
   $t = debug_backtrace();
   $args = func_get_args();
   ob_start();
   echo basename($t[0]["file"]).":{$t[0]["line"]} > ";
   var_dump($args);
   $data = ob_get_contents();
   ob_end_clean();
   file_put_contents(LOG_FILE,$data,FILE_APPEND);
   if(end($args) === 1) die;
}
?>


Post your feedback, and if it's positive we'll push this on the svn and it will work automatically in the end...
User avatar
mad_ady
Developer
 
Posts: 4568
Joined: Fri Nov 05, 2010 9:08 am
Location: Bucharest, Romania

Re: Plugin: YouTube Subscriptions + new subscription videos   

Postby biomedica » Tue Nov 16, 2010 2:06 pm

Great job mad_ady. I would like to ask you if you know how to increase the number of videos being loaded per channel subscribed. Right now, on my box I can only get 20 videos per channel and I would like to increase that number. Also, is your box updating the videos without having to restart the box?
biomedica
DLX'er
 
Posts: 83
Joined: Tue Oct 12, 2010 9:14 pm

PreviousNext

Return to UMSP Media Server

Who is online

Users browsing this forum: No registered users and 1 guest