Sign up here and you can log into the forum!

Webend remote view

Feature requests / enhancements discussion regarding the web interface

Webend remote view   

Postby mad_ady » Mon Jan 07, 2013 11:39 pm

I've been contacted in a PM by a user who wanted to know if I was still working on a way to remotely view the OSD of the WDTV inside the webend (similar to how WDTV Remote for 1.02 works). Original thread here: http://forum.wdlxtv.com/viewtopic.php?f=46&t=6800&p=57012&hilit=remoteview#p57010

Well, I haven't worked on it since I was hoping of getting a unified method that would work on 1.02 and 1.05, but I haven't totally scrapped the idea.

This thread is intended to collect and share whatever information I gathered so that others with more free time (or even myself) can make use of later on.

So, the idea is to have a script/program on the WDTV that will request a screen refresh and save it in a temporary file and have a web page (integrated into the webend) periodically request this image.

For 1.02 the wdtv remote code (written in C# - http://sourceforge.net/projects/wdtvtelnetremot/) that decodes the image is this:
Code: Select all
        public Bitmap GetScreenReadArea()
        {
            ConnectSocket();
            int count = 10;
            byte[] data = new byte[1024 * 1024];
            while (count-- > 0)
            {
                byte[] cmd = new byte[1];
                cmd[0] = 7;
                skt.Send(cmd);

                int read = 0;
                int totalRead = 0;

                SocketError so;

                skt.ReceiveTimeout = 1500;
                do
                {
                    read = skt.Receive(data, totalRead, data.Length - totalRead, SocketFlags.Partial, out so);
                    totalRead += read;
                } while (read > 0 && totalRead < data.Length);



                if (totalRead != data.Length)
                    continue;

                totalRead = 0;


                MemoryStream ms = new MemoryStream(data);

                ms.Position = 0;
                BinaryReader sr = new BinaryReader(ms);
                ScreenInfo si = new ScreenInfo();
                si.Read(sr);

                if (si.dataSize <= 0 || si.dataSize != si.bpp * si.cols * si.rows / 8)
                {
                    Debug.WriteLine(string.Format("Failed read screen, reson = {0}", si.lockRes));
                    continue;
                }

                data = new byte[si.dataSize];

                Stopwatch sw = Stopwatch.StartNew();

                do
                {
                    read = skt.Receive(data, totalRead, data.Length - totalRead, SocketFlags.Partial, out so);
                    totalRead += read;
                } while (read > 0 && totalRead < data.Length);

                PixelFormat format = PixelFormat.Format32bppPArgb;
                Bitmap bmp = new Bitmap(si.cols, si.rows, format);
                BitmapData bits = bmp.LockBits(new Rectangle(0, 0, si.cols, si.rows), ImageLockMode.ReadWrite, format);
                Marshal.Copy(data, 0, bits.Scan0, data.Length);
                bmp.UnlockBits(bits);
                bmp.Save("c:\\screen.png", ImageFormat.Png);

                return bmp;
            }
            return null;
        }


The image itself sent by the wdtv comes in RGBA format and can be converted back into a png using imagemagick (some ideas here: http://www.imagemagick.org/discourse-server/viewtopic.php?f=7&t=20581)

recliq had some more info on how to get wdtvext to generate an image, but sadly I didn't have time to look into it:
Code: Select all
(20:05:42) recliq: eg. to call command_get_screen_info() you can do this
(20:05:43) recliq: echo -e "\005" | ncat 192.168.1.12 3490 | hexdump
(20:05:43) recliq: 0000000 0500 0000 02d0 0000 0020 0000 0000 0000
(20:05:43) recliq: 0000010
(20:06:26) recliq: you should replace the IP or on WD itself you can use localhost
(20:10:17) mad_ady: thanks recliq!
(20:10:24) mad_ady: I'll have a look tomorrow
(20:10:51) mad_ady: I take it the binary output is the image itself?
(20:12:30) recliq left the room (quit: Ping timeout).
(20:13:27) mozilla.se.eu.dal.net: (notice) ChanServ invited recliq into channel #wdtv-dev
(20:13:27) recliq [recliq@wdlxtv.de] entered the room.
(20:13:27) mode (+o recliq) by ChanServ
(20:13:27) recliq: dalnet is driving me crazy today
(20:13:27) recliq: |19?11?38| <@recliq> not sure i think it's compressed
(20:13:27) recliq: |19?11?48| <@recliq> haven't dug into the code much
(20:13:27) recliq: |19?12?03| <@recliq> but there's a compression lib supplied
(20:13:27) recliq: |19?12?11| <@recliq> so i guess it's compressed
(20:14:03) recliq: |19?12?22| <@recliq> that's what calimero and pibos discussed about in the thread i can't find i think
(20:15:48) mad_ady: I see


For 1.05 firmware I remember hearing b-rad say that if you send a PrintScreen key code to the OSD, it will write a screenshot somewhere - but my memory is failing and I don't remember any specifics.
Last edited by recliq on Tue Jan 08, 2013 9:54 am, edited 1 time in total.
Reason: added url tags - now the sourceforge link actually works ;)
User avatar
mad_ady
Developer
 
Posts: 4522
Joined: Fri Nov 05, 2010 9:08 am
Location: Bucharest, Romania

Re: Webend remote view   

Postby recliq » Tue Jan 08, 2013 3:50 am

Firmwares >1.02.21 (eg. 1.05.04) use DirectFB instead of NanoX as render engine.
Here's how to make screenshots using DirectFB (quoted from directfb README.screenshots)
How to make DirectFB screenshots
--------------------------------

There are two ways to generate DirectFB screenshots. The easy way is
to set the "screenshot-dir" parameter in the DirectFB configuration
file directfbrc or to pass it as a command-line option. See the
directfbrc man-page for more details. You can then generate screen
dumps in the PPM format by pressing the PrintScreen key. The PPM
files can easily be converted to others formats using for example the
netpbm tools.

The hard way to do screenshots is to read directly from the frame
buffer device. This works for all applications that use the frame
buffer device, not only for DirectFB applications.

The resulting data is then converted to a more convenient format using
the netpbm graphics conversion tools. If the frame buffer is not
running at 24 bit depth, the data has to be propagated to 24bit RGB
before netpbm can handle it. The tools directory contains the source
for two small utilities that do just this:

raw16toraw24 is a small tool that reads 16bit RGB565 data from stdin,
converts to 24bit RGB888 data and writes it to stdout.

raw15toraw24 is a small tool that reads 15bit RGB555 data from stdin,
converts to 24bit RGB888 data and writes it to stdout.

raw32toraw24 is a small tool that reads 32bit ARGB data from stdin,
converts to 24bit RGB888 data and writes it to stdout.


The following steps have to be performed to take screenshots:

Step 1 - Log in from another computer using ssh or telnet. Start your
application and stop it by pressing Ctrl+C in the remote
terminal.

Step 2 - Read data from /dev/fb0 and write it to a file.

examples:
[15bit] raw15toraw24 < /dev/fb0 > raw24.tmp
[16bit] raw16toraw24 < /dev/fb0 > raw24.tmp
[24bit] cat /dev/fb0 > raw24.tmp
[32bit] raw32toraw24 < /dev/fb0 > raw24.tmp

Step 2 - Convert data to ppm using rawtoppm and specify the resolution of
the frame buffer.

example: rawtoppm 800 600 raw24.tmp > ppm24.tmp

Step 3 - Convert ppm to png using pnmtopng.

example : pnmtopng <ppm24.tmp >screenshot.png


Thats it!
­WDLXTV Project Maintainer
-:] If you like my contributions feel free to donate for a beer or a new flash drive. ...and always remember: RTFM! (README, FAQ, WIKI) [:-
User avatar
recliq
WDLXTV Team
 
Posts: 5513
Joined: Thu Apr 15, 2010 8:09 am
Location: Kiel, Germany

Re: Webend remote view   

Postby Darren » Tue Jan 08, 2013 8:12 am

Thanks for the handy info! I'd been searching a while for a method to get a screen image via the wdtv directly.
After some experimenting I was able to request the screen image and convert it to a usable format.
Here's what I've deduced so far, as tested on the 1.02.21 firmware.

You need to send commands to the wdtvext at its address and port:
001 = get uncompressed image
005 = get image header (width, height, bit depth)
007 = get compressed image

Here's how I've been able to get the screen image and convert it:

Code: Select all
echo -e "\001" | nc localhost 3490 > screen.rgba
convert -size 1280x720+20 -depth 8 -separate -swap 0,2 -combine screen.rgba -quality 92 screen.jpg

Some notes on the above:
- Convert to jpg takes about 9 seconds with no other cpu load, about 1 second longer to process than convert to bmp.
- The wdtvext outputs the image in a BGRA format, so you need to swap the Red and Blue channels for a compatible RGB format.
- The "+20" in the -size option skips the image header. The rest is read in as raw image data using the specified dimensions.
- When using convert, the ".rgba" extension of the input file is needed to identify the file format (or use the -channel rgba option).
- Sometimes the OO1 command does NOT return the uncompressed image (!) and instead it returns just the image header. The 001 command seems a bit random in this respect. Sending the 005 and/or 007 commands will eventually cause the 001 command to start working.

And some questions too...
- What's the secret to (reliably) getting the uncompressed image via the 001 command?
- What format is the 007 command's compressed image in? I haven't found a way to convert this format yet, and it might be preferable since it seems to always be returned at command.
- Is there more info available on what other commands the wdtvext can respond to, and their usage? That could be quite interesting!
- I tried googling for "command_get_screen_info()" but there were no results. Is this function a part of existing code I could access as a reference?

recliq: eg. to call command_get_screen_info() you can do this
recliq: echo -e "\005" | ncat 192.168.1.12 3490 | hexdump
Darren
DLX'er
 
Posts: 69
Joined: Sat Jun 19, 2010 3:35 pm

Re: Webend remote view   

Postby recliq » Tue Jan 08, 2013 9:53 am

Unfortunately there is neither source nor any internal documentation for WDTVExt.
What I told mad_ady back then was based on experiments and the source of WDTVRemote. The later might give you more hints.
http://sourceforge.net/projects/wdtvtelnetremot/
All the other infos I read were on the old wdtvforum.com which seems to be dead right now. WDTVExt development started there and pibos and calimero were discussing things over there back then. Maybe it's just temporarily down (haven't been there for quite some time) then it's def. worth searching that forum when back online.
­WDLXTV Project Maintainer
-:] If you like my contributions feel free to donate for a beer or a new flash drive. ...and always remember: RTFM! (README, FAQ, WIKI) [:-
User avatar
recliq
WDLXTV Team
 
Posts: 5513
Joined: Thu Apr 15, 2010 8:09 am
Location: Kiel, Germany

Re: Webend remote view   

Postby mad_ady » Tue Jan 08, 2013 10:30 am

Great work!

Regarding the regeneration of the image, i too have noticed that wdtvext does not always return something. It seems to do this when there is no input from the remote for a while. Sending remote key events seems to make it work again...
User avatar
mad_ady
Developer
 
Posts: 4522
Joined: Fri Nov 05, 2010 9:08 am
Location: Bucharest, Romania

Re: Webend remote view   

Postby recliq » Tue Jan 08, 2013 12:49 pm

related to screensaver?
­WDLXTV Project Maintainer
-:] If you like my contributions feel free to donate for a beer or a new flash drive. ...and always remember: RTFM! (README, FAQ, WIKI) [:-
User avatar
recliq
WDLXTV Team
 
Posts: 5513
Joined: Thu Apr 15, 2010 8:09 am
Location: Kiel, Germany

Re: Webend remote view   

Postby Darren » Tue Jan 08, 2013 6:32 pm

Ah, you're right, sending keys does make it work, but not always. So maybe its not the sending of keys that makes it work directly, but rather maybe it only will send the uncompressed image if there is a change in what's displayed onscreen? And it seems like it needs to be more than just a minor change. If I just up-arrow in a list then it might not send the image, but if I hit enter or exit then it does.

So adding a simple check would help:

Code: Select all
echo -e "\001" | nc localhost 3490 > screen.rgba
bytes=`stat -c %s screen.rgba`
if [ "$bytes" -eq "3686420" ]; then   ### the normal uncompressed image file always has the same number of bytes
   convert -size 1280x720+20 -depth 8 -separate -swap 0,2 -combine screen.rgba -quality 92 screen.jpg
fi


Btw I've got my screensaver disabled.
Darren
DLX'er
 
Posts: 69
Joined: Sat Jun 19, 2010 3:35 pm

Re: Webend remote view   

Postby Darren » Tue Jan 08, 2013 9:57 pm

On a hunch I took a look inside the wdtvext binary (libext.so) and found a few answers to my questions:
- The compressed image appears to be using QuickLZ. I don't know of a way to decompress that directly on the wdtv.
- Inside there I also found the command_get_screen_info() I was wondering about. I had seen a mention somewhere that pibos may have released the source code for the wdtvext? That would help a whole bunch to see that!
- I also found other commands that the wdtvext will respond to, mostly screen and buffer related.
Darren
DLX'er
 
Posts: 69
Joined: Sat Jun 19, 2010 3:35 pm

Re: Webend remote view   

Postby mad_ady » Tue Jan 08, 2013 10:58 pm

Seems quicklz has sources available (http://www.quicklz.com/download.html) and you could probably cross-compile them for the wdtv. I wonder if you can get better performance by getting the compressed image -> uncompressing it -> recompressing it as jpg than from asking for the uncompressed image. There might be a performance issue when wdtvext hands you the uncompressed buffer compared to when you get the uncompressed one, but this needs testing.

Regarding the wdtvext sources, you should contact b-rad - he might have some older version still. Otherwise, you should contact pibos (who has been missing in action for about 6 months) who should also have them...
User avatar
mad_ady
Developer
 
Posts: 4522
Joined: Fri Nov 05, 2010 9:08 am
Location: Bucharest, Romania

Re: Webend remote view   

Postby Darren » Wed Jan 09, 2013 3:41 am

In my testing, getting the uncompressed image is about twice as fast as getting the compressed one. This makes sense, as the quicklz compression takes some extra time to perform. As long as you don't have to deliver the uncompressed image over the wire, then it would be faster to use that one while processing the image on the box.
And we're only talking a difference of 250ms vs 500ms, so retrieving the image is pretty quick in either format.

I discovered a quirk in convert that added about 5 seconds to the process. For some reason using the needed -depth option caused slowness. To not need that option you'd have to have an image header on the input file that convert can understand, to tell it what format the data is in. So I chose the miff format, native to convert, since its easy to implement.

Previously it took 9 to 10 seconds to get the screen image and convert to jpg. With this new method it now only takes 4 to 5 seconds, so its much better!

Code: Select all
echo -e "\001" | nc localhost 3490 > screen.rgba
bytes=`stat -c %s screen.rgba`
if [ "$bytes" -eq "3686420" ]; then   # the normal uncompressed image file always has the same number of bytes
   echo Screen update: Yes
   echo -en "id=ImageMagick columns=1280 rows=720 depth=8 matte=true\n\f\n:\032" > screen.miff      # this header must be 60 bytes long, to match dd command
   dd if=screen.rgba of=screen.miff ibs=20 skip=1 obs=60 seek=1 2> /dev/null
   convert -separate -swap 0,2 -combine screen.miff -quality 92 screen.jpg
else
   echo Screen update: No
fi

Btw, if I could find an RGBA image header that was only 20 bytes long then it would reduce the total time to 3 or 4 seconds, since you wouldn't need to copy all that data with dd, and the header could be written directly inside the original screen image.
Darren
DLX'er
 
Posts: 69
Joined: Sat Jun 19, 2010 3:35 pm

Next

Return to Webend Web Interface

Who is online

Users browsing this forum: No registered users and 2 guests