Institute for the Explanation of Everything

G'Day,

my son is almost 6 years old and now needs a PC, he says. Educational aspects will not be covered here, so we immediately start to discuss the technical side.

The point is that even my beloved wife does not own a PC. She has a business laptop provided by her company. Having this in mind as well, I started to wonder about whether a fully featured PC for my son is really needed or not. And the answer is, of course: no. On my wife's Win7-PC, for example, I installed Xming, pulseaudio and opened ports 6000 and 4713 (the truth is: Not I did that but the admins in my wife's company - my wife only has a normal user-account with very restricted rights). Now, whenever she wants to perform private things with her machine, she opens up her X-server and connects to my server via xdmcp.

For the kid, this would be a perfect setup as well, and making backups and such is much easier, as all data is stored in my server's /home. Contrary to my wife, however, my son does not have a machine which could serve as an xdmcp-client. Yes, I could by him a small PC, install Linux and map his home-directory via NFS, but first, I do not want to administer one more machine and second, I do have an old T30 with a broken harddisk inside - time for a thin client, hm?

The setup is relatively easy, in theory. The T30 can boot from network using PXE, so all to do is to serve an OS. In theory. In pratice, some configuration is needed. To begin with, I have one standard-router working provided by my ISP and that device is of course not configurable. The SuSE-shipped ISC DHCP server can be configured to only respond to pxe-requests and thereby does not get in conflict with the standard-server in my router, which means, it can be run in parallel. So, on the Xhost (the machine running the xdmcp-server) I started dhcpd with the following config:

Beside a proper router-definition, as usual, the crucial entry is next-server. It points to the tftp-server.

So next is tftpd. On SuSE, installation of the packet is sufficient. The server can be configured with yast, which also adds a user tftp, and leaving the defaults just untouched is fine. Manual configuration is done by adding the user (useradd tftp) and tweaking /etc/sysconfig/atftpd. The defaults point to /srv/tftpboot/, so we go there and add three directories pxelinux.cfg, boot, and image, which we need later.

No. 3 is syslinux. Install that package. On SuSE, syslinux' location is /usr/share/, and what we need from there is pxelinux.0. We copy it to tftpd's directory:

Before I forget: If you run a firewall, then the following information is probably of importance: dhcpd listens on port 67, tftpd on port 69.

We now switch to pxelinux.cfg and create a file default there with the following content:

The file is pretty self-explanatory. It tells the pxe-client that there is a kernel in boot (relative to /srv/tftpboot/) and an initial ram-disk (in the same directory). As regards the arguments for initrd, ramdisk_size matches my T30's RAM, and kiwiserver is something we will come to in a minute.

And now the question is: Where do we get a kernel from? An initial ram-disk? And by the way: Is that enough to provide a running system?

No. We know that from normal grub-booting: Kernel and initrd get loaded, a very small system within initrd provides basics like mounting devices, hopefully finds a filesystem and finally hands over to /sbin/init. That is, even in case we can get hold of a kernel and an initrd, we don't have a filesystem to boot from.

For the impatient: You will certainly know damn small linux (DSL). There is a pxe-bootable DSL out there, from which you need the files linux24 and minirt24.gz, to be copied to /srv/tftpboot/boot/linux and /srv/tftpboot/boot/initrd, respectively - and you're done. The filesystem we just talked about is included in DSL's initrd. If you unpack initrd, you will find a KNOPPIX-cd, which contains a filesystem and which is mounted by DSL's initrd to /dev/ram1. Tricky, yes.

That remembers me that I should write a few words about un- and repacking an initrd … later perhaps.

Topic no. 4 is KIWI. The DSL-way is fine but has its limitations. You can't kill the X-server in order to start it again with -query or -broadcast, which is what I need for xdmcp. In addition, I need pulseaudio, which in turn means that I need alsa as well, and plugins, and compat-modules, and a working config, and so on. What I need is an appliance, a pre-configured thin client.

KIWI can produce exactly this. Install it. Using KIWI consists in performing two steps: Preparing and building. For the preparation, one needs to provide a template in which the appliance is configured. KIWI comes with such templates, but neither of them satisfied my needs. Moreover, KIWI (which is just a perl-script), has bugs. I will not report about the hours it took me to produce a thin client using the shipped KIWI, I will only present what did the trick for me.

The usual way of invoking kiwi is like this: You decide which template to use, say we take the one in /usr/share/doc/packages/kiwi/examples/suse-11.4/suse-pxe-client/. You then edit config.xml in that directory and change it to fir your needs and then you do

the linux32 directive is important in my case, because I am on 64 Bit, but the appliance to build should be 32 Bit. If all goes well, four files will be available in /tmp/kiwi/result, I will come to these soon.

Unfortunately, I came across multiple problems in the first step (prepare). Although I mentioned pulseaudio in the package-section, KIWI didn't install it. Same with other software. As I was unable to fix the probpems and went to SuSE Studio. I was hoping to use SuSE's build service to get PXE-images, but SuSE does not provide such builds. So I did the following. I went with a SuSE-Live-CD with X, which will produce a minimal desktop-system with IceWM (instead of KDE or GNOME).

I did not change the configuration (that will be done later) and just downloaded it. What I mean is this: In the last step you will be asked for the format to download,

and as you can see, there is no .pxe available. The reason is hidden under the link "Read more about formats …"

but this statement can be safely ignored in my case. I am not using atftpd and everythings works fine, but well, my setup is simple. We will go a more flexible path anyway. As I said, I did not download an image, but just the config. There is a link on SuSE Studio

with which we download our template. Now, we choose some place where to store it, unpack it and can now use kiwi. In detail: The config comes as a .tgz, containing a directory source and a shell-script create_appliance.sh. We don't want to start the script, as it will produce a cd-image, which we don't want. We want pxe-images. We only unpack source and edit config.xml in that directory. In the original .xml, we find a section preferences saying

This section must be replaced by

This is the essential step and I will come back to this later. The second change is to edit the repositories (if needed, that depends on the config SuSE Studio hase created)

and to insert something sensefull:

For my particular needs, I also edited the section <packages> and added pulseaudio and changed the image-type:

If this is done, building requires the creation of an MD5-checksum (the image is located in /root/tmp/kiwi/<name as defined in config.xml>/)

and a preparation afterwards:

This is the place to alter the build. In my case, I copy my version of default.pa to build/etc/pulseaudio, but more important, I replace build/etc/init.d/xdm with a code which starts an X-server doing xdmcp-broadcasts instead of loading X and a local displaymanager afterwards. To keep it simple, I have just replaced build/etc/init.d/xdm with

The second step of KIWI is the create step.

That should it be … but I mentioned bugs, didn't I? When invoking the two commands above, you will at least see one error-message saying that MD5-checksums do not match. This will prevent a complete build. The solution is to edit the kiwi-script. There is a function checkImageIntegrity, which checks the wrong location:

and as a dirty hack, I hardcoded the correct location:

This is probably a bad idea, but I wasn't in the mood to trace invokation of the subroutine yet.

There is a second, more subtile bug, which you don't meet during image-creation. You will see it when the thin client boots: It does not find /sbin/init on the just mounted image. It is there, but it is a symlink to /bin/systemd and because linuxrc inside initrd checks permissions first, it complains about /sbin/init not being as expected. This bug is known and requires you to edit KIWILinuxRC.sh accordingly (on my, system, it is in /usr/share/kiwi/modules/).

Having said this, let us now see what we have. In the image directory you will find what you need: 1. a kernel, 2. an initrd, and 3. a root-filesystem. These files must be copied to their respective directories under /srv/tftpboot/, that is

This has been mentioned before and now we are there: We not only have a kernel and an initial ramdisk, we also have an image containing a compressed filesystem (clicfs). However, up to this point, the image does not differ largely from what you can download from SuSE Studio. The point is that we need to tell the boot-process how to mount the clicfs-image. This has already been done in config.xml in section preferences (see above). As a result of what has been declared there, we find one more file relevant for us, which needs to be installed as well.

Here, < MAC > ist the MAC-address of the pxe-client. The content is precisely what has been said in config.xml.

Let me say something to netboot, as this hasn't been mentioned before, but it is important. In our modified config.xml we have a statement saying boot="netboot/suse-11.4. It directs KIWI to /usr/share/kiwi/image/netboot/suse-11.4 and contains images and configs for booting using pxe. You must use this directive to get a proper pxe-boot.

And now we are there. Below you can see how the boot-process of my thin client looks like. To be able to record it, I fired up a vmware virtual machine, stopped normal startup and selected to pxe-boot via the first virtual nic (yes, VMware can pxe-boot).

Get the Flash Player to see this player.

(3237981 KB / 110 sec = approx. 233 KBit/s)

Is this client thin? No. Compared to DSL's 50MB-KNOPPIX-image, my 200MB-image is large. And that is less than one third of the truth, because my image is a compressed clicfs. It's uncompressed version turns out to be as fat as 750MB (and gows up to 1GB during the build). Remember: It's a SuSE-Live-CD. It calls itself minimalistic, but as a live-CD, it must run on almost all hardware, which requires it to include tons of drivers. Alone glibc-locale eats up to 100MB. For me, however, it makes things much more easier. Using the kiwi-shipped templates, for example, I was unable to install and configure the radeon-driver needed in a T30. In addition, my goal was not only to provide an xdmcp-client, but to play audio coming from the xhost as well. This requires the proper installation of pulseaudio, alsa and all dependant packages, modules and plugins. No thanks.

The main advantage of a live-system is the built-in logic how to probe for devices and to install them on the fly. This logic (provided as scripts) is part of SuSE's base system, which is huge, yes, but allows my so created appliance to run on a large amount of hardware. Now, if I was going to dig into this deeper (which I most probably will not do), I would start to customize the kiwi-created images. A look into the kiwi-script, I assume, will show that tweaking the build directly shouldn't be too complex. Basically, it creates a chroot-environment, switches there and then makes heavy use of zypper and the standard SuSE-install-procedures, which are part of the daily yast-business anyway. There must be some point (unknown to me) at which one should be able to chroot to the environment and to customize the build, most likely by using yast-procedures or even yast directly. The final image contains yast (and tons of other stuff not needed for a this client), so I am quite sure that making the client significiantly smaller and preserving its flexibility at the same time is doable. What discourages me is that I fear there will be too much hassle with checksums, dependencies, bla. I don't know. My project is finished.

I usually use ffmpeg to create AVC/AAC content in an mp4-container. As mplayer's resp. menconder's capabilities are strong, I combine both tools and do the pre-processing with mplayer, write to a named pipe and let ffmpeg read from it. This has been described in a former post (see here under Audio and Video).

By ripping a DVD however, menus and chapter marks will get lost, together with subtitles. This posting is about how to embed chapter marks and subtitles in to the just created mp4.

To begin with, we first need to get hold of the chapter-information and the subtitles. The most convenient way to grab the chapters is done using dvdxchap from the ogmtools-package and the syntax is

where -t provides the title and <dvd-device> is the device-node (/dev/sr0 in my case). The content of the file looks like this,

and is not appropriate for our needs. In an .mp4, the chapters can have names, so what we do is to reformat the file using an editor so that it looks like this:

That is all to be done for the chapters. Subtitles are more complicated. The reason is that on a DVD, the subs are pictures, multiplexed as one more stream into the existing MPEG2-stream. Modern containers allow subtitles to be simple text (in UTF-8) and will choose an own or a system-font to render the subs. The capabilities of these soft-subs go far beyond a simple subtitle, as they allow transforming and coloring the text and even more (animations). This will not be covered here.

The biggest obstacle is therefore to turn the pics into text. Avidemux contains a small OCR-tools and we will come back to this in a second, but before, we must extract the subtitles-stream from the DVD. Avidemux can do this,

but that requires you to know in advance in which .vob on the DVD the stream you're interested in resides. So, what I do is to rip the stream under consideration first and to store it as, say, film.vob. Afterwards, the above dialogue opens a new window, in which I tell Avidemux where to find my film.vob and then, I provide the IFO-file on the disk.

On clicking OK, his will produce two files, film.sub.sub and film.sub.idx, and now we go back to Avidemux for the ocr-stuff. From the same menu as above (Tools), we choose OCR (VobSub -> srt),

and will be confronted with this:

(yes, it says "ouput" - ever realised?). We click on Select idx file: and get one more dialogue,

where we click on Select .idx and choose our just created .idx file. After clicking OK, we then tell Avidemux where to store the .srt file and one more click on OK makes Avidemux's mini-ocr work.

As we have not provided a glyph-file, we need to enter the appropriate letter for every glyph Avidemux finds. That is, the work to be done is limited to at most 26 letters and 10 digits, eventually with additinal characters like ", -, !, … or, in my case, umlauts.

This, when ready, has produced a file as given above (e.g. film.srt) with content like this:

We now take MP4Box from the Gpac-package and invoke it with the following switch:

which produces a .ttxt file like this:

and it is this file which can be muxed into an existing .mp4, together with the chaps file from above. The command is

Notes:

There are some alternatives like mp4chaps, but they don't work reliably. Use MP4Box, which is ISO-conform.

Streaming media to a smartphone is relatively easy - in theory. In practice, success depends on your phone and your streaming server.

First streaming. I use VLC to do this and in principle, the following command line suffices:

Here, the codecs are essential, because mp4v and aac are exactly what the phone expects, at least for rtsp-streaming. To be honest, I am not quite sure about the combination mp4v/aac, as I know that there are Androids out there being able to handle other audio-codecs as well. For my phone, the mentioned combination is the only working one. As regards the container, the above command line will produce am MPEG2-transport-stream and will stream it via rtsp. So far, so easy.

Now, I know that people are (quite desperately) looking for a streaming-client on Android. The point is, it's already there. It can be accessed using the (built-in) browser. The only step to take is to type a URL into the browser's address-field (using rtsp as the protocol) and the browser will automatically open a media-player. So, in order to access the streaming content, type

where hostname is the name of the server on which VLC is running in the LAN. I have also managed to use the public IP-address provided by my ISP, when the phone was connected by GSM (or G3) rather than via WLAN. This requires port-forwarding on the router and will not be covered here.

Now, in theory, this is all to do. In practice, there is a minor and a major problem.

The minor problem is that typing the URL is annoying and setting a bookmark is not possible. Android always assumes http as the protocol to use. If you don't believe, then try to add a bookmark rtsp://whatever and you will realise that Android's browser will try to contact http://rtsp://whatever. This can be worked around by creating a simple HTML-file (locally or on a webserver), which automatically redirects your request and to bookmark that one. It will look like this:

The major problem is <content> from the VLC-command-line from above. It is major, because whether the client is able to play the stream heavily depends on VLC as the streaming server. As an example, local content is quite easy to handle, that is, where <content> is a file. Incoming IPTV-streams behave like this as well. The problem is with DVB (-T in my case). Whatever options I use, the above command will not provide a stable stream. This not only affects the phone, another VLAN-client in my LAN has the same poblem: it will eventually play the stream, but in general, it will fail.

So, for example, the following two lines work smoothly,

whereas the next isn't:

What do I miss?

On searching for a large harddisk I came across a 2 TB drive from Seagate and purchased it, having in mind that Seagate, contrary to other vendors I have had experiences with so far, always satisfied my needs in terms of stability and longevity. The drive was assembled as an external one with one USB and one eSATA connection for easy handling, the case was black with an illuminated Seagate-logo on the front, blinking during i/o.

Shortly after connecting the drive to a host, clicking noises appeared and my thinking was that this noise is quite normal while the disk wakes up from parking. To my disappointment, the frequency of that click increased up to the point where the drive was totally unuseable and freezed the host. Searching the net I found myself among hundreds of users with the same problem (google for 'click of death' and you'll see). Some of them solved the problem by using a better USB-cable (one with a dongle to lower interferences), which didn't work me, some of them suggested to connect the drive via SATA/eSATA, which didn't work either.

I was on the verge of disposing the crap, but then decided to open the case. And that resolved my problem. The drive is now connected to the host without a case, works smoothly, and is silent. I am still using USB, but will put it into a PC and connect it via SATA.

Ekiga is a softphone (see their website), which is running quite well under Linux and always had the potential to become my favourite voip-appliance. Unfortunately, the developers are neither responsive nor do they pay attention to their user's needs. An example is the way Ekiga handles network interfaces. If there are multiple NICs on a system, Ekiga will make use of all of them, eventually listening on eth0 and initiating calls on eth1, or vice versa.

As I am running Asterisk on the same host, it is vital that Ekiga uses one and only one interface. For my setup, the application must listen on eth0 (which it does), but it also must initiate calls on the same NIC, otherwise I can talk, but don't hear anything (or the other way). As a result, Ekiga clears the call after a few seconds, because it realises that there are connection problems. This problem is known, as can be seen from hundreds of complaints found in forums and blogs. But the developers don't care and say that this is the way Ekiga is supposed to work.

Thanks to this very helpfull piece of software, provided by Roman in his blog, it is possible to force an application to only see one NIC. Exactly what is needed for Ekiga.

Before we start

This has become a rather long contribution. The reason is that I own a 'Pulse' from T-Mobile Germany for 4 months now and have played with it now and then, and everytime I found something I thought might be worth to document, I wrote some lines. As a result, the following chapters have no linear structure and information may be outdated.

I am also aware that a 2.1 ROM is available for the Pulse, yet I am quite happy with the device as it is now with its 1.5 Android, so I will update, but not soon.

Table of contents:

I. Rooting it
II. Replacing the SD-card
III. T-Mobile Pulse as a Gateway or Modem
IV. SMS
V. MMS
VI. Audio and Video
VII. Virtual Keyboards
VIII. Android VNC Server
IX. Theming the Pulse (aka De-Magenta it)
X. Android Sources and a Common Kernel
XI. Bluetooth
XII. Accessing the internal Modem with Bluetooth
XIII. Eclipse and Android-SDK
XIV. Ubuntu
XV. MoDaCo's Android 2.1 v6r1 leaked beta longstory ROM

I. Rooting it

Again: T-Mobile UK has released a 2.1 update for the Pulse and attempts are being made these days to create a customised ROM based on it and to watch its performance, see here. However, I am still at 1.5. For a custom ROM based on the old stock-ROM, you may start reading here or here. The ROM I use at the moment is from here. Thanks to Paul.

II. Replacing the SD-card

An easy one, he?

Er, what can I say … this topic happened to be one of the most challenging, to my surprise. In my naivity, I was planning to dd the entire content of my old card to the new one, and then to take gparted. Problems began where I needed to realise that my card reader does not support SDHC-cards. You can however use the phone as a card reader. In recovery mode, enable USB-MS and whatch your kernel recognise /dev/sdX (for me, X=g). At this point, I dd'd my old (non-HC) card (in my card-reader) to the new one (in the phone). The phone should start with the new card left inside, as it should think that the old card is still inserted. I didn't try. Instead, I used gparted, moved /dev/sdg2 (the ext2-partition) to the end and doubled its size from 512 MB to 1 GB. Then I enlarged /dev/sdg1 to occupy the remaining space (15 GB). With this work being done, the phone got stuck in a boot loop, with logcat saying that it can neither access the vfat- nor the ext2-partition.

Since I left my old card unchanged, all data was safely stored, so I tried to partition the card using the recovery shell. After more than 3 hrs of watching the phone printing dots on the screen, I gave up. Into recovery mode again, I partitioned the card manually. To do so, open a terminal with adb shell and run

In the parted-shell, say help or print, remove partitions or create new ones using

And that's it. The next step was to copy the contents of the old partitions to the new ones. I did this using adb while being still in recovery-mode. The vfat-partition is easy as it just contains user-data. The ext2-partition however holds the cache (which can be erased without danger), and in particular it also has the app-directoy, in which ap2sd-apps are stored. So, I copied all contents file by file, rebootet the phone and it in fact bootet!

But something still seemed to be corrupted, as I was getting tons of messages like this:

Yes, it says "seachable" and using google you will find exactly nothing about this message.

I therefore tried some more things, like removing all files in dalvik's cache. By doing so and rebooting, I lost Google Maps and Titanium backup (I know the reason and its the file ownership). In recovery mode, I tried to fix uids and to repair ext2, everytime loosing even more apps.

Fortunately, I did make a nandroid-backup from the almost working setup (which was quite okay beside the database-warnings) and upon restoring it, recovery mode complained about md5 mismatches. At that point, I was lost.

I took the opportunity to play with my phone as it was messed up anyway. I even tried to re-apply the stock-T-Mobile ROM, because my phone has a little HW-problem which should be covered by its warranty. That didn't work, however, for some reason I no more remember.

Finally, I took an older nandroid-backup (including ext2), still stored on my old card, applied it, and voila: Everything is running smoothly now.

For you guys out there reading this it clearly means the following:

- Make a nandroid-backup (including ext2) before switching the card.
- Partition the new card using parted in recovery mode.
- Copy the contents of the old vfat-partition to the new one, including the nandroid-subfolders
- Apply a restore

So, yes, changing the card is easy, if you know how …

III. T-Mobile Pulse as a Router or Modem

For me, as a mobile worker, using the phone as a modem is vital. It is of course shipped with a Windows-application (Dialup), but the question was: How does that software work? Respectively, how do I use the device as a modem under Linux?

Don't get confused. As regards the question we're handling here, the phrase 'tethering' became quite popular, but in my opinion, it introduces too much misunderstanding. For me, tethering means to use the phone as a router, so that it forwards IP-packets and does some NAT on the outgoing interface (/dev/rmnet0). The internal interface can be wifi (where the phone either acts as a client or as an access point), an IP-interface over a serial Bluetooth line, or over USB. As regards BT and USB, we normally think of point-to-point-connections, yet we're not restricted to them. In all cases, the device can serve as a network access point, but that is beyond the scope of this posting.

Using the phone as a modem is something different, as by using this phrase, I really mean that old-fashioned AT-command-businnes with Hayes-compatible devices. But let's start with its router-capabilities.

Searching the Web leads to azilink, which works perfectly. The solution is to install an app on the Android, proving a non-encrypting VPN-server with NAT to which a host can connect (this may be called USB-tethering). This solution has one big advantage: It works independently of how the phone is connected. If a known WLAN-AP is reachable, it goes online using the access point rather than using GPRS. That is, at home, this setup just serves as a gateway to my LAN. And without any reconfiguration on my laptop, it allows me to use it as a router outside.

What is needed for this approach is that the device is connected via USB to the host. Then, one needs

The app is installed on the phone by either using the market or by pushing it from the host using

Establishing the connection is then done by starting azilink on the phone and to select check service active there. On the host, do

The connection can be terminated by killing openvpn with ^C.

As this approach makes the Pulse a router, I wondered about whether the phone is a simple Hayes-compatible machine as well. And actually it is. This can be deduced from sniffing the USB-port while "Dialup" (the Windows-app shipped with the phone) connects to the net. One clearly sees AT-commands passing the line. So, under Linux, the solution is to modprobe usbserial

(this establishes a serial line over USB) and that's it. If the device is, say, /dev/ttyUSB0, then you will see a modem there, and by using the following two scripts,

you can easily dial into your provider's network:

Technically, I prefer to use the device as a router rather than as a modem, but that is pointless, because I have a data-flatrate anyway. On the other side, I wonder why providers do it this way at all. All I can imagine is an accounting reason: In fact, your laptop gets its IP-address by establishing a PPP-connection to the provider using the phone's modem. And that of course differs from the IP your phone already has.

I am fine with that, because, if I do have networks to connect to, I can do that with my laptop without using the phone. The challenge begins where you are alone, with only your GSM-connection left. You can then use the internal modem via USB, as described. The Bluetooth-story is similar, but goes into a separate chapter (see below).

One final note: adb will not work when usbserial is loaded. Before adb-usage, remember to rmmod usbserial. In addition, if you get weired messages saying that the server cannot be started, restart it by typing

as root.

IV. SMS

Well. The point why I wanted to try Hayes-commands is exactly that I was hoping to send SMSs from my laptop or desktop rather than using the annoying phone's virtual keyboard. And that is perfectly working as well. As I've modprobed usbserial anyway, I only use this script:

Sending an SMS is then done by

in a shell. The crucial point here is that the phone needs to be informed about the SMSC (SMS Center) to use. For T-Mobile in Germany it is +491710760000, more information about SMSCs can be found using a search engine. It is worth to talk to the phone using some terminal as the error messages usually contain useful hints.

V. MMS

So that was about sending SMS. Using the built-in app SMS/MMS however does more in that it organises the messages by using threads. In those threads, it also stores short and multimedia messages sent by the owner. As we were sending with the native Hayes-functionality, we cannot expect our SMS to be monitored by the app. What we can do, however, is to pull out all messages from the phone and store them locally. This task is very easy, because all information is properly stored in the app's database.

This is the script I use to fetch all messages from all threads thereby keeping the order by putting everything into appropriate directories. Nothing special about it beside the fact that I needed to understand the relations between the tables first. And some recalc of the dates, stored in epoch-time, was necessary. Nothing special, so don't expect too much.

VI. Audio and Video

There is nothing special about audio and video. As regards audio, the device is capable of playing .mp3 and .ogg and probably a few more like .aac … I didn't test yet and I am not going to test as long as there is no need.

The more interesting part is the built-in (= T-Mobile provided) Music-App, which frequently scans the content of the sdcard and so is aware of all media-files.

This is interesting. The app precisely knows where your media files are, yet choosing an arbitrary file of them as a ringtone will most likely not work - if you can set it at all.

Point 1: The stock ringtones are stored as .ogg in /system/media/audio/ringtones (and the same is true for alarms and notifications). Choosing Settings -> Sound & display -> Phone ringtone will allow you to choose from Android System only, which is simply offering the contents of /system/media/audio/ringtones.

Point 2: Installing a 3rd party app like Rings Extended will allow you you to choose any file accessible by a file manager - but: It will not work. Doing so, logcat will claim that it cannot access the file, saying something like

The message is misleading in that it complains about a location /settings/system/ringtone, which of course is not the location of the file to be played but rather the entry in the system's settings settings.db. This database in turn is part of com.android.providers.settings and is located in /data/data/com.android.providers.settings/databases/settings.db. The database contains a table named system and an attribute ringtone which points to the media-file. And the issue is that if the entry does not point to /media/external/audio, then the above error message will appear.

This clearly is a bug (surf the net for this, the bug is documented), but the workaround is quite easy: Put your files to /sdcard/media/ringtones/, use a 3rd party app like Rings Extended and choose your tone from there.

And now video. Question was: What type of video (containers, codecs) is the smartphone able to play? Well, the format recorded by the phone when using the camera is something like this:

but that doesn't say much. The device is capable of playing MP4-containers with an AVC/AAC-multiplexed stream up to a size of at least 512×288 and probably beyond, but the physical resolution of the screen is 480×320, so anything larger than 480×270 and 384×288 for 16:9 resp. 4:3 videos is overkill, because the device must scale the video then. On the other side, compression will work better when using 512×288 for 16:9 frames. The phone is quite happy with several profiles (baseline, main, etc.). The following options provide very good quality of the video, but transcoding will not be done in real-time. Note, however, that CABAC must be disabled for the Pulse.

If the video is located anywhere in the LAN (and you have some player being aware of SMB-shares), the phone will not play it immediately and caches it first (files stored locally will not suffer this problem, of course). To start playing without downloading the whole file, it must be made "streamable", which means that hints need to be inserted and, even more important, it must not have b-frames. Note: It is not suffcient to tell x264 that B-frames must not be used as reference-frames (that is, not to use +bpyramid), instead, no b-frames are allowed at all. You can however leave +bpyramid in the command line, but then -bf must be added and followed by 0. To insert hints, there are various tools available, my preferred one is MP4Box:

The quality of the video is steered by the options -bitrate, -maxrate, and -qmax. This is not the place to discuss best possible parameters, if there is such a place at all. Common videocasts often show a bitrate somewhere at 250 KBit/s, but that is way too less for watching a video on the phone in high quality. Though it depends on the material which is going to be transcoded, I recommend bitrates at 400 - 600 KBit/s with qmax between 25 and 35 (max quantization). If no B-frames are used, the size of the video-file will increase (or the quality decreases).

If pre-processing of the video is needed, I use mplayer to do that. First named pipes are created, into which mplayer writes its processing (like deinterlacing, smoothing, and resizing) in raw format:

and then ffmpeg reads from the y4m-pipe using appropriate switches:

Here, I use a constant rate factor (crf) instead of setting the bitrate.

VII. Virtual Keyboards

I am adding this section because I came across some complaints saying that ConnectBot is not usable on the Pulse. In fact, the following screenshot (the one on the left hand side) shows that there is no "quick connect box". The second screenshot shows that, on addition, the virtual keyboard has no RETURN-key, so that one can enter an address, but that cannot be confirmed.

The solution is to switch to another keyboard. To do this, hold the touch-pad's button pressed while touching the virtual keyboard. Then, a set of options pops up, from which one chooses "input method" and select "Android Keyboard" rather than TouchPal.

The result looks like this:

which means that we now have a RETURN-key ("Fertig" at the right bottom).

VIII. Android VNC Server

We all know VNC, but on a phone it's really cool. It can be used to capture videos from what you do on the phone, but it can also be used just for fun. Get it from here, install it,

run it,

and start a client with

or, without adb-forwarding, use the address to which the phone is connected via WLAN.

The following capture has been taken a few days after I got it, so it's still German and pink (magenta).

Get the Flash Player to see this player.

(328×506 px, 4.5 MB / 110 sec = 347 KBit/s overall bitrate)

Let me say here that the option -t "/dev/input/event0" makes it possible to use the mouse on the client-side to behave like the phone's trackpoint. In addition, ESC maps to the device's BACK button, Android's Home is Home on the PC and F1 on the host is Android's END.

IX. Theming the Pulse (aka De-Magenta it)

Well, theming is possibly a too strong word for what we're going to do, because we will only modifiy the existing (stock-) theme. But that is always the first step of any theming-approach. I am also well aware that there is Metamorph out there, yet MM will not do what we're going to do now, for several reasons.

The very first step was to switch to English.

So, we are going to de-magenta the phone in the sense that we try to get rid of the annoying T-Mobile standard color. The theme is okay and I am quite happy with it … if there only was a possibility to change the T-color easily …

From what we're going to do here, setting the wallpaper is the easiest task. Taking into account the relatively dark background of the standard-theme (which I will not change), some green would please my eyes, so I searched the web for a nice wallpaper, found this, turned it to green and ended up with this:

Thanks to the creator Danny Cohen for this really nice wallpaper. Now, the standard wallpaper size of the phone is 640 x 960 (2:3), as can be seen from several drawables in the resources of the installed apps. However, what you get to see is only a part of the picture and that in addition depends on how your Home is setup. The standard launcher provides 2×3 desktops. That can be extended to 3×3 by placing icons appropriately, but I will stay with 6. And this means that from the 640 x 960 picture only the below sketched area will ever be seen (note: I am talking about portrait mode, as the standard launcher does not provide landscape mode).

So, I scaled the original wallpaper to my needs and placed it into a black template as above, and the upper middle and the lower left and the upper right desktops now look like this:

We now understand what the additional lines in the above picture stand for.

Now let's see. Everything seems to be green, but on clicking somewhere, T-Mobiles's branding pops up:

That's a clear no-go and we're now going to make it look like this:

What I am describing here is the manual (and seriously laborious) way. It will however show how to handle things in general and will lead us to even more complex problems, being interesting in themselves.

The resources an application uses is stored in its res-folder. As an example, we pull T-Mobile's standard launcher from the phone,

unzip it, and find the following icon in res/drawable/mainmenu_press.png

which is exactly one of the suspects. The question, of course, is: Can we just change it? And the answer is: Yes. The precise answer is: No, because it needs some work, but let's face it. Fortunately, we're not talking about an odex'd ROM (which I have) and we're not talking about signing applications! The task is in fact as easy as to replace the icon by another one, say, this:

Once this has been done, we repack the apk by saying

and transfer this zip back to the phone,

To tell the truth, this sounds easy, but the limited space in /dev/block/mtdblock1 (mounted as /system) probably needs a reboot, just like this: We first remove the file, then we reboot,

The reason is that even after removing the file, you'll be unable to replace it by the modified version, because although the file has been removed, no additional space has been gained. And this in turn is due to the fact that the file is still in use (as the application is still running). So, we could either kill the app or reboot. After that, as soon as the "android"-logo appears, do

This works, because the space occupied by the original file has now been released and we're ready to push the modded file. Keep in mind that if you're not uploading the file at this stage, then the phone will probably not start correctly (depending on the app).

Again: It is as easy as this. Difficult tasks like de-odexing services.jar are covered elsewhere. There also is no need to resign the apps, it's just this. There are only two things to remember:

Apps /system/framework/framework-res.apk and /system/app/Community_B116SP01.apk must not be compressed. This is the reason for the -0 switch in the zip-command above. Other APKs you modify can be compressed. By just un- and re-zipping (where zip is now told to compress), I was able to gain some more space in /system without any loss in performance. This can be done for almost all APKs, beside the two just mentioned.

While writing this, it makes me laugh when I am thinking of all those Windows-users, claiming to have the solution by saying that they did this replacement using their favorite zip-GUI … as if that would change anything. There is no need to resign, nothing serious to consider during the zip-process. However, there is zipalign out there and I strongly recommend to use it after all work on the .apk has been done. Again: Regardless of how you've modified an .apk, zipalign is the last step. See below for a command line.

Second, all files must be placed with the original filenames at the original location (links do not work, which is precisely the reason for app2sd's existence, which I will not cover in this post).

If everything is so easy, where is the drawback, then?

The first problem is that T-Mobile's magenta appears almost everywhere. On a stock Pulse, I find not less than 33 APKs tweaked by the vendor (marked with names like in the above example: …-B116SP01.apk or whatever). The drawables contain up to more than 800 icons and placeholders (per APK!) with tons of opportunities to change something. And now that could be changed automatically (gimp in batch-mode is your friend, see below), but then there still remain some severe issues and these are stretchable PNGs.

If you only change the hue of the icons and re-zip, you will end up with something like this:

The reason for this is that during the conversion of the icon, information got lost about how the PNG is stretched properly to fit a given geometry. This however only affects a subset of the modified PNGs. These are exactly the stretchable ones and they always have an ending .9.png.

You may want to start here reading about 9-patches. The solution is to use a tool from the Android-SDK, called draw9patch. The just converted icon must be renamed from icon-name.9.png to icon-name.png. You then open draw9patch, which looks like this:

A quick side-note: draw9patch is a Java-application and does not work with Java 1.5, it insists on 1.6. This is funny, because the Android-sources insist on 1.5 to compile correctly … but well, changing the default Java-version appears to be easy on today's distributions (see below under installation of the sources).

Fine, now you either drag your icon icon-name.png and drop it to draw9patch's desktop or open it via the menu, to see an icon to which a 1-pixel transparent border has been added. It is this border in which you draw your guides which indicate how an icon will be stretched. I am not going into the details here as the best method to understand draw-9-patches is to use the tool and to play around with it.

So, let us assume that we've added the above guides and saved the image as icon-name.9.png (the tool will automatically rename the icon and add .9 to the filename). Then another step is necessary (see below) and after replacing the orignal icon in the .apk, we end up with this:

Note that I did only convert icons which show T-Mobile's magenta, so the icon on the left hand side is the original one where the one on the right hand side is mine … Something went wrong and it's precisely the guides. The correct ones to use are:

And now we understand why the above original icon rendered so badly. That is, it renders badly only if the icon is stretched vertically, which is not supposed to happen, if the correct guides on the right and the bottom are added. After this correction, everything is okay:

When I say laborious, then it is because we're talking about approximately 100 stretchable icons which need new guides after conversion. Now, the situation isn't too complex, because most of the icons tweaked by T-Mobile are in fact the standard icons from the Android-sources, so the guides can be taken from there. For the remaining few, I added them manually.

What I did, however, was to take care of future modifications, so I stored the quides (only the guides) to a template-folder and add them automatically everytime I run my script to re-color the icons. This is possible, because the tool will help you on finding the correct guides, but all it does, finally, is to write a file with dimensions (orig-width + 2) x (orig-height + 2) with the quides included. So there is no need to use draw9patch everytime, because for inserting the modified icon to the template I just take ImageMagick:

Am I going to say that I've managed to fully automate the process? No, because what we get, rather than the desired icon (below left), is the one on the right hand side below:

Unfortunately, this is the point, at which we need to intervene manually, using Eclipse. The magic is to copy the just created icons (with the guides) to some Android-project's resources and to export the project to an APK. The so created APK will then include the icons with the (visible) guides removed! The structure of the patches is still included in the PNGs, but they look just as if they where "normal" PNGs.

So, create a new Android-project in Eclipse, locate its folder on your harddisk, copy your .9.png to the respective res/drawable folder, clean your projects in Eclipse, so that the new files are recognized, and export the project as an unsigned application (right-click on project's root, choose Android Tools -> Export Unsigned Application Package).

My desktop in action:

Get the Flash Player to see this player.

(640×512 px, 3.9 MB / 50 sec = 650 KBit/s overall bitrate)

This is a reasonable amount of work to be done, isn't it? I can now switch to a default-color I like within minutes. That's good.

The following APKs need modifications:

from which the following contain .9.png files

I probably went too far …

Here, we see that changing an icon's hue globally will affect nearby colors as well and so, not only the meganta play-button turns to green, but also the rest of, say, the music-icon. But in fact, only ShortCard_B116SP01.apk has this problem, so I needed to fix three icons manually (music, notepad, and picture):

For those of you being interested, this is the GIMP-script I use to modifiy the colors:

Just copy and paste the code to your editor and the correct formatting will reappear. The code converts the hue of the magenta-channel (6) to 60 and the red channel (1) to the same value, thereby turning them to green, which of course will also affect nearby colors, like in the example just given. However, the just mentioned icons are the only ones which show this problem, as all the others either contain magenta only or magenta with grey, for which the script does a perfect job.

To turn the colors into orange, I use

which now modifies the master-channel. Invokation is done by

where COLOR is green or orange (or whatever you name your code). Note that there is one line testing whether the image is RGB or indexed and if it is indexed, it turns it to RGB, allowing HUE-operations (btw.: it is this conversion from indexed to RGB which blows your framework-res up; mine went from 2.5 to 3 MB).

At this point, we've changed a few .apks and as announced above, we're now going to zipalign them, making sure that the directories in the ZIPs begin at easily accessible boundaries.

zipalign is part of the Android-SDK and located in the tools folder.

Okay, now that all icons have been replaced, then what is the following?

What we see here is that we cannot steer everything by simply changing icons …

So, let's have a more sophisticated look at APK-contents. You may have realized that beside the drawable-* folders there are many others inside an APK, e.g. layout. In general, what we are interested in is everything that influences the look and feel. This includes fonts, default colours, animations, etc. These informations are stored in various .XML files, which unfortunately are saved in a binary format (the technical phrase is WBXML, iirc) rather than in human-readable form. But there is a possibility to turn them into ASCII-XML using this code:

This will output the XML-contents to stdout. For example, framework-res.apk's res/layout/status_bar.xml contains, beside others, this section:

It says the obvious, namely that the status_bar's default text-color is 0xFFFFFFFF. This presentation differs from the usual 3-byte RGB representation in that it consists of 4 bytes. The additional byte steers the opacity and it is the first one of the quadrupel. So, 0xFFFFFFFF stands for white text, being fully opaque, and 0x00FFFFFF is white text being fully transparent. 0x7F0000FF is blue text, half transparent, and so on.

Of course this number will be found in the binary-XML as well, you only need to know where to look at. The reason is that the just decoded XML cannot be re-converted to its binary form. This is done at creation time by Eclipse.

So, the question is, where is the location of this resource:

In fact, this bar appears quite often, not only in the Meridia-player above, but also with YouTube:

The orange button in the right hand picture is framework-res/res/drawable/btn_media_player_pressed.9.png. I know that because I came across it and did not change it because it has no magenta - and the gray handles can be identified as well … so where does the color come from?

The answer is that what we see is the standard Android progress dialogue. It's the same on all Androids. And in fact, the drawable folder in framework-res contains some interesting XMLs:

There are some more, yes, but they don't contain color-information. Using the above ASCII-view, one quickly finds that progress_horizontal.xml defines three shades of magenta:

which, when tinted with (or hue'd to) green as above using gimp, produces this:

Therefore, the procedure is to hexedit progress_horizontal.xml and to replace all appearences of 0xC8086B with 0x94c808, 0x99004F with 0x709A00, and 0xFF128C with 0xBFFF13 (two times each).

Side-note: I am using ghex2 or okteta. If you do so as well, you will need to search for 4F 00 99 rather than 99 00 4F etc.

The same as above applies to the circular progress dialogue, for which the remaining 4 XMLs are responsible. They all contain a certain shade of magenta, namely 0xE60085, which I change to 0xB8E600. The result is this:

For my orange theme, color-mappings are 0xC8086B -> 0xC88B08, 0x99004F -> 0x9A6900, and 0xFF128C -> 0xFFB413 in progress_horizontal.xml and 0xE60085 -> 0xE68D00 in the 4 other XMLs.

What remains to do is to fix the browser's URL-bar (the background is magenta, of course), To do this, edit browser_progress_drawable.xml in the browser's res/drawable folder. Here, the color-mappings are
0x99004F -> 0x709A00, 0xFF128C -> 0xBFFF13, and 0xCD096E -> 0x98CD09. For orange it is 0x99004F -> 0x9A6900, 0xFF128C -> 0xFFB413, and 0xCD096E -> 0xCD8F09.

There is another file indicating a magenta, which is browser_seekbar_horizontal.xml. The color defined there (0xAE206A) will be changed to 0x88AE20 for green and to 0xAE8120 for orange.

At the first sigh, my phone now seems to be magenta-free and in fact, after playing around for a day or so I couldn't see any more magenta … until I realised this:

This resource is located in resources.arsc, which is again a binary (compressed) file, included in framework-res.apk. To read all resources of an APK, we take

That gives a rather long output, but we know what we're looking for. We're after something that steers the appearances of text, so let's lookup "text" and after a short search, we find the following section:

And there we are. Want to have a look at 0xFE118C? It's this:

Bah … My gimp-script will map it to this one:

that is, the annoying color 0xFE118C maps to 0xBEFE12 (green theme) resp. 0xFEB312 (orange). So, we modify the file resources.arsc using a hex-editor as above, by looking up 0xFE118C and changing it to the new value. Proof:

While we are at it: If you look for a HUAWEI-ROM which has no branding (no branding is the wrong saying, it has less), then go to their website and watch for ROMs they release. Say, go to here and look for ROMs. I cannot provide a static link, as they tend to change their download-area every day. In addition, bandwith is poor.

What if we need to change a resource that is not located in the ROM? Let's take ServoSearch as an example. I don't use it, but it is highly magentaised by T-Mobile. The application is located in /system/sd/app/com.db4o.servo.search.apk and tweaking it is performed as above. But contrary to the examples shown above, the .zip must be signed before deployment and before its installation, the old version must be removed. The steps to take are as follows:

keytool and jarsigner are part of usual distros. After that, do

Now, is it safe to say that the phone is now magenta-free? No. I know that there are still shades of magenta defined somewhere. An example is camera.apk. In res/drawable/camera_zoombar_* and in res/drawable-finger/video_timebar_* we still find magentas, but I haven't come across those in daily usage. On the other side, there is this, what you see while cropping a picture:

And I am not aware where this color is coming from.

Just as reminder for me (need to remember that I must try the following):

X. Android Sources and a Common Kernel

Initially, there was no reason for me to install the sources, but the limited BT-functionality of the phone forced me to do so. Here's how.

What I did first was to download the Android Sources together with a kernel. To do so, I created a directory android, went there and followed the steps at http://source.android.com/download. The last make will not work without further config: Android does not work with Java 1.6, but with 1.5. Java 1.5 is present on my machine, but I had to tell SuSE to use 1.5 as default. So, on SuSE, do

in a console and chose from the menu. Other distributions provide other approaches. I now got (staying in the same directory) the kernel,

Btw.: The entire repository is here.

The kernel is now located in common (a subfolder of android). According to source.android.com/download, a simple make will do the job, but it doesn't as long as you don't meet the dependencies. The current sources work with 32 bit tools, so make sure that not only the needed libs are installed but also their 32 bit variants together with the respective development-packages (aka sources).

With this setup, you can compile the kernel, modules and userspace-apps for the Pulse.

I did the complete compile and it took ages (= hours, so do that at night and go to bed). I also compiled the kernel, for which I needed the toolchain in prebuilt/linux-x86/toolchain (relative to android). I went to common (location of the kernel) and did

This will put an initial kernel-config to the kernel-directory. Cross-compilation of the kernel can then be done using the just mentioned toolchain

Here, it is very important to change the config as follows: Goto System Type and choose system type to be Qualcomm MSM7X00A. This must be done for the Pulse, because it is an ARMv6 system. The differences between v5 and v6 may be small, but for kernel modules, the system on the phone will complain about the wrong system number and deny to insert a module.

The correct magic string is 2.6.27-perf preempt mod_unload ARMv6.

I then could make my own kernel using

For example, in menuconfig I marked bnep.ko to be compiled as a module for insertion to the kernel on the phone.

XI. Bluetooth

I went this stony way, because I came along a tutorial saying that one needs bnep.ko as a module and pand and iptables as binaries. What I realised was that my MCR-ROM from MoDaCo already has bnep.ko built into the kernel. Same for iptables, which came together with the ROM. I only needed a binary copy of pand, which I compiled as mentioned above.

That is, to be precise, the tools in external/bluetooth/bluez/compat (paths are always relative to the Android-sources) do not compile out of the box, because there is no Android.mk located in that directory. That needed to be created first.

To begin with,

worked of course, so I started pand on the phone

and on the host I did

to give, beside other information,

Next on the host I did

This action caused both the phone and the host to provide a network interface named bnep0. Perfect! For a detailed description of pand see here. It is usually contained in the bluez-compat-package of your distribution, but, as said above, for the phone I needed the compiled binary.

Now, of course I could set the devices up, giving them IP-addresses and let them communicate, so that the host and the phone are connected, but the phone would still not forward IP packets and will not do masquerading.

As we all know, making a Linux-box to be a router first requires

and iptables. What we normally do is to

and it is exactly this what is failing. Trying to run iptables gives

which simply means that ip_tables.ko has not been loaded (see also here). The problem is that it cannot be inserted. As I can make my own modules, I built ip_tables.ko, but dmesg complains again:

and some more. The solution is of course to resolve the dependencies, but insmod x_tables.ko immediately forces the phone to boot. Strange? Well, post #5 in this thread reports about this problem as well, so that seems to be a difficult one.

If Kernel-NATing is not available, then what about userspace-NAT? There are two solutions I am aware of. The first one is a tool called minewt, project page is here (though site is down) and it's downloadable from here. I did not manage to integrate it into the Android-sources, because the dependencies are a pain. The tool is delivered with an own (and quite old) version of libnet, libpcap, and tomcrypt. As libpcap and tomcrypt are part of the Android-sources, I tried to install a newer version of libnet first and to compile minewt against it, but that failed with tons of errors.

The same is true for vde (virtual distributet environment), which is the second method to do NAT in userspace (though vde provides a much more general approach): I gave up translating the Makefile to an Android.mk, again due to too much dependencies.

We're not talking about one-click-solutions anyway, so let's have a look at a possibility to get the mentioned tools to work. This is provided by installing Ubuntu on the phone (see below).

Once within the Ubuntu-environment, I installed some development packages like gcc, make, bison, etc. And then I went the usual way to compile a package, namely, via ./configure and make. Both mentioned packages compile without any probem (though I needed to meet some more dependendies). The build-process for minewt in fact unpacks the shipped libraries (see above) and compiles against them and, at least, the binary starts.

Now, the general usage of minewt is as follows (and you're running it from within the chroot-env).

provided the tool can figure out the routers (192.168.1.4) MAC-address itself. So, let's assume that we have one standard-interface over which the phone is connected (this is either rmnet0 for GSM or eth0 for WLAN). Let us also say that as above, bnep0 is our BT-interface, which we got by running pand. Then the above command will serve as a NAT-gateway for connections coming in from bnep0. By running tcpdump against eth0, I figured out that the destination-MAC of the Ethernet-frames consists of zeroes only, thereby being cancelled. It is exactly this what made me try to add the MAC to the command-line:

And the only answer from running the tool was bus error. Well, the source of minewt is a single C-file, so I had a look at it and hardcoded my router's MAC. The bus error was away, and minewt actually did some NATing! Yeah!

Unfortunately, it's not useable this way. A simple ping shows up to 70% packet loss, response times are two seconds and even more and you can clearly see from the output of minewt that it drops more than 70% of the requests.

Going with vde revealed the same. In principle, the three lines

should do the job, but as soon as a packet arrives at bnep0, slirpvde fails with - guess? - a bus error.

I mean, that's interesting. It seems as if everytime the actual NATing is being done (must be done for both the IP and the MAC, of course), both tools fail with exactly the same message. strace does not show any problems - until the bus error.

At this point I am really interested in compiling the code for the native Android (not chrooted), hoping to see something more … say a reboot?

XII. Accessing the internal Modem with Bluetooth

So that was about tethering again, this time BT-tethering (didn't I say I don't like the wording?). Again, I am not really sure about why I should tether at all. If my phone can connect, my laptop can do so as well, so where is the benefit? For me, the wireless solution would be to establish a serial line via Bluetooth and to use the phone as a normal modem afterwards. That's fine in the sense of convenience. Most of the time, the device will be connected USB anyway, you know, for charging reasons …

Starting pand is good, but, as mentioned, needs NATing. There is another daemon, dund, which serves as a dial up server (PPP) rather than an access point - with the same drawback: It starts pppd (if present on the system), lets you connect and provides an IP, and here, you need iptables again.

Now, as we know that there is a modem, why don't we use it? Well, the first question is: Where is it?

Answer: The device's internal modem is at /dev/smd0.

This is the way to proceed:

Install rfcomm on your device. The command is part of Android's source (see below). On the phone, say

This creates a device /dev/rfcommX (where X is the device-number and Y is a channel number) and to which a host can connect to using

where now Z is the device-number (/dev/rfcommZ) choosen on the host and <bt-address> is the phone's address. A serial line is established, without any further traffic.

The idea is to bridge /dev/rfcommX and /dev/smd0, hoping that the host will then see a modem on /dev/rfcommZ. Bridging two serial ports is easy, as we only need to cat the output of one device to the other, and vice versa, just like this:

That in fact works, but automatic handling is complicated. See below: We want rfcomm watch to start our bridge and to cancel it after termination of the line. This can be done, of course, but needs some special care of which cat to kill when, including pid-handling. There is another way:

While watching the system-load being online via USB, I realised that such a bridge already exists. It is /system/bin/port-bridge and is started by init as a service:

Obviously, the tool takes Unix device nodes as arguments, and clearly, the service is used to bridge the internal modem with the USB serial port. So, I did the self-evident

which gives

and in principle, this suffices to see a Hayes-compatible modem on /dev/rfcommZ on the host. This in fact works somehow, but using a terminal like minicom or kppp appears to be very slow and unreliable, and often enough, characters just disappear.

The problem is the already running bridge for /dev/ttyHSUSB0. That is, we now have two mappers running, both delivering the output of /dev/smd0 to their targets (/dev/ttyHSUSB0 and /dev/rfcommX, respectively). As my phone was attached to my host's USB as well (for adb-reasons) I could in fact see that the traffic arriving at the USB-port did not make it to /dev/rfcommZ, and vice versa..

The obvious action to take is to shutdown the running bridge, but that is not possible as it immediately gets respawned by init. That could be changed of course, but I wanted to be flexible, so I went another way, moved the binary /system/bin/port-bridge to /system/bin/port-bridge-orig and killed the process.

On the phone, I then ran

Option watch tells rfcomm to restart listening when a connection is finished. As soon as the host connects (see above), rfcomm creates the device node /dev/rfcommX and invokes /system/bin/port-bridge-orig /dev/smd0 /dev/rfcommX.

I was testing this while the phone was still attached via USB, so as a final test, I unplugged the device, and the bluetooth connection could no more be restarted.

I don't have a source code of the bridge, so I started it with strace (yet another binary to compile using the SDK) and that clearly revealed the cause:

Setting the state manually with echo "USB_STATE_CONFIGURED" > /sys/devices/platform/msm_hsusb_periphera/state does not work, as the state is corrected immediately by the system.

The solution is to modify the binary with a hex-editor and to replace the location
/sys/devices/platform/msm_hsusb_periphera/state with a string of exactly the same length, I choosed /data/local/tmp/fake_sys_dev_platf_msmusb_state, saved the program as /data/local/bin/port-bridge-fake, and echoed the correct state (USB_STATE_CONFIGURED) into the file fake_sys_dev_platf_msmusb_state. The complete procedure is as follows, assuming that /system/bin/port-bridge has been copied to /system/bin/port-bridge-orig before (not moved, as above).

A few more notes: The first line unpairs my headset (of course, dbus-send is not part of a stock or customized ROM, I compiled it as I did for the other binaries, like rfcomm, sdptool, etc.). If I don't unpair first, an incoming call will break both the headset's and the host's connection. If I unpair first and pair the headset again after the host and the phone are connected, everything works dandy. I think I know the reason for this, but have not yet found the time to investigate. The switch -A tells rfcomm to ask for authentification, which means that the PC and the phone get paired after both provide the same passkey.

Then I took 17 as the channel to communicate, hoping that it is ununsed by the system. It is, as can be seen using sdptool browse on the host and this in turn also means that we need to tell the phone's bt-subsystem to offer an appropriate service (last line).

Turn off:

Dial-in on the host is then done as in the USB-case (see above), now using /dev/rfcomm0 instead of /dev/ttyUSB0.

A final comment: Data rate is very poor with BT. A dial-up provides an approximate
average of 40 KB/s (with rare peaks at 60 KB/s) downstream, where the USB-connection is 10 times faster (stable 2.5 Mbit/s). What makes it even worse, are parallel phone calls using a BT-headset. The voice-connection alone eats up to 35 KB/s, allowing the online-channel to download with not more than 5 KB/s. However, the connection stays stable and recovers after the headset-channel has been terminated. This setup works perfectly for Windows as well, of course (just setup a BT-modem).

I went the above way with Ubuntu 9.something and SuSE 11.1 (64 Bit). Be careful: 11.1's bluetooth is broken (not seriously, but broken), that is, something is wrong with either bluez4 or bluez3 … I don't know and I don't care.

The specific problem with SuSE 11.1 is that the small BT-icon in KDE's (and Gnome's) task bar does not do anything. Okay, that is better than what WinXP does, namely talking too much, but I need the icon for pairing. If I unpair the connection to my host, an attempt to go online will force the phone to ask for a PIN, and so is the host expected to do - which doesn't work.

Pairing is stressing my nerves anyway, whatever OS, so all I've ever admired was something scriptable; and there is. The name of the tool is simple agent, which is pure understatement. It can be found at git.kernel.org. Do this on the host:

and in another shell just say

and watch it asking for the PIN. Let's be smart and close our connection when done (i.e., after killing the above pppd):

Btw.: When I say 'scriptable' then I am aware of that simple agent requires interaction. I use expect for full automation.

XIII. Eclipse and Android-SDK

In order to install applications, one needs a development-environment, and for Android, this is Eclipse with a plugin. This section is about setting it up.

Before using Eclipse for Android, one needs to set it up correctly. Start it, goto Window -> Preferences -> General -> Capabilities and enable CLassic Update.

We are now going to install the ADT-plugin, see also here. To setup the remote site, goto Help -> Software Updates -> Find and Install -> Search for new features to install and click on New Remote Site.... Enter a name (Android, say) and as URL https://dl-ssl.google.com/android/eclipse/. If that doesn't work, use http rather than https. Then click on Finish and restart Eclipse.

When starting Eclipse, it looks like this (in my case):

Go to Window -> Preferencies -> Android and provide the location of your local SDK:

You'll see this:

Connect your device via USB and got to Window -> Open Pespective -> Other.

Choose DDMS and watch your device being recognised. At this point, you can also take screenshots.

Unplug your device,

and choose File -> New -> Project:

choose Android Project and provide your details,

and you are here:

Watch your code,

and resolve problems like "R cannot be resolved". In this case, load the class from com.zarrabi.first after clicking on the red icon.

Now click on run, choose Android Application,

and a virtual device pops up and boots:

It is now ready,

and shows the installed applications, including the just deployed small app.

While the virtual device is open, watch again the DDMS-view to see that it now shows this phone instead of the real one before.

XIV. Ubuntu

Installing Ubuntu on the phone is very easy thanks to Paolo Sammicheli, because we are already running a Linux-Kernel, so all we do is to unpack an already compiled chroot-environment. Go here, download, unzip and start (instructions are given on the project's website).

I am not backing up Ubuntu with nandroid, as it wastes too much space on /sdcard/nandroid. Instead, the backup is stored on a host.

Start TightVNC on the phone:

And watch on the client:

Beware, starting X exhausts the phone's capabilities in terms of performance.

XV. MoDaCo's Android 2.1 v6r1 leaked beta longstory ROM

Again: T-Mobile provides a 2.1-version (see above). The following has been written while only a leaked version was available, so the information is outdated. I will keep it for documentation reasons.

As of this writing, it's here. The forum discussing it is here, so you will better go to the second link first.

I started with a smoothly running Pulse, almost completely de-magentaized and tweaked slightly as described above (Bluetooth). I first backed up my existing nandroid-backups by saying the follwoing, and I also removed dalvik's cache:

One small side note here, although it's offtopic: Removing the Dalvik-cache is not only harmless, I strongly recommend to do so from time to time (reboots are a good opportunity). If you encounter problems with the market, for example, then go to the phone, remove all files in /system/sd/dalvik-cache/ and reboot. The cache will be rebuilt, of course, and you can install market-apps again.

I left sdcard's vfat partition unbackuped, pushed the ROM, and went into recovery mode:

There, I did a full nand and ext backup:

Then, I powered the device off, took out the card, inserted it into my card-reader and dd'd the content to my PC, just in case something goes wrong. And remember: I did not perform a backup of the vfat-partition.

Turned the card back into the phone, started it in recovery mode. There, I went to wipe, and choosed data/factory reset and ext2 wipe.

go to "Flash zip from sdcard" and choose r4-2.1-pulse-signed.zip", reboot afterwards and expect problems.

At this time, I was able to adb to the phone and saw it doing something (top, ps ) … but there was no locat and the device got stuck in android. The reason was unknown to me (because there was bo logcat to study), but a manual mount revealed that there was no /system/sd folder, so I created one saying

The boot-loop was away, the circular t-mobile-logo appeared, but still no logcat. The devices boots okay with only very few force closes. I found it quite stable. Device id changed to vendor=0x12d, product=0x1502 from vendor=0x12d1, product=0x1501 before, so needed to rescript sendsms etc.

Issues: No logcat, no vibration, no ddms (so no screenshots).

The logcat-issue could be resolved: Look at /sys/kernel/logger/log_main/enable. It returns 0, so set it to 1. Also look at /sys/kernel/logger/log_main/priority. It returns 0 as well, set it to 2. Now logcat works. The information provided in this blog is from here.

What also changed is that the green button goes to call log. All comments I saw relative to this complained about the behaviour, but I like it, as I got used to this by my old phone.

On tweaking APKs as above, I found that Contacts must not be compressed (replaces Community in 1.5).

Being at 2.1, I tried PDANET as it promises to do userspace NAT (see above). After pdanet has started, it offers a dial-up service on channel 3. So, on the host do

and then do

where /etc/ppp/peers/pdanet is just like above for gprs (USB) with noccp uncommented (important!). The chat-script contains

The results where pretty much the same as for minewt and vde (see above). Most of the packets are dropped and the longer I think about it, the more I am convinced that this is a problem of routing between the interfaces. Contrary to kernel-NAT, userspace-NAT does (probably, I don't know about pdanet's internals) have the disadvantage that the kernel is not aware of that incoming packets in response to a previously NATed packet are valid and therefore drops them. I will perhaps dig into this later.

The short investigations I took also showed that the port-bridge still exists and that I can go online as descried above. However, all theme-tweaking from above must be done again, as the icons have changed dramatically. I began to de-magenta this 2.1-version but decided to stay at 1.5 for now (for other reasons), so I stopped making the phone green.

And this is basically all I tested with 2.1. I found it quite fast and responsive, but when heavily loaded, using the graphical interface is a pain. Better not to use live wallpaper (hehe).

I went back to 1.5. Before, I initiated a nand backup of 2.1, and in fact, by just restoring it I can quickly switch to 2.1 again for testing purposes, but I am running 1.5 for now.

As an official 2.1 is out now, I frequently watch MoDaCo's forum and will definitely switch to it later, but that will then go into a separate posting.

Als Zuspieler kann man den MR300, der seit einem Jahr bei uns herumsteht, nicht gerade bezeichnen. Zwar ist sein Innenleben einigermaßen hochwertig und sollte in der Lage sein, alle gängigen Container und Codecs abzuspielen, aber sein Hersteller hat die Box ordentlich dicht gemacht. Es gibt Projekte, die sich mit dem Flashen des Geräts beschäftigen, aber mir fehlt einfach die Zeit für solche Frickeleien. Wenn man die Box also nicht anfassen möchte, muss man ihr geben, was sie immer abspielt, nämlich einen MPEG-Transportstrom; und das ist mit relativ wenig Aufwand hinzubekommen.

Auf meinem Gateway, das über einen SP 300HS am Internet hängt, läuft ein igmpproxy und macht das System zu einem Multicast-Router. Das ist wichtig, denn natürlich ist die naheliegende Lösung zwar, einfach einen eigenen MPEG-TS ins Lan zu streamen, aber da man dem Media-Receiver nicht neue Adressen nennen kann, auf denen er lauscht (die Programmliste bekommt er regelmäßig von T-Home), muss man einen der Multicast-Streams auf dem Gateway blockieren. Die einfache Zeile

verhindert für die Zukunft den Empfang von "rbb Brandenburg". Genauer: Bei Anwahl des Senders erscheint das Programm für die 10 Sekunden, die es über unicast hereinkommt, und friert dann ein - wie gewünscht. Ich habe zunächst auch die Quelladresse mit angegeben, weiss aber heute, dass die irrelevant ist (und sich auch öfter mal ändert).

Nun ist es so: Ich besitze einen Billig-Switch, der nicht IGMP-fähig ist, weswegen hereinkommende Transportströme das gesamte Lan fluten, aber selbst bei 2 verschiedenen HD-Sendungen und einem SD-Kanal, die, wenn ich sie gleichzeitig empfange, meinen VDSL-Anschluss mit 25 MBit/s fast auslasten (2 x HD + 1 x SD ergeben etwa 20 MBit/s an Traffic), verhalten sich alle Geräte normal und auch die zwei Mediaboxen kommen mit der Flut klar. Da muss man doch im Lan einen TS an die MCast-Adresse 239.35.20.15 streamen können, oder?

Das Naheliegendste ist natürlich den VLC zu nehmen. VLC kann prima an Multicastadressen streamen und in der Tat zeigt ein zweiter VLC im Lan ihn auch an. Nur MR300 und X300T sind unbeeindruckt. Erster Gedanke: Es könnte an den Programm-IDs liegen. Es könnte überhaupt daran liegen, dass der vom VLC erzeugte TS nicht das ist, was die Box erwartet. Lösung: Einige Sekunden den "rbb Brandenburg" aufzeichnen und per VLC stream, dann sollten ja zumindest die Stream-Parameter stimmen. Kein Erfolg.

Was macht man in so einer Situation? Man sieht sich mal den UDP-Strom an. Und zwar kann ich vom Gateway aus bequem beide Interfaces (eth0 und vlan8) sniffen. Zwar dringt der Strom von vlan8 nicht zu eth0 vor, aber er kommt ja trotzdem an, weil der igmpproxy der entsprechenden Gruppe beigetreten ist. Der Unterschied ist einigermaßen umfangreich:

Zunächst unterscheidet sich natürlich die Source-IP. Das kann man auf dem streamenden Host localhost ändern mit

es ändert aber nichts. Die Mediaboxen zeigen nichts an. Es unterscheiden sich auch einige Flags, wie das DSP-Feld des IP-Headers, das beim Originalstrom den Wert 0x80 hat. Und es gibt noch einen interessanten Unterschied. Beim TS von T-Home wird im RTP-Header das 3. und 4. Byte bis 0x8000 hochgezählt, auf 0 zurückgesetzt und wieder hochgezählt, usw. Ich nehme an, das dient der Synchronisierung zwischen dem Unicast- und dem Multicast-Stream, aber das ist reine Spekulation. Jedenfalls leistet der VLC das nicht, bei ihm ist der Wert immer 0×00.

Am auffälligsten ist allerdings die SSRC (Bytes 9 - 12). Die ist nämlich immer konstant, so wie im Standard vorgesehen, und im Originalstrom immer 0x1501F734. Beim VLC ist sie von Sitzung zu Sitzung anders (nämlich zufällig), aber eben wieder konstant. Ich habe natürlich keine Muße, den VLC neu zu kompilieren, auf der anderen Seite ist er wegen seines Interfaces auch von Remote zu bedienen und er ist flexibel, was seine Formate angeht. Also soll der VLC nur den Transportstrom erzeugen, gestreamt wird anders.

Zum Streamen benutze ich testMPEG2TransportStreamer aus den testProgs von live555. Dort ist im Sourcecode die Multicast-IP, auf die gestreamt werden soll, fest auf 232.255.42.42 gesetzt, was ich einfach geändert habe auf 239.35.20.15 (und etwas weiter unten die Portnummer von 1234 auf 10000). Die SSRC ändert man in liveMedia/RTPSink.cpp von

auf

Und dann kann es auch schon losgehen:

wobei stream.ts gerade der vorher gedumpte Originalstream ist. Ergebnis: Läuft! Also, mit Einschränkungen. Man wählt den Sender auf der Box, dann kommt der Unicast, dann friert das Bild ein, dann flackert es, aber irgendwann läuft es schön stabil.

Nächster Test: Streame auf diese Weise die Aufnahme eines anderen Senders, z.B. Arte HD. Läuft. Dann also richtig: In einer Shell wird der TS gestreamt,

in der anderen schreibt VLC in die Pipe:

und herauszufinden ist nun, welche Formate MR300 und X300T akzeptieren.

Das natürliche Format ist ein MPEG2-TS (.m2ts) als Container mit einem h264-kodierten Video- und einem ac3/a52-kodierten Audio-Strom. Es ist "natürlich", weil dies die TSe sind, die von T-Home kommen. Solche Dateien kann man also z.B. und wie erwähnt erhalten, wenn man einen (nicht verschlüsselten, also öffentlich-rechtlichen) Sender aufnimmt. Vollkommen unproblematisch sind auch MP4-Container mit einem h264/aac-gemultiplexten Inhalt. VLC muss diese nur in einen TS muxen, was kaum messbare Last erzeugt. Genauso stressfrei ist ein TS mit mpeg2/ac3- oder mpeg2/mp2-Kombinationen.

Ich habe zunächst die Auflösungen getestet und festgestellt, dass die Boxen mindestens alles abspielen, was als Standardformat daherkommt. Das heisst, es ist unerheblich, ob die Auflösung nun 720×576, 1280×1024, 1920×1080 oder auch 576×432 oder noch exotischer ist. Wenn der VLC sagt, gewisse Auflösungen seien für gewisse Container nicht zulässig, dann verstößt man zwar gegen den Standard, aber die Boxen spielen das Material trotzdem ab. Es ist auch egal, ob es sich um i50, p25 oder um p50 handelt. Es gibt dennoch einige Restriktionen und Spezialitäten:

Erstens sieht der mp4-Standard kein ac3 als Tonspur vor. Ffmpeg weigert sich auch, eine avc/ac3-Kombination in einen mp4-Container zu schreiben. Mit avidemux2 geht es aber (mit Warnung) und die Mediaboxen spielen den Strom auch ab - klar: es ist das Format, was sie normalerweise erwarten.

Zweitens verstehen die Boxen keine hochauflösenden mpeg2-Ströme. Das ist nicht ganz unwesentlich. Ich werde natürlich nicht alle meine Videos nach mp4 kodieren, nur um sie streamen zu können. Das bedeutet, dass proprietäre Formate wie wmv am besten on the fly kodiert werden sollten, aber das Transkodieren nach h264/aac kann ich mit meiner Hardware nicht in Echtzeit durchführen. Solche Dateien kann ich also nur nach mpeg2 konvertieren und wenn diese in HD vorliegen, muss ich skalieren. Unschön, aber wenn es wirklich stört, kann ich immer noch vorher eine Nicht-Echtzeit-Kodierung vornehmen.

Drittens, und das ist der wesentliche Punkt: Die Mediaboxen sind enorm empfindlich, wenn es um ihren DPB (decoded picture buffer) geht. Dieser ist sehr klein und verbietet damit eine hohe Anzahl an Referenzbildern bei gleichzeitig hoher Bitrate. Dies hat nicht direkt mit der Auflösung zu tun, aber bei Bitraten um 3 Mbit/s kann man bis zu 7 Ref-Frames benutzen, bei Bitraten um die 8 MBit/s hingegen nur noch maximal 3. Ich rate, HD-Material evtl. etwas mehr Bitrate zu spendieren, dafür die Referenzframes auf 3 zu begrenzen.

Viertens: Die Boxen mögen keine B-Frames als Referenzbilder, man benutze also nicht -flags2 +bpyramid als Option für ffmpeg, wenn man eine Videodatei fürs Streamen vorbereitet.

Fünftens: Wenn man irgendein Video nach mp4 (avc/aac) kodiert, dann sollte man dieses in einem Rutsch übersetzen, meint, man wähle nicht 2-pass. Der Vorteil ist, dass man bei Angabe einer durchschnittlichen Bitrate keine großen Sprünge derselben erhält - und das ist gut: Streaming-Clients sind an diesem Punkt sehr wählerisch. So wundert es auch nicht, dass T-Home bei 720p bereits mit 8 MBit/s streamt, was eigentlich gar nicht nötig ist, weil bei 720p wesentlich kleinere Durchschnittsbitraten möglich sind.

Die verbreiteten mkv-Container sind ein einziges Ärgernis, weil sie Kombinationen weit ab vom Mainstream erlauben. Das ist schön für den Austausch von Videos, aber Streaming-Clients sind anspruchsvoll. Ich habe keine Möglichkeit gefunden .mkv in Echtzeit nach m2ts (ob mit mpeg2 oder avc) zu kodieren, weswegen man hier auf jeden Fall vorher übersetzen muss. Ich empfehle avidemux2, um mkv ohne Rekodieren in einen mp4-Container zu muxen, denn avidemux2 erlaubt auch, wie erwähnt, ac3 in den Strom einzubetten.

Was die weitverbreiteten avi-Files angeht, hilft nur ein Blick in ihr Inneres. Der Container ist nämlich kein Problem (im Gegensatz zu mkv mit seinen Clustern), aber ob nun ein vorheriges Umkodieren nötig ist, kann man so allgemein nicht sagen. In aller Regel ist das der Fall, weil die am häufigsten verwendeten xvid-Inhalte von den Boxen nicht verstanden werden.

Das gilt kurioserweise auch für wmv. Kurios deswegen, weil MR300 und Co. Windows-Büchsen sind. Wenn also ein Transkodieren in Echtzeit nötig ist, dann gehe ich so vor:

Sind die T-Home-Boxen also HD-Zuspieler? Ja bzw. Nein, je nach Sichtweise.