This is something i posted on Google+ , and shared via ifttt :

This is something i posted on Google+ , and shared via ifttt :

Well, it’s that time of year again – my annual geek-week at linux.conf.au. Every day there were many interesting talks to attend and many I had to miss. I am currently catching the ones I missed by watching the LCA 2012 videos.

Keynotes and Open Source DSP

The Bruce Perens keynote had many good points on why open source is becoming essential to our security and well-being in the 21st century. These themes were expanded by the other keynote speakers. Bruce stated that “open software is the only credible producer of software”, we can choose to be “slaves to tools or the people who control the tools”. Watch the talk for more information on these memes.

I am interested in “the art” of presentation (as distinct from the content) so was also interested in Bruce’s presentation style from that perspective. He appeared on stage in a suit, which is uncommon in a geeky crowd. It raised a few eyebrows. However Bruce then explained that “a suit at LCA” was a theatrical device to underscore the point that we, as an open source community, should be facing outward. Open source communities are good at talking to people within our community, but our image outside of that community is poor and misunderstood (e.g. not many people understand their email is relayed via open source, or how it helps security problems and can help preserve democracy). Our external image must be improved.

Another great point by Bruce was how Open Source is now solving tough, previously opaque problems that were traditionally considered too hard due to patents or specialised knowledge. All you need is one guy to really get into and understand the problem. Suddenly the voodoo evaporates. People then know it’s possible – a problem that our peers have solved always appears easier. This one guy publishes source and shares what he has learnt. Others start to hack the code. Bruce, much to my embarrassment (!), actually cited my DSP work as an example, for example Oslec for echo cancellation and projects like Codec 2. When Oslec started I had many people tell me “it can’t be done”, “you need a DSP chip to do it in hardware”, or “it’s all covered by patents”. The truth is that Open Source DSP algorithms are now out performing closed source competitors. For example the Opus guys have developed a world-beating open source audio and voice codec. More on that below.

Actually I really enjoyed all of the keynotes. Paul Fenwick spoke on how our mind works, including topics like “decoy choices” and the “planning fallacy”, and how playing Tetris can help with traumatic experiences. I also recommend you watch the keynotes by Jacob Applebaum and Karen Sandler.

It was great to see Jean-Marc Valin and Timothy B. Terriberry in person, presenting on the Opus Codec. I wonder if this will be the “last” audio codec. It’s open source, royalty and patent free, will be an IETF standard, codes speech and music signals from 6,000 bit/s up, and outperforms other codecs like MP3.

Codec 2 talk

As I mentioned above I am interested in how conference presentations work. Like a lot of my work, I am inclined to experiment. Try something different. Hack it. A presentation on Codec 2, like many presentations at LCA, has a strong technical component that is hard for the average LCA attendee to follow. My Codec 2 work is an extension of my PhD research in speech coding. It took me 3 years to get my head around speech coding for the PhD. So how do I communicate Codec 2 topics to a smart, but non-speech-coding aware audience?

One way to handle this is “tutorial” style. You spend about half the talk bringing people up to speed on your technical topic. Enough to explain in the second half how you applied Linux or open source to this field. This is a common approach at LCA. It can work well, but also means a lot to absorb for the audience. This can be a challenge after a day (let alone a week) of great ideas and intellectual stimulation at a conference like LCA.

Instead of the tutorial approach I hit on a different idea. Rather than confine my talk to Codec 2 and DSP theory, I tried approaching Speech Coding from a variety of tangential topics that matter to LCA attendees. I talked about codec patents, how Ham radio relates to Open Source, and finally a really easy to grasp graphical explanation of how the sinusoidal model used in Codec 2 works. I left out a bunch of DSP topics, and didn’t even put up a block diagram of the codec. I wanted the audience to walk away knowing 3 or 4 things about speech coding really well, rather than try to cover the entire, technically deep, acronym rich subject at a shallow level.

This worked well. Really well in fact – my Codec 2 talk was voted the best of the conference and I was asked to repeat it later in the week. Wow! This was especially amazing for me as the voting is done by the attendees. A nice way to start 2012 for me, after working through some tough personal issues in 2011. Here are the slides for my Codec 2 LCA 2012 talk in Open Office format.

How Good Was Your Conference Talk?

It is important to me that my talks are well received. For me it’s part of my job and I take it seriously. Here is what I look for. This applies mainly to conferences with multiple parallel threads where people have a choice in what they attend:

• Did I fill the room (or nearly so)?
• Were people still asking questions at the end of your allocated time? Extra points if people come up to you afterwards and ask more questions. Even better if you get hustled off by the conference organisers because the next speaker is overdue to start.
• Did some of the more popular speakers/major contributors to the conference attend the talk?
• Was the applause loud and enthusiastic?

Oh, and I also like to make my talks short and leave more time than usual for questions. For example in a 50 minute slot I will time my talk to be 30 minutes rather than the nominal 40, allowing 20 minutes for questions. I feel strongly that the audience should drive a good chunk of the talk through their questions. This feels much better to me than running over time and not allowing enough time for questions.

21 Second Lightning Talk

Lightning talks are a fun part of LCA. These usually last 5 minutes. I had an idea for a lightning talk on my Electric Car that I wanted to try. I figured I could get my talk done in 10-15 seconds. Yes, I was experimenting with presentation styles again. I wanted to use lots of slides connected with just a few words (normally we are encouraged to do it the other way around). This year I managed to get a lightning talk slot and presented the talk. You can see it on the lightning talk video starting at 50:20. From when I start to when I stop talking is 21 seconds, not quite the sub-15 seconds I was aiming at. I always was a bit talkative. However the applause was pretty loud so I think the idea worked!

Codec 2 Hacking

While at LCA Jean-Marc did some great LSP vector quantiser work for Codec 2 and explained some of the techniques involved. This was very useful, and will be part of Codec 2 soon. Thanks Jean-Marc!

At the end of the conference Bruce Perens (left), Timothy and Jean-Marc (right), came to stay for a few days at Nuria’s (centre) house. It was really nice to have them all, we did some good work on Codec 2, and the dinner-time conversation (fuelled by Nuria’s fine lasagne and BBQ) was fascinating. As Bruce pointed out, there is great value in the small number of open source speech coding guys meeting face to face.

I was a bit nervous travelling in the same car with Jean-Marc and Timothy. People working on open source voice codecs are rare – so we figured we had 60% of the world’s open source codec guys in one car!

All my linux.conf.au 2012 photos have been uploaded to Flickr, within a week of the conference. (that's gotta be a record! :-)

At Icelab, we love Markdown. We use it wherever we can for text formatting. In a web app, the obvious place for it is in large text areas, where we can allow complete freedom of formatting. Headers, paragraphs, lists, it’s all good.

What about the formatting of text in single-line text fields? If our form entry is a single line, that’s usually how its text will be displayed in our interface. In this case, we probably want to avoid all the block-level elements that Markdown will let the user create.

This is easy to do using Redcarpet, a fantastic Markdown renderer for Ruby. It is fast and, importantly, it is modular: it allows us to define our own custom renderers. We can use this to create a renderer that ignores all the block-level Markdown elements. Put this somewhere in your Rails app (e.g., lib/redcarpet_renderers.rb):

module Redcarpet
  module Render
    class HTMLWithoutBlockElements < HTML
      include SmartyPants

      def initialize(opts = {})
        opts[:tables] = false
        super(opts)
      end

      # Regular markdown, just ignore all the block-level elements

      def block_code(code, language)
        code
      end

      def block_quote(quote)
        quote
      end

      def block_html(raw_html)
        raw_html
      end

      def header(text, header_level)
        "#{text} "
      end

      def hrule
        " "
      end

      def list(contents, list_type)
        " #{contents}"
      end

      def list_item(text, list_type)
        "* #{text}"
      end

      def paragraph(text)
        text
      end

      # Span-level calls

      def linebreak
        " "
      end

      # Postprocessing: strip the newlines

      def postprocess(document)
        document.gsub("\n", ' ').strip
      end
    end
  end
end

Now we can safely render the content from our single-line text fields using markdown. The user can still make things _emphasised_ or **bold** or even [linked](http://icelab.com.au/) and we don’t have to worry about unwanted block elements messing up our page layouts.

To use this renderer, just throw something like the following into your app/helpers/application_helper.rb file:

module ApplicationHelper
  def markdown(text)
    renderer = Redcarpet::Render::HTML.new({
      :filter_html => true,
      :hard_wrap => true
    })
    markdown = Redcarpet::Markdown.new(renderer, {
      :autolink => true,
      :no_intra_emphasis => true
    })

    markdown.render(text).html_safe
  end

  def markdown_line(text)
    renderer = Redcarpet::Render::HTMLWithoutBlockElements.new({
      :filter_html => true,
      :hard_wrap => true
    })
    markdown = Redcarpet::Markdown.new(renderer, {
      :autolink => true,
      :no_intra_emphasis => true
    })

    markdown.render(text).html_safe
  end
end

And it’s now a helper ready for use in our page templates. For example:

<div class="item">
  <h3 class="name"><%= markdown_line(item.name) %></h3>
  <div class="description">
    <%= markdown(item.description) %>
  </div>
</div>

The more Markdown we can use, the happier we are!

For Ticketscout’s payment processing, we use a custom Rails action with an HTTP streaming response. It shows the user a spinner and a “please wait” message, then periodically sends empty strings to the browser until the credit card transaction is completed. This streaming response keeps the request connection alive longer than the 30 second timeout period that is the default on the Heroku Cedar stack. The technique works well and ensures we don’t ever lose track of any slow-to-process credit card payments.

During development, we found that the streaming responses on Heroku did not work if we had the New Relic add-on enabled. Instead of seeing the page content incrementally delivered while the request ran, we’d see nothing until the request completed, then all the content delivered at once.

To deliver the feature on time, we simply disabled the add-on and released. Everything worked smoothly, but it was unnerving to run the app “blind” without New Relic’s useful metrics. This became especially problematic now that we’re wanting to pay some more attention to improving the app’s performance. I finally contacted New Relic support, and it turns out that a working solution is fairly simple: disabling the real user monitoring.

Turns out that the real user monitoring is currently incompatible with streaming responses. I disabled the automatic instrumentation in my newrelic.yml config file:

browser_monitoring:
  auto_instrument: false

I also switched off the “Enable end user monitoring” setting in the “Settings > Application” page. Then, after fresh deploy to Heroku, I re-enabled the add-on and we had both the metrics collection and streaming responses working together!

Granted, we’ve lost access to some useful real user metrics (like network load, page rendering and DOM processing times), but it’s better to have some metrics than none at all!

Down the track, we can probably get the best of both world’s by manually enabling the real user monitoring instrumentation in all the pages layouts except those used for the streaming responses. Hopefully further down the track, the automatic instrumentation will be compatible with streaming responses out of the box. In the meantime, it’s good to have our charts back!

I haven’t written much about my EV for a while as nothing much has happened. It just goes and goes, and has just clocked over 28,000 electric km since it was converted. This post is a collection of notes from my EV driving in 2011.

It’s been two years since I installed Lithium batteries and they have operated faultlessly. I haven’t touched them. It’s a bit deceiving actually, guess I should check the terminals or something. But when it just goes and goes you get a bit complacent. I don’t even have an ammeter or voltmeter at the moment.

This picture says it all – spider webs where the petrol used to go!

However I realised I hadn’t taken the car in for a “service” since I finished the conversion to electric drive 3 years ago. This is because there isn’t much to service in an EV, no oil, water, spark plugs, timing belts, heads to crack, water pumps, hoses, fuel pumps, or exhaust systems to corrode. The only wearing parts are brakes and tyres. So I took it down to my friends at Woodville Park Autos and they rotated the tyres, checked the brakes and replaced the wiper blades. That’s the maintenance for 3 years and 28,000 km on an EV!

Earlier this year I did 108km of city driving on one charge, and the car still felt just fine. So I am not sure what the range actually is. One of those things I don’t really want to find out!

The 6.7 inch Advanced DC motor is adequate for a 60 km/hr commuter vehicle on the Adelaide plains. It’s quick off the line on the flat but struggles up hills at say 80 km/hr. OK for our terrain but I think I would put a 8 inch motor in next time. However this would require bigger Lithium batteries to provide enough current.

The feeling of “electric cruising” on a warm summer night is quite magical. Windows down, a warm breeze in your face and no motor noise.

My daughter turned 16 recently and had her very first driving lesson in the EV:

It’s much like an automatic to drive so a nice easy way to start. I am sure EVs will play a much bigger part in her life time than ICE vehicles.

• Registered for conference, accommodation and buses... Check
• Paid for above... Check
• Flights booked and paid for... Check
• Set up TripIt for January trip... Check

All set to go. Bring on January!

These instructions are for programming a Yamaha RX-V361 AV receiver remote control so it can control a Panasonic TV:

1. Press and hold the DTV/CBL input button on the remote
2. While still holding that button press and hold the AV POWER button for about 3 seconds – long enough for the DTV/CBL message to go away and the sound setting to be displayed (e.g. Music Enh. 2ch). If you don’t hold it down long enough it will say “RemoteSetup NG”. In that case, try again. If you are successful it will just display the sound setting.
3. Enter the four-digit code using the green numeric buttons at the bottom of the remote. The code for the TV is found in the back of the AV receiver manual, which can be found online. After the fourth button is pressed the display should say “RemoteSetup OK”. The first number listed for Panasonic worked for me for an old CRT TV.
4. The TV POWER button on the Yamaha remote should now turn on/off the TV (provided the standby button on the TV is already depressed). Other buttons should also work, e.g. TV INPUT, TV MUTE, TV CH, TV VOL, etc.

Key points that are not made clear in the manual: you must use the DTV/CBL input when programming the remote to control a TV (you don’t need to have this input selected to control the TV once it is programmed though); the RemoteSetup NG/OK/display stuff as outlined above.

This took me a while to get right the first time, and I neglected to write it down. Hopefully this will help someone out.

On a friend’s recommendation, I purchased a Logitech Harmony universal remote. Some quick searching online suggested that the unofficial community Linux support for this device was decent.

It took a bit of extra futzing around to get it working under Linux (Fedora 14 in my case), but I got there in the end. Amazingly it supports all my devices (TV, PVR, DVD, RX) despite some of them being ancient and obscure, for example a cheap and cheerful Telefunken DVD player.

Logitech has a web site for configuring and downloading the settings needed to control up to five of your AV devices with the Harmony remote. Unfortunately, the current configuration site balks with an ‘unsupported operating system’ / missing plugin error when visited from a browser running on Linux. Time to head off the beaten track…

The ‘congruity’ package provides a graphical setup for the Logitech Harmony. The process still involves configuring the device from a Logitech website, but at an alternative URL, and some preparation is required:

1. Install the ‘congruity’ package. In Fedora: sudo yum install congruity
2. Configure the web browser to ask what to do with download links. In Firefox: Edit->Preferences->General->Downloads->Always ask me where to save files

Now go through the following process:

1. Plug in the remote via the provided USB cable.
2. Visit the URL mentioned above.
3. Create an account (why do I need to provide my name to program a remote control?)
4. Skip/ignore the “you need to update your software” steps, and eventually a download prompt appears.
5. Choosing ‘open’ rather than ‘save’ impressively results in the Congruity graphical setup up launching.
6. Step through the setup boxes as prompted.

However, in my case the remote control detection phrase failed. This was overcome by updating the concordance package from 0.21 to 0.23 and trying again. See further below for the sordid details.

I was then able to select my four devices and program the Harmony remote control unit.

I encountered some teething issues when attempting some tasks with the remote, e.g. the wrong AV input was selected on the TV or the DVD player wouldn’t switch off, but I pressed the ‘help’ button on the remote and it tried some things and made it work!

The only thing I couldn’t get to work was changing the band (AM/FM) on the radio tuner of the AV receiver (Yamaha RX-V361).

Now, provided the devices are in standby mode, it is a one touch action to have everything setup to watch TV, then another to watch a DVD and so on. This saves a lot of fiddling and lengthy explanations to visitors of how correctly set all the various remotes to watch TV, not to mention removing some clutter.

It’s a shame that the Linux configuration is not smoother. Perhaps it could work better by separating out the concordance device identification data, and provide a mechanism for updating that alone, without having to recompile or rebuild the whole package.

It’s also a bit unsettling that future reprogramming of the remote is dependent on Logitech’s website being operational. It feels all too inevitable that bit-rot will set in once Logitech’s priorities shift onto newer products.

Nevertheless, it’s working just well enough for now.

Sordid details:

In different attempts the following errors occurred during the detection phase within the Congruity graphical app:

LibConcordException: libconcord function 'init_concord' failed with error code 11 ('Error connecting or finding the remote')

…and…

LibConcordException: libconcord function 'get_identity' failed with error code 1 ('Unknown error')

Now for the investigative diagnostics:

Running the ‘concordance’ command line app to display detected info also failed to find the device in my case: sudo concordance -i

Concordance 0.21
Copyright 2007 Kevin Timmerman and Phil Dibowitz
This software is distributed under the GPLv3.

ERROR: failed to requesting identity
Requesting Identity: Failed with error 1

Digging a bit deeper reveals the relevant USB device ID information, and shows that at some level the operating system is aware of the device:

lsusb | grep -i logitech

Bus 005 Device 006: ID 046d:c122 Logitech, Inc. Harmony 700 Remote

dmesg

[ 8041.000104] usb 5-1: new full speed USB device using uhci_hcd and address 6
[ 8042.176170] usb 5-1: New USB device found, idVendor=046d, idProduct=c122
[ 8042.176181] usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 8042.176189] usb 5-1: Product: Harmony Remote 0-0.2.0
[ 8042.176195] usb 5-1: Manufacturer: Harmony Remote 0-0.2.0

Interestingly, while my particular unit was marketed as a “Logitech Harmony 600 Five-In-One Universal Remote”, lsusb reports it as a Logitech Harmony 700 Remote.

Some web searching turned up a useful thread on troubleshooting Harmony software setup with Congruity and Concordance:

i had the same problem.
I bought a Harmony 650, and concordance (0.21) doesnt detect it.
Easy solution: install concordance 0.23″

Now, I tend to prefer the clean solution to the easy one. In Fedora that involves building and upgrading relevant RPM packages (as opposed to downloading the tarball and doing a make install as root, which doesn’t have the traceability that RPM packaging gives).

Here is an outline of the steps:

1. Remove the existing packages to avoid conflicts later on: sudo yum remove congruity
2. Install build dependencies: sudo yum install libusb-devel python-devel
3. Download standard Fedora source packages for libconcord and concordance in order to get the .spec files: yumdownloader --source --disablerepo=ifedora --enablerepo=fedora libconcord concordance
4. Download the concordance-0.23.tar.bz source tarball in a web browser.
5. Edit libconcord.spec and comment out the patch lines, and replace references to version 0.21 with 0.23:

   #Patch0: libconcord-0.21-mime-type-def.patch
   #%patch0 -p1
   # Comment out mime file
   #%{_datadir}/mime/packages/%{name}.xml
   

   The patch lines need to be removed to avoid errors during the rpmbuild phase like the following:

   Reversed (or previously applied) patch detected!  Assume -R? [n] n
   
   RPM build errors:
       File not found: /home/makerpm/rpmbuild/BUILDROOT/libconcord-0.23-1.fc14.x86_64/usr/share/mime/packages/libconcord.xml
   

6. Change to “makerpm” user su - makerpm and copy the spec files in the SPECS directory and the tarball into the SOURCES directory.
7. Build the libconcord rpm package: rpmbuild -ba libconcord.spec
8. Install the generated rpm: sudo yum localinstall --nogpgcheck /home/makerpm/rpmbuild/RPMS/x86_64/libconcord-python-0.23-1.fc14.x86_64.rpm
   If this is not done at this point then the latter rpm building will fail with something like:

   error: Failed build dependencies:
   	libconcord-devel >= 0.23 is needed by concordance-0.23-1.fc14.x86_64
   

9. Edit concordance.spec and change all occurrences of the version 0.21 to 0.23
10. Build the concordance rpm package: rpmbuild -ba SPECS/concordance.spec
11. Install the resultant packages: yum localinstall --nogpgcheck /home/makerpm/rpmbuild/RPMS/x86_64/libconcord-python-0.23-1.fc14.x86_64.rpm
yum localinstall --nogpgcheck /home/makerpm/rpmbuild/RPMS/x86_64/concordance-0.23-1.fc14.x86_64.rpm
12. Install congruity again, now that the backend has been upgraded: sudo yum install congruity

Note that I’ve missed some steps above, e.g. some variant of rpmbuild --rebuild xyz.srpm which is left as an exercise for the reader.

Now the concordance tool will be cleanly installed and should show up the device:

[root@orange etc]# concordance -i
Concordance 0.23
Copyright 2007 Kevin Timmerman and Phil Dibowitz
This software is distributed under the GPLv3.

Requesting Identity: 100%                 done
  Model: Logitech Harmony 700 (Molson)
  Firmware Version: 0.2
  Hardware Version: 1.1
  Config Flash Used: 12% (473 of 3904 KiB)

The Ubuntu forum thread also mentioned the following which may be required in some cases:
echo 'SYSFS{idVendor}=="046d", SYSFS{idProduct}=="c122", MODE="666"' |tee /etc/udev/rules.d/custom-concordance.rules
Also tried changing SYSFS to ATTR, based on warnings in /var/log/messages

At this point, the web configuration steps outlined earlier in the post should work, with congruity successfully detecting the Harmony remote, allowing subsequent software setup.

So you've got a new computer and you realise that you GPG keys are on the old one.

gpg --armor --export > keys.asc
gpg --armor --export-secret-keys >> keys.asc
gpg --armor --export-ownertrust > trust.asc

Then some sleight of hand to move the two .asc key files to the new computer. [1]

gpg --import keys.asc
gpg --import-ownertrust trust.asc

---
[1] And yeah, read that with irony. Moving keys around is one of the "hard" problems of everyday cryptography, and the hassle to do it right can be large (and thus expensive, for example a "hand-to-hand courier" where the key material never leaves someone's immediate possession).

I have the opportunity to use a Google Nexus S for a period of time, so I figured I would use my time with it to document my subjective comparison of webOS and Android.

Before I do so, I’ll make some introductory statements, so this blog entry is not misinterpreted or used in a way for which it is not intended.

Firstly, a smartphone operating system (like a desktop operating system) is a tool, not a religion.

Blindly advocating one smartphone operating system without having any real exposure to it’s competitors is like believing in one religion without having any exposure to any of the others, or saying that your country is the best in the world without ever leaving it’s borders or having an awareness of foreign places and cultures.

A good craftsman is able to subjectively compare tools from different vendors unemotionally, and choose the best heterogeneous set of tools for their own personal toolbox, rather than just blindly purchasing all their tools from the one vendor. However, compatibility between tools and suitability of a particular tool to a particular user may require a less than optimal single item to be purchased for the overall benefit of the efficacy of the complete set.  Life is full of trade-offs and compromises, and ideals are rarely achieved.

Smartphones in particular are very personal items, and therefore subject to the whims of individual tastes, prejudices, legacies and routines. What works best for one person may be unusable for another, and vice versa.

Secondly, in general I don’t care what your opinion about smartphone operating systems is, and usually prefer not to get involved in arguments over subjective opinions. I choose the set of tools that work best for me, not the set of tools that work best for you or for anyone else. I don’t care whether one smartphone operating system has a higher number of users than another – I care only whether it works best for me. I don’t care whether one smartphone operating system has a higher number of available applications than another – I care only whether it has the applications that I need to use.  I don’t care what market analysts think – the goals that they have when making smartphone operating system comparisons are very different from mine. I’m a developer, and in general if my need for some application or feature is strong, then I’ll either write it myself or choose to live without it if that is not possible. So I don’t necessarily need to follow the herd, but may choose to follow the path of least resistance to achieve my goals for the task at hand.

Please don’t waste your time trying to convince me that my subjective opinions are wrong, because they are not.  Subjective opinions are just that – subjective, and opinions. Subjective opinions only apply to the person who made them, and cannot be used as some sort of objective truth that can be applied to another person. They also cannot be used by a second party as some sort of weapon against a third party – subjective opinions are not transitive.

Thirdly, most folks who read this will know that I founded and lead an open source development group called WebOS Internals. When people put a lot of time and effort in to the creation and growth of something, they are very likely to have a strong bias towards that thing. I am no different. My biases will affect my subjective opinions, and that’s fine – they are subjective opinions formed by me, not objective truths that apply to you.

Finally, to put to rest any conspiracy theories that may result from this post: doing a subjective comparison between two smartphone operating systems is not an indication of the death of one smartphone operating system, or the supreme superiority of another. Nor is it an indication of any change in direction of any development group. Nor is it a vote of confidence or no-confidence in any particular vendor. Nor is it a threat, a weapon, or a sell-out.

Put simply, if you find yourself wanting to use this post for any reason of than to appraise yourself of Rod Whitby’s subjective opinion about Android and webOS (as deployed on specific hardware as mentioned), then you should stop now and take a good hard look at yourself, cause you didn’t really read anything I wrote above.

With all that said, let’s begin the comparison.  The items that are being compared are Android 2.3.4 running on a Google Nexus S and webOS 2.1.0 running on a Palm Pre 2.

[To be continued ...]

A while ago I documented how to configure a anycast DNS service. It turned out to be a popular idea, as it breaks the nexus between the address of the service and its location in the network. This is more useful then using a real IP address, having circumstances force you to change the location of the server, and having to find clients which have hard-coded the server's IP address (rather than using DHCP or SLP). It also works fine for other services, such as RADIUS.

Anyway, that paper included the best practices for DNS. As did a AusCERT advisory I wrote some years ago. Those practices included a bogon filtering list.

With the exhaustion of the IANA pool of IPv4 addresses we can give one last issue of the bogon list, containing only the "impossible" networks. This list superceeds that in the two papers.

// RFC5735 Special use IPv4 addresses
// All other address space has been assigned to unicast routing by IANA,
// although some blocks have been held back for special purposes by the
// various regional internet registries.
acl "bogon" {
   0.0.0.0/8;          // Null, RFC1122
   10.0.0.0/8;         // Private network, RFC1918
   127.0.0.0/8;        // Loopback, RFC1122
   169.254.0.0/16;     // Link-local network, RFC3927
   172.16.0.0/16;      // Private network, RFC1918
   192.0.0.0/24;       // Protocols, RFC 5736
   192.0.2.0/24;       // Documentation, RFC1166
   192.88.99.0/24;     // 6to4 can't be src_addr, RFC3068
   192.168.0.0/16;     // Private network, RFC1918
   198.18.0.0/15;      // Benchmarking, RFC2544
   198.51.100.0/24;    // Documentation, RFC5737
   203.0.113.0/24;     // Documentation, RFC5737
   224.0.0.0/4;        // Multicast can't be src_addr, RFC3171
   // 240.0.0.0/4         Class D is for future use, maybe routable, RFC3171
   255.255.255.255/32; // Limited broadcast, RFC919
}; 

options {
   blackhole {
      bogon;
   };
};

As with all these lists, you should read it first. Maybe you want to allow 127.0.0.1/32 when seen from the machine's own interfaces, so you can do "dig @localhost" when investigating issues (and if you want to do that, then BIND has a nice "localhosts" in-built ACL which includes all of the IP addresses of all of the interfaces on the machine [1]). Maybe you use private addressing, and so don't want to filter all of those.

As a side note, a researcher contacted me explaining that I had the first published use of the work "bogon" in a routing context and if I knew its origin. I'm afraid not. It's use was widespread in computing and electrical engineering talk at the time: a spike on a voltage would be described as being caused by a "bogon particle". In a routing context, the mix of "bogus" and "Vogon" caught my imagination (badly behaved routes arriving from beyond the solar system, with poor poetry to boot). As it did for a lot of people at about the same time. The Bogon List from Team Cymru turned it from a quip to a usage, so they are really the etymologcal source.

---
[1] Cisco and Juniper take note. This in-built ACL would make writing ACLs to limit access to the control plane of the router a thousand times simpler.

Event; AHARS Symposium

Topics; Amateur radio, Construction

Speakers; Drew Diamond VK3XU, Phil Harman VK6APH and more

Venue; Belair Community Centre, Corner of Sheoak Road and Burnell Drive, Belair

Cost; $20 at the door for morning tea, lunch and afternoon tea.

Date; 19 September, 2010

Time; 930am - 5pm

Kim has been tinkering with analog electronic music. Over the Christmas break he has put together an 8 step rotary sequencer and atari punk console.

You won't see anything new here! Try http://rusty.ozlabs.org instead.

So two weeks back was the Official Handing Over Of The SAMBA Team T-shirt! Since then I have done my first serious push to the git tree, and received spam from the build farm about it (false positives, AFAICT).

I'm still maintinging virtio and the module and parameter code of course. But the kernel has slowly morphed into a complicated and hairy place. Formality has crept in, and the pile of prerequisites grows higher (eg. git, checkpatch.pl, Signed-off-by). This is maturity, but it raises the question: when will some neat lean OS without all this baggage come along?

SMP, micro-optimizations, multithreading and extreme portability are responsible for much of the added coding burdens, but also hyper-distributed development means many coders shy away from changes which would break APIs. The suboptimality accretes and this method of working becomes the new norm. BUG_ON() for API misuse is now seen as unduly harsh, but undefined APIs make the next change harder, and WARN_ON() tends to stay around forever.

SAMBA has some brilliant ideas which coding a joy (talloc chief among them, but there are other gems to be found). Hell, it even has a testsuite! But of course it has its own issues; the SAMBA 3/4 split, lack of the kernel's massive human resources and the inevitable code quality issues. Ask me again in a few years to do a comparison...

After the Preware 0.9.4 release, which included Applications, Themes, and Patches, and offers over 670 homebrew packages for installation, we (webos-internals.org)  started getting reports of the Palm App Catalog “hanging” the user interface for 30 seconds or more when the installation of a new application is initiated, but only when the Package Manager Service (the service which does all the Linux-level work for the Preware front-end) was installed.

After some analysis, I found that disabling all the feeds in Preware reduced the “hang” from more than 30 seconds to less than a second.

Looking through the output of ‘dbus-util –capture’ showed that the “hang” was during the call to the queryInstallCapacity method of the com.palm.appinstaller service, the routine that the Palm App Catalog application uses to determine whether there is enough space to install a new application from the Palm App Catalog.  Yes, this is the method which is at the heart of the dreaded “Too many applications” errors that people are seeing when they have a number of homebrew applications installed and try to install a new application from the Palm App Catalog.

Watching the output of ‘ps -ef’ during calls to this method (you can call it manually using luna-send) showed that palm was calling “ipkg -o /var list”.  Curious.  Why would you want to know the *complete* list of all *available* applications when determining whether there is room to install one known new application.  I suspect that Palm should be calling “ipkg -o /var list_installed” instead (which just lists the installed applications).  Note that Palm doesn’t use feeds the way that Preware does, so for Palm’s official use of ipkg, list and list_installed would return the same thing in their testing, but list_installed is definitely what they should be using to determine the size of installed applications.

The plot thickens when you use strace (which Palm conveniently includes in the official firmware image) on the running LunaSysMgr process.

It seems that LunaSysMgr calls “ipkg -o /var list” to get a list of installed packages (the probably intend to just get the list of installed packages, but when you have Preware installed and have feed configuration files in /var/etc/ipkg/*.conf, it actually returns a list of all *available packages).

LunaSysMgr then does an execve of “/usr/bin/du -s /var/usr/palm/applications/package” for each package in that list.  (BTW Palm, you seem to have a bug in the logic of that code, cause it’s running du on random garbage strings after the end of the real package list)

Yes, that’s right.  A call to queryInstallCapacity spawns a new program (“du”) once for each package returned by “ipkg -o /var list”.  No wonder the UI hangs for 30 seconds or more!

A single “du -s /var/usr/palm/applications/*” would be a far more efficient way to get exactly the same information, but again, Palm would not see this difference in testing because they do not support the third-party Preware application usage of ipkg feeds.

You can imagine that this behaviour is probably related to the app install limit that many people are experiencing too.  Unfortunately, I’ll have to reduce my /var partition size down from it’s current 2GB size (courtesy of the WebOS Internals Meta-Doctor) to be able to investigate this one.

Now the webos-internals.org developers need to develop a new method of installing homebrew applications so that this bug in Palm’s appInstaller service is not triggered.

In the meantime, the work-around is to go into the Preware Preferences screen, hit the “Feeds” button in the top-right corner, and disable all the feeds while you use the Palm App Catalog application in another card.  No need to exit the Feeds screen, just toggle all the button to “Off”, and then toggle them back to “On” when you’re finished with the App Catalog.

For the solution to this problem, see Update #2, below.

I’ve created a thread on PreCentral where this issue can be discussed.  As I uncover more information, I’ll publish my finding here.

Update #1: I’ve now webOS Doctored my Pre in the name of science, and have tested the limits of installing applications.

If you run “du -s /var/usr/palm/applications/*”, and add up all the numbers in the first column, then as soon as you hit the 62367 1K blocks limit of the addition of the sizes reported by that “du” command and the size of the app you with to install, you will get the dreaded “Sorry, Not Enough Memory” error from the Palm App Catalog application (and any other installer, like fileCoaster or PreLoad, which uses the palm appInstaller API).  It doesn’t matter whether you have 192MB free in your /var partition, it will max out at just under 64MB of application usage.

Update #2: I have now created a Linux Application called “Fair Dinkum App Limit” (org.webosinternals.fairdinkum), which removes both the “hang” and the arbitrary application limit.  You can find it in Preware.  Just install it (no need to even run anything – if it is installed, it’s working), and you’re ready to install more applications than you can poke a stick at …

Fair Dinkum App Limit works by simply putting a couple of wrapper scripts in /usr/local/bin, which returns a size of zero when du is called, and returns the output of “ipkg -o /var list_installed” when “ipkg -o /var list” is called.  In the future, the wrappers will be made much more sophisticated than they are right now to prevent operation outside of the specific cases where they need to fool LunaSysMgr, and to also include a safety buffer so that users do not fill the /var partition.  This is a tactical response to a problem that people using homebrew applications are experiencing.  Hopefully, Palm will provide the long term solution for limits on application installation in a release in the near future.

Notes for Palm, if you happen to read this:

1) We fully appreciate that the usage of the ipkg database in /var for homebrew applications is a choice that the homebrew community has made, and is not supported by Palm.

2) We fully agree that the use of “ipkg -o /var list” instead of “ipkg -o /var list_installed” would work perfectly fine for the way that Palm is officially using the ipkg database in /var/usr/lib/ipkg, but we contend that the “list” and “list_installed” commands have clear intended usage, and the one to find the list of installed applications for checking available disk space should be “list_installed”.

3) We fully realise that the initial version of the FairDinkum scripts are unsafe.  Returning a zero value for du is a temporary solution while we work out how to achieve the same result safely.  The intention is to only return false values when du is being called from LunaSysMgr, and to make sure that a safety buffer is kept so that users do not fill the /var partition.

4) I would be very happy to discuss these issues with anyone at Palm (Chuq and Google both have my email address), and would hope that we can together architect a solution for supporting homebrew application installation which does not require these work-arounds.

5) We have designed these work-arounds in a way which does not collide with Palm OTA Updates, and in a way that we can update them automatically using the Preware installer application, and in a way that we can cause them to self-remove when Palm releases a long term solution.

Update #3:

It seems that there is yet another limit on application installation imposed by LunaSysMgr.  Once the used space on /var crosses around 60%, LunaSysMgr will once again refuse to install applications.

I’m going to need to webOS Doctor my Pre yet again (to reallocate 2GB to /var) to determine whether this limit is a fixed percentage, or a fixed number of blocks.

Update #4:

The limit is 102400 free blocks.  Mystery solved.  That also means the Fair Dinkum App Limit cannot cause your /var to be filled.

Update #5:

Thanks to Carl Thompson, an improved version of Fair Dinkum App Limit which does not alter the operation of ‘du’ has been released.

If you've ever written a stack of SQL inserts using Perl and wondered if there was a better way to do it, and that gets around these issues:

1. Doesn't need a multi-line string with quoted variables,
2. Isn't a page long handling the query with transactions, and
3. Checks that the Perl variable contains data that matches the database column's field type,

then this piece of code will help:

use strict;
use DBI qw(:sql_types);

my $dbname = "christest";

my $dbh = DBI->connect("dbi:Pg:dbname=$dbname", "", "",
                       { RaiseError => 1, AutoCommit => 0 });

# using the DBI statement handler's bind_param() feature, bind the
# given SQL parameters to the prepared query.
#
# To be eval'ed from sql_insert()
#
# Args:
#       - reference to statement handler (i.e. \$sth)
#       - remaining args are arrary references representing
#         value and type pairs - e.g. ["fred", SQL_VARCHAR]
#
# Returns:
#       nothing
#
sub bind_query {
read more

St George supply a shared library for RedHat Linux 9 at https://www.ipg.stgeorge.com.au We want it to run it using FreeBSD for various reasons, the least of which is that RH9 hasn't been supported for more than a year. (Actually, the real reason is that we have a client using FreeBSD that doesn't want to use the Java API.)

The current version (released Jan 2004) will work under FreeBSD 4 with a few steps:

1. Install the linux emulation ports:
   
emulators/linux_base-rh-9
devel/linux_devtools
devel/linux-glib2


2. Install the kerberos5 and openssl libaries using RPMs.
   
RPMFLAGS="--root /compat/linux --dbpath /var/lib/rpm --nodeps --replacepkgs --ignoreos --ignorearch"
rpm -U $RPMFLAGS krb5-libs-1.2.7-10.i386.rpm
rpm -U $RPMFLAGS openssl-0.9.7a-20.2.i386.rpm


3. Copy the webpay library to both usr/lib directories:
   

   read more