LJ Archive

UpFront

diff -u: What's New in Kernel Development

Zack Brown

Issue #237, January 2014

A recent bug hunt by kernel developers ended up identifying a long-standing bug in GCC. The indications were there from the start, but it took some investigation to nail it down.

Originally, Fengguang Wu reported a kernel oops, and used “git bisect” to identify the specific patch that revealed the problem. It was an optimization suggested by Linus Torvalds and implemented by Peter Zijlstra that aimed at freeing up a hardware register by using the “asm goto” instruction in the kernel's modify_and_test() functions.

The first indication that the problem might boil down to a compiler bug was that the patch just seemed correct to folks. Neither Peter nor Linus were able to see anything wrong with it, so they suggested trying to reproduce the oops on kernels compiled with different versions of GCC, and Linus suggested disabling “asm goto” directly to see if that had any effect.

At first, Fengguang found that earlier compilers made no difference. He'd started off using GCC 4.8.1, but 4.6.1 also produced a kernel that would reproduce the oops. But as Linus suspected, disabling “asm goto” in the kernel code did fix the problem. After a while, Fengguang also discovered that the older GCC version 4.4.7 also produced a working kernel, because that compiler had no support for “asm goto”.

Gradually, other folks began to be able to reproduce the problem on their own systems. Originally, the issue seemed to affect only 32-bit Linux systems, but ultimately, Linus was able to reproduce the problem on his own 64-bit system. It was harder to trigger on a 64-bit system, but it boiled down to being the same problem. As the scope of the problem began to reveal itself, Linus remarked, “It makes me nervous about all our traditional uses of asm goto too, never mind the new ones.”

Jakub Jelinek opened a Bugzilla ticket against GCC, and folks started thinking about workarounds for the kernel. Even after GCC got a fix for this particular bug, it wouldn't do to allow the kernel to miscompile on any version of GCC, if it possibly could be avoided.

A workaround did end up going into the next Linux kernel release candidate, and a fix went into GCC 2.8.2. Shortly afterward, Greg Kroah-Hartman also adopted the kernel workaround in the 3.11.x stable tree.

The reason the kernel needed a workaround in spite of the fact that a real fix went into GCC was because the kernel needs to support the widest possible dispersion of host systems. Anyone, anywhere, with any particular hardware setup, using any particular versions of the various development tools, should be able to build and run the kernel. In some cases that ideal can't be reached, but it remains an ideal nonetheless.

Traditionally, software could mount a filesystem only after registering it with the kernel, so the kernel would know its name and a bit about how to manage it. This has been true even for internal filesystems like ia64, pfmfs, anon_inodes, bdev, pipefs and sockfs. But, Al Viro recently said there was no longer any reason to require registration for these filesystems, and he submitted a patch to take out the requirement.

First of all, he and Linus Torvalds agreed that there probably isn't any user code that actually looks up those filesystems in the registry. There's just no reason anyone would want to.

As Al explained on the mailing list, there used to be a need to register all filesystems. But about a decade ago, the kern_mount() call changed to take only a pointer to the filesystem, rather than needing to look it up by name.

Ever since then, the need to register these internal filesystems has been minimal. The only remaining dependency was a single data structure initialized by register_filesystem() that was needed by all filesystems. But, Al said that even this dependency was eliminated a couple years ago, when the data structure was optimized no longer to need register_filesystem(). By now, Al said, “there's no reason to register the filesystem types that can only be used for internal mounts.”

With this change, /proc/filesystems would no longer list internal filesystems. And as Linus pointed out, those filesystems wouldn't reliably be listed anywhere on the system. Even /proc/modules, Linus said, would list those filesystems only if they'd been compiled as modules.

So, with some mild trepidation, Linus accepted the patch. If no one howls, it'll probably stay.

Zedge, for All Your Annoying Ringtones!

Shawn Powers

Issue #237, January 2014

I really don't understand folks who use songs as their ringtones. Isn't it annoying or confusing when the song comes on the radio? If it's your favorite song, don't you get desensitized to it when you listen to the CD (or digital equivalent of CD)? Nevertheless, you probably hear dozens of ringtones every day. Those probably vary from “super annoying” to “what a cool ringtone”. With Zedge, you can be the person annoying your fellow subway passengers—or making them jealous.

Screenshot from the Google Play store

Zedge is a free app in the Google Play store, and the ringtones (and notification sounds and alarm sounds) are completely free as well. I currently use the “WHAAAT?!?!??!” sound from the minions on Despicable Me as a notification sound (which is clearly super cool and not annoying). My ringtone, which I hear much less often than in years past, is one I made myself from pasting together sound clips from Star Trek the Next Generation. Somehow, my homemade ringtone ended up on Zedge. I know it's mine, because I pasted together sounds that don't actually occur together on the show. I'm terribly proud of my ringtone, and if you'd like to hear it for yourself, search for “Incoming Subspace Signal”, it should pop right up. If Star Trek isn't up your alley, there are thousands of other options from which to choose. With Zedge, installing them is simple and, of course, free.

Due to its incredible selection, seamless integration and amazing price tag, Zedge is this month's Editors' Choice winner. Check it out today at https://play.google.com/store/apps/details?id=net.zedge.android.

Blu-ray Encryption—Why Most People Pirate Movies

Shawn Powers

Issue #237, January 2014

I get a fair amount of e-mail from readers asking how a person could do “questionable” things due to limitations imposed by DRM. Whether it's how to strip DRM from ebooks, how to connect to Usenet or how to decrypt video, I do my best to point folks in the right direction with lots of warnings and disclaimers. The most frustrating DRM by far has been with Blu-ray discs.

Unless I've missed an announcement, there still isn't a “proper” way for Linux users to watch Blu-ray movies on their computers. It's hard enough with Windows or Macintosh, but when it comes to Linux, it seems that turning to the dark side is the only option. In the spirit of freedom, let me point you in the direction of “how”, and leave it up to you to decide whether it's a road you want to travel.

When ripping a movie from Blu-ray, I know of only one program that can do the job. MakeMKV is a cross-platform utility that will extract the full, uncompressed movie from most Blu-ray discs. Unfortunately, you have to download the source code and compile it. You need both the binaries and the source download files, and then follow the included directions for compiling the software. Yes, it's a bit complex.

Once you compile MakeMKV, you should be able to use it to extract the Blu-ray disc to your computer. Be warned, the file is enormous, and you'll most likely want to compress it a bit. The tool for that thankfully is much easier to install. Handbrake has been the de facto standard video encoding app for a long time, and when paired with MakeMKV, it makes creating playable video files close to painless. I won't go through the step-by-step process, but if the legally questionable act of ripping a Blu-ray disc is something you're comfortable doing, www.makemkv.com and www.handbrake.fr are the two software packages you'll want to explore.

Non-Linux FOSS: Persistence of Vision Raytracer (POV-Ray)

Shawn Powers

Issue #237, January 2014

Back in the mid-1990s, a college friend (hi Russ!) and I would put our old 8088 computers to work rendering ray-traced images for days—literally. The end result would be, by today's standards, incredibly low resolution and not terribly interesting. Still, the thought of a computer system creating realistic photos from nothing more than math equations was fascinating. As you probably already guessed, Russ and I weren't terribly popular.

This image is completely computer-generated, created by Gilles Tran, released into public domain.

All these years later, the same ray-tracing software we used back then is now up to version 3.7, and it has been released as free, open-source software. The developers kindly have created a downloadable Windows installer for those folks stuck on a Microsoft operating system. If you think the world is nothing more than math, and you'd like to prove it with ray-traced images, head on over to www.povray.org and download your copy today. I can't promise it will make you popular, but at least by my standards, it will make you cool!

Taking Fractals off the Page

Joey Bernard

Issue #237, January 2014

Fractals are one of the weirder things you may come across when studying computer science and programming algorithms. From Wikipedia: “A fractal is a mathematical set that has a fractal dimension that usually exceeds its topological dimension and may fall between integers.” This is a really odd concept—that you could have something like an image that isn't made up of lines or of surfaces, but something in between. The term fractal was coined by Benoit Mandelbrot in 1975.

A key property of fractals is that they are self-similar. This means if you zoom in on a fractal, it will look similar to the way the fractal looked originally. The concept of recursion also is very important here. Many types of fractal algorithms use recursion to generate the values in the given set. Almost everyone has seen computer generated images of classic fractals, like the Mandelbrot set or the Cantor set. One thing about all of these classic images is that they are two-dimensional (or actually greater than one and less than two-dimensional, if you want to be pedantic). But there is nothing that forces this to be the case. Fractals can be any dimension, including greater than two. And with modern 3-D graphics cards, there is no reason why you shouldn't be able to examine these and play with them. Now you can, with the software package Mandelbulber (www.mandelbulber.com).

Mandelbulber is an experimental, open-source package that lets you render three-dimensional fractal images and interact with them. It is written using the GTK toolkit, so there are downloads available for Windows and Mac OS X as well as Linux. Actually, most Linux distributions should include it in their package management systems. If not, you always can download the source code and build it from scratch.

If you want some inspiration on what is possible with Mandelbulber, I strongly suggest you go check out the gallery of images that have been generated with this software. There are some truly innovative and amazing images out there, and some of them include the parameters you need in order to regenerate the image on your own. The Mandelbulber Wiki provides a large amount of information (wiki.mandelbulber.com/index.php?title=Main_Page). When you are done reading this article, check out everything else that you can do with Mandelbulber.

When you first start up Mandelbulber, three windows open. The first is the parameters window (Figure 1). Along the very top are the two main buttons: render and stop. Below that is a list of 12 buttons that pull up different panes of parameters. You get an initial set of default parameters that will generate a 3-D version of the classic Mandelbrot set. Clicking on the render button will start the rendering process. If you have multiple cores on your machine, Mandelbulber will grab them to help speed up the calculations.

Figure 1. The main window gives you all parameters that control the generation of your fractal.

The rendered plot will be drawn in its own window (Figure 2). The third window shows you some measures of how the rendering progressed (Figure 3). You get two histograms describing the number of iterations and the number of steps.

Figure 2. This is what the default 3-D fractal looks like.

Figure 3. Histograms of the Rendering Progression

To generate new images, more than 70 examples are included with the installation of Mandelbulber that you can use as starting points. Clicking on the button Load example pulls up a file dialog where you can load one of them. For example, you could load “menger sponge.fract”. Clicking the render button will generate a 3-D Sierpinski sponge (Figure 4). Although technically, the set is only one topological dimension that encloses zero volume (aren't fractals weird?).

Figure 4. A Sierpinski sponge has infinite surface area and zero volume.

What can you change in Mandelbulber? Clicking on the fractal button pulls up the pane where you can set the parameters for the fractal itself (Figure 5). You can select from several different types of fractal formula types, such as mandelbulb, quaternion or menger sponge. You can set several options, depending on exactly which fractal type you choose. For example, if you select the iterated function system (IFS), you then can click on the IFS tab to set several different parameters.

Figure 5. There are several different fractal types from which to choose.

One of the issues is coming up with truly unique, yet aesthetically pleasing, sets of equations with which to experiment. To help in this regard, Mandelbulber has a hybrid option in the list of fractal types. When you select this option, you then can choose the hybrid button and set up to five different fractal equations (Figure 6). With this option, you can create very complex and sophisticated fractals to render.

Figure 6. You can create a hybrid system made from a mix of up to five different fractal types.

Mandelbulber doesn't just generate static images of these higher dimensional fractals. There is an option to generate animations of how these images change when some parameter is swept over. To start, you need to click on the Timeline button at the bottom of the view pane. This pulls up a timeline window where you can set the parameters used to generate your animation. The record button puts parameters into the actual keyframe number (Key no. field on the right). It then loads and renders the next keyframe if it is not the last keyframe.

Then, you can add new keyframes with the “insert after” button or delete keyframes with the Delete button. To modify a given keyframe, you can double-click it to set the parameters, and then you can click on record to render the keyframe.

Interpolation between the keyframes is handled by Catmull-Rom splines. Once you have the keyframes handled, you will need to render the full animation. Clicking on the Animation button in the main window brings up the parameters you can set. These include things like the number of frames to render from the keyframes, as well as the start and end frame numbers. You then can click on the Render from key-frames button to generate the animation. On my netbook, this is a pretty long process. For image generation, you also have control over camera position, lighting and shader options. You should be able to generate the exact image or animation that you want.

If you are looking to generate some amazing 3-D landscapes or unique shapes for something science-fictiony, you definitely should check out Mandelbulber—just be prepared to lose several hours as you start playing with all of the parameters available.

Make Peace with pax

James K. Lowden

Issue #237, January 2014

pax is one of the lesser known utilities in a typical Linux installation. That's too bad, because pax has a very good feature set, and its command-line options are easy to understand and remember. pax is an archiver, like tar(1), but it's also a better version of cp(1) in some ways, not least because you can use pax with SSH to copy sets of files over a network. Once you learn pax, you may wonder how you lived without it all these years.

pax has four modes: list, read, write and copy. Reading and writing are controlled by the -r and -w options, repectively. In combination, -rw, pax acts a little bit like cp -R. If neither is used, pax lists the contents of the archive, which may be a file, device or a pipe.

By default, pax operates as a filter: it reads from standard input and writes to standard output, a feature that turns out to be very useful. But usually these days, the target is an archive file, the familiar tarball. Let's start by creating one:

$ cd /tmp
$ mkdir paxample
$ touch paxample/foo
$ pax -wf paxample.tar paxample

The -w option means “write”—that is, create an archive. The -f option provides the name of a file to which to write the archive. If desired, pax can gzip or bzip the file at the same time:

$ pax -wzf paxample.tar.gz paxample

Like most tar implementations, pax, by default, uses the Posix ustar file format. Because pax was born of a desire to unify archive file formats, many other formats also are supported, but in practice, they're seldom used. Likely as not, any .tar.gz file you download from the Internet actually will be a ustar archive:

$ pax -wzf paxample.tar.gz paxample
$ file paxample.tar*
paxample.tar:    POSIX tar archive
paxample.tar.gz: gzip compressed data

The first thing you nearly always want to know about any archive is what's in it. Listing the contents is the default action in the absence of either a -r or -w option:

$ pax -f paxample.tar
paxample
paxample/foo

Note that the archive retains the directory name you specified on the command line. That comes into play later when you read it.

To read an archive, use -r:

$ mkdir t
$ cd t
$ pax -rf ../paxample.tar

What did that do? Let's look at the source and target directories:

$ cd /tmp
$ find paxample t # traverse both trees
paxample
paxample/foo
t
t/paxample
t/paxample/foo

When pax read the paxample.tar archive, it created files in the current directory, t. Because the archive included a directory name, paxample, that directory was re-created in the output.

Copying Sets of Files

To my mind, pax's -r and -w options make more sense than their -x and -c equivalents in tar—reason enough to switch. But, pax can do more than tar: it can copy files too:

$ rm -rf t
$ pax -rw paxample t
$ find t
t
t/paxample
t/paxample/foo

Unlike cp(1), pax is an archive utility. Its job isn't to make copies, but to archive files. When pax creates a file, it preserves the file's metadata from its input. The form of the input doesn't matter. In this case, the input isn't from an archive, it's the file itself:

$ ls -l paxample/foo t/paxample/foo
-rw-r--r--  1 jklowden  wheel  0 Sep 22 15:45 paxample/foo
-rw-r--r--  1 jklowden  wheel  0 Sep 22 15:45 t/paxample/foo

Yes—two identical files with two identical timestamps. The permission bits and ownership can be controlled too, if desired. Take that, cp(1)!

Perhaps you don't want to re-create the directory, or perhaps you want to change it in some way. One option is not to mention the input directory on the command line, but instead provide filenames:


$ rm -rf t/paxample/
$ (cd paxample/ && pax -rw * ../t/)
$ find t
t
t/foo

That's usually easiest. But if you need something more sophisticated, the -s option rewrites the path—actually, any part of the filename—using a regular expression:

$ rm -rf t/*
$ pax -rw -s ':paxample:my/new/path:g' paxample/ t
$ find t
t
t/my
t/my/new
t/my/new/path
t/my/new/path/foo

The -s option is handy, for instance, when unpacking a tarball that doesn't have version information in the directory name.

What Could Go Wrong?

If you give the wrong filename to write, you just get an archive by the wrong namemdash;no harm no foul. If you mistype an input archive filename though, you'll find yourself in 1985:

$ pax -rf paxample.whoopsie
pax: Failed open to read on paxample.whoopsie (No such file 
or directory)

ATTENTION! pax archive volume change required.
Ready for archive volume: 1
Input archive name or "." to quit pax.
Archive name >

This is an idea that outlived its usefulness before it was implemented. You could type in the filename here, again, without readline support or tab completion. Well, at least it says what to do:

Archive name > .
Quitting pax!

How exciting!

As mentioned previously, pax uses standard input and standard output by default. That is a feature, but the first time you forget to provide a filename, you may think pax is very, very slow:

$ pax  -r paxample.tar

Oops! No -f. Also no message and no prompt. pax is ignoring the archive filename argument and reading standard input, which in this case, is the keyboard. You could type ^D, for end-of-file, but that forms invalid input to pax. Better to send up a smoke signal:

^C
pax: Signal caught, cleaning up.

It's even worse the first time you accidentally write to standard output while it's connected to your terminal. You heard it here first: don't do that.

Putting Standard Input to Work

Standard input and standard output do have their uses, and here pax really comes into its own. For one thing, you can verify the effect of the -s option without creating an archive or the files:

$ pax -w -s ':paxample:my/new/path:g' paxample/ | pax
my/new/path
my/new/path/foo

Absent the -f option, pax -w writes to standard output. So rewrite the pathname with -s, and pipe the output to pax again, this time using its “list” mode, with neither the -r nor -w option. By default, pax reads from standard input and, in “list” mode, prints the filenames on the terminal.

That can save a lot of time, not to mention a mess on the disk, when there are thousands of files.

Suppose you want to copy the paxample directory to another machine. One approach would be to create a tarball, copy to the target, log in to the target and unpack the tarball:

$ pax -wf paxample.tar paxample
$ scp paxample.tar oak:/tmp/
paxample.tar             100%   10KB  10.0KB/s   00:00
$ ssh oak
oak[~]$ cd /tmp
oak[tmp]$ pax -rf paxample.tar
oak[tmp]$ ls paxample/
foo

But there's a much easier way. Invoke pax on both machines, and connect the output of one to the input of the other:


$ pax -w paxample | ssh oak 'cd /tmp/ && pax -r && find paxample'
paxample
paxample/foo

pax -w writes to standard output. ssh reads standard input and attaches it to whatever utility is invoked, which of course in this case is pax again. pax -r reads from standard input and creates the files from that “archive”.

pax is one of the lesser known utilities in a typical Linux installation. But it's both simple and versatile, well worth the time it takes to learn—recommended.

Stream and Share Your Media with PlexWeb

Shawn Powers

Issue #237, January 2014

Plex is one of those applications I tend to write about a lot. It's not because I get any sort of kickback or even a discount, but rather it's just an incredible system that keeps getting better. For this piece, I want to talk about PlexWeb, which functions much like the Android app I've mentioned before, but works completely inside a Web browser—almost any Web browser, on any operating system.

You can access PlexWeb by surfing to my.plexapp.com and logging in with your free account. (If you have a static IP at home, you also can connect directly to your home server by bookmarking the URL generated by plexapp.com.) You will be redirected to your home server, and you'll be able to transcode and stream your movies to any computer, anywhere.

I freely admit that I wish Plex was open source. Thankfully, however, its proprietary code does't mean Linux users are excluded. Whether you're using the Plex app on your Android device, installing Plex Home Theater on your Linux machine or even streaming video to your Aunt Edna's Web browser while visiting over the holidays, Plex is an incredible tool that keeps getting better. PlexWeb is free, but if you're interested in experiencing the latest and greatest Plex has to offer, a PlexPass subscription will get you access to features like Cloud Sync before anyone else gets to see them! To get started with Plex, visit the Web site at www.plexapp.com.

They Said It

Never let the future disturb you. You will meet it, if you have to, with the same weapons of reason which today arm you against the present.

—Marcus Aurelius Antoninus

Temptation rarely comes in working hours. It is in their leisure time that men are made or marred.

—W. N. Taylor

We turn not older with years, but newer every day.

—Emily Dickinson

The human tendency to regard little things as important has produced very many great things.

—Georg Christoph Lichtenberg

Getting fired is nature's way of telling you that you had the wrong job in the first place.

—Hal Lancaster

LJ Archive