LJ Archive

UpFront

diff -u: What's New in Kernel Development

Zack Brown

Issue #257, September 2015

Boot times can become slow on systems with many CPUs, partly because of the time it takes to crank up all the RAM chips. Mel Gorman recently submitted some patches to start up RAM chips in parallel instead of one after the other. One of the main problems with trying to implement such a feature—and one of the main reasons such patches haven't made it into the kernel before—is the need to avoid slowing things down for smaller systems.

Mel's patches modified the kswapd code to give each CPU its own RAM initialization thread. On smaller systems this theoretically would amount to no change at all, while larger systems could see dramatically reduced boot times.

An initial test by Waiman Long reported a 25% reduction in boot time on his 12 terabyte system—from 404 seconds to 298. And when Peter Zijlstra and Mel asked if this made a worthwhile difference to him, Waiman replied:

Booting 100s faster is certainly something that is nice to have. Right now, more time is spent in the firmware POST portion of the bootup process than in the OS boot. So I would say this patch isn't really critical right now as machines with that much memory are relatively rare. However, if we look forward to the near future, some new memory technology like persistent memory is coming, and machines with large amount of memory (whether persistent or not) will become more common. This patch will certainly be useful if we look forward into the future.

And Scott J. Norton also added, “100 seconds really does matter and is a big deal. When businesses have one of these large machines go down, their business is stopped (unless they have a fast failover solution in place). Every minute and second the machine is down is crucial to these businesses.”

There was a bit of a push by Andrew Morton for Mel to simplify his code, but Mel felt that Andrew's suggestions could make things worse, such as forcing the kernel to rely on user-space code. And so long as systems keep getting bigger, patches like these seemed destined for eventual acceptance.

Intel has invited Linux kernel engineers to assist in development for chips that are so new that their in-house developers must code on software simulations of the eventual hardware.

The patches, released by Dave Hansen, wouldn't run for anyone outside Intel—since no one else has those chips—but he was hoping for feedback on their implementation of Memory Protection Keys for user space.

The underlying idea involves utilizing previously unused bits from existing registers and introducing new registers and associated assembler instructions to secure system memory on a page-by-page basis. Essentially, this gives users the ability to enable a particular set of actions on a given set of pages, while prohibiting others.

When Ingo Molnar asked Dave to list some potential use cases for this chip feature, Dave replied that there were various things that a user might want to protect, such as the following: “Data structures like logs or journals that are only written to in very limited code paths, but that you want to protect from 'stray' writes.” Or: “a database where a query operation will never need to write to memory, but an insert would. You could keep the data R/O during the entire operation except when an insert is actually in progress.”

And, Alan Cox also suggested:

You also can use it for certain types of emulator trickery, and I suspect even for things like interpreters and controlling access to “tainted” values.

Other obvious uses are making it a shade harder for SSL or ssh type errors to leak things like key data by reducing the damage done by out of bound accesses.

Ingo asked if there could be any issues surrounding this feature existing on some CPUs but not others. And Dave replied, “It's always a problem with new CPU features.” He then went on to say:

I've thought a bit about trying to “emulate” the feature on older CPUs using good ol' mprotect() so that we could have an API that folks can use today, but that would get magically fast on future CPUs. But, the problem with that is the thread-local aspect.

mprotect() is fundamentally process-wide and protection keys right are fundamentally thread-local. Those things are going to be hard to reconcile unless we do something slightly extreme like having per-thread page tables.

The discussion got technical, but clearly the main question is how to support the new chip features, rather than whether to support them at all.

Luis R. Rodriguez has extended module signing to support signing firmware as well. Eventually, he figures it should be possible to sign user data too. This seemed to be a natural extension of existing features and not very controversial. But, there are certain differences between the firmware signing code and the module signing code; for example, Luis' code introduces separate files to contain the firmware signatures as a means to better handle licensing issues.

Luis' patches also “do not taint the kernel in the permissive [firmware] signing mode due to restrictions on the firmware_class API; extensions to enable this are expected, however, in the future.”

Android Candy: Copay—the Next-Generation Bitcoin Wallet

Shawn Powers

Issue #257, September 2015

When I hear the word “copay”, I think of the doctor's office. Thankfully, the Copay app from the folks at Bitpay doesn't cost you anything, and it keeps your Bitcoin healthy and secure. I've mentioned many Bitcoin wallet applications and cloud solutions during the past few years, but Copay truly is different. It has features other wallets can't touch, such as:

  • Truly cross-platform with availability for Android, iOS, Windows, OS X, Linux and Chrome.

  • Fast Bitcoin communication with Bitcoin network, no blockchain download.

  • Payment verification (BIP-0070-0073).

  • Shared wallets (multi-party transfer approval).

  • Wallet backups, multi-device access of same wallet.

  • 100% open, downloadable source code hosted on GitHub.

I'll admit the first time I tried Copay, I didn't quite understand the hype. It feels like every other app accessing a cloud-based Bitcoin wallet (for example, Coinbase). But Copay is an actual wallet, with private keys stored only where you back them up. Thankfully, it allows simple backup of your wallet keys, so you can access your Bitcoin from multiple locations. In fact, you really really really need to have your wallet backed up and/or accessed from multiple locations. If you lose your phone and don't have a backup of your wallet, there's no way to recover your Bitcoin.

(Photo Credit: https://copay.io)

Thanks to its devotion to multi-platform open-source development and attention to security while never compromising flexibility, Copay gets this month's Editors' Choice award. If you want to manage your own Bitcoin without trusting an on-line cloud provider, Copay puts the control in your hands. Check it out today at https://copay.io.

BitTorrentSync: Dropbox for Nerds

Shawn Powers

Issue #257, September 2015

It's not really fair to compare Dropbox directly with BitTorrentSync. First of all, my title implies Dropbox is somehow inferior. To be honest, I haven't found anything that works as smoothly as Dropbox when it comes to sync reliability and ease of installation. That said, although it has incredible strengths, it also has a few shortcomings. One, it's not free, and if you have a lot of data, it can be expensive. Two, your data is stored on the Dropbox servers. The second one is a showstopper for many folks.

BitTorrentSync uses the BitTorrent protocol to keep multiple locations in sync. Everything is stored on your own machines, and you're limited only by your own storage limitations. The basic features of BitTorrentSync are free, but like Dropbox, there is a paid version. For me, quantity of storage is the most important feature with a syncing app, and with BitTorrentSync, that's unlimited. If you've been in the market for a Dropbox replacement, I urge you to check out BitTorrentSync (or more properly just “Sync” as it's been rebranded, probably due to the bad press “torrents” get). It's matured to the point that I trust it with my data, and its cross-platform nature makes it easy to distribute. Grab a copy at www.getsync.com.

Non-Linux FOSS: PlexConnect

Shawn Powers

Issue #257, September 2015

It's no secret that I'm a huge fan of Plex. It might be a secret, however, that I live in a house with quite a few Apple products. That said, I find the Apple TV to be one of the most limiting, frustrating set-top boxes to work with. (I'm sure most readers would agree.) I prefer to be a lover, not a hater, so I searched long and hard to find a way to make the Apple TV suck less. Thanks to PlexConnect, I succeeded.

The Apple TV is still not rootable (if you see claims that it is, you're likely being bamboozled). PlexConnect works around the walled garden of iOS by hijacking an official Apple app (the Trailers app specifically) and allowing access to a Plex server.

The open-source PlexConnect is really just a brilliant translation layer that hijacks DNS (pointing trailers.apple.com to the PlexConnect server IP) and feeds the Apple TV data formatted like it expects. Rather than showing a listing of recent movie trailers, however, PlexConnect shows a direct interface with your Plex media server. And to be honest, the interface is actually surprisingly pleasant to use.

If you're stuck using an Apple TV for your living-room media playing, or if you'd simply like to hop over that walled garden just because you can, check out PlexConnect today. It's open source and available on GitHub: https://github.com/iBaa/PlexConnect.

iTVMediaCenter: Scam or Brilliance?

Shawn Powers

Issue #257, September 2015

The folks at iTVMediaCenter recently contacted me about their one-stop-shop solution for cord-cutters. For $14.99, they sell a program that consolidates tons of on-line media into a central location so you can watch it on demand. The problem is, it looks like the application does little more than open the same Web sites you can open with a browser. Also, the “one-time fee” is rumored to be an annual fee. I can't verify whether the fee is recurring, but honestly, I don't recommend it either way. What I can recommend, however, is to visit their Web site!

Although the application is seemingly a bust, the Web site is free, and it does a decent job of linking to dozens and dozens of on-line streaming sites. If you're looking for some free on-line media, but don't know where to begin, go to itvmediacenter.com and check out the on-line offerings. I urge you to be cautious about the application, but I found the free on-line catalog to be very helpful.

Finite-Element Methods for PDEs

Joey Bernard

Issue #257, September 2015

One of the common classes of equations that is encountered in several branches of science is partial differential equations. So in this article, I look at a software package called FreeFem++ that is designed to help you calculate these partial differential equations (www.freefem.org).

One popular method of solving these types of equations, and the one FreeFem++ uses, is the finite-element method (https://en.wikipedia.org/wiki/Finite_element_method). The basic idea with this method is to take the entire problem domain and subdivide it into a mesh of smaller regions. You then apply a simplified version of the partial differential equations that still is locally valid. This makes the problem a tractable one that actually can be solved in a reasonable amount of time.

FreeFem++ is available in several different flavors. The earlier versions were named freefem, freefem3D and freefem+. The latest version is called FreeFem++ and is written in C++. It can be compiled and runs on all three major operating systems, Windows, Mac OS X and Linux. Since this is Linux Journal, I focus on Linux here. On Debian-based distributions, installation is as easy as:

sudo apt-get install freefem++

The other versions also are available as the freefem and freefem3d packages.

Although I am specifically discussing solving electromagnetic fields in this article, FreeFem++ is a general finite-element method solver. This means it should be able to deal with most partial differential equations.

Once it is installed, you will want to start using it. The first thing to be aware of is that FreeFem++ is designed to be used for “production-level” work—meaning active, high-level research work. As such, it does not have a pretty front end to help new users through their first attempts at using it. It is best to think of FreeFem++ as a programming language that you use to write a program to solve your problem.

The first step is to define the geometry of your problem. This step is usually called meshing, or building a mesh. FreeFem++ does not include any CAD functionality to build geometries directly. You can, however, generate meshes automatically from a set of border descriptions. FreeFem++ can take a set of equations describing your geometry and generate a mesh based on the Delaunay-Voronoi algorithm. As a simple example, let's say you wanted to build a square object. You could create the related mesh with these commands:

int sides = 8;
mesh Th = square(sides, sides);

From this first example, you should notice that the language FreeFem++ uses is very similar to C/C++. Variables are typed and can store values only of that type. The language is also polymorphic, so commands and operations will do different things based on the types of the parameters or operands.

You can define more complex shapes with border functions. An example looks like this:

border aa(t=0, 2*pi) {x=cos(t); y=sin(t);}

You then can hand these types of objects in to the buildmesh() function to generate the same type of mesh you would have received from the higher-level square() function.

If you want to see what these equations actually look like (to verify that you have it correct), you can use the plot() function to visualize them. You can hand in the border functions you may have created or even the mesh object you get from buildmesh().

Once the mesh is created, you need to create a set of two-dimensional spaces from this mesh in order to solve your problem. This is done with the fespace function:

fespace Vh(Th, P1);
Vh u,v;

The P1 parameter tells fespace what type of finite element you want, whether it is continuous or discontinuous, smooth, linear or with a bubble. (A rather large set of possibilities is available that will be left as an exercise for the dear reader.)

Next, you need to define the finite-element functions within this newly generated space—in this case u and v.

Now that you have all of the background scaffolding built, you need to define your problem and solve it within your problem geometry. You can use the problem type to define a more complex problem. As an example, you might want to look at the cooling of a hot plate. The following would set up the problem for you:

mesh Th=square(30,5,[6*x,y]);
fespace Vh(Th, P1);
Vh u=u0, v, uold;
problem thermic(u,v)=int2d(Th)(u*v/dt + k*(dx(u) 
 ↪* dx(v) + dy(u) * dy(v)))
              + int1d(Th,1,3)(alpha*u*v)
              - int1d(Th,1,3)(alpha*ue*v)
              - int2d(Th)(uold*v/dt) + on(2,4,u=u0);

The variable name thermic is now a function call. When you issue the command thermic, FreeFem++ will go ahead and solve this problem that you defined. The purpose for this method is to be able to define your problem and make alterations and adjustments before actually solving it.

If the problem is simpler to define, you can use the solve command to define your problem and do the solving step immediately. For example, if you wanted to model motion on a membrane, you could use something like this:

solve Laplace(phi,w)=int2d(Th)(dx(phi)*dx(w) 
 ↪+ dy(phi)*dy(w))
          - int2d(Th)(f*w) + on(Gamma1, phi=z);

where the appropriate finite-element space and functions have been defined. Once FreeFem++ has solved your problem, you can use plot with the finite-element functions to visualize the actual results of this numerical solution.

Although being able to visualize the results of your work immediately is important, you need to have a way of saving this work so you don't need to repeat any calculations unnecessarily. You can save meshes that are generated with the savemesh function. You simply need to hand in the mesh to save and a filename to use:

savemesh(Th, "my_mesh.msh");

You can reload this mesh at a later time with the readmesh command, for example:

mesh Sh = readmesh("my_mesh.msh");

Outputting results is a bit more of a hassle. You have access to the standard C++ input/output streams, specifically cin and cout, so you can dump out the numerical results that way. You also can create a new output stream with ofstream to write things out to a specific file, rather than to what standard output is redirected to. In this way, you have full control over what data gets saved, and what format this file and its data uses.

Now that you've read this introduction to FreeFem++, you should take a look at other tutorials and documentation on the Web. Several good examples are available that should give you at least a starting point to solve the specific problem you are investigating. If the problem you are trying to solve is especially large, FreeFem++ also has MPI support available. In this way, you can spread your calculations over potentially hundreds of CPUs and hopefully get even more work done.

They Said It

The highest reward for man's toil is not what he gets for it, but what he becomes by it.

—John Ruskin

The greatest conflicts are not between two people but between one person and himself.

—Garth Brooks

I can't give you a sure-fire formula for success, but I can give you a formula for failure: try to please everybody all the time.

—Herbert Bayard Swope

Who begins too much accomplishes little.

—German Proverb

It is impossible to live without failing at something, unless you live so cautiously that you might as well not lived at all. In which case, you've failed by default.

—J. K. Rowling

LJ Archive