An Interview with Andreas Leimer, CTO of Inalambrica.net

Back row from left to right: Geraldy Castillo, Esteban Barrientos (Pato), Alfredo Delgado (Alf), Andreas Leimer (Andy), Alberto Brealey (Beto) and Paulo Campos. Front row: Randall Aguilar, Vera Sanchez, Alina Castro, Agustin Guevara and Federico Figueroa (Fede).

I recently interviewed Andreas Leimer, Chief Technical Officer of Inalambrica.net. Inalambrica, based in Costa Rica, is using Linux to offer inexpensive, reliable internet connectivity around the world. Here is their story.

Phil Start from the beginning. What got you interested in Costa Rica?

Andreas My wife is Costa Rican. She got pregnant, and she decided to come home to have the baby.

I came to use the Internet here, and it just didn't work. Dial-up was very bad and communication was very bad. So I met some Costa Ricans and told them we need to start a company to bring better communications, better solutions, here to Costa Rica. At the time of course, the OS of choice was NT.

Phil What year?

Andreas Five years ago—1997. We tried many different solutions. We tried compression software that worked on NT proxies, and that just didn't work. It sort of worked in the office, but when we went to demonstrate it, it just didn't work. And then we had to diagnose it for a couple of weeks to figure out how it worked or why it didn't work. At the time, all this was based on dial-up connectivity.

Then we learned about direct tier-1 type satellite connectivity with DVB. We were using NT, so it was expensive. Easy to configure but hard to diagnose—it just didn't work.

This led us to a Harmonic Data Enterprise BR-501.

It was a very badly configured Linux system, which almost kept us from switching to Linux. The drivers were especially bad. They had problems with their configuration. They had no simple way to use interfaces. It was a very complicated thing. That happened about two and a half years ago, about a year and half after we failed completely with any Windows applications that we had tried to use.

Phil So that was really your introduction to Linux. It was a pain to configure, but at least it had the potential of working.

Andreas Yes. We installed a couple of these systems, and we started testing. I learned of a technician at a local university that knew quite a bit about Linux. He said he wanted a job, and I hired him. That's how this mess started, and that was approximately two and a half to three years ago. We removed the drivers out of that box and built our own Linux box and tested it. We told Harmonic Data in San Diego what we'd done, and they told us we weren't allowed to use the drivers. We said okay, we'll respect that, we won't use them (they're not open-source drivers). Right around that time, Telemann came out with a card called the SkyMedia 200D, and they had open-source drivers. They actually gave us the source code.

Phil This is the card to connect to the dish?

Andreas Right, this is the card that plugs into the PC running a Linux distribution, with a satellite dish. We also use the PC as a proxy caching engine, and then we built some caching engines to give us multicaching over the one-way satellite. The rest is history. We started installing systems. We had customers with different requests to install the system. The customers said we can't use Linux; we can't do anything with this; we don't know how. So that's what gave us the idea to start with PHP.

We hired some PHP programmers and started building an interface, to change gateways, for example, and that was the beginning of the interface of INATS. Each customer had a different need—I'll buy this if you do this or if you'll do that. And that's how INATS was created and developed, through my experience with going to the customers.

That was two and a half years ago. This last year we've made our greatest developments. We've had a little more money to spend, and we've gotten to a stripped-down distribution. Down to a very small size, we've created our own base. It was based on Slackware, believe it or not. Our first base was a stripped-down Slackware distribution. [See the article “A Conversation with Alfredo Delgado of Inalambrica.net” in the January/February 2002 issue of our sister publication, Embedded Linux Journal.]

Phil Then you added your own packaging and administration system?

Andreas Yes, our own packaging and administration system called INATS (Internet Networking Administration Tool Software). INATS controls each of the modules. That's it in a nutshell.

Phil When did you start the development of this intelligent interface?

Andreas About three months after our first satellite installation. That was at the Ulatina University here and Earth University in Guapiles. By the way, in Earth it rains 320 days a year. We had some problems with the 76cm dish. That's when we learned about the 1.2m dishes and Squid; we really liked the way Squid worked. If we couldn't get the packets over the satellite during a fade, then Squid would revert back to its internet channel to pull the packets. So we really liked Squid, just for that simple feature.

Phil So all of this is one-way satellite?

Andreas One-way, everything simplexed—to bring broadband down. It's only for downloading. We've found other products like cellular CDPD, where we'd connect it through a serial port, and it would be able to reach speeds of 512Kb on the satellite down. CDPD is a 12Kb connection, which is very slow and bad to use for internet surfing. But we found with the dish and the caching engines at multilevel, it was very fast. Also, we could do some video streaming on the satellites with a handshake on the cellular. Of course it's UDP multicast, but it works very well—high quality.

Phil Well, one other thing that you've got is compression. Is that inspired by CDPD?

Andreas No, compression was inspired by the fact that I still have phone dial-up at my house, and I don't have a wireless, high-bandwidth connection. I wanted to be able to get information from the Web faster, so we built a compression engine, using Squid, of course, to do the caching part of it, and then compressing the images and transmitting through phone lines at a high speed. I do have a 28.8K modem at the house, and with compression, it outperforms my neighbor's 56K modem.

Phil I know few people reading this magazine are going to understand how Costa Rica works as far as communications.

Andreas It's a total monopoly. I think it's one of the only telecommunication monopolies in the world—owned by the government. Costa Rica is completely controlled by the ICE and RACSA. RACSA is the primary internet provider. They consider themselves an internet access provider; there are no ISPs in Costa Rica. ICE is also an internet access provider. They are the same company with two different names. ICE owns the infrastructure, and RACSA sells on top of the infrastructure.

Phil And what they are selling mostly is dial-up connectivity?

Andreas Dial-up and dedicated lines. Currently in Costa Rica there are about 900 dedicated lines. ICE is working on a DSL project that could become very popular here. [This is covered in Phil's interview with Guy de Téramond in the January 2002 issue of Linux Journal.] The cable company here does have some cable modems installed, but they have many problems. One of the many problems in Costa Rica is mold, of course, and earthquakes. We had 3,000 earthquakes last year, 70 of which we felt. Power lines go down, cables get broken, fiber gets broken. They are talking about constructing some fiber around the central valley to connect all of the DSLs to all of the homes. We've done some low-earth orbit satellite technology here, but it really doesn't work. Iridium went out of business; Orbcomm is another company that has LEOs. We can use that frequency, but the satellites are at a lower altitude, have very low bandwidth, and the satellites hide behind the mountains. We get some activity for two hours, and in the end it just doesn't work.

Phil That's really the problem with LEOs—they aren't accessible.

Andreas So, then we went to the fixed satellite two and a half years ago, and it works absolutely perfectly. One of the things that we engineered and take pride in is having first-level backbone connectivity, putting a Squid caching engine at the teleport. Then, parenting with a satellite receive cache engine on a multilevel over a simplex system—we're actually pioneers of that. We've actually written and filed a patent, and it looks like it's going to be accepted. It's already been filed for two years, and they haven't found anybody with the same type of patent.

Phil It's essentially multilevel caching over...?

Andreas A simplex satellite. We don't own any patent on the caching; we own the patent on the idea of the simplex satellite multilevel caching, and we are using Squid.

Phil With all of these systems, with the broadband download, what's the other way?

Andreas Just a regular connection. It could be a dedicated line, a DSL, cable, cell phone or dial-up. Whatever it takes to make a connection to the backbone. When I say the backbone, I'm talking about the Costa Rican backbone that connects us to the Internet. It all travels through the local monopoly, all the packets on the outbound. The inbound over the satellite basically is to speed up your HTTP, HTTPS and FTP over HTTP. Basically that protocol is all that we handle over the satellite. Caching web content on level one in the US.

Figure 1. INATS Diagram

Phil How does INATS fit in to all this?

Andreas The idea in principle and the path of the packets are simple to understand in the diagram form [see Figure 1].

When it came to the server, nobody seemed to understand a Linux server, or very few people understood. This made our market very small. So, we determined it was time to start building a Linux-in-Windows or Windows-in-Linux-type machine—button pressing. That's what people like to do, press buttons, add IPs and basically not have to diagnose a server requiring stability and ease of use, like an appliance. That's how INATS started.

We started originally with the gateway program. All that we had was a customer that had two different gateways, two different internet connections; if one were to fail they would switch to the other one. Their main criterion was to have connectivity 24 hours.

So from the network managers' level, they didn't have control of the router or didn't know how to program it. They needed something simple so they wouldn't have to change 150 computers from one network to the other, which takes hours to do. We built the simple PHP interface and programmed everything. It was just a button that would switch from one network to another. Obviously, we had set all of the parameters inside of the Linux machine, and through their web browser on their Windows machine—they just press a button.

From that we developed using it as a proxy, adding routing, subnets and control reports. The control aspect of it was nice because they were tired of the sales department going to the wrong sites. You know, XXX sites, whatever. And they wanted to account for it, saying “hey, stop doing this.” If they wouldn't stop, then they actually would deny those sites. So that was one of the programs that we wrote, what we called our control program, I guess. It has three different levels: unlimited use, denied sites and allowed sites.

“Denied” means that we deny these sites on the subnet or the whole net, or we only allow certain sites. “Allowing” is meant for schools, and we have some high schools that use the program. They wanted to give Internet to the classroom, but they were worried about students going to the wrong sites; you know how you can type in something random and a sex site shows up; you don't have to do it on purpose, it just happens.

Phil Whitehouse.com vs. Whitehouse.gov is a perfect example.

Andreas Exactly. So what the teacher can do is program into a database the sites that children are allowed to go to. They can't click through; there are no ways to go to any other pages—only those pages. Two things happen when this occurs. One is the pages cache, and then if you have 40 people in a classroom, they have a cached web page at 1.6 megabits per second. It's wonderful. Saves bandwidth for us and we can reduce the price for them for their connectivity to the Internet.

Phil What are your goals, who are potential customers and where are you going?

Andreas Costa Rica has a very small market. But per capita, it's a very technologically advanced market. We currently are working with large companies here, and we are doing some pilot projects with the PTT.

Worldwide, or at least Latin America-wide, we've signed a contract with a large company called Bismark International, which has subsidiaries in Latin American countries including Brazil, Peru, Chile, Ecuador, Colombia, Venezuela, Uruguay, Panama and Mexico.

Currently, in Mexico we have a test system we are doing with Telcel there. And, they signed a contract with us for us to provide them with the software platform that will integrate hardware items off the shelf. And that's basically what INATS does using Linux as its OS. With different open-source products, plus some of our own that we created, we're operating different network appliance-type servers. The contract that they've asked for is approximately 30,000 different installations over a three-year period, and it could lead us to bigger and better things.

Phil Do you see Latin America as the primary market?

Andreas We'd like to take it into other markets and are looking for distributors in other markets. Currently, yes; in the future, no.

Phil This is sort of a Costa Rican question: Why is this the right answer, as opposed to, for example, decent fiber connectivity, which they're beginning to get?

Andreas Well correct, but even the fiber seems to saturate during the day, and the speed of the response directly off the one hop from the US backbone seems to compete very well with the fiber. It's amazing actually. We've been running tests these last few weeks. The fiber, when it's empty, is very fast. But when it's being used, it tends to slow down. The speed of the satellite, especially in remote areas where there is no fiber, is almost the same speed as the fiber. With the one-way dish, we're averaging approximately 370 milliseconds per round-trip. With clean fiber we're averaging about 140-180 milliseconds per round-trip. When fiber is saturated during the day here in Costa Rica (the country, by the way, has 90MB coming in) our fiber connection goes up to 500-600 milliseconds. And as we are like a highspeed motorcycle out to the US backbone, we are like a tractor-trailer down over the satellite. Our response times are very good over the satellite. And, it is used for areas where you can't get large fiber interconnectivity. There are areas in Costa Rica that have dedicated lines, and the copper can't go above a 384 or a 128 or even a 64Kb connection in some areas. We can compliment that dedicated line with a dish to receive more bandwidth in areas that can't have a larger than 64Kb connection.

Phil As far as rural, I know in the US, for example, that with DSL connections at least, Qwest specs them so that if you're more than three miles from the central office, you're screwed. And it seems that in this country there are 240 telephone offices in the whole country.

Andreas And everybody lives further than three kilometers from that office. So those areas have limitations. The 802.11 spectrum is very saturated. Even on the 802.11 that isn't saturated, we've complimented it with a dish to the point where we can use the 802.11 as our backbone connection and use the dish for the downloading directly off the US. When it's local connectivity, we don't use the dish but we do see about 70% of the traffic coming off the US backbone. And that is for sites and information from companies that you want to be able to get quickly. So for a business, this is a great item to have. Of course the caching makes it a lot faster. The caching helps; it makes the speed incredible. Without the multilevel caching, we wouldn't have this kind of speed.

Phil I assume your customer base is primarily business?

Andreas Yes, small and large businesses. We are working on doing some home-user-type applications where a condominium actually would share one of these satellite dishes and wire it into each apartment. And, with the INATS software we can give good quality service to each apartment.

This brings me back to the OS that we're installing into INATS so we can give certain service-level agreements to each customer. If they're sharing a 512Kb when there is a 20 to 1 ratio, the minimum would be a 25Kb SLA. And we can guarantee that, using a 2.4 Linux kernel with an OS built into it.

Phil As opposed to inventing all this for...?

Andreas Exactly. And making it cost-effective for the customers. Total cost of ownership is probably the lowest in the world.

Phil That brings up one thing. Other than reliability, you mentioned quality of service. What else in Linux has made this an easier task than trying to do it with brand M or something?

Andreas MRTG, multirouter traffic grapher, is an excellent program giving us knowledge of how much bandwidth is being used and how much is needed. Or, if we can reduce connections in different businesses. Some businesses over-purchase. We actually can help save money by having that MRTG and using it in our administration tool. It's wonderful. It reads the traffic that's going through the NIC cards; or if it's a router card that's installed, it reads packets going through the router cards. It gives administrators management of their network; if they are saturated and need more bandwidth they are going to know.

The Squid Analysis Report Generator (SARG) is another great program written by Orso [orso@onda.com.br]. SARG is a tool that allows you to view where your users are going to on the Internet. It generates reports in HTML using the access_log file, with fields such as: users, IPAddresses, bytes, sites and times. It's very nice tool for control or accountability.

Using DHCP, of course, is a nice feature, as is using the routing, the proxying video streaming. The Darwin streaming server is wonderful with the IceCast radio streaming; you can plug in a radio or tape for viewing or learning and the Darwin streaming actually takes videos, compressing them into a digital format so that they can multicast onto the network and people can use that for viewing or learning.

Phil So essentially...?

Andreas All open-source products.

Phil What you're doing is taking a whole bunch of open-source stuff and putting it together, and your added value is that you made it so Joe Moron can come in and take all of this open-source software and turn it into something that will solve this problem.

Andreas You don't have to learn a new operating system, but you get the benefit of using one.

Phil Okay, what did I miss?

Andreas I think that's pretty good. This same type of service is available using Cisco equipment—that is, routing, a caching engine and QoS. To do everything that we're doing would cost about $130,000 in Cisco hardware.

Phil And what does this cost from you?

Andreas Small network, 20 people, 20 bucks a month (if they furnish the hardware). If not, we can finance that (Costa Rica only at this time). It's off-the-shelf items. Try to buy Cisco products in different countries; you have to order it because there are limited supplies. Our idea is to use off-the-shelf items; go to the store, get a PIII or motherboard or an IDE hard drive and plug it in. And on Flash disks too. We are finishing that.