The 19th Century Meets the 21st

I live at the foot of the Brooklyn Bridge, in a 150-year-old building that used to be the headquarters of the Brooklyn Railroad. Before the bridge was built, barges and ferries docked along the piers. Trains brought people and goods into Brooklyn and beyond. The bridge killed the ferries, the railroad and, eventually, the neighborhood—the heart of Brooklyn in Walt Whitman's day.

After a century of decline, the neighborhood, Fulton Ferry Landing, is being reclaimed by artists and people like me who work anywhere, thanks to the Internet. Disconnected for a hundred years, due to a network of roads that ignored it, Fulton Ferry Landing changed little. Today, it is quickly being reconnected to the rest of the planet, as a result of the most efficient transportation network ever devised.

Figure 1. View of Brooklyn Bridge from Apartment Window

Wall Street—just across the river—is one of the most wired neighborhoods on the planet. The rest of New York, however, lags behind most of North America in terms of connectivity options: few cable modems, little DSL, and ISDN being more expensive than digital leased lines. Most people make do with a modem. After seven years of dial-up access, I decided I could no longer wait for the dissolution of our local telephone and cable monopolies—it was taking too long.

Thanks to Linux, some volunteer work and a bit of luck, the old headquarters of the Brooklyn Railroad is now one of the most wired old buildings on the planet. Each apartment has two data ports connected to a router in the basement. The router is connected to the Internet through a digital leased line. We now have high-speed connectivity, 24 hours a day, for less than it used to cost us to maintain dial-up accounts.

A few months ago, I asked the building's residents to let me wire the building with CAT5 in order to set up a building network, because it made sense both economically and organizationally. Sharing resources, after all, is the whole point of packet switched networking—not wiring an apartment building is perverse.

Three years ago, I used to walk into many businesses that had each desktop computer connected to the Internet through a dial-up connection. Today, most of those businesses have connected their entire intranet to the Internet though a single, large pipe. Dial-up connections are expensive and inefficient; no IT organization would dream of setting things up that way. Apartment building dwellers, however, have barely begun to question the way they approach the problem. Although they enjoy high-speed, permanent connectivity at work, they don't question the AOL dial-up ritual at home.

To date, no one is offering residential network management service. If you want a building network, you will need an on-site expert to set up and maintain it.

In the U.S., the local telephone company is responsible for wires to the building. Residents are responsible for wires in their apartment. The building owner is responsible for the wiring within the rest of the building. I suspect it will take at least another two to three years for people to realize that IP is as fundamental as telephone service. At that point, they will start making noise about wiring capable of carrying data from the basement to their apartments. Network equipment vendors will start building and pricing hardware for this market, and of course, residential network management companies will be formed. In the meantime, we have to plan and build everything ourselves.

My building was not prepared to provide a network infrastructure. I guessed that trying to convince a majority of the shareholders to do so would be a waste of time, so I offered to pay for it all and resell the service to anyone who wanted it. Everyone with a computer has joined. While I would have preferred not to absorb all up-front costs, I have enjoyed the privilege of making decisions without running them by a committee.

The most significant up-front costs are the wiring, the router and the computer providing name, mail and web services. Running CAT5 wires from each apartment to a central hub—in our case, the basement—is never going to be cheap. In most cases, however, it will cost far less than it cost me. The age of the building worked against me.

In the early 20th century, services were run as if they would never require replacement. Electrical wires were buried in plaster walls. Telephones were wired directly to the building's exterior. Telephone jacks were a 1950s innovation, an early example of plug-and-play. Today, architects frequently design electrical systems to be accessible without the help of a demolition crew. Those who are truly forward-thinking will design easily accessible, parallel conduits: one for electricity and one for data. Today, “data” usually consists of telephone and cable television wires. Tomorrow, those two will be joined by computer network wires, which soon enough will also carry telephone and television data.

In our building, nothing is straightforward. Throughout the years, conduits have been run through the wood and concrete floors to carry electrical, telephone, intercom and cable TV wiring. None were large enough to accommodate additional wires. Running a new conduit was estimated to cost almost $1000 per apartment. That expense was impossible to justify at that time.

Figure 2. View of New York from Apartment Window

While I was mulling over what to do about this network wiring problem, another arose. The building ran out of telephone wires. Whoever did the capacity planning when the central wires were installed never considered fax lines, dial-up lines and two or three voice lines per unit. Also, the wires were old—many broke due to corrosion and many were static-filled. Clearly, I had another project on my hands.

Actually, I was lucky the building reached the end of its telephone network lifetime when it did. Any earlier, and I would not have had the foresight to run network lines in parallel with the new telephone lines. Any later, and I would probably have invested in a high-speed solution for myself and would not even have considered doing the work on a communal scale. The incremental cost of running the network wires was negligible, so I decided to go ahead and do it.

The great irony, of course, is that everyone has now canceled the lines they had for their dial-up service. Under the old system, we would now have plenty of lines.

Besides the wiring, the router and central computer turned out to be the other big cost in this sort of operation.

The router is expensive because each apartment needs its own subnet. I asked Cisco what they sold that could do the job. They literally answered that buying a router from them would cost me “both arms and both legs”. They did, however, suggest a “cheap” alternative: a low-end router and a switch, a solution that would have cost me about $3,500. I was not willing to spend half that much to solve this piece of the puzzle. I was fairly sure I could build what I needed using Linux.

My neighbor, a Linux guru, assured me I could. Before long, he and I had done the research and mapped out a strategy that worked.

The hardware we needed was free. Businesses all over town have mountains of 486s gathering dust in their storerooms. They are thrilled to give them away! As you know, the operating system we decided to use was also freely available.

Since our router was going to be a general purpose computer, we decided to run all of the shared services on the same computer. This simplified a lot of management issues. It also made disaster recovery relatively straightforward. We built a second, identical machine that can be swapped in for the first at a moment's notice. This sort of approach is practical only if a single machine is involved.

Figure 3. Apartment Building

At the start of the project, I had one overriding goal: keep the architecture as simple as possible. I could not guarantee a networking wizard would be available when things failed. In fact, our backup system administrator is a 12-year old resident who knows little about computers; I figured she would be easier to train than most adults. Knowing I was going to have to write thorough documentation about everything I implemented helped me stick to my goal.

We knew Linux could support multiple Ethernet interfaces. We were not sure where to find a card with Linux drivers that could interface with our DSU. A bit of Net research turned up a Canadian vendor, Sangoma Technologies, that seemed to be selling exactly what we needed. Five minutes on the phone with one of their Linux guys convinced me their WAN pipe product would do the job. At $550, it was the most expensive piece of hardware I had to buy, and it certainly beat Cisco's “cheap” solution.

I now had all the pieces: a frame-relay line from the outside, a DSU, a router, a hub, a general purpose computer, wires and a willing alpha tester. I just had to work out the details.

We originally planned to isolate each apartment behind an Ethernet interface. Of course, that seemed ridiculous for those with a single Windows 95 box. We then considered putting all the single machine apartments on their own segment. This presented an evolutionary problem. Whenever anyone bought a second machine, we would have to change IP addresses, physical connectivity, etc. We were stuck between over- and under-engineering the network, until my neighbor remembered some work he'd done earlier for a client in Atlanta.

He remembered Linux supports something called Ethernet aliasing. This allows a single interface to support multiple networks. For example, a single Ethernet card can be configured to support ten apartments, each of which is assigned its own subnet. This turned out to be the perfect compromise. We could logically isolate each apartment without having to use many Ethernet cards and several computers.

If an apartment grows into needing more thorough isolation, we can upgrade it to its own Ethernet board! By the time all available slots are used in our current 486, it will have to be replaced in order to deal with the Y2K issue. By then, maybe the router vendors will be selling solutions with more down-to-earth prices.

When I first began discussing the network idea with other residents, security seemed to be at the top of their list of concerns.

We worked out a few security schemes using proxy and masquerading facilities. Whatever we ultimately decided to do had to be configurable on an interface-by-interface basis. I personally wanted access to my computers from the outside world. Luckily, Linux supports that sort of granular security.

One day, I happened to mention the various options to a relatively computer-savvy neighbor who runs a local area network in her apartment. She was horrified that I would consider implementing a security scheme at the building level. She wanted control over her own security so that she could access her machines from anywhere on the Net. After a bit of discussion, we realized the original requests for high security were all from people who used Windows 95 to dial up through AOL.

It turns out the concerns were the result of alarmist articles in the local papers—security threat articles fail to put the subject in perspective. The least savvy are most easily frightened, even though they are least at risk since they use operating systems with few services that can be abused.

Having come to that realization and remembering our “keep it simple” goal, we decided to leave security up to the individual apartment. After all, AOL does not provide any special security to the lone PC connecting through its network.

We toyed with the idea of allowing everyone to register their own domains, but finally decided against it as this would have created too much work. Instead, we registered a domain for our building, 8OldFulton.com, which is related to our physical address. This is one of the few cases in which I think geographic addressing of any kind makes sense. Given the choices we made, the administrative burden of adding a machine or cluster of machines is relatively light.

Mail service is not yet settled. At the moment, we run a POP3 server, because it is essentially administration-free. POP3 is not, however, particularly friendly for people who travel a lot or use multiple computers. Therefore, it is very likely I will eventually bring up an IMAP4 or web-based mail server.

Anyone who wants a more flexible e-mail system immediately in place needs to set up and maintain their own.

It is tempting to offer web hosting services for everyone in the building. This would, however, run counter to our “keep it simple” goal. Although there is little complicated about allowing people to set up and maintain home pages, the peripheral support involved is potentially significant. As people become more sophisticated and web development and management software becomes easier to use, my policy will probably change.

Currently, the web server we run serves only private building information: contact information, bylaws, house rules, meeting minutes, etc. I am sure some public information (e.g., directions) will eventually find its way onto the server.

As with special mail servers, anyone wanting to run their own web server is free to do so, on their network segment.

Figure 4. Apartment Building Showing Proximity of Bridge

I find it hard to quantify the difficulty involved in setting up the network. My neighbor and I both have done quite a bit of UNIX system administration. Tasks that seem easy to us, like configuring sendmail or name service, might require quite a bit more effort for a beginner. Luckily, the Linux community is extremely supportive. Before embarking on a project like this one, anyone unfamiliar with system administration should make sure they know how to deal with the following issues:

We found the most difficult task was setting up the WAN pipe. Because it is not a common router, the telephone company and ISP tend to blame it for every problem—Sangoma is used to this. They ship excellent debugging tools with their hardware, and their installation support personnel are top notch.

Having the network in place is a great first step. We now have something very solid to build on. Immediately, we all had better Internet access.

We are currently evaluating the purchase of a RaQ from Cobalt Networks. It would provide a more flexible e-mail system and would allow each apartment to maintain its own web site. Under the hood, the RaQ runs on Linux, of course!

Within a few months, I suspect most people will have given up their fax lines. They were often justified because they were shared with the computer. Now that they are stand alone, it probably makes more sense to use the JFax or efax services. It is cheaper (JFax) or free (efax), and more flexible than a dedicated phone line.

When we can buy IP telephones that look and act like telephones, we will buy them. I can easily imagine this building buying no local lines from Bell Atlantic within five years. Between IP telephones and the incredible calling plans offered by our national cellular providers, local lines might not make any sense.

Installing a building security camera will now cost us about $800—the price of an IP camera.

We will likely bump the network up from 10BASE-T to 100BASE-T within the next two years. I suspect a gigabit network will become necessary once we all start using net-based video broadcasts. If that turns out to be impossible over copper, we will run fiber through the old telephone wire conduits. The wire was left in place so that it would be easy to pull the fiber.

Acknowledgments