Usenet is Still a Strange Place

Who ever said that Usenet is dead? It most certainly is not. Even though Usenet has changed its face over the last, say, ten years, the simple idea of exchanging ideas and having discussions among a large number of people by posting articles on a global messageboard has perpetuated. Yes, for those of you who still don't know what this is all about, we are talking about newsgroups, an important part of the Internet, albeit one heavily neglected by the media who usually equate Internet with Word Wide Web. The spread of the Internet, the increased availability to more and more households, has not only influenced the way we shop, trade stocks, etc., but has also transformed the old messageboard system of Usenet. But rest assured, no matter how much things have changed over the last few years, Usenet is still a strange place.

This strange place has, for a long time, been regarded as inhabited only by full-time geeks, nerds with Coke-bottle glasses trying to download obviously fake pictures of naked celebrities when they are not busy discussing the technological details of the Starship Enterprise's Warp-Drive.

But while these people certainly exist, there is another side to Usenet: with more and more people getting on the Internet it is true that on the one hand the signal-to-noise ratio has been lowered, which is why some people might say that “Usenet is dead”. On the other hand, more and more people are exchanging ideas, helping each other out and having highly interesting discussions that take place every day in this community. If you invest a little time, I'm sure I can convince you that Usenet is quite alive.

The history of Usenet is so closely linked to the history of the Internet itself and the history of UNIX (and thus Linux) that it is worth while to spend some time finding out how things started. Let's take a trip back to the year 1976 (a good year, I might add)--the place, how could it be any different, AT&T Bell Laboratories, where a new utility called “UNIX To UNIX Copy Program” (UUCP) had just been developed. UUCP was designed as a simple and efficient way of copying files between computers via phonelines, and while nowadays UUCP has been superseded by TCP/IP based protocols, it provided the foundation for what became known as “the poor man's ARPANet”.

ARPANet (Advanced Research Projects Agency Network), created by the US Department of Defense in 1969, was originally limited to computer scientists with Department of Defense contracts—not everybody could just go “willy-nilly over the ARPANet”. Aside from these connections, in order to join ARPANet one needed quite a bit of cash—assumed numbers ranged up to $100,000.

In 1979, two Duke University grad students started using UUCP to enable people to exchange information by uploading (posting) a message to a designated subject area called a newsgroup. Following a simple bulletin-board approach, subsequent messages would appear in the same newsgroup and messages regarding other topics would appear in a different newsgroup. This message system became known as Usenet News (UNIX Users Network). Using UUCP, everybody, even poor students, were able to use this system without being connected to the ARPANet.

The shell-scripts used by these two students, Tom Truscott and Jim Ellis, were later rewritten in C for public distribution and named the “A” release of news. If you consider how much traffic goes through an average full-feed newsserver nowadays, it is amusing to know that this A release was designed for not more than a few articles per group per day.

This small number of messages posted each day evolved into what became its own culture of shared information and support as more and more groups of core sites were linked together. In the year 1981, UUCP and ARPANet, up until then still independent, were linked together by Berknet (at the University of California at Berkeley). Other universities and colleges developed their own networks, such as BITnet (Because It's Time net—a geek's affection for witty acronyms has a long tradition), which was based on the IBM protocol and developed by Yale and the City University of New York.

As these various networks were linked together, Usenet had the form of a graph, and Network maps were drawn. The news software was rewritten and became the “B” release. Networks were developed and linked, and in brief, things were changing quickly. More and more forums were established, and it became clear that more structure was needed—it was time for the Great Renaming.

Figure 1. An Early Map of Usenet, ca. 1980

The first forums, or newsgroups, to be established were net.xxxx, and dept.xxxx; several mailing lists from the ARPANet that fed into this UNIX Users Network formed the “fa” hierarchy. As traffic increased, several hierarchies were instantiated, and Usenet was now connecting people all over the globe. However, transmission was still very expensive, so that universities in Europe were not willing to pay for “the fluff groups like net.religion and net.flame”. A new hierarchy, the “talk.*” hierarchy was proposed, and the Great Renaming began.

In the time between 1986 and 1987, the main top hierarchies were created, including comp, misc, news, rec, sci, soc and talk. Brian Reid, as many other Usenet users, was still not quite happy with this structure—the Backbone Cabal, for example, was unwilling to create rec.drugs and planned on dropping net.flame. In 1987, Brian Reid came up with the idea for an alternative distribution system that would not use these backbone links, creating the new top-level hierarchy “alt.*”; alt.sex and alt.rock-n-roll followed a year later:

Eventually, with the advent of the TCP/IP based Network News Transport Protocol (NNTP) all these groups were also transmitted to Europe and all over the world. Fewer people were using UUCP; NNTP became more and more popular; the Internet itself spread around the globe; Backbone Cabal became more and more silent and eventually was declared dead. Usenet, however, was quite alive and thriving, attracting more and more people every day—ultimately leading to what some old-timers refer to as “the September that never ended”.

During the late 80s and early 90s, Usenet became quite popular, especially more so, quite understandably, among students. Every year in September when semesters started and Freshmen got access to the Internet, many so-called “newbies” did not bother to follow the Netiquette but started posting to the various groups in rude and inexperienced ways. In the days before 1993 the majority of people on Usenet were regulars, following the newsgroups for years and (sometimes more, sometimes less) willing to help out newcomers, teaching them Netiquette. But in 1993 AOL gave Usenet-Access to its customers, opening the doors for a flood of technically inexperienced newbies to swamp the forums demonstrating (sometimes more, sometimes less) firmly that they did not know how to deal with this new medium, and much worse, they demonstrated limited willingness to learn. Some people insist that ever since that September in 1993 the overall quality of discussions in Usenet has gone down.

It is hard to deny that the merging of two such different groups of people, the tech-savvy geeks and the average Joe-Sixpack with an AOL-account led to conflicts and changed the way of the discussions held in the various newsgroups. However, this did not mean the end of Usenet, as some old-timers would have it—in the end, all it meant was a larger variety of opinions to be discussed. More groups were created, old silent groups deleted and new structures laid in. For every possible topic nowadays there is a newsgroup; all one needs to do is take a look at the impressive hierarchy of Usenet to get an idea how information is flowing around the globe and into everybody's home.

As mentioned earlier, Usenet is structured hierarchically. The main top-level hierarchies are alt, comp, misc, news, rec, sci, soc and talk. Furthermore, there are hierarchies for each country, the country code of which precedes the international equivalent. For example, de.comp.os.linux.misc would be the German speaking equivalent to comp.os.linux.misc, just as fr.comp.lang.c is the French equivalent to comp.lang.c. In addition to the country (or rather language) specific newsgroups, there are now a large variety of other top-level hierarchies such as free.*, gnu.* and news.*, as well as commercial or “company-related” newsgroups.

The “Master List of all Newsgroup Hierarchies” is available at faqs.org/faqs/usenet/hierarchy-list.

These hierarchies are highly functional and very well structured. Occasionally, however, some restructuring is necessary. If one discussion group, for example, becomes overwhelmed with too many question about a certain topic, the group may split into foo.general (or foo.misc) and foo.bar. If such a decision needs to be made, a posting is sent into this group as well as related groups to call for a vote. Every person reading the newsgroup can then cast his or her vote for or against the creation of another, more specific group—a very democratic process.

Because every newsgroup has its own Charta, in which the purpose and the topic of this group is clearly stated, it is easy to post your article to the right group. Furthermore, this Charta specifies the rules that apply to the particular group. For example, it is generally frowned upon to send HTML-formatted postings into Usenet; however, a Charta may indicate that this behavior is acceptable or even desired in a certain newsgroup.

Some newsgroups are moderated, meaning that the posts are not posted to the group directly, but rather to a person (or a group of people) who is in charge of filtering out noise and inappropriate posts. Anything that is not on topic in these newsgroups, anything that is of offensive nature or anything that already has been answered in the newsgroup or in its Charta or FAQ, is weeded out by the moderator(s) and will not appear in the group.

The third kind of newsgroup is the private newsgroup. These may or may not be moderated, or have specific rules, but in general the only difference from the “normal” newsgroups is that they are not available on all news servers, and only certain people have access to these groups. Most universities have some local newsgroups that can only be accessed from within the local network; the same goes for some ISPs. Other companies, for example, sell a certain service and restrict the access to their full-feed to paying customers only.

Since network news articles certainly resemble mail messages (or e-mail), many people think of them as being the same, which is why they often request a program that handles mail and news instead of looking for one single program for each of these rather different tasks. A news article is broadcasted to all interested hosts by the news server, which keeps one local copy for a certain amount of time, before it expires. This type of article contains a lot of important information in its headers, among them the unique Message ID identifying each article and the Message IDs of the articles it refers to (see Resources), if any.

The structure of Usenet and the way these articles are copied from news server to news server require the posting agent to be as standards-conforming as possible. To ensure that a newsreader follows the standards, The Good Net Keeping Seal of Approval (GNKSA), a document intended to describe minimal standards for decent Net behavior, was created. It specifies the requirements a good news client should meet, some of which are rather essential, others of which are merely a very good idea.

Note, however, that this is no official document, and that there are plenty of newsreaders out there that do not carry this Seal and may still be functional. In general, a newsreader should follow all of GNKSA's musts—otherwise broken postings might be generated.

While the content of the article is certainly in the hands of the user, the posting agent must ensure that it does everything possible to create standards-compliant postings and does not encourage behavior that is considered to be bad Netiquette (see below).

The question as to which is the best newsreader cannot be answered. It would be equivalent to asking which Linux distribution is the best. It all depends on your personal preference: some people prefer a GUI, while others are die-hard, command-line-interface fans. The following is an alphabetical list of the most common newsreaders available for Linux:

To evaluate each and every one of these newsreaders would certainly go beyond the scope of this article; however, I feel that I should point out a few brief details about some of these applications. For a more detailed discussion of the various newsreaders see Resources.

For all of you hard-core (X)Emacs fans out there, you probably already made your decision: this is the Newsreader for you. From all of the above listed applications, this is the only one that combines News and Mail in one application and can still be recommended without any reservations. It is highly configurable and written in Emacs LISP for GNU Emacs and XEmacs.

Figure 2. A Screenshot of Gnus

Pan is a very powerful gtk/gnome-based newsreader. It can be used in off-line mode as well as in online-mode, which makes it very attractive to the poor souls out there not using DSL or a Cable-Modem. It recently scored the highest GNKS rating of all UNIX/Linux newsreaders and simplifies downloading binaries by automatically grouping together multipart binary attachments and support of multiple connections.

The developers at Superpimp are focusing on creating a functional GUI newsreader that still can be operated entirely with the keyboard. The compliance with all of GNKSA's rules (at the time of this writing, Pan would receive a perfect score if re-evaluated) ensures that the user does not accidently create a broken posting.

One of the most important things about Pan is its exemplary development: daily CVS snapshots are available for everybody who is interested; developers are actively following the users mailing list and submitted bug reports and patches are considered and/or applied within days. It is a great example of how and why open source works.

Figure 3. A Screenshot of Pan

Slrn (“s-lang read news”) is a text-interfaced newsreader, very similar in appearance to mutt. It has been ported to a large variety of operating systems and can be easily extended using the s-lang macro language. If you don't need or don't want to use a GUI, this is most certainly the fastest and most functional console newsreader you can get. For example the many configuration options offered through the s-lang macro language make it more customizable than Pan.

Figure 4. A Screenshot of Slrn

“On the Internet, nobody knows you're a dog.” While this certainly is true, there are a few simple rules that everybody should follow. These rules are referred to as Netiquette (from network etiquette), and they ensure that Usenet does not drown in noise. Even though you may ask yourself in the beginning why things should be a certain way, rest assured that once you spend some time on Usenet, you will understand and appreciate these rules.

One thing to always keep in mind is that Usenet is not a free support service, but people having discussions in their free time. If you have a question and you get it answered quickly, remember this.

What it boils down to is basically common sense and common courtesy. One of the core-rules of Netiquette is to always remember the human on the other side. All other rules follow logically as long as you keep this in mind.

Of course it is important to find the correct newsgroup for your question—not only would it be quite unlikely that somebody in japan.animal.penguin might know the answer to a networking-related question, it is also considered rude to post off-topic messages into a newsgroup.

In order to make it easier for people to find the right newsgroup to post their message to, almost all newsreaders are able to show you a brief summary: a one line description of most groups when you subscribe to a newsgroup. However, occasionally you might want to subscribe to a newsgroup without having to search through the list of all groups available on your server. Since all Linux newsreaders use the same file format to keep track of the available and subscribed groups, it is easy to search through this file (~/.newsrc in general) for a specific newsgroup using grep or egrep.

For example, say you wish to find a list of all Linux-specific groups from the “alt” or “comp” hierarchy:

egrep '^(comp|alt).*linux' ~/.newsrc | more

If you want to see the list of groups regarding news-related software and announcements, you might try

egrep '^news (soft|announce)' ~/.newsrc | more

Before you start posting into a newsgroup, it is generally considered a good idea to

This way you get an idea of what the general tone in this newsgroup is or if your question has been answered already or is even part of the FAQ. In addition, it is also generally expected that you searched through a news archive like Deja.com before you post into the group. Remember, Usenet is not free tech support!

It is important to remember that some people are reading hundreds and hundreds of messages each day; most likely, they are best qualified to help you out, so you want to make it easy for them to read your posts. Many people just read through the subject lines and determine from there, if they are going to to read the entire article, so try to keep the subject line short but significant. “HELP!” is a bad, bad example. Remember that writing in all caps is considered to be screaming; furthermore, “multiple exclamation marks are a sure sign of a sick mind.”

As mentioned above, nobody knows who you are, and sometimes you may not even want to reveal your identity. However, many people prefer answering questions when asked politely by somebody with a real name. Your name is like your outer appearance in cyberspace—if you post under the name “K3wl d00d”, your postings might not be taken seriously. Likewise, if you choose a rather offensive name, people might killfile you right away, without even looking at your posts. Just use your real name and your real e-mail address.

When you reply to another message, quote only the relevant portions of the message you are replying to. Don't quote the whole message if it is not necessary; especially delete previous signatures. Be careful to not misattribute something. Leave in the attribution line that most newsreaders provide. Observe the attribution lines and read posts carefully. Replied text goes below the quoted text! For a more detailed discussion on the “Art of Quoting”, see Resources.

Many people are still using a dial-up connection to connect to the Internet. These people are literally paying for every byte of information that goes through their modem. Let this be just one reason why you should never ever post an attachment into a newsgroup that does not specifically state in its Charta that it is desired. Usually, attachments are only appropriate in the *binaries* newsgroups.

Once you have spent some time on Usenet, you will see that many people have a signature that is appended to every posting. Some people use it to provide, for example, a link to their web site, others have a funny quote in it. If you want to have a signature, keep in mind that there are rules for these, too: first of all, a signature should not be any longer than four lines. This is again an argument of bandwidth. As your article is copied hundreds and hundreds of times, one more line does make a difference. Why four? Why not five? Well, basically, because. The original purpose of the signature file was to provide your business information. There has to be a limit somewhere; since, generally speaking, all information such as an address, phone number and e-mail address can fit into four lines, it's four.

Another thing about the signature is the correct delimiter. Usually, your newsreader will provide the correct delimiter itself, but in case you want to supply your own, please notice that the only correct delimiter is “­”. The delimiter (or “sigdash”) allows other programs to recognize when the body of your message is finished and where the sig begins—this is useful for automatically removing the sig when quoting for a followup, as well as coloring it a different color, archiving and various other things.

Many people use their signature not to provide their contact information, but rather to express their individuality to a certain extent; after all, it's called their signature. While you may often find short ASCII-art, or a link to the owner's home page in the signature, many people use shell scripts ranging from very simple to rather elaborate to generate a unique and/or funny signature for every post they send. For example, you will always be able to impress a newbie by having your signature display the current uptime of your Linux box or feed it with some quotes from the popular “fortune” program:

#!/bin/bash
echo -e "Cheers,  n Yourname  n  n- " > ~/.signature
uptime » ~/.signature
fortune -s » ~/.signature

Just make sure you use the -s switch for fortune—otherwise your signature might get too long. To make your signature even more individual, you might want to create a few fortune files yourself and specify them in the script. Then put this script into your crontab and run it every, say, five minutes. Voil<\#224>--you have your own personal sig.

Unfortunately, not all of the people posting to Usenet read and follow the Netiquette. Every now and then you will find that some people enjoy lurking around in newsgroups and posting offensive material and/or rude replies. It's usually a good idea to simply ignore these people—a task that can easily be automatically accomplished by your newsreader. Most (if not all) Linux-based newsreaders provide the capability of assigning different levels of importance to the different articles, depending on various criteria such as Author, Number of Lines, Subject, etc. The act of deleting a certain person's articles is referred to as “killfiling”, while other actions (such as highlighting, sorting by importance, marking as “read” while still displaying the article, etc.) are referred to as “scoring”. Again Gnus, Pan and Slrn stand out among the other readers with respect to this capability.

A more severe drawback from posting to newsgroups is that you expose your e-mail address to the world, and special spiders or bots are harvesting high traffic newsgroups for e-mail addresses, resulting in increased Spam in your mailbox. It is important to provide a valid e-mail address in some way when posting to Usenet, so that people can reply if you desire. Many people are “munging” their e-mail address or are simply entering an invalid address in the From-Header and a short note in their signature explaining how to retrieve the original valid e-mail address.

However, one should not simply use an invalid e-mail address because this way it either puts more of a load on one's system or on the originating mailserver. In general, it is considered polite to use only the suffix “.invalid” for your e-mail address, so as to make it clear that your address is in fact not a genuine one.

As you can, did and/or will see, Usenet is a very interesting place to spend some (plenty? all?) of your free time. But it is also a particularly large place—a topic that can hardly be covered adequately within limited space. In this article I tried to give an overview over the main aspects of Usenet, touching various topics only briefly.

Some topics that are of particular interest and that need further, more detailed attention include real spam protection through a mail-filtering program such as procmail, and the installation and maintenance of a local newsserver such as leafnode (for a small local network) or inn (for a full-blown professional network or newsservice). Since these topics are too extensive to be covered on here, I might elaborate on them in future articles.

In the mean time, I would like to encourage the reader to follow the referenced links and, of course, explore Usenet themselves so. But remember: “This game lends itself to certain abuses.”--Bill Watterson's Calvin

Resources