How to build a synchronization server and keep your data where you want it.
Like most families these days, our family is extremely busy. We have four boys who have activities and appointments. My wife and I both have our own businesses as well as outside activities. For years, we've been using eGroupware to help coordinate our schedules and manage contacts. The eGroupware system has served us well for a long time. However, it is starting to show its age. As a Web-based groupware system, it's pretty well polished, but it doesn't hold a candle to Kontact or Thunderbird. Also, my wife finds that she needs to access her calendar from her Android phone, and eGroupware just isn't very mobile-friendly. Sure, we can set up calendar synchronization, but eGroupware seems to have added synchronization as an afterthought, and it really doesn't work as well as we'd like.
So, I started looking for a new groupware system that would allow us to access our calendars and contacts seamlessly from our smartphones, a Web browser or our favorite desktop PIM. Sure, we simply could have uploaded all of our information to a Google server. However, I may be paranoid, but I just don't want an outside corporation having personal information like who my friends are, my wife's recipe for cornbread or what I'm doing next Tuesday at 3:00pm; it's just none of their business. By hosting my own groupware server, I maintain my privacy and don't have to worry about arbitrary changes in service.
The ownCloud system has a calendar, address book, task manager, bookmark manager and file manager, among other features. These services can be accessed from any Web browser. However, ownCloud also supports the calDAV, cardDAV and webDAV standards, so synchronization with other clients should be pretty straightforward.
In practice, there was a slight learning curve, but synchronization works very well. The ownCloud system also allows you to integrate third-party modules (apps) in order to add features. Apps are available that provide music and video streaming, file encryption, e-mail and feature enhancements for existing functions.
In order to install ownCloud, you need PHP, a Web server and a database server. The installation documentation walks you through configuring the Apache, Lighttpd, Nginx, Yaws or Hiawatha Web servers for use with ownCloud. For a database server, you can choose from MySQL, PostgreSQL, or SQLite. It's pretty hard to have a system that doesn't meet these requirements.
The installation process is well documented, so I won't go into too much detail here. Essentially, you download and extract the tarball into a subdirectory under your Web server's htdocs directory. Then you make the Web server configuration changes indicated in the manual and restart the Web server.
Basically, you're setting permissions and enabling cgi execution. Once this is done, you point a Web browser at the new installation and follow the installation wizard. I purposely neglected to make some of the file permission changes, and the wizard notified me that the permissions weren't right. The installation is really pretty straightforward.
After all of the installation is complete, you won't able to access your new ownCloud installation. To resolve this problem, you have to edit ./config/config.php and comment out the trusted_domains line. This is a security setting that determines which domains clients are able to connect from, and by default, limits access only to localhost. I happen to think the default values are a bit strict.
After the installation is complete, point a Web browser at your ownCloud server and log in. You will be greeted with a page resembling what is shown in Figure 1. As you can see, the interface is simple. From here, you can access the calendar, contact manager, task list and so on. All of the tools are intuitive to use, but not polished enough that you'd want to use them every day. The intent is that you'd point your favorite PIM at the server and use it as an interface to your shared information.
Figure 1. ownCloud Web Interface
The initial configuration should be done in a particular order. Since my initial intent was simply to test this system, I managed to do everything in the wrong order. If I had known I would be end up using the system as a permanent solution, I would have put more thought into its initial implementation. I still ended up with a usable system, but I've made things more complicated and harder to manage than they should have been. Let me share what I did wrong and how it could have been done better.
As soon as you get logged in as the administrator user, you should start creating users and groups. First, I would create the groups. You'll want to create a group for every group of users who need unique access capabilities. For example, I created a group for our family as a whole, and a separate group for each of our businesses. This way, when I create a calendar or address book, I can share it to just my company group, and my wife doesn't have to look at it on her PIM. I initially made the mistake of simply creating a family group and sharing everything to it. But when I created a chore list for the kids, I discovered that they also were able to see my company's calendar, which isn't what I wanted. The moral of the story is to spend the time and keep your groups as granular as possible because users in the same group can see everything shared to it. Once you've got your groups created, you can create users and assign them to the appropriate group(s) from the pick list. In my case, I created the users first, then I had to go back and assign them to groups, which was tedious.
Next, you should start creating calendars. I thought I'd be clever and log in as the administrative user and create a family calendar, a calender for each of our businesses and a private calendar for each family member. This sounds reasonable until you discover that each user gets created with his or her own default calendar, which is now redundant. So, use the administrative account to create entity calendars and address books, but let each of your users share their assets themselves.
Then, create a shared documents folder. This is a pretty straightforward process. However, I would recommend that once you've created the shared space, you also create as much of the directory structure as you can reasonably foresee. Otherwise, you end up with a hodge-podge, and no one will be able to find anything when they need it, and that defeats the purpose of a shared file space.
One of the goals of this project is to be able to access the system from the LAN or from the Internet. To make this work from the LAN side, I logged in to my router, which is running OpenWRT, and configured a static hostname, which it was happy to serve to every DNS client on the network. Then, I went to my DNS registrar and configured the same FQDN, but with my router's outside IP address. Then, it was simply a matter of configuring iptables to port-forward TCP/80 to the machine that's hosting ownCloud. A reverse proxy might be more secure, but this works quite well.
I have successfully synchronized my ownCloud with Kontact, Thunderbird, Evolution, my Android phone and our iPad.
Kontact is the easiest to set up. In order to configure address book synchronization, you simply create a new cardDAV address book and point it at http://server.example.com/owncloud/remote.php/carddav/. Kontact happily will discover every shared address book to which your login has access. Similarly, by creating a calDAV calendar and pointing it at http://server.example.com/owncloud/remote.php/caldav/, you'll be able to get all of your calendars configured in one step.
Thunderbird and Evolution are the next easiest clients to configure. However, in these cases, you have to point the client to each individual asset. For example, if you have a calendar named “family”, you have to point these clients to http://server.example.com/owncloud/remote.php/caldav/calendars/username/family/. You have to do this for each calendar and address book with which you want to synchronize.
To make matters a bit worse, the structure of the URL changes if the asset was shared by another user. Fortunately, ownCloud will tell you what the correct URL is for each asset. To get this information, simply edit the asset. You will see an icon that looks like a globe. If you click on that, you will be provided with the correct URL.
In order to get the iPad to synchronize, you simply create an account under settings, where it says “Mail, Contact, Calendars”, and point it to the same URL mentioned above. This is pretty easy to get working even for a non-Apple user like myself. I don't have an iPhone, but I'm assuming the process is the same.
Synchronizing to the Android device requires additional software. For contact synchronization, I used “CardDAV-Sync free beta”. For calendar synchronization, I used “Caldav Sync Free Beta”. Once the software is installed, you simply create a corresponding account for each application under Setup. However, you have to point the software to the individual assets, just as you do for Thunderbird and Evolution. There are two potential gotchas though. Automatic synchronization isn't turned on by default, so you have to turn it on and perform an initial synchronization before you will see your calendars and contacts. Also, the Android calendar application supports multiple calendars, but you have to select which ones will be displayed. It doesn't do any good to have a perfectly functioning synchronization system that simply isn't turned on, and don't ask me how I know.
The ownCloud Web site indicates that there is a custom client available that costs $.99. I installed it to see how it works. I was a little disappointed to find that it was simply a webDAV client. I guess I was hoping it would be an integrated calendar, contacts and file client. However, once it was configured, I was able to share files from my Android directly to my file space on my ownCloud server. I did find that the client occasionally lost its configuration and had to be reconfigured, which is a bit tedious. Otherwise,the ownCloud client rounds out almost all of the synchronization features of ownCloud.
I say “almost” because ownCloud also offers a Firefox browser synchronization function. This function is supposed to allow you to synchronize your bookmarks and browser history across multiple machines. However, with the latest version of Firefox, there is no way to point Firefox to the ownCloud server. Perhaps this will be fixed with the next upgrade.
Once everything is configured, there are some operational issues. The obvious issue stems from making concurrent changes to an asset. This results in a conflict, and the various clients handle conflicts differently. To avoid problems, simply synchronize the asset before you modify it, and then re-synchronize when your changes are complete. This will ensure that everyone has the same version of each asset on their client.
I also discovered that it is very difficult to move assets from one calendar or address book to another. The various clients don't seem to do a very good job of this. So far, my attempts at organizing my contacts has resulted in duplicate contacts in different address books. I think the solution is going to involve adding the assets in question to a category, exporting the assets in that category, deleting the assets in that category and the re-importing the assets into the appropriate calendar or address book. This seems like the long way around the block, so I'm going to hold on doing it this way until I know for sure there isn't an easier way to do it.
The rest of the difficulties involve file security. The first problem is that when a user uploads a file into his or her cloud space, that file will be owned by the Web server user. This is okay as long as you don't want to access the file from the filesystem directly or via a Samba share. In these cases, you either have to change the user name that the Web server runs as or the name that the Samba server uses to access the files. Either way, you still won't be able to access the files directly. I've not yet decided on if or how I intend to fix this. I'll probably just access the files via a Samba share or NFS mount.
The ownCloud system supports server-side encryption that can be turned on and off on a per-user basis. This leads to more problems than it's worth, in my opinion. For example, what happens when a user encrypts his or her files and then shares a directory with a user who does not? I happen to know that you get a warning from ownCloud, but I didn't spend the time to find out what actually happens, because I stumbled upon another problem. Server-side encryption pretty much breaks any possible means of file access besides webDAV. I guess that's the point of server-side encryption, but it doesn't work for the way I want/need to access my files. I ended up turning off encryption and decrypting my existing files, which was done seamlessly for me by ownCloud.
The better solution might be to use an encrypted filesystem like Encfs to protect your files. With this solution, you still will be able to use Samba and NFS to access the plain-text files on the filesystem. Also, you'll be able to upload the encrypted files to another cloud provider, such as Dropbox, as a means of backing up your files without giving up your privacy.
I have found ownCloud to be a very capable and easy-to-manage synchronization server. The actual installation process is pretty simple, so I've spent most of this article pointing out as many of the potential pitfalls as I could. Now that I have it properly configured, I am able to share calendars, contacts and files with other members of my family, no matter where they are or what client they choose to use...and I maintain complete control over my information.
