Introducing DevAssistant

If you ever have started writing a new piece of software or checked out someone else's, you certainly know there's a lot more to be done than just opening the file and starting coding. You need dependencies. You need the proper directory structure. You need the proper things put on your $PATH and countless others considerations. To seasoned developers, these are issues that only somewhat slow them down; for new people, these often are obstacles that make them give up and try something else altogether. A few of my team members and I decided we wanted to fix the problem for both of these kinds of developers. Easier said than done, you think? It depends. About two years ago, we started discussing concrete ways of solving these small, frequent and annoying problems, and we've come up with something that gradually became DevAssistant (usually shortened to DA) today (www.devassistant.org).

What we created is, essentially, a simple but powerful tool that sets up your entire environment for developing a particular software project with only a few clicks in a GUI or a single line on the command line, and that you can extend almost without effort. DevAssistant consists basically of three things: the main program, which is packaged under the name devassistant on the Python Package Index and in Fedora; the scripts called Assistants that implement the particular workflows; and the DevAssistant Package Index (DAPI), which is where you install the Assistants from.

This is usually where people start asking questions. Truly, at first glance, DevAssistant may seem redundant. After all, there's support for most of these things in IDEs, so that's sorted out already, right? And dependencies are specified in the package management software for that particular language, aren't they? Technically, yes. Actually, no. The main problem with IDEs is the very fact that you actually have to use them, and by “them”, I mean only the specific one that supports what you need. Every single IDE has a ton of neat features, but what about users of different IDEs? When we designed DevAssistant, we deliberately wanted to avoid this kind of dependence on a particular development suite, so that we are able to cater to all users, regardless of whether they use a particular IDE, a standalone text editor or write machine code into standard input.

Figure 1. DevAssistant's main window—you can select which Assistant to run from here (Node.js selected, cursor not visible).

The other major objection we usually get is that there are dependency mechanisms already well established across languages, and there's no reason to have another. That assertion is almost, yet not exactly, true. There certainly are mechanisms to manage dependencies that are necessary to run the program or use the library. On the other hand, there is usually little that can be done if you want to make sure that you have everything needed to develop the code. And even if there is, it rarely takes care of settings other than dependencies. What if you need a working Web server to test your app, open a port in a firewall or set up SELinux? None of the established systems takes care of such trouble for you. Furthermore, nobody solves the dependencies for you if you're not really writing a program—like writing documentation in Markdown and wanting to see how your text will look when converted to HTML or another language.

In addition to all this, there is the question of publishing your code. Since you're using free/open-source software, we bet that you want to publish your code as well. To make this easier for you, DevAssistant is fairly tightly integrated with GitHub, so if you want to create a new repository and push your sources there, you can do so without even opening your browser. The same goes for forking existing repositories. Conventionally, it means this: going into your browser, logging in, typing the repo's full address, clicking “Fork”, waiting for the forking to finish, selecting the address for cloning, then going into your console and typing git clone and pasting the address. With DevAssistant (and the Assistant called “custom”), you run this:

da prepare custom --gh-repo olduser/repo --gh-fork newuser

DevAssistant then performs all the steps mentioned above, in a single command. You can do the exact same thing in the GUI with very few clicks as well. More specifically, the GUI and the command-line interface both share the same Assistants that contain instructions and dependencies specifications, and that are analyzed and carried out by the main program.

The previous four paragraphs possibly made you think “Wouldn't it be better if beginners learned by setting up everything in the first place, instead of having DA do it all for them?” Of course it would. However, ask yourself honestly, when you were starting whatever you are doing now, did you dig deep into each step of the installation instructions and analyze it thoroughly before moving on? Or did you just stumble your way through it, having a vague idea of what the instructions meant, impatient to get to the “Hello world!” example at the end? We are willing to bet that nine out of ten cases were the latter, and most developers learned only much later how the language worked. What our team did was simply build a tool to lower the learning curve for these developers.

Let's talk about what you can automate through various Assistants. Be aware that at the time of writing of this article, there are two supported versions of DevAssistant: 0.9.3 and 0.10.0. The first one is a bug-fix release, which still includes some Assistants in its core distribution and is available in Fedora 21 by default. To install Assistants for this version, you need to download and unpack the packages manually (more on that later). In version 0.10.0, no Assistants are included, and they can be installed via the command line by running:

da pkg install NAME_OF_PACKAGE

With Assistants installed, you can run them either from the command-line interface (CLI) or the GUI. In the CLI, an invocation looks like this (the example works with the Assistant called “python”, available on the DAPI):

da create python flask -n my_flask_app

The same can be achieved in the GUI by selecting Create project→Python→Flask and typing “my\_flask\_app” as the project name. Running this Assistant creates a new Flask Web application named “my\_flask\_app”, installs all necessary dependencies, creates the required directory structure in a directory named like the app itself, and modifies the boilerplate files to conform to your project (for example, by changing paths, names and settings). Once DA's run is finished, you then can just run:

./manage.py runserver

in the new project's root directory, and you'll have a running Web app only seconds after launching DA. During the run, basic information is logged either to the standard output or the log window (depending on which interface you are using), so you can keep track of what is being done. If you prefer more verbose feedback or need to figure out what the problem is if something went wrong, you can select the debug option and see all the information that either DevAssistant itself or the shell underneath it outputs.

Figure 2. Project creation window—here is where you specify parameters for the project you want to create.

Generally speaking, there are four main kinds of Assistants: create, tweak, prepare and extras. This is the naming used in version 0.10.0 and all future versions; older releases of DevAssistant use “modify” instead of “tweak” and “task” instead of “extras”. Each of the Assistant types corresponds to the action you want to perform. Some Assistants help you create new projects from scratch. Some tweak existing code by adding new headers or generating new files from existing ones. The extras (or “task”, in version 0.9.3) Assistants are used for all other workflows that are not covered by the previously mentioned categories.

The prepare type is a bit special, as it is used for preparing the environment for a specific project, not just a generic one. That means that if you create a Web application in Django, add some dependencies, have some specific database settings that work just for your app and don't make sense for other Django projects, you can write a prepare Assistant that will set up users' computers for them. Users then can develop your application without making any further configuration changes. This is especially helpful in cases where the project you created is very complex and you want to make it easy for people to submit patches to you. Consider that if a patch takes technically adept users five minutes to write, and they have to spend an hour setting up their environment just to verify that their patch works, they simply won't write it. With DevAssistant, they spend maybe a minute installing a prepare Assistant for your app and five minutes writing that patch. Everybody wins.

Out of the box, DevAssistant version 0.9.3 supports many popular languages including Python, Java, C(++), PHP, Ruby, Node.js or Perl. For most of these languages, sub-Assistants exist for particular frameworks or apps. So under Python, you can find both the general Python library support as well as Django. If necessary, more Assistants implementing Python workflows easily can be added without affecting the existing ones. Both the command-line arguments and the GUI screens then are generated automatically. As a side note, DA comes with bash completion in Fedora, which also is automatically generated right until the last parameter. All the mentioned Assistants are available on the DAPI for installation into DevAssistant 0.10.0 as well.

As you may have noticed, all I've mentioned up until now was happening on the user's machine, on bare metal. At this very moment, we are working on having the support for DevAssistant workflows in Docker containers as well, which means that once you set up a development environment in Docker with DA, you easily can share it with your teammates or just publish the Dockerfile for anyone to rebuild. Note that at the moment, DA is not the tool to deploy that very Docker image in production, but we may add that functionality in the future.

As previously mentioned, one of the key parts of the DevAssistant ecosystem is the DevAssistant Package Index, or DAPI. This is a server for developers to upload their Assistants in packages, which are called DAPs (DAP is short for DevAssistant Package). It's worth mentioning that this mechanism is quite new (made for use with DevAssistant version 0.10.0), so more content and features are to come. In the aforementioned version 0.10.0, users can install these packages directly with DA itself by issuing the command:

da pkg install FOO BAR

Running this particular command would install the packages FOO and BAR with all the DAPs on which these two depend. Similarly, users can un-install and search directly from DevAssistant too. At the moment, this functionality is available only in the command-line interface, but version 1.0 of DevAssistant will feature a re-designed GUI with the same features in a “clickable” form as well.

If you are using version 0.9.3 of DevAssistant and want to install a new Assistant, you need to do so yourself. First, you need to download the DAP file, presumably from the DAPI Web interface found at https://dapi.devassistant.org. Once downloaded, you must unpack the file (it is technically a gzipped tarball) into ~/.devassistant while stripping the topmost directory. On the command line, this is done by running:

tar xzf DAP_NAME --strip-components=1

In a graphical archive tool, you just select the second-level directories (assistants, snippets, icons and so on) and extract those. Furthermore, it is necessary to rename the directories “twk” or “extra” to “mod” or “task”, respectively, because of the older layout used by version 0.9.3. The reason for the apparent lack of user-friendliness is that due to significant architectural changes between versions 0.9.3 and 0.10.0, it takes a disproportionate amount of effort to backport the package installation mechanism from 0.10.0 to the previous version, and we'd rather spend it on fixing the new version and implementing new features for you. For users of Fedora 21 who want to benefit from the DAPI integration, we recommend using the COPR repository located at https://copr.fedoraproject.org/coprs/tradej/DevAssistant, where version 0.10.0 is packaged.

Developers who want to upload a package to the DAPI need to log in either with their GitHub account or Fedora's FAS account. Once uploaded, the DAP is tested for validity, and if it passes, saved to the repository. Obviously, this means there soon will be hundreds and hundreds of DAPs in the index, and users will have to make sense of them somehow. We have taken that into consideration, giving users the ability to search by tags (filled in by the authors of DAPs) and by quality, represented by a simple one-to-five-star system (as voted on by logged-in users). There is, of course, a mechanism to report malicious packages as well.

So far, I have been talking mainly about the user perspective only. If you don't want to delve deeper in the technical side of things, feel free to skip this section.

On the inside, DevAssistant consists of the core, which is written in Python, and the Assistants accompanied by auxiliary files (which are usually copied to the final project's directory structure). The Assistants are written in a domain-specific language based on YAML. We are expecting quite a few eyebrows will raise right now. We are perfectly aware of the ramifications of this selection, and we didn't make it by rolling dice or doing an eenie-meenie. Instead, we have done our research, and although it wasn't the obvious or even the first choice, we settled with YAML due to a number of reasons.

Figure 3. A log window with a finished successful run—when you see this, your project is ready.

First and foremost, it isn't associated with any other widely used language. In the past, we repeatedly got a lot of negative feedback when deciding to support only one particular language for writing modules for programs. People using interpreted languages don't like compiled. Users of static typing refuse to use dynamic, and so and so on. For this reason, a markup language was the obvious choice. We didn't select simpler formats like INI due to technical constraints, and we didn't select XML because we think that the scripts should be written in something that is readable by humans. YAML, a mature enough language used widely in systems orchestration, which is fairly close to what DevAssistant does, emerged as a victor of this challenge.

However, for those who do not want to write their whole script in YAML and prefer to use their language of choice, there is a nice surprise in version 0.10.0. You guessed it, DevAssistant has an API for running scripts in other languages. For each language that you want to use for running an Assistant, you need bindings for the DevAssistant PingPong protocol (DAPP). At the moment, these bindings are available for Python only in the form of a reference implementation. When using the bindings, you need a minimal YAML Assistant file to specify dependencies, arguments and the script, which then does all the heavy lifting itself.

The content of the YAML-only Assistant is fairly simple. There are metadata like name and description, declarative dependencies, argument and files sections, and an imperative run section that contains the actual commands and shell calls that perform the Assistant's task. Assistants are inheritable and support an inclusion mechanism; therefore, you easily can do things like support Eclipse integration with all your Java projects or add a specific Vim config to only a select few. The invocation of DAPP-enabled scripts is done in the run section too, and it requires only the script to be specified in the “Files” section.

As far as dependencies go, we tried to make the mechanism as bulletproof as possible, which means that you can specify dependencies for a particular packaging format (RPM, pip, pacman and so on) in a descending order. So if DA cannot use a particular package manager, it moves on until it finds one that is available or runs out of package managers altogether. To answer a question we get quite often, right now we do not support versioned dependencies due to the fact that it is difficult to make sure that the underlying package managers report accurate information to DA, but we're in the process of finding alternative solutions.

You can read further descriptions of the Assistants and related concepts at docs.devassistant.org.

At the moment, we have several initiatives going on, which largely will define where DevAssistant goes from here. Apart from the aforementioned Docker container support across Assistants, we aim to improve support of certain project management workflows that take place after the project has been created. However, this support is rather specific because we do want to avoid duplicating the IDE functionality as much as possible.

On the integration level, DevAssistant is a part of a larger ongoing effort to make Fedora Linux more accessible and attractive to developers, so although DA does run on other Linux distros (and, mind you, even runs on Mac OS), integration with Fedora is our main focus. That means, among others things, automatically creating RPM SPEC files for your projects, building on the Fedora infrastructure (especially COPR), possibly deploying on FedoraHosted and the like.

In version 1.0, DA will have a new, more user-friendly graphical interface, which also will support installing, removing and looking up Assistants directly within the application. We still aren't quite sure about some aspects of the GUI, so expect more to come.

If you want to try DevAssistant, you can get it in several ways. For one, it is present in the Fedora 21 Workstation installation by default (that is true for version 0.9.3). This way, you get seamless installation, bash completion and automatic updates as part of the system update routine. Version 0.10.0 for this version of Fedora is accessible as a COPR repository (for installation instructions, see the Download section of the DevAssistant home page: https://devassistant.org/download). Another option is downloading DevAssistant from the Python Package Index. If your system is running pip, you need to run:

pip install devassistant

Bear in mind, though, that for some dependent Python libraries, you will need certain underlying C header files, so the installation may not be straightforward, and the program may not behave the way it is intended to. It generally should be safer to use the Python 3 version of DA, installed by invoking the Python 3 variant of pip. Still, we can't tell if it will work as expected on your system. The last option is downloading the source code from GitHub, which also is a perfectly valid way to go, especially if you intend to contribute to DevAssistant.

We hope this article makes DevAssistant interesting to you, and that you will check it out (if you pardon the pun). Both DA and DAPI are free and open source (licensed under GPL v2+ and AGPL v3+, respectively). We're eager to get more feedback, because DevAssistant is mainly about helping you, and we can't do that if we don't know what you need as precisely as possible—whether it's a message sent to the mailing list, an IRC question, a bug report or a pull request, we are grateful for all that you tell us. Let DevAssistant help you start developing with ease.