The Hackerlab at regexps.com

Introducing arch

up: arch
next: System Requirements

arch is a source code management, revision control, and configuration management tool.

Advantages of arch

What makes arch better than other revision control systems?

There are minor advantages, and a a few major advantages.

The minor advantages are things like: regular, clean, interfaces; small code size; on-line help; automated ChangeLog generation; features for browsing change sets with a web browser; software tools architecture. Some of these are arguably matters of taste -- worth mentioning but not arguing for.

There are at least six major advantages to arch: a file tree library of revisions , renaming , distributed repositories , robust and easy operation , configurations , and sophisticated branching and merging .

"a file tree library of revisions" means that all revisions can be made accessible in a (space efficient) library of revision trees. That means that you can use ordinary tools like diff , find , and grep to explore past revisions and revisions on other branches.

"renaming" means that you can rename files and directories and arch keeps up just fine: it rearranges the patches between revisions in the corresponding way. And it's convenient: you can use arch in a mode that doesn't require you to run add , remove , or move commands every time you add, delete, or rename a file or directory.

"robust and easy operation" means that arch repository transactions have ACID properties and repositories are stored as ordinary unix files. Commits may be concurrent with reads and are atomic. There is no overhead from having to administer a relational or hash-table database. The transient locks held by arch never become permanently wedged (requiring by-hand repair): they can be broken remotely to recover from interrupted commands.

"Distributed repositories" means that you never need write access to a repository in order to start branches from projects stored in that repository. You can store branches in any repository. That means every programmer can have private repositories for day-to-day work or as scratch areas for working out complex merges. Every sub-group working on a project can have their own repository. If two groups want to fork a project, but still loosely cooperate, they can each have their own trunk in their own repository -- selectively merging changes between the two trunks. Without wishing to put too much hype on it, as far as arch is concerned, there is exactly one repository database for the entire world -- we all share it.

"configurations" are a mechanism for defining "meta-projects" that are a combination of multiple "sub-projects". For every meta-project, you need to be able to define configuration rules, which explain which revisions from what branches to check out to get a particular revision of the meta-project. arch provides a flexible mechanism for that, allowing you to pick and choose pieces from various branches and repositories with varying degrees of specificity.

"Sophisticated branching and merging" has to do with arch's support for asynchronous development on branches, coordinated by a shared trunk. Commonly, people arrange branches in a "star topology". There is a trunk in the middle, surrounded by branches, as in this (hypothetical) example:

        patch-reviewer A                    patch-reviewer B
          (who merges patches               (another patch reviewer)
           contributed by people           /
           who don't have write-access    /
           to gcc-main)                  /
                \                       /
                 \                     /
                   gcc-main (the trunk)
                  /                   \
                 /                     \
                /                       \
        C 99 team                       Fortran team
          (who work on features          (who work on fortran)
           required by the new
           standard)

And we might imagine many more branches than are actually drawn here.

Ideally, each of the surrounding branches will sometimes be merged into gcc-main . And as gcc-main changes, it wll sometimes be merged back into the surrounding branches.

When two versions each merge with the other repeatedly, forming a merge without creating spurious conflicts is a tricky problem. (For a more detailed description of the problem, see Star Topology Branching and Merging.)

In CVS, the problem is solved by using tags and not generating conflicts for changes that appear to be redundant. Unfortunately, tags are expensive, work only within one repository, and worst of all, are complicated to use correctly. If you forget to create a tag at the right time (or worse, if your tagging operation is interleaved with other operations) you're hosed.

In arch , history sensitivity is taken to the next level. arch knows how handle back-and-forth merging in a star topology automatically.

This is an advance in revision control similar to the invention of lockless operation in CVS. With lockless operation, you have a star topology with an archived development path in the middle, surrounded not by branches, but by working copies. Programmers can commit from the working copies to the trunk or merge from the trunk into working copies repeatedly, and everything works beautifully. The trunk is used to syncronize the multiple working copies.

arch takes the next step. It allows those satellite working copies to be archived as long-lived branches. The trunk is used to syncronize the surrounding branches.

arch automates the merge operation (with the larch star-merge ) command and gives you control over the precedence-ordering of change sets (which branch takes priority, and which has changes that are rejected if they conflict). You don't have to use tags, and you don't have to figure out the right revision arguments to pass to an update command.

Each satellite has its own log history. Each can be used for progress tracking. A satellite can be used to maintain an alternative distribution that tracks the trunk, but sometimes takes the lead with additional features.

In addition to star topology merging, arch provides some other fancy merging options too (e.g. see Multi-Branch Merging -- The reconcile Command.)

Global Revision Control Done Right

The foundation of arch is two commands: mkpatch and dopatch .

mkpatch computes a patch set describing the difference between two trees. dopatch applies a patch set to a tree, gracefully handling the cases when a patch doesn't apply cleanly.

Conceptually, mkpatch is similar to diff -r and dopatch is similar to patch . Unlike diff and patch , though, mkpatch and dopatch can handle the addition or removal of files and directories, the renaming of files and directories, files which are symbolic links, changes to file permissions, and binary files.

An arch repository is a collection of full-source revisions, and patch sets. Brand new trees are represented as full-source revisions. Modified trees are stored as patch sets. Any revision can be reconstructed by retrieving a full-source revision, intermediate patch sets, and applying those patches.

arch repositories have globally unique names and every revision in a repository has an easy to understand, easy to type name. Putting the two names together, arch provides a global namespace for revisions.

On the basis of that global namespace, branches and merge operations can span repository boundaries. As far as arch is concerned, all of the repositories you can access over the Internet are integrated into one gigantic repository -- seemlessly integrated.

In arch , there is no such thing as a "working directory". That is to say, there is no distinction between a tree that you download as a source distribution, and a tree that you check out from a repository. arch is happy to work with both.

Every tree that arch works with has a "patch log" -- a record of all of the patches (in the global namespace) that have ever been applied to the tree, along with a record of the full-source revision that the tree started from. In arch , a tree never belongs to a specific branch in some specific repository -- instead: a source tree is considered to be a part of every branch in every repository for which it has a patch log. At any time, any tree can join any branch with which it has a common ancestor.

arch is agile and flexible at handling patch sets. It provides update -- a patch set manipulation operation that is logically equivalent to the traditional update operation of CVS ; it provides replay -- a history-sensative merge operation equivalent to the update operation of Subversion (as documented in the Subversion manual); arch also provides two rather more sophisticated merge operations -- reconcile and i-merge -- which handle very complex (yet quite realistic) branching and merging scenarios gracefully.

In general, the design philosophy of arch is to be a very good librarian for whole-tree patch sets, and a very good mechanic for manipulating whole-tree patch sets. arch is designed to stay out of your way while you hack, but come to your aid when you commit , review, update , or merge. The aim is simplicity, clarity of function, and flexibility.

Introducing arch Project Trees

arch manages "project trees". A project tree is a file system tree, usually containing the source code for a project.

What distinguishes a project tree from an ordinary tree is the presence of "arch control files", primarilly stored in a top-level subdirectory called {arch} .

The control files include information needed to keep an inventory of the tree, a "patch log" documenting the history of the tree, various default values for arch commands applied to the tree, and a local cache of information to speed up some arch commands.

When you distribute a tree, usually you will include all of the arch control files -- they are useful to others.

arch has no special concept of a "working copy" (in other revision control systems, a working copy is a tree checked out from the revision control database, as contrasted with a tree from any other source). Any tree that contains arch control files can be used as a "working copy". If you download a distribution for a project managed with arch and unpack that distribution -- you have a working copy.

For more information, see arch Project Trees.

Introducing arch Inventories

arch keeps track of an inventory of the files in a project tree. For example, it can distinguish the files that are officially part of the tree from other files, such as scratch files or editor back-up files. The command:

        % larch inventory --source

prints an inventory of files in a tree.

Every file has two names: its "location" (a path relative to the root of the tree) and its "tag" (a logical, location-independent name for the file). The command:

        % larch inventory --source --tags

prints an inventory of files in a tree, showing the logical name of each file.

arch uses tags to keep track, for example, of when files are renamed.

If you are being extra careful, you can use the command:

        % larch set-manifest

to record a list of all the files that are supposed to be in a tree and the command:

        % larch check-manifest

to see if any unexpected files have been added, or expected files removed.

For more information, see arch Project Inventories.

Introducing arch Patch Sets

Experienced programmers should be familiar with the standard command diff -c -r , used to create a (standard) "patch set" describing the changes made between two copies of a tree. And they are familiar with the standard command patch -- used to modify a tree according to the description of changes in a patch set.

Standard patch sets have limitations: they do not cleanly handle file or directory additions, removals, or renames, symbolic links, file permissions, or binary files.

arch provides mkpatch and dopatch , similar to diff -r and patch , but without the limitations.

For more information, see arch Patch Sets.

Global Namespaces

arch implements a global namespace of projects, taking into account the organization publishing a project, branching of projects, and versioning of branches.

arch implements a global namespace of patch sets, building on the global namespace of projects.

Those namespaces give rise to the idea of a "development path". For example, the very first revision of the project arch , might be called:

        lord@regexps.com--arch/arch--0.1--base-0

The next three revisions might be called:

        lord@regexps.com--arch/arch--0.1--patch-1
        lord@regexps.com--arch/arch--0.1--patch-2
        lord@regexps.com--arch/arch--0.1--patch-3

Each of those revision names is the name of a patch set that describes what changed in that revision, compared to the previous revision.

If arch forked into a separate development paths, say "intl" (for internationalizing the code), there might be revisions such as:

        lord@regexps.com--arch/arch--intl--0.1--patch-1
        lord@regexps.com--arch/arch--intl--0.1--patch-2

arch implements a global namespace of all user's of arch (layered on top of email addresses). Every patch set has an associated log entry, with a Creator: line that contains a user's arch id.

For more information, see The arch Global Name-space of Users, The arch Global Name-space of Projects, and Basic Revision Control.

Introducing arch Archives

arch can manage repositories of revisions, storing those revisions as compressed tar files of patch sets or compressed tar files of complete trees.

arch archives can be used remotely if they are made accessible by an ordinary FTP server (there is no need for a special arch -specific server).

You can (safely, atomically) add a new revision to a repository (larch import or larch commit ). You can reconstruct an arbitrary revision from the files in a repository (larch get ). You can "branch" a development path to create a new, related development path (larch tag ). Branches can cross repository boundaries, and to the user, arch appears to integrate the two repositories into one.

Repositories can be migrated and, for read-only access, replicated.

arch repositories have an easy-to-understand format, amendable to browsing by hand or with special-purpose interface programs.

For more information, see Archives.

Introducing arch Patch Logs

In every project tree, arch keeps a detailed "patch log": a record of what the original source tree was, along with a record for every patch applied to the tree (regardless of what branch the patch came from).

arch log entries use RFC822-style formatting. Automatically generated headers record what files were changed by each patch, what repository the patch came from, what user created the patch, etc. The patch log is a very useful source of information for programmers.

arch can automatically generate and maintain a GNU-style ChangeLog from its patch log.

When you "merge" two branches (combine the changes made in those branches), arch uses the information in the patch log to avoid redundantly applying patches.

For more information, see Patch Logs and ChangeLogs.

Cheap Branching and Smart Merging

Creating a branch in arch is inexpensive in both space and time.

The arch commands for merging have many subtle features to help a merge go smoothly.

For more information, see Basic Branching and Merging, Multi-Branch Merging -- The reconcile Command, and Idempotent Merging.

What Does It All Mean?

Putting all those features together, arch is an elegant and more featureful replacement for older systems like CVS .

For example, using arch , every programmer can conveniently have a private repository for day-to-day work rather than burdening a shared repository with per-user branches.

A project can be "multi-homed" -- stored in multiple repositories -- with different branches in each repository.

It is easy and convenient, when using arch , to improve and clean-up projects by reorganizing the files and directories they contain: something that is quite awkward with CVS .

The fancy branching and merging commands of arch make it convenient to do more development than ever in feature-specific branches, merging features as they are completed, generating a clean, complete, and isolated patch set for each new feature.

arch is written in a software-tools style: it is made up of many small programs, each of which does one job well. The commands have very regular option and argument syntax and input and output formats. arch is an excellent foundation for process automation and for layering under fancy graphical interfaces. arch is self-documenting and extensible.

arch: The arch Revision Control System
The Hackerlab at regexps.com