Which team do you belong to? Mono-repo or Multi-repo?

In case you are from Multi-repo, or you are thinking about it, explore WEST.

In case you don’t know what this article is about… you are lucky! May be your software stack has not exploded yet to parallel development; Thinking about multiple repositories is not a question in your mind yet.

Multi Repo

Some times a single Git repository is not enough for your code base. You split them logically it into multiple different repositories. What is logical is up to your product. The topic of how you split your code base in into multiple repositories would take a dedicated discussion. But for example, may be you choose to split into separate repository for documentation, portable API Layer, core library, non portable hardware specific code, tests, etc.

Here are some of the reasons for splitting your code into multiple repositories.

Access and Authority Control

  • E.g. API layer would need a stringent scrutiny for change, while tests and documentation would see relatively relaxed attention.

  • The set of engineers approving any change to core library would be different from those approving changes to tests.

  • Any change in library would need to pass through some level of testing before being approved for mainline.

Reuse

  • Same sub repository can be re-used across various different projects and teams.

  • In some cases, the repository itself is external to you and you are using it as-is.

  • Instead of manually copying files from one location to other, collaboration/re-use can be leveraged directly by Git.

    If you have done this many times, you will know what are the pitfalls of such manual copy.

WEST

WEST is a Python based project / tool / library / utility. It can be used for managing multiple git repositories (and more).

Extensive documentation for west is available as a part of the Zephyr Project

How to set it up?

Generally, it starts with creating a west.yml in your top level repo. Something like this:

 1---
 2manifest:
 3    # The version of this manifest
 4    version: '1.2'
 5    # Default project attributes
 6    defaults:
 7        remote: c6h6-demo
 8    # Short names for project URLs
 9    remotes:
10        - name: c6h6-demo
11          url-base: git@github.com:c6h6-demo
12    # A list of projects managed by west
13    projects:
14        - name: sub-r1
15          description: West Demo sub repo r1
16          repo-path: west-sub-r1
17          revision: main
18    self:
19        path: _top

You want to try?

Do you want to see it / use it for yourself? There’s a demo site for you to play.

Install west

1pip install --user west

Initialize a repo

Run the command:

1west init --manifest-url git@github.com:c6h6-demo/west-top.git

The output would be:

1=== Initializing in /Users/c6h6/data/p/west-demo/try2
2--- Cloning manifest repository from git@github.com:c6h6-demo/west-top.git
3Cloning into '/Users/c6h6/data/p/west-demo/try2/.west/manifest-tmp'...
4--- setting manifest.path to _top
5=== Initialized. Now run "west update" inside /Users/c6h6/data/p/west-demo/try2.

The tree would look like:

1.
2└── _top
3    ├── README.md
4    └── west.yml

As you can see, it contains the west.yml and README.md. But it still does not contain the dependent repositories.

Bring the dependent sub repositories

Run the command:

1west update --keep-descendants

The output would be:

1=== updating sub-r1 (sub-r1):
2--- sub-r1: initializing
3Initialized empty Git repository in /Users/c6h6/data/p/west-demo/try2/sub-r1/.git/
4--- sub-r1: fetching, need revision main
5From github.com:c6h6-demo/west-sub-r1
6 * branch            main       -> FETCH_HEAD
7HEAD is now at 36fc942 Initial commit
8HEAD is now at 36fc942 Initial commit

The tree would look like:

1.
2├── _top
3│   ├── README.md
4│   └── west.yml
5└── sub-r1
6    └── README.md

Now, we have sub-r1 and it’s contents.

Check for changes

Run the command:

1west status

The output would be empty.

But, if we modify sub-r1/README.md, the output would be:

1=== status of sub-r1 (sub-r1):
2HEAD detached at refs/heads/manifest-rev
3Changes not staged for commit:
4  (use "git add <file>..." to update what will be committed)
5  (use "git restore <file>..." to discard changes in working directory)
6    modified:   README.md
7
8no changes added to commit (use "git add" and/or "git commit -a")

For little silent output:

1west status -s

The output would be:

1=== status of sub-r1 (sub-r1):
2 M README.md

Verbose status check

For no changes, if we run the command:

1west -v status

The output would be:

1=== status of manifest (_top):
2On branch main
3Your branch is up to date with 'origin/main'.
4
5nothing to commit, working tree clean
6=== status of sub-r1 (sub-r1):
7HEAD detached at refs/heads/manifest-rev
8nothing to commit, working tree clean

west and manifest-rev

Run the command:

1git -C sub-r1 status

The output would be:

1HEAD detached at refs/heads/manifest-rev
2nothing to commit, working tree clean

As you can see, west has created a local reference commit of manifest-rev to refer to the revision as selected by the manifest.

Other commands

If you want to run some command in all git repositories, run the command similar to:

1west forall -c "git fetch"

The output for above command would be:

1=== running "git fetch" in manifest (_top):
2=== running "git fetch" in sub-r1 (sub-r1):
3From github.com:c6h6-demo/west-sub-r1
4 * [new branch]      main       -> c6h6-demo/main

Switch away from manifest-rev

As you can see with west output of west -v status, the working tree refers to manifest-rev. If you want to switch to top of the branch you can take help of the following command:

1west list -f "git -C {path} checkout {revision}" | grep -v " HEAD"

It uses the built in command west list to generate a command to check out directly the branch for that repo.

As of now the output would be only:

1git -C sub-r1 checkout main

But if you have more repositories, the output would change.

How does it compare to git repo?

Git Repo is a tool that is extensively used for the Android project.

Key difference between both the tool is that repo is xml based, while west is yaml based. But both the tools try to solve a similar problem.

Read more at WEST document.

How does it compare to git submodules?

Well, in simple words, west and repo exists because git submodules is not enough for decent sized work flows.

Read more at WEST document

What could be better?

west is a nice tool, and I think for quite some time I will stick to it, but here are few things that I feel west potentially can do better after some initial run.

  • As compared to git-repo, here are few improvements that I feel west could do better.

    • repo does not use a default upstream repository, neither should west

      • i.e. repo init will fail. But, west init will clone Zephyr.

        If west wants to be true generic / general project, it should not be coupled with Zephyr

    • repo can work in parallel, it would be nice if west could too.

  • Shell Completion

    Shell completion as reported in official documentation, using west completion, which helps for completion using bash/zsh/fish, is part of zephyr repo. Having this outside of zephyr would help users.