Gear/Introduction: Difference between revisions

From ALT Linux Wiki
m (→‎Commiting the stuff: - more real-life example)
m (+interlang: ru)
 
(14 intermediate revisions by 8 users not shown)
Line 1: Line 1:
This is introduction to [[gear]] for the people who already have the RPM packaging experience and know how to use git.
This is an introduction to [[gear]] for the people who already have the RPM packaging experience and know how to use git.


== Purpose of gear ==
== Purpose of gear ==


gear provides a mean to build RPM packages from git repository, by providing
Gear is a set of tools for maintenance and release management of RPM packages based on a git repository. A gear-enabled git repository of a software package combines
* format of rules file, describing the steps to generate source package from repository
* software development history
* modifications necessary to produce an RPM package
* rules to bind software releases to specific RPM-based distribution releases
 
Developers using Gear need to understand
* a format of the rules file describing transformation procedures that convert a given git changeset and branches to a number of source files for a RPM package
* utilities for building packages and working with source code (importing upstream sources and managing patches)
* utilities for building packages and working with source code (importing upstream sources and managing patches)


== RPM-specific stuff ==
== RPM-specific stuff ==


RPM source packages are built from the following files:
RPM source packages are built using the following files:
* .spec file, which is roughly debian/* without debian/patches/*
* .spec file, which is roughly debian/* without debian/patches/*
* several source files/tarballs
* a number of source files/archives
* and several patch-files.
* optionally patches containing local fixes to the upstream source code


gear provides the means to build/generate latter two types of files from
Gear gives a way to automate generation of patches and source files off the git repostiory.
the git, so repository is not required to contain them in the form
digestible by rpmbuild.


== Working with upstream source code ==
== Working with upstream source code ==


Upstream source code is generally stored in the separate branch.
A source code of a project is generally tracked in a separate branch that could be synchronized with a corresponding upstream repository by traditional git tools. When the upstream of the project does not use a VCS that allows direct import into git, the source code can still be managed with archives and source RPM packages.
 
There are two main sources of source code: upstream tarballs or upstream
VCS, plus importing SRPMs for keeping older packaging history.


=== SRPMs ===
=== SRPMs ===
Line 32: Line 32:


will import the given SRPMs to the branch ("srpms" by default) in the
will import the given SRPMs to the branch ("srpms" by default) in the
repository (like git-import-dsc).
repository (This operation is similar to <tt>git-import-dsc</tt> from <tt>git-buildpackage</tt>).


Importing SRPM creates git layout similar to the "Like SRPM" scenario
Importing SRPM creates a git layout similar to the "Like SRPM" scenario
described below.
described below.


Line 43: Line 43:
  ...
  ...


will update the subdirectory foo in git repository with contents of
will update the subdirectory foo in the git repository with the contents of
tarball. This operation is similar to the <tt>git-import-orig</tt> from
the tarball. This operation is similar to <tt>git-import-orig</tt> from
<tt>dpkg-buildpackage</tt>. As SRPMs may contain several sources, the source code usually stored in the subdirectory.
<tt>git-buildpackage</tt>. As SRPMs may contain several sources, the source code is usually stored in a subdirectory of the repository.


=== Upstream VCS ===
=== Upstream VCS ===


This should be obvious: just fetch/pull if upstream uses git, or use
This should be obvious: just fetch/pull if the upstream uses git, or use conversion tools (e.g. svn2git, cvs2git) if it has not yet migrated to git.
converter if it does not.


== Packaging and patching ==
== Packaging and patching ==


There are three major scenarios for keeping changes in the repository:
There are three major scenarios for keeping changes in the repository:
* like SRPM
* Work with changes much like in SRPM era
* patches in one branch
* Keep patches in one branch
* patches in topic branches
* Keep patches in specialized topic branches


=== Scenario 1. "Like SRPM" ===
=== Scenario 1. "Like SRPM" ===


This is least useful, but simple scenario. git is used just for keeping
This simple scenario where git is used just for keeping history, and the usual maintainer's workflow is not changed.
history, and the usual maintainer's workflow is not changed.


Branches in repository:
Branches in repository:
Line 75: Line 73:
  foo-another-fixed.patch  (in master)
  foo-another-fixed.patch  (in master)


git repository is generated by the importing SRPMs or by creating from scratch.
git repository is generated by importing SRPMs or by creating from scratch.


==== Importing tarball to the upstream branch ====
==== Importing tarball to the upstream branch ====


By using master branch to make it a bit easier
Use master branch to make it a bit easier


  $ mkdir foo
  $ mkdir foo
Line 85: Line 83:
  $ git init foo
  $ git init foo
  $ gear-update ../foo-1.0.tar.gz foo
  $ gear-update ../foo-1.0.tar.gz foo
  $ git checkout -b upstream
  $ git branch upstream


==== Adding spec and patches ====
==== Add spec and patches ====


  $ vi foo.spec
  $ vi foo.spec
Line 93: Line 91:
  $ vi foo-another-fixed.patch
  $ vi foo-another-fixed.patch


==== Adding gear/rules ====
==== Add gear/rules ====


The .gear/rules will have the following contents:
The .gear/rules will have the following content:


  copy: foo-something-fixed.patch
  copy: foo-something-fixed.patch
Line 101: Line 99:
  tar.gz: foo
  tar.gz: foo


This will add both patches and tar.gz'ed directory foo/ to the buildroot
This sample rules file describes a source tarball from a <tt>foo</tt> directory and two patches.  
where the package is built.


Specfile is picked up automatically if there is only .spec in the root
Specfile is picked up automatically if there is only .spec in the root
Line 109: Line 106:
==== Commiting the stuff ====
==== Commiting the stuff ====


gear is picky about it - it only handles the stuff which is in repo.
Gear is relying on the whole history being recorded by git. This means that any changes you'd like to see in RPM, must be committed to the repository.


  $ git add .  # Commit entire workdir
  $ git add .  # Commit the entire workdir
  $ gear-commit # this is wrapper around git-commit which uses the last
  $ gear-commit # this is a wrapper around git-commit which uses the last
               # changelog entry as a commit message.
               # changelog entry as a commit message.


=== Scenario 2. "Small fixes" ===
=== Scenario 2. "Small fixes" ===


This is scenario for the packages which need the small non-overlapping
This scenario is useful for packages which deviate from upstream with a small number of non-overlapping
fixes here and there. This is also easiest scenario to use.
fixes here and there. This is also the easiest scenario to use.


Branches in repository:
Branches in repository:
Line 132: Line 129:


  $ git checkout master
  $ git checkout master
  $ vim ... # hack-hack
  $ vim ...     # hack-hack
$ git add ... # add new files if needed (optional)
  $ git commit
  $ git commit


Line 145: Line 143:


This will generate foo.tar, containing upstream source code, taken from
This will generate foo.tar, containing upstream source code, taken from
the tag <tt>v${version}</tt>, where version is parsed from the <tt>foo.spec</tt>, and the diff with the difference between the directory foo in <tt>v${version}</tt> tag
the tag <tt>v${version}</tt>, where version is parsed from the <tt>foo.spec</tt>, and the diff is containing the difference between the directory foo in <tt>v${version}</tt> tag and current content of the directory foo.
and the current directory foo.


gear does not use real git tags, but instead it uses the tags stored in <tt>.gear/tags</tt>.
Gear does not use real git tags, but instead it uses the tags stored in <tt>.gear/tags</tt>. The reason for that is to keep packages reproduceable over time as tags may move but SHA1 labels recorded at the time of run of <tt>gear-update-tag</tt> will be able to address original changesets properly.


This has the reasons: as tags may move over time, to achieve
The <tt>gear-update-tag</tt> utility writes down SHA1 lables of all tags referenced in package process to <tt>.gear/tags</tt>.
reproducibility of builds, tag position need to be stored in the git
tree. There is utility for maintaining this set of tags:


  $ gear-update-tag --all
  $ gear-update-tag --all
Line 160: Line 155:
  $ git-commit .gear/tags -m 'tags updated'
  $ git-commit .gear/tags -m 'tags updated'


==== Rebasing to new upstream version ====
==== Rebasing to a new upstream version ====


  $ git merge upstream
  $ git merge upstream
Line 166: Line 161:
  $ gear-commit
  $ gear-commit


That's all! As patches are stored in git branch, it's easy to update to
That's all! As patches are stored in a git branch, it's easy to update to
new upstream version: merge will detect many incosistences. As an
a new upstream version: merge will detect most incosistences. As an
additional bonus, if upstream uses git as well, patches forwarded
additional bonus, if the upstream uses git as well, patches forwarded
upstream and accepted there will not generate conflicts on merge.
upstream and accepted there will not generate conflicts on merge.


Line 180: Line 175:
=== Scenario 3. "Full-blown development" ===
=== Scenario 3. "Full-blown development" ===


This is scenario for the packages which need the lot of work downstream
This is a scenario for the packages where downstream maintainers do a lot of work (e.g. kernel) or they are upstream themselves. In this case each separate feature is maintained in a separate branch and gear rules are used to automatically produce patches for RPM.
(e.g. kernel).


Branches:
Branches:
Line 193: Line 187:
  foo/        (in upstream, patched in topic-*, master)
  foo/        (in upstream, patched in topic-*, master)
  .gear/rules (in master)
  .gear/rules (in master)
  foos.pec   (in master)
  foo.spec   (in master)


==== Working on package ====
==== Working on package ====


All the work is done in topic branches, each is dedicated for some topic:
All the work is done in topic branches, each is dedicated to some feature:


  $ git checkout topic-A
  $ git checkout topic-A
Line 205: Line 199:
  ...
  ...


As patches may overlap, some conflict resolution need to be
As patches may overlap, some conflict resolution needs to be
involved. The naive approach would be using the following branching
involved. The naive approach would be using the following branching
scheme:
scheme:
Line 215: Line 209:
  \--> master
  \--> master


And then generating patch-per-topic by using .gear/rules. This will not
And then generating patch-per-topic by using .gear/rules. This will not always work, as the generated patches might conflict.
work, as the generated patches will conflict.


Instead, the following scheme is used:
To address the issue branching off each feature topic could be used:


  * upstream
  * upstream
Line 225: Line 218:
         \-> master
         \-> master


And gear-merge(1) utility merges the branches as described in
And gear-merge(1) utility merges the branches as described in .gear/merge file:
.gear/merge file:


  merge: upstream topic-A
  merge: upstream topic-A
Line 232: Line 224:
  merge: topic-B master
  merge: topic-B master


After using this utility (and resolving all conflicts occured), master
After using this utility (and resolving all possible conflicts), master branch gets a cumulative change.
branch gets a cumulative change.


==== .gear/rules ====
==== .gear/rules ====


The .gear/rules file will be exactly the same as for previous scenario,
The .gear/rules file will be exactly the same as for the previous scenario,
so the gear-update-tag(1) is also required:
so the gear-update-tag(1) is also required:


Line 243: Line 234:
  diff: diff: v@version@:foo foo
  diff: diff: v@version@:foo foo


==== Rebasing to new upstream version ====
==== Rebasing to a new upstream version ====


  $ ... # obtain the new upstream code in upstream branch
  $ ... # obtain the new upstream code in upstream branch
Line 261: Line 252:
  $ gear-hsh
  $ gear-hsh


Here the .gear/rules is consulted and the following is performed:
<tt>gear-hsh</tt> first uses .gear/rules file to produce all needed parts of a SRPM. After that gear will supply a created SRPM to a [[hasher]] build environment.
 
* pkg.tar package is constructed, according to the .gear/rules file
* hasher is run with the given source package.


There are also another build-commands: gear-rpm (uses <tt>rpmbuild</tt> instead of [[hasher]]) and gear-remote-hsh/gear-remote-rpm, which use remote host for actual building (communicating over SSH).
There are also other build commands: gear-rpm (uses <tt>rpmbuild</tt> instead of [[hasher]]) and gear-remote-hsh/gear-remote-rpm, which use remote host for actual building (communicating over SSH).




[[ru:Руководство по gear]]
[[Category:Sisyphus]]
[[Category:Sisyphus]]
[[Category:Devel]]
[[Category:Devel]]

Latest revision as of 22:22, 18 February 2015

This is an introduction to gear for the people who already have the RPM packaging experience and know how to use git.

Purpose of gear

Gear is a set of tools for maintenance and release management of RPM packages based on a git repository. A gear-enabled git repository of a software package combines

  • software development history
  • modifications necessary to produce an RPM package
  • rules to bind software releases to specific RPM-based distribution releases

Developers using Gear need to understand

  • a format of the rules file describing transformation procedures that convert a given git changeset and branches to a number of source files for a RPM package
  • utilities for building packages and working with source code (importing upstream sources and managing patches)

RPM-specific stuff

RPM source packages are built using the following files:

  • .spec file, which is roughly debian/* without debian/patches/*
  • a number of source files/archives
  • optionally patches containing local fixes to the upstream source code

Gear gives a way to automate generation of patches and source files off the git repostiory.

Working with upstream source code

A source code of a project is generally tracked in a separate branch that could be synchronized with a corresponding upstream repository by traditional git tools. When the upstream of the project does not use a VCS that allows direct import into git, the source code can still be managed with archives and source RPM packages.

SRPMs

$ gear-srpmimport some-0.1.src.rpm
$ gear-srpmimport some-0.2.src.rpm
...

will import the given SRPMs to the branch ("srpms" by default) in the repository (This operation is similar to git-import-dsc from git-buildpackage).

Importing SRPM creates a git layout similar to the "Like SRPM" scenario described below.

Tarballs

$ gear-update foo-0.2.tar.gz foo
$ gear-update foo-2.0.tar.gz foo
...

will update the subdirectory foo in the git repository with the contents of the tarball. This operation is similar to git-import-orig from git-buildpackage. As SRPMs may contain several sources, the source code is usually stored in a subdirectory of the repository.

Upstream VCS

This should be obvious: just fetch/pull if the upstream uses git, or use conversion tools (e.g. svn2git, cvs2git) if it has not yet migrated to git.

Packaging and patching

There are three major scenarios for keeping changes in the repository:

  • Work with changes much like in SRPM era
  • Keep patches in one branch
  • Keep patches in specialized topic branches

Scenario 1. "Like SRPM"

This simple scenario where git is used just for keeping history, and the usual maintainer's workflow is not changed.

Branches in repository:

master
upstream

Tree layout:

foo/                      (in upstream, master)
.gear/rules               (in master)
foo.spec                  (in master)
foo-something-fixed.patch (in master)
foo-another-fixed.patch   (in master)

git repository is generated by importing SRPMs or by creating from scratch.

Importing tarball to the upstream branch

Use master branch to make it a bit easier

$ mkdir foo
$ cd foo
$ git init foo
$ gear-update ../foo-1.0.tar.gz foo
$ git branch upstream

Add spec and patches

$ vi foo.spec
$ vi foo-something-fixed.patch
$ vi foo-another-fixed.patch

Add gear/rules

The .gear/rules will have the following content:

copy: foo-something-fixed.patch
copy: foo-another-fixed.patch
tar.gz: foo

This sample rules file describes a source tarball from a foo directory and two patches.

Specfile is picked up automatically if there is only .spec in the root directory fo git.

Commiting the stuff

Gear is relying on the whole history being recorded by git. This means that any changes you'd like to see in RPM, must be committed to the repository.

$ git add .   # Commit the entire workdir
$ gear-commit # this is a wrapper around git-commit which uses the last
              # changelog entry as a commit message.

Scenario 2. "Small fixes"

This scenario is useful for packages which deviate from upstream with a small number of non-overlapping fixes here and there. This is also the easiest scenario to use.

Branches in repository:

master
upstream

Tree layout:

foo/        (in upstream, patched in master)
.gear/rules (in master)
foo.spec    (in master)

Working on package

$ git checkout master
$ vim ...     # hack-hack
$ git add ... # add new files if needed (optional)
$ git commit

All the changes are applied just on top of the upstream source code.

.gear/rules

The .gear/rules file will be of the following form:

tar: v@version@:foo
diff: diff: v@version@:foo foo

This will generate foo.tar, containing upstream source code, taken from the tag v${version}, where version is parsed from the foo.spec, and the diff is containing the difference between the directory foo in v${version} tag and current content of the directory foo.

Gear does not use real git tags, but instead it uses the tags stored in .gear/tags. The reason for that is to keep packages reproduceable over time as tags may move but SHA1 labels recorded at the time of run of gear-update-tag will be able to address original changesets properly.

The gear-update-tag utility writes down SHA1 lables of all tags referenced in package process to .gear/tags.

$ gear-update-tag --all

Don't forget to commit the changes after updating:

$ git-commit .gear/tags -m 'tags updated'

Rebasing to a new upstream version

$ git merge upstream
... Fix conflicts, update changelog, remove unnecesary patches etc ...
$ gear-commit

That's all! As patches are stored in a git branch, it's easy to update to a new upstream version: merge will detect most incosistences. As an additional bonus, if the upstream uses git as well, patches forwarded upstream and accepted there will not generate conflicts on merge.

New package revision

$ vim ... # hack-hack, update changelog
$ gear-commit

Also simple.

Scenario 3. "Full-blown development"

This is a scenario for the packages where downstream maintainers do a lot of work (e.g. kernel) or they are upstream themselves. In this case each separate feature is maintained in a separate branch and gear rules are used to automatically produce patches for RPM.

Branches:

upstream
topic-A
topic-B
...
master

Tree layout:

foo/        (in upstream, patched in topic-*, master)
.gear/rules (in master)
foo.spec    (in master)

Working on package

All the work is done in topic branches, each is dedicated to some feature:

$ git checkout topic-A
$ vim ... # hack-hack-hack
$ git commit
$ git checkout topic-B
...

As patches may overlap, some conflict resolution needs to be involved. The naive approach would be using the following branching scheme:

* upstream
|
\--> topic-A
\--> topic-B
\--> master

And then generating patch-per-topic by using .gear/rules. This will not always work, as the generated patches might conflict.

To address the issue branching off each feature topic could be used:

* upstream
\-> topic-A
    \-> topic-B
        \-> master

And gear-merge(1) utility merges the branches as described in .gear/merge file:

merge: upstream topic-A
merge: topic-A topic-B
merge: topic-B master

After using this utility (and resolving all possible conflicts), master branch gets a cumulative change.

.gear/rules

The .gear/rules file will be exactly the same as for the previous scenario, so the gear-update-tag(1) is also required:

tar: v@version@:foo
diff: diff: v@version@:foo foo

Rebasing to a new upstream version

$ ... # obtain the new upstream code in upstream branch
$ gear-merge
... Fix conflicts, update changelog, remove unnecessary patches etc...
$ gear-commit

Due to gear-merge, it's only necessary to fix the conflicts once.

New package revision

$ vim ... # hack-hack, commit
$ gear-merge

Builds

$ gear-hsh

gear-hsh first uses .gear/rules file to produce all needed parts of a SRPM. After that gear will supply a created SRPM to a hasher build environment.

There are also other build commands: gear-rpm (uses rpmbuild instead of hasher) and gear-remote-hsh/gear-remote-rpm, which use remote host for actual building (communicating over SSH).