LilyPond — Contributor’s Guide

1.1 Help us

We need you!

Thank you for your interest in helping us – we would love to see you get involved! Your contribution will help a large group of users make beautifully typeset music.

Even working on small tasks can have a big impact: taking care of them allows experienced developers to work on advanced tasks instead of spending time on those simple tasks.

For a multi-faceted project like LilyPond, sometimes it’s tough to know where to begin. In addition to the avenues proposed below, you can send an e-mail to the lilypond-devel@gnu.org mailing list, and we’ll help you to get started.

Simple tasks

No programming skills required!

• Mailing list support: answer questions from fellow users. (This may entail helping them navigate the online documentation; in such cases it may sometimes be appropriate to point them to version-agnostic URL paths such as ‘latest’ or ‘stable’, which are automatically redirected.)
• Bug reporting: help users create proper Bug reports, and/or join the Bug Squad to organize Issues.
• Documentation: small changes can be proposed by following the guidelines for Documentation suggestions.
• LilyPond Snippet Repository (LSR): create and fix snippets following the guidelines in Adding and editing snippets.
• Discussions, reviews, and testing: the developers often ask for feedback about new documentation, potential syntax changes, and testing new features. Please contribute to these discussions!

Advanced tasks

These jobs generally require that you have the source code and can compile LilyPond.

Contributors using Linux or FreeBSD may also use Lilydev, but if they prefer their own development environment, they should read Working with source code, and Compiling.

Begin by reading Summary for experienced developers.

• Documentation: for large changes, see Documentation work.
• Website: the website is built from the normal documentation source. See the info about documentation, and also Website work.
• Translations: see Translating the documentation, and Translating the website.
• Bugfixes or new features: read Programming work.

1.2 Overview of work flow

Git is a version control system that tracks the history of a program’s source code. The LilyPond source code is maintained as a Git repository, which contains:

• all of the source files needed to build LilyPond, and
• a record of the entire history of every change made to every file since the program was born.

The ‘official’ LilyPond Git repository is hosted by the GNU Savannah software forge at https://git.sv.gnu.org. The server provides two separate interfaces for viewing the LilyPond Git repository online: cgit and gitweb.

However, the main development takes place at https://gitlab.com/lilypond/lilypond/, which also hosts the project’s issues. Automatic mirroring ensures that ‘important’ branches (such as master and stable/*) are up to date on the ‘official’ repository at GNU Savannah, so you can also base your development on a clone from there.

Compiling (‘building’) LilyPond allows developers to see how changes to the source code affect the program itself. Compiling is also needed to package the program for specific operating systems or distributions. LilyPond can be compiled from a local Git repository (for developers), or from a downloaded tarball (for packagers). Compiling LilyPond is a rather involved process, and most contributor tasks do not require it.

Contributors can contact the developers through the ‘lilypond-devel’ mailing list. The mailing list archive is located at https://lists.gnu.org/archive/html/lilypond-devel/. If you have a question for the developers, search the archives first to see if the issue has already been discussed. Otherwise, send an email to lilypond-devel@gnu.org. You can subscribe to the developers’ mailing list here: https://lists.gnu.org/mailman/listinfo/lilypond-devel.

1.3 Summary for experienced developers

If you are already familiar with typical open-source tools, here’s what you need to know:

• ‘Official’ source repository: hosted by GNU Savannah

  https://git.savannah.gnu.org/gitweb/?p=lilypond.git
  

• Development platform: hosted by GitLab; also includes the issue tracker (see Issues)

  https://gitlab.com/lilypond/lilypond/
  

• Environment variables: needed by many maintenance scripts, and many instructions in this guide rely on them; see Environment variables.
• Mailing lists: given in Contact.
• Git branches:
  • master: always base your work on this branch, but never push directly to it.
  • dev/foo: feel free to push any new branches under dev/ and use GitLab to create Merge Requests (MR), which eventually get merged into master after they have passed automatic testing (see below).
• Regression tests: also known as “regtests”. A collection of around two thousand .ly files that are used to track LilyPond’s engraving output between released stable and unstable versions as well as checked for all patches submitted for testing.

  If a patch introduces any unintentional changes to any of the regtests it is very likely it will be rejected (to be fixed) – if you expect any regression test changes, always make sure that they are explained clearly as part of the patch description when submitting for testing. For more information, see Regression tests.

• Reviews: after finishing work on a patch or branch you should do the following.
  1. Commit the changes and create a Merge Request. See Uploading a patch for review for more information.
  2. A Merge Request is usually automatically tested within an hour of submission. Once it has passed the basic tests – make check, make, make doc – the tracker will be updated and the patch’s status will change to Patch::review for other developers to examine.
  3. Every third day, the “Patch Meister” will examine all Merge Requests currently under review, looking for any comments by other developers. Depending on what has been posted, the patch will be either “moved on” to the next patch status (Patch::countdown), set back to Patch::needs_work, or if more discussion is needed, left at Patch::review. In all cases the Merge Request will be updated by the Patch Meister accordingly.
  4. Once another three days have passed, any patch that has been given Patch::countdown status will be changed to Patch::push, the Merge Request is updated, and the developer can now rebase and merge to the master branch (or ask one of the other developers to merge it for you).

  Advanced note: This process does means that most patches will take about a week before finally being merged into master. With the limited resources for reviewing patches available and a history of unintended breakages in the master branch (from patches that have not had time to be reviewed properly), this is the best compromise we have found.

1.4 Mentors

We have a semi-formal system of mentorship, similar to the medieval “journeyman/master” training system. New contributors will have a dedicated mentor to help them “learn the ropes”.

Contributor responsibilities

1. Ask your mentor which sections of the CG you should read.
2. If you get stuck for longer than ten minutes, ask your mentor. They might not be able to help you with all problems, but we find that new contributors often get stuck with something that could be solved or explained with two or three sentences from a mentor.
3. If you have been working on a task much longer than was originally estimated, stop and ask your mentor. There may have been a miscommunication, or there may be some time-saving tips that could vastly simplify your task.
4. Send patches to your mentor for initial comments.
5. Inform your mentor if you’re going to be away for a month, or if you leave entirely. Contributing to LilyPond isn’t for everybody; just let your mentor know so that we can reassign that work to somebody else.
6. Inform your mentor if you’re willing to do more work – we always have way more work than we have helpers available. We try to avoid overwhelming new contributors, so you’ll be given less work than we think you can handle.

Mentor responsibilities

1. Respond to questions from your contributor(s) promptly, even if the response is just “sorry, I don’t know” or “sorry, I’m very busy for the next 3 days; I’ll get back to you then”. Make sure they feel valued.
2. Inform your contributor(s) about the expected turnaround for your emails – do you work on LilyPond every day, or every weekend, or what? Also, if you’ll be unavailable for longer than usual (say, if you normally reply within 24 hours, but you’ll be at a conference for a week), let your contributors know. Again, make sure they feel valued, and that your silence (if they ask a question during that period) isn’t their fault.
3. Inform your contributor(s) if they need to do anything unusual for the builds, such as doing a make clean or make doc-clean, or switching git branches (not expected, but just in case).
4. You don’t need to be able to completely approve patches. Make sure the patch meets whatever you know of the guidelines (documentation style, code indentation, whatever).
5. Keep track of patches from your contributor. Either open Merge Requests by yourself or help and encourage them to upload the patches themselves.
6. Encourage your contributor to review patches, particularly your own! It doesn’t matter if they’re not familiar with C++, Scheme, the build system, or the documentation guidelines – simply going through the process is valuable. Besides, anybody can find a typo!
7. Contact your contributor at least once a week. The goal is just to get a conversation started – there’s nothing wrong with simply copying & pasting this into an email:

   Hey there,
   
   How are things going?  If you sent a patch and got a review, do
   you know what you need to fix?  If you sent a patch but have no
   reviews yet, do you know when you will get reviews?  If you are
   working on a patch, what step(s) are you working on?
   

2.1 LilyDev

“LilyDev” is a custom GNU/Linux operating system which includes all the necessary software and tools to compile LilyPond, the documentation and the website (also see Website work).

While compiling LilyPond on macOS and Windows is possible, both environments are complex to set up. LilyDev can be easily run inside a ‘virtual machine’ on either of these operating systems relatively easily using readily available virtualization software. We recommend using VirtualBox as it is available for all major operating systems and is very easy to install & configure.

LilyDev comes in two ‘flavours’: containers and a standard disk image. Windows or macOS users should choose the Debian disk image (to be run in a virtual machine), that is the file named LilyDev-VERSION-debian-vm.zip. GNU/Linux users are recommended to choose one of the containers (currently Debian or Fedora), which are smaller in size, lightweight and easier to manage. The Fedora disk image has currently not been released, you can create it from the sources located in the /mkosi subdirectory of the LilyDev repository, however.

Download the appropriate file from here:

https://github.com/fedelibre/LilyDev/releases/latest

If you are not familiar with GNU/Linux, it may be beneficial to read a few “introduction to Linux” type web pages.

2.2 Compiling with LilyDev

LilyDev is our custom GNU/Linux which contains all the necessary dependencies to do LilyPond development; for more information, see LilyDev.

Preparing the build

To prepare the build directory, enter (or copy&paste) the below text. This should take less than a minute.

cd $LILYPOND_GIT
sh autogen.sh --noconfigure
mkdir -p build/
cd build/
../configure

Building lilypond

Compiling LilyPond will take anywhere between 1 and 15 minutes on most ‘modern’ computers – depending on CPU and available RAM. We also recommend that you minimize the terminal window while it is building; this can help speed up on compilation times.

cd $LILYPOND_GIT/build/
make

It is possible to run make with the -j option to help speed up compilation times even more. See Compiling LilyPond

You may run the compiled lilypond with:

cd $LILYPOND_GIT/build/
out/bin/lilypond my-file.ly

Building the documentation

Compiling the documentation is a much more involved process, and will likely take 2 to 10 hours.

cd $LILYPOND_GIT/build/
make
make doc

The documentation is put in out-www/offline-root/. You may view the html files by entering the below text; we recommend that you bookmark the resulting page:

firefox $LILYPOND_GIT/build/out-www/offline-root/index.html

Installing

Don’t. There is no reason to install LilyPond within LilyDev. All development work can (and should) stay within the $LILYPOND_GIT directory, and any personal composition or typesetting work should be done with an official release.

Problems and other options

To select different build options, or isolate certain parts of the build, or to use multiple CPUs while building, read Compiling.

In particular, contributors working on the documentation should be aware of some bugs in the build system, and should read the workarounds in Generating documentation.

2.3 Now start work!

LilyDev users may now skip to the chapter which is aimed at their intended contributions:

• Documentation work
• Translating the documentation
• Website work
• Regression tests
• Programming work

These chapters are mainly intended for people not using LilyDev, but they contain extra information about the “behind-the-scenes” activities. We recommend that you read these at your leisure, a few weeks after beginning work with LilyDev.

• Working with source code
• Compiling

3.1 Setting up

The authenticity of host 'gitlab.com' can't be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Warning: Permanently added 'gitlab.com' (ECDSA) to the list of known hosts.

Permission denied (publickey).
fatal: The remote end hung up unexpectedly

git clone git@gitlab.com:lilypond/lilypond.git

cd lilypond
git remote add fork git@gitlab.com:your-username/lilypond.git

git remote -v

git config --global user.name "John Smith"
git config --global user.email john@example.com

git config --global color.ui auto

git config --global core.editor "gedit --wait --new-window"

export PS1="\u@\h \w\$(__git_ps1)$ "
export GIT_PS1_SHOWDIRTYSTATE=true
export GIT_PS1_SHOWUNTRACKEDFILES=true
export GIT_PS1_SHOWUPSTREAM=auto

3.2 Git cheat sheet

The intent of this section is to get you working on LilyPond quickly. If you want to learn about Git, see Further Git documentation resources.

Also, these instructions are designed to eliminate the most common problems we have found in using Git. If you already know Git and have a different way of working, great! Feel free to ignore this advice.

Pulling recent changes

As LilyPond’s source code is continously improved, it is wise to integrate recent changes into your local copy whenever you start a working session. On the master branch (this term is explained below), run:

git pull

Viewing the history

Each change is contained in a commit with an explanatory message. To list commits starting from the latest:

git log

Press Enter to see more or Q to exit.

Start work: make a new branch

The Git workflow is based on branches, which can be viewed as different copies of the source code with concurrent changes that are eventually merged. You start a contribution by creating a branch, freezing the initial state of the source code you will base your work onto. Ultimately, your branch will be merged into master. This latter, special branch centralizes all features developed simultaneously and is the source for unstable releases.

Let’s pretend you want to add a section to the Contributor’s Guide about using branches. To create a new branch for this:

git branch cg-add-branches

Switching branches

Switching branches is somehow like “loading a file”, although in this case it is really “loading a directory and subdirectories full of files”. The command to use is git switch.¹

git switch master
git switch cg-add-branches
git switch origin/release/unstable

Branches that begin with origin/ are part of the remote repository, rather than your local repository, so when you check them out you get a temporary local branch. Therefore, do not commit to these either. Always work in a local branch.

Listing branches

To list local branches:

git branch

If you want remote branches too:

git branch -a

In the output, the current branch is prefixed with a star.

Staging and committing files

Now edit files. To show a summary of your edits:

git status

For every file that you modified or added, first preview your changes:

git diff file

If everything looks right:

git add file

Then commit your changes:

git commit

A text editor window appears for you to write a commit message. See Writing good commit messages.

Amending and reverting changes

To add some more changes to the latest commit, stage them using git add, then run:

git commit --amend

This also works for rephrasing the commit message.

You might want to use git add -p instead of git add; this allows you to add changes incrementally in an interactive way.

To revert changes to a file that has not been committed yet, use git restore:²

git restore filename

You might want to use git reset -p instead of git restore; this allows you to revert changes incrementally in an interactive way.

To get back to the last commit, discarding all changes:

git reset --hard HEAD

If the commit to edit is not the top one, you need to perform an interactive rebase with git rebase -i $(git merge-base master HEAD). The full functionality of git rebase -i is not covered here; please try it and follow Git’s instructions or read any tutorial on the Web.

Uploading your branch for review

To upload the current branch on the remote repository:

git push -u fork cg-add-branches

This sets the remote branch so subsequent pushes are simpler:

git push

The next section covers how to create a Merge Request³ from your branch.

In response to review comments, you may need to amend your changes. Do not close your merge request and open a new one; instead, amend your commits, which can be done with git commit --amend or git rebase -i as explained above. Note that Git will by default refuse a push when you have amended your commits. This is because this kind of push is a destructive operation: once it is done, the old commits are no longer available on the remote branch. Git prevents this as a safety measure against deleting commits added by someone else without you realizing it. Do not follow Git’s advice to do git pull (which would try to integrate the remote changes into the local ones); instead, just force it with

git push --force-with-lease

Also note that due to the way GitLab compares successive revisions of a merge request, it is preferable if you do not mix catching up with master and changing your commits. In other words, use git rebase -i $(git merge-base master HEAD) rather than git rebase -i master. Alternatively, first rebase on master and push, then do the interactive rebase and push again.

Deleting branches

After the merge request has passed testing and was merged to master, or after a failed experiment, you can delete your local branch.

git switch master
git branch -d cg-add-branches

As a safety measure, this will fail if the commits of cg-add-branches are not present in master. This can be because you used GitLab to rebase your branch, which modifies the commit data and changes the hash. If you are sure that the branch is not needed anymore, replace the -d on the final line with a -D instead.

Over time, remote branches of accepted merge requests may accumulate in your local repository. If you want to delete these and get back to the state of the official repository, run

git fetch -p origin

(short for --prune) once in a while.

3.3 Lifecycle of a merge request

PATCHES: Countdown for February 30th

3.4 Writing good commit messages

Your commit message should begin with a one-line summary describing the change (no more than 50 characters long), and if necessary a blank line followed by more explanatory text (wrapped at 72 characters). Here is how a good commit message looks like:

Doc: add Baerenreiter and Henle solo cello suites

Added comparison of solo cello suite engravings to new essay with
high-resolution images.  Fixed cropping on Finale example.

Closes #1234.

The “Closes” part is specially recognized by GitLab. See the documentation for closing issues automatically.

Commit messages often start with a short prefix describing the general location of the changes. Commits affecting the documentation in English (or in several languages simultaneously) should be prefixed with “Doc:”. When the commit affects only one of the translations, use “Doc-**:”, where ** is the two-letter language code. For the website, this is “Web:” or “Web-**”. Commits that change CSS files should use “Web: CSS” or “Doc: CSS:”. Finally, changes to a single file are often prefixed with the name of the file involved.

The imperative form, e.g. “Include this in that”, is strongly preferred over the descriptive form “That is now included in this”.

See also this blog post for details on how to write good commit messages.

3.5 Commit access

New contributors are not able to push branches directly to the main repository – only members of the LilyPond development team have commit access. If you are a contributor and are interested in joining the development team, contact the Project Manager through the mailing list (lilypond-devel@gnu.org). Generally, only contributors who have already provided a number of patches which have been merged to the main repository will be considered for membership.

If you have been approved by the Project Manager, navigate to https://gitlab.com/lilypond and ‘Request access’ to the group. Make sure that your account can be related to your activity on the mailing list. If in doubt, please post the user name after requesting access.

Note that you will not have commit access until the Project Manager activates your membership. Once your membership is activated, LilyPond should appear under the heading “Groups” on your profile page. When this is done, you can test your commit access with a dry run:

git push --dry-run --verbose

3.6 Further Git documentation resources

The following page on the Git website provides links to the Pro Git book and a variety of tutorials, as well as the official man pages (also available with man git …).

https://git-scm.com/doc

The GitLab user documentation contains tutorials on using Git and GitLab:

https://docs.gitlab.com/ee/tutorials/#use-git

3.7 Repository directory structure

The following is a verbatim output of the file ROADMAP, which can be found in the top-level directory of LilyPond’s git repository.

Prebuilt Documentation and packages are available from:

    http://www.lilypond.org

LilyPond development is hosted at:

    https://gitlab.com/lilypond/

Here is a simple explanation of the directory layout for
LilyPond's source files.


.                        Toplevel READMEs, files for
|                          configuration and building, etc.
|
|-- Documentation/       Top sources for most of the manuals
|   |
|   |
|   |   INDIVIDUAL CHAPTERS FOR EACH MANUAL:
|   |     Note: "Snippets" and "Internals Reference" are
|   |     auto-generated during the Documentation Build process.
|   |
|   |
|   |-- en/contributor/  Contributor's Guide
|   |-- en/essay/        Essay on automated music engraving
|   |-- en/extending/    Extending the functionality of LilyPond
|   |-- en/included/     Doc files that are used more than once
|   |-- en/learning/     Learning Manual
|   |-- en/notation/     Notation Reference
|   |-- en/usage/        How to run the programs that come with LilyPond
|   |-- en/web/          Website files
|   |
|   |
|   |   TRANSLATED MANUALS:
|   |     Each language's directory can contain...
|   |       1) translated versions of:
|   |          * "en/*" sources for manuals
|   |          * individual chapters for each manual
|   |       2) a texidocs/ directory for snippet translations
|   |
|   |-- ca/              Catalan
|   |-- de/              German
|   |-- es/              Spanish
|   |-- fr/              French
|   |-- it/              Italian
|   |-- ja/              Japanese
|   |-- zh/              Chinese
|   |
|   |
|   |   MISCELLANEOUS DOC STUFF:
|   |
|   |-- bib/             Bibliography files for documentation
|   |-- css/             CSS files for HTML docs
|   |-- logo/            Web logo and "note" icon
|   |-- ly-examples/     .ly example files for the webpage
|   |-- misc/            Old announcements, ChangeLogs and NEWS
|   |-- pictures/        Images (eps/jpg/png/svg) for the webpage
|   |   `-- pdf/         (pdf)
|   |-- po/              Translated build/maintenance scripts
|   |-- snippets/        Auto-generated from the LSR and from ./new/
|   |   `-- new/         Snippets too new for the LSR
|   |-- topdocs/         AUTHORS, INSTALL
|   `-- tex/             TeX and texinfo library files
|
|
|   C++ SOURCES:
|
|-- flower/              A simple C++ library
|   `-- include/         C++ header files for basic LilyPond structures
|-- lily/                C++ sources for the LilyPond binary
|   `-- include/         C++ header files for higher-level stuff
|
|
|   LIBRARIES:
|
|-- ly/                  .ly \include files
|-- mf/                  MetaFont sources for Emmentaler fonts
|-- ps/                  PostScript library files
|-- scm/                 Scheme sources for LilyPond and subroutine files
|
|
|   SCRIPTS:
|
|-- config/              Autoconf helpers for configure script
|-- m4/                  Files used while generating the configure script
|-- python/              Python modules, MIDI module
|   `-- auxiliar/        Python modules for build/maintenance
|-- scripts/             End-user scripts (--> lilypond/usr/bin/)
|   |-- auxiliar/        Maintenance and non-essential build scripts
|   `-- build/           Essential build scripts
|
|
|   BUILD PROCESS:
|   (also see SCRIPTS section above)
|
|-- make/                Specific make subroutine files
|
|-- docker/
|   |-- base/            CI Docker files used for running `make`
|   |-- ci/              Support for continuous integration (CI) on gitlab
|   `-- doc/             CI Docker files used for running `make doc`
|
|-- release/             Scripts to generate and upload release packages
|   |-- binaries/        Scripts to build binaries
|   |   |-- ansible/     Ansible playbooks for building binaries
|   |   |-- lib/         Auxiliary files for building binaries
|   |   `-- relocate/    Relocation data for lilypond binary
|   `-- doc              Scripts to build documentation
|
|   REGRESSION TESTS:
|
|-- input/
|   `-- regression/      .ly regression tests
|       |-- abc2ly/      .abc regression tests
|       |-- lilypond-book/
|       |                lilypond-book regression tests
|       |-- midi/        midi2ly regression tests
|       |-- musicxml/    .xml and .itexi regression tests
|       `-- other/       regression tests without graphical output
|
|
|   MISCELLANEOUS:
|
|-- elisp/               Emacs LilyPond mode and syntax coloring
|-- vim/                 Vi(M) LilyPond mode and syntax coloring
`-- po/                  Translations for binaries and end-user scripts

4.1 Overview of compiling

Compiling LilyPond from source is an involved process, and is only recommended for developers and packagers. Typical program users are instead encouraged to obtain the program from a package manager (on Unix) or by downloading a precompiled binary configured for a specific operating system. Pre-compiled binaries are available on the Download page.

Compiling LilyPond from source is necessary if you want to build, install, or test your own version of the program.

A successful compile can also be used to generate and install the documentation, incorporating any changes you may have made. However, a successful compile is not a requirement for generating the documentation. The documentation can be built using a Git repository in conjunction with a locally installed copy of the program. For more information, see Building documentation without compiling.

Attempts to compile LilyPond natively on Windows have been unsuccessful, though a workaround is available (see LilyDev).

4.2 Requirements

sudo dnf install texlive-xetex

sudo apt-get install texlive-xetex

sudo zypper install texlive-xetex

sudo apt-get install texlive-xetex

4.3 Getting the source code

Downloading the Git repository

In general, developers compile LilyPond from within a local Git repository. Setting up a local Git repository is explained in Setting up.

Downloading a source tarball

Packagers are encouraged to use source tarballs for compiling.

The tarball for the latest stable release is available on the Source page.

The latest source code snapshot is also available as a tarball from the GNU Savannah Git server.

All tagged releases (including legacy stable versions and the most recent development release) are available here:

https://lilypond.org/download/source/

Download the tarball to your ~/src/ directory, or some other appropriate place.

Unpack the tarball with this command:

tar -xzf lilypond-2.25.21.tar.gz

This creates a subdirectory within the current directory called lilypond-2.25.21/. Once unpacked, the source files occupy about 66MB of disk space.

Windows users wanting to look at the source code may have to download and install the free-software 7zip archiver to extract the tarball.

4.4 Configuring make

cd lilypond/
mkdir build/
cd build/

../autogen.sh --noconfigure

../configure --help

../configure

ERROR: Please install required programs:  foo

WARNING: Please consider installing optional programs:  bar

../configure --prefix=$HOME/usr

4.5 Compiling LilyPond

make

make help

make -j3

4.6 Post-compilation options

make install

sudo make install

su -c 'make install'

make doc

make -C Documentation out=www pdf

make LANGS='en' doc

make LANGS='de fr' doc

make info

No rule to make target 'X', needed by 'Y'

cd build/
cd Documentation/
touch ../../Documentation/en/contributor.texi
make out=www out-www/en/contributor.pdf

make -j2 CPU_COUNT=2 doc

make install-doc

make install-info

make WEB_TARGETS=online doc

make WEB_TARGETS="offline online" doc

make help

./autogen.sh   # ignore any warning messages
cp GNUmakefile.in GNUmakefile
make -C scripts && make -C python
nice make LILYPOND_EXTERNAL_BINARY=/path/to/bin/lilypond doc

make out=www WWW-post

make test

4.7 Problems

For help and questions use lilypond-user@gnu.org. Send bug reports to bug-lilypond@gnu.org.

Bugs that are not fault of LilyPond are documented here.

Compiling on macOS

Here are special instructions for compiling under macOS. These instructions assume that dependencies are installed using MacPorts. The instructions have been tested using Mac~OS~X 10.5 (Leopard).

First, install the relevant dependencies using MacPorts.

Next, add the following to your relevant shell initialization files. This is ~/.profile by default. You should create this file if it does not exist.

export PATH=/opt/local/bin:/opt/local/sbin:$PATH
export DYLD_FALLBACK_LIBRARY_PATH=/opt/local/lib:$DYLD_FALLBACK_LIBRARY_PATH

At this point, you should verify that you have the appropriate fonts installed with your Ghostscript installation. Check ls /opt/local/share/ghostscript/fonts for: ’c0590*’ files (.pfb, .pfb and .afm). If you don’t have them, run the following commands to grab them from the Ghostscript SVN server and install them in the appropriate location:

svn export http://svn.ghostscript.com/ghostscript/tags/urw-fonts-1.0.7pre44/
sudo mv urw-fonts-1.0.7pre44/* /opt/local/share/ghostscript/fonts/
rm -rf urw-fonts-1.07pre44

Now run the ./configure script. To avoid complications with automatic font detection, add

--with-fonts-dir=/opt/local/share/ghostscript/fonts

FreeBSD

To use system fonts, dejaview must be installed. With the default port, the fonts are installed in usr/X11R6/lib/X11/fonts/dejavu.

Open the file $LILYPONDBASE/usr/etc/fonts/local.conf and add the following line just after the <fontconfig> line. (Adjust as necessary for your hierarchy.)

<dir>/usr/X11R6/lib/X11/fonts</dir>

International fonts

On macOS, all fonts are installed by default. However, finding all system fonts requires a bit of configuration; see this post on the lilypond-user mailing list.

On Linux, international fonts are installed by different means on every distribution. We cannot list the exact commands or packages that are necessary, as each distribution is different, and the exact package names within each distribution changes. Here are some hints, though:

Red Hat Fedora

    taipeifonts fonts-xorg-truetype ttfonts-ja fonts-arabic \
         ttfonts-zh_CN fonts-ja fonts-hebrew

Debian GNU/Linux

   apt-get install emacs-intl-fonts xfonts-intl-.* \
        fonts-ipafont-gothic  fonts-ipafont-mincho \
        xfonts-bolkhov-75dpi xfonts-cronyx-100dpi xfonts-cronyx-75dpi

Using lilypond python libraries

If you want to use lilypond’s python libraries (either running certain build scripts manually, or using them in other programs), set PYTHONPATH to python/out in your build directory, or …/usr/lib/lilypond/current/python in the installation directory structure.

4.8 Concurrent stable and development versions

It can be useful to have both the stable and the development versions of LilyPond available at once. One way to do this on GNU/Linux is to install the stable version using the precompiled binary, and run the development version from the source tree. After running make all from the top directory of the LilyPond source files, there will be a binary called lilypond in the out directory:

<path to>/lilypond/out/bin/lilypond

This binary can be run without actually doing the make install command. The advantage to this is that you can have all of the latest changes available after pulling from git and running make all, without having to uninstall the old version and reinstall the new.

So, to use the stable version, install it as usual and use the normal commands:

lilypond foobar.ly

To use the development version, create a link to the binary in the source tree by saving the following line in a file somewhere in your $PATH:

exec <path to>/lilypond/out/bin/lilypond "$@"

Save it as Lilypond (with a capital L to distinguish it from the stable lilypond), and make it executable:

chmod +x Lilypond

Then you can invoke the development version this way:

Lilypond foobar.ly

TODO: ADD

- other compilation tricks for developers

4.9 Build system

Version-specific texinfo macros

• made with scripts/build/create-version-itexi.py and
  scripts/build/create-weblinks-itexi.py
• used extensively in the WEBSITE_ONLY_BUILD version of the website (made with website.make, used on lilypond.org)
• not (?) used in the main docs?
• the numbers in VERSION file: MINOR_VERSION should be 1 more than the last release, VERSION_DEVEL should be the last online release. Yes, VERSION_DEVEL is less than VERSION.

5.1 Introduction to documentation work

Our documentation tries to adhere to our Documentation policy. This policy contains a few items which may seem odd. One policy in particular is often questioned by potential contributors: we do not repeat material in the Notation Reference, and instead provide links to the “definitive” presentation of that information. Some people point out, with good reason, that this makes the documentation harder to read. If we repeated certain information in relevant places, readers would be less likely to miss that information.

That reasoning is sound, but we have two counter-arguments. First, the Notation Reference – one of five manuals for users to read – is already over 500 pages long. If we repeated material, we could easily exceed 1000 pages! Second, and much more importantly, LilyPond is an evolving project. New features are added, bugs are fixed, and bugs are discovered and documented. If features are discussed in multiple places, the documentation team must find every instance. Since the manual is so large, it is impossible for one person to have the location of every piece of information memorized, so any attempt to update the documentation will invariably omit a few places. This second concern is not at all theoretical; the documentation used to be plagued with inconsistent information.

If the documentation were targeted for a specific version – say, LilyPond 2.10.5 – and we had unlimited resources to spend on documentation, then we could avoid this second problem. But since LilyPond evolves (and that is a very good thing!), and since we have quite limited resources, this policy remains in place.

A few other policies (such as not permitting the use of tweaks in the main portion of NR 1+2) may also seem counter-intuitive, but they also stem from attempting to find the most effective use of limited documentation help.

Before undertaking any large documentation work, contributors are encouraged to contact the lilypond-devel mailing list.

5.2 \version in documentation files

Every documentation file which includes LilyPond code must begin with a \version statement, since the build procedure explicitly tests for its presence and will not continue otherwise. The \version statement should reference a version of LilyPond consistent with the syntax of the contained code.

Since the \version statement is not valid Texinfo input it must be commented out like this:

@c \version "2.19.1"

So, if you are adding LilyPond code which is not consistent with the current version header, you should

1. run convert-ly on the file using the latest version of LilyPond (which should, if everybody has done proper maintenance, not change anything);
2. add the new code;
3. modify the version number to match the new code.

5.3 Documentation suggestions

Small additions

For additions to the documentation,

1. Tell us where the addition should be placed. Please include both the section number and title (i.e. "LM 2.13 Printing lyrics").
2. Please write exact changes to the text.
3. A formal patch to the source code is not required; we can take care of the technical details.
4. Send the suggestions to the bug-lilypond mailing list as discussed in Contact.
5. Here is an example of a perfect documentation report:

   To: bug-lilypond@gnu.org
   From: helpful-user@example.net
   Subject: doc addition
   
   In LM 2.13 (printing lyrics), above the last line ("More options,
   like..."), please add:
   
   ----
   To add lyrics to a divided part, use blah blah blah.  For example,
   
   \score {
     \notes {blah <<blah>> }
     \lyrics {blah <<blah>> }
     blah blah blah
   }
   ----
   
   In addition, the second sentence of the first paragraph is
   confusing.  Please delete that sentence (it begins "Users
   often...") and replace it with this:
   ----
   To align lyrics with something, do this thing.
   ----
   
   Have a nice day,
   Helpful User
   

Larger contributions

To replace large sections of the documentation, the guidelines are stricter. We cannot remove parts of the current documentation unless we are certain that the new version is an improvement.

1. Ask on the lilypond-devel mailing list if such a rewrite is necessary; somebody else might already be working on this issue!
2. Split your work into small sections; this makes it much easier to compare the new and old documentation.
3. Please prepare a formal git patch.

Contributions that contain examples using overrides

Examples that use overrides, tweaks, customer Scheme functions, etc. are (with very few exceptions) not included in the main text of the manuals; as there would be far too many, equally useful, candidates.

The correct way is to submit your example, with appropriate explanatory text and tags, to the LilyPond Snippet Repository (LSR). Snippets that have the “docs” tag can then be easily added as a selected snippet in the documentation. It will also appear automatically in the Snippets lists. See Introduction to LSR.

Snippets that don’t have the “docs” tag will still be searchable and viewable within the LSR, but will be not be included in the Snippets list or be able to be included as part of the main documentation.

Generally, any new snippets that have the “docs” tag are more carefully checked for syntax and formatting.

Announcing your snippet

Once you have followed these guidelines, please send a message to lilypond-devel with your documentation submissions. Unfortunately there is a strict ‘no top-posting’ check on the mailing list; to avoid this, add:

> I'm not top posting

(you must include the > ) to the top of your documentation addition.

We may edit your suggestion for spelling, grammar, or style, and we may not place the material exactly where you suggested, but if you give us some material to work with, we can improve the manual much faster.

Thanks for your interest!

5.4 Texinfo introduction and usage policy

@node Foo
@unnumberedsubsubsec Foo

@node Foo
@subsection Foo

@subsubsubheading Foo

not:
@node @code{Foo} Bar
@subsection @code{Foo} Bar

but instead:
@node Foo Bar
@subsection @code{Foo} Bar

@menu
* Foo Bar::
@end menu

@node Foo Bar
@subsection Foo: Bar

@menu
* Set and unset
@end menu

@node Set and unset
@subsection @code{\set} and @code{\unset}

@menu
* foo::
* bar::
@end menu

(define (foo x)
  "The @code{\\foo} command..."
  ...)

5.5 Documentation policy

make check-xrefs
make fix-xrefs

5.6 Tips for writing docs

In the NR, I highly recommend focusing on one subsection at a time. For each subsection,

• check the mundane formatting. Are the headings (@predefined, @morerefs, etc.) in the right order?
• add any appropriate index entries.
• check the links in the @morerefs section – links to music glossary, internal references, and other NR sections are the main concern. Check for potential additions.
• move LSR-worthy material into LSR. Add the snippet, delete the material from the .itely file, and add a @lilypondfile command.
• check the examples and descriptions. Do they still work? Do not assume that the existing text is accurate/complete; some of the manual is highly out of date.
• is the material in the @knownissues still accurate?
• can the examples be improved (made more explanatory), or is there any missing info? (feel free to ask specific questions on -user; a couple of people claimed to be interesting in being “consultants” who would help with such questions)

In general, I favor short text explanations with good examples – “an example is worth a thousand words”. When I worked on the docs, I spent about half my time just working on those tiny lilypond examples. Making easily-understandable examples is much harder than it looks.

Tweaks

In general, any \set or \override commands should go in the “select snippets” section, which means that they should go in LSR and not the .itely file. For some cases, the command obviously belongs in the “main text” (i.e., not inside @predefined or @morerefs or whatever) – instrument names are a good example of this.

\set Staff.instrumentName = "foo"

On the other side of this,

\override Score.Hairpin.after-line-breaking = ##t

clearly belongs in LSR.

I’m quite willing to discuss specific cases if you think that a tweaks needs to be in the main text. But items that can go into LSR are easier to maintain, so I’d like to move as much as possible into there.

It would be “nice” if you spent a lot of time crafting nice tweaks for users… but my recommendation is not to do this. There’s a lot of doc work to do without adding examples of tweaks. Tweak examples can easily be added by normal users by adding them to the LSR.

One place where a documentation writer can profitably spend time writing or upgrading tweaks is creating tweaks to deal with known issues. It would be ideal if every significant known issue had a workaround to avoid the difficulty.

See also

Adding and editing snippets.

5.7 Scripts to ease doc work

scripts/auxiliar/doc-section.sh MANUAL SECTION

scripts/auxiliar/doc-section.sh notation pitches

build/tempdocs/pitches/out/pitches.html

scripts/auxiliar/cg-section.sh SECTION

scripts/auxiliar/cg-section.sh doc-work

scripts/auxiliar/node-menuify.sh FILENAME

5.8 Docstrings in scheme

Material in the Internals reference is generated automatically from our source code. Any doc work on Internals therefore requires modifying files in scm/*.scm. Texinfo is allowed in these docstrings.

Most documentation writers never touch these, though. If you want to work on them, please ask for help.

5.9 Translating the documentation

./autogen.sh

./autogen.sh --prefix=$HOME

make ISOLANG=MY-LANGUAGE new-lang

@node Foo bar
@section_command Bar baz

@node Foo bar
@section_command translation of Bar baz

@node Foo bar
@section_command translation of Bar baz

@untranslated

@c KEEP LY

@lilypondfile[<number of fragment options>,texidoc]{filename.ly}

doctitlees = "Spanish title baz"
texidoces = "
Spanish translation blah
"

cp Documentation/en/learning.tely Documentation/LANG/learning.tely
cp Documentation/en/learning/tutorial.itely Documentation/LANG/tutorial.itely

make ISOLANG=MY_LANGUAGE check-translation

make ISOLANG=MY_LANGUAGE check-translation | less -R

make TRANSLATION_FILES=MY_LANGUAGE/manual/foo.itely check-translation

make ISOLANG=MY_LANGUAGE check-translation | grep -n 'diff --git'

make ISOLANG=MY_LANGUAGE NO_COLOR=1 check-translation

make ISOLANG=MY_LANGUAGE update-translation

make TRANSLATION_FILES=MY_LANGUAGE/manual/foo.itely update-translation

make po-update

make ISOLANG=MY_LANGUAGE snippet-update

git rev-list HEAD |head -1

cd Documentation/LANG/texidocs
sed -i -r 's/[0-9a-z]{40}/NEW-COMMITTISH/' *.texidoc

6.1 Introduction to website work

The website is not written directly in HTML; instead it is autogenerated along with the documentation using Texinfo source files. Texinfo is the standard for documentation of GNU software and allows generating output in HTML, PDF, and Info formats, which drastically reduces maintenance effort and ensures that the website content is consistent with the rest of the documentation. This makes the environment for improving the website rather different from common web development.

If you have not contributed to LilyPond before, a good starting point might be incremental changes to the CSS file, to be found at https://lilypond.org/css/lilypond-website.css or in the LilyPond source code at ./Documentation/css/lilypond-website.css.

Large scale structural changes tend to require familiarity with the project in general, a track record in working on LilyPond documentation as well as a prospect of long-term commitment.

The Texinfo source file for generating HTML are to be found in

Documentation/en/web.texi
Documentation/en/web/*.texi

Unless otherwise specified, follow the instructions and policies given in Documentation work. That chapter also contains a quick introduction to Texinfo; consulting an external Texinfo manual should be not necessary.

Exceptions to the documentation policies

• Sectioning: the website only uses chapters and sections; no subsections or subsubsections.
• @ref{}s to other manuals (@rnotation, @rlearning, etc): you can’t link to any pieces of automatically generated documentation, like the IR or certain NR appendices.
• The bibliography in Community->Publications is generated automatically from .bib files; formatting is done automatically by texi-web.bst.
• …
• For anything not listed here, just follow the same style as the existing website texinfo files.

6.2 Uploading website

Overall idea

The website is generated by converting the Documentation/*/web.texi files to HTML, and reorganizing the resulting files into out/website-root/. This is controlled from toplevel GNUmakefile and Documentation/GNUmakefile.

To build the website, run make website. This leaves the website in out/website-root/.

The website is deployed onto lilypond.org in the following steps:

• Run the manual job to build the website, either for the merge request you want to deploy or for the latest pipeline on master at https://gitlab.com/lilypond/lilypond/-/pipelines, by clicking the play button.
• This runs make website and stores the result in a website.zip artifact.
• On lilypond.org, the downloader https://gitlab.com/lilypond/infrastructure/-/blob/master/website/main.go is run every 2 hours, from a systemd timed job. If a newer website.zip is found, it is unpacked into the website directory on lilypond.org.

6.3 Debugging website and docs locally

• Install Apache (you can use version 2, but keep in mind that the server hosting lilypond.org runs version 1.3). These instructions assume that you also enable mod_userdir, and use $HOME/public_html as DocumentRoot (i.e., the root directory of the web server).
• Build the online docs and website:

  make WEB_TARGETS="offline online" doc
  make website
  

  This will make all the language variants of the website. To save a little time, just the English version can be made with the command make WEB_LANGS='' website or the English and (for example) the French with make WEB_LANGS='fr' website.

• Choose the web directory where to copy the built stuff. If you already have other web projects in your DocumentRoot and don’t need to test the .htaccess file, you can copy to ~/public_html/lilypond.org. Otherwise you’d better copy to ~/public_html. It’s highly recommended to have your build dir and web dir on the same partition.
• Add the directory for the online documentation:

  mkdir -p ~/public_html/doc/v2.19/
  

  You may want to add also the stable documentation in ~/public_html/doc/v2.18/, extracting the contents of the html directory present in the tarball available in All. Just in case you want to test the redirects to the stable documentation.

• Copy the files with rsync:

  rsync -av --delete out-website/website ~/public_html/
  cp out-website/.htaccess ~/public_html
  rsync -av --delete out-www/online-root/ ~/public_html/doc/v2.19/
  

6.4 Translating the website

As it has much more audience, the website should be translated before the documentation; see Translating the documentation.

In addition to the normal documentation translation practices, there are a few additional things to note:

• Build the website with:

  make website
  

• Some of the translation infrastructure is defined in python files; you must look at the ### translation data sections in:

  scripts/build/create-weblinks-itexi.py
  scripts/build/website_post.py
  

  Do not submit a patch to add your language to this file unless make website completes with fewer than 5 warnings.

• Links to manuals are done with macros like @manualDevelLearningSplit. To get translated links, you must change that to @manualDevelLearningSplit-es (for es/Spanish translations, for example).

7.1 Introduction to LSR

The LilyPond Snippet Repository (LSR) is a collection of LilyPond examples. A subset of these examples are automatically imported into the documentation, making it easy for users to contribute to the documentation without learning Git and Texinfo.

7.2 Adding and editing snippets

General guidelines

When you create (or find!) a nice snippet, and if it is supported by the LilyPond version running on the LSR, please add it to the LSR. Go to LSR and log in – if you haven’t already, create an account. Follow the instructions on the website. These instructions also explain how to modify existing snippets.

If you think a snippet is particularly informative and should be included in the documentation, tag it with ‘docs’ and one or more other categories, or ask on the development list for somebody who has editing permissions to do it.

Please make sure that the LilyPond code follows our formatting guidelines, see LilyPond formatting.

If a new snippet created for documentation purposes compiles with the LilyPond version currently on LSR, it should be added to the LSR, and a reference to the snippet should be added to the documentation. Please ask a documentation editor to add a reference to it in an appropriate place in the docs. (Note – it should appear in the ‘snippets’ document automatically, once it has been imported into git and built. See LSR to Git.)

If a new snippet uses new features that are not available in the current LSR version of LilyPond, it should be added to directory Documentation/snippets/new/, and a reference should be added to the manual.

Snippets created or updated in Documentation/snippets/new/ must be adjusted and copied to directory Documentation/snippets/. This should be done by invoking the makelsr.pl script – after you have compiled LilyPond. Assuming that your LilyPond build is in the top-level subdirectory build/, a proper invocation is

cd /your/lilypond/git/top/dir
scripts/auxiliar/makelsr.pl --new

See The makelsr.pl script, for more details.

Be sure that ‘make doc’ runs successfully before submitting a patch, to prevent breaking compilation (see Generating documentation).

Formatting snippets in Documentation/snippets/new/

When adding a file to this directory, please start the file with the following template …

\version "2.xx.yy"

\header {
  % Use existing LSR tags other than 'docs'; the names of the
  % `*.snippet-list` files in `Documentation/snippets/` give the
  % tags currently used.
  lsrtags = "rhythms, expressive-marks"

  % The documentation string must use Texinfo syntax.  In
  % addition, `\` and `"` must be written as `\\` and `\"`,
  % respectively.
  texidoc = "
This snippet demonstrates @code{\\foo} ...
"

  % The snippet title string must be formatted similar to
  % `texidoc`.
  doctitle = "Snippet title"
}

<LilyPond code starts here>

… and name the file snippet-title.ly.

It is important that the version number you use at the top of the example is the minimum LilyPond version that the file compiles with: for example, if the LSR is currently at 2.22.2, your example requires 2.23.4, and the current development version of LilyPond is 2.25.21, use \version "2.23.4".

Particular attention is also necessary for the lsrtags and doctitle fields: the tags must match tags used in the documentation, and the doctitle must match the file name (makelsr.pl shows a helpful error message if it doesn’t).

The order of \version, \header, and the LilyPond code must be as shown above, otherwise makelsr.pl aborts with an error. The same holds for the order of the lsrtags, texidoc, and doctitle fields within \header.

7.3 Approving snippets

The main task of LSR editors is approving snippets. To find a list of unapproved snippets, log into LSR and select “No” from the drop-down menu to the right of the word “Approved” at the bottom of the interface, then click “Enable filter”.

Here is a checklist of the necessary tasks.

1. Does the snippet make sense and does it what the author claims that it does? If you think the snippet is suited to be included into the LilyPond documentation, add the ‘docs’ tag and at least one other tag.
2. If the snippet is tagged with ‘docs’, check whether it matches our formatting guidelines, see LilyPond formatting.

   Also, snippets tagged with ‘docs’ should not be explaining (or replicating) existing material in the documentation. They should not refer to the documentation; the documentation should rather refer to them.

3. If the snippet uses Scheme code, check that everything looks good and there are no security risks.

   Note: Somebody could add code like #'(system "rm -rf /") to a snippet, which would cause catastrophic results if executed! Take this step VERY SERIOUSLY.

4. If all is well, check the box labeled “approved” and save the snippet.

7.4 The makelsr.pl script

As you might have guessed already, makelsr.pl is a Perl script. Obviously, you need Perl to execute it, which you should now install in case it isn’t already available on your system.

There is a dependency on the Pandoc program, which the script uses to convert LSR’s snippet documentation strings (which are formatted in HTML) to Texinfo. This must be installed, too.

Furthermore, makelsr.pl needs a few additional modules that are not Perl core modules (tested with Perl version 5.36):

• File::Which
• IPC::Run3
• MySQL::Dump::Parser::XS
• Pandoc
• Parallel::ForkManager

Either install missing modules with your package manager (if available) or use the cpanm command.⁵

A typical call might me

cpanm --sudo Parallel::ForkManager

to download, compile, and install module ‘Parallel::ForkManager’.⁶

Finally, it needs to find the convert-ly script from the current LilyPond development build.

By default, executing makelsr.pl performs the following actions.

• Download a current MySQL dump of the LSR database (the dump is regenerated once a day).
• Delete all snippet and snippet list files in directory Documentation/snippets/ (but not in Documentation/snippets/new/).
• Extract all snippets from the LSR database that have the ‘docs’ tag set, convert their documentation parts from HTML to Texinfo with the pandoc program, run the script convert-ly to update their LilyPond code parts to current syntax, and store them in Documentation/snippets/.
• Create snippet list files named winds.snippet-list or connecting-notes.snippet-list that list the snippets grouped by tags assigned in the database. These files are used to structure LilyPond’s ‘snippets’ documentation.
• Convert all snippet files in Documentation/snippets/new/ with convert-ly and output them to Documentation/snippets/, possibly overwriting existing files.

This flow of actions can be adjusted; say ‘scripts/auxiliar/makelsr.pl --help’ to get a detailed description of the provided command-line options and used environment variables.

7.5 LSR to Git

Introduction

Snippets used in the documentation are in $LILYPOND_GIT/Documentation/snippets/. This directory contains a set of all snippets in the LSR that are tagged with ‘docs’. An import is done with the makelsr.pl script, which downloads a complete database dump of the LSR to update this directory.

Snippets that are too new to be run on the LSR (which uses a stable LilyPond version) are put into $LILYPOND_GIT/Documentation/snippets/new/. Once the LSR gets upgraded to a LilyPond version that can actually compile them, they are transferred to the LSR and deleted from snippets/new/.

‘Git’ is the shorthand name for LilyPond’s Git repository, which contains all the development code. For further information on setting this up, see Working with source code. An alternative to setting up a Git repository for people wanting to do LSR work is to get the source code from https://lilypond.org/development.html. However, we don’t recommend this since it doesn’t allow easy submission of patches as merge requests.

Importing the LSR to Git

1. Make sure that the convert-ly script is a bleeding edge version – the latest development release, or even better, freshly compiled from Git master, with the environment variable LILYPOND_BUILD_DIR correctly set up (see Environment variables) in case your build directory isn’t $LILYPOND_GIT/build/.
2. Check the other prerequisites necessary for executing the makelsr.pl script (see The makelsr.pl script).
3. If you are using a git repository, create and check out a branch, for example

   git checkout -b lsr-import
   

4. From the top source directory, execute

   scripts/auxiliar/makelsr.pl
   

   Say ‘scripts/auxiliar/makelsr.pl --help’ to find out how to modify this call; for example, command-line option --dump file makes the script use a locally stored dump file.

5. Carefully check the output of the script for warnings and errors, then carefully check the file differences in the git repository. ‘git diff’ is your friend.
6. Rebuild the documentation. If some snippets from Documentation/snippets/ cause the documentation compilation to fail, try the following steps to fix it.
   • Look up the snippet file name foo.ly in the error output or log file, then fix the file Documentation/snippets/foo.ly to make the documentation build successfully.
   • Determine where it comes from by looking at its first two lines, e.g., run

     head -2 Documentation/snippets/foo.ly
     

   • If the snippet comes from the LSR, also apply the fix to the snippet in the LSR and send a notification email to an LSR editor with CC to the development list, see Adding and editing snippets.

     Note that the failure may sometimes not be caused by the snippet in the LSR but by LilyPond syntax changes that convert-ly can’t handle automatically. Such files must be added to the new/ directory.

   • If the snippet comes from Documentation/snippets/new/, apply the fix in Documentation/snippets/new/foo.ly and run makelsr.pl as follows:

     scripts/auxiliar/makelsr.pl --new
     

     Then, inspect Documentation/snippets/foo.ly to check that the fix has been well propagated.

   • If the build failure was caused by a translation string, you may have to fix some Documentation/lang/texidocs/foo.texidoc files instead.
7. When you are done, commit your changes to your Git branch and create a merge request (see Lifecycle of a merge request).

7.6 Renaming a snippet

Due to the potential duality of snippets (i.e., they may exist both in the LSR database and in Documentation/snippets/new/), this process is a bit more involved than we might like.

1. Send an email to an LSR editor, requesting the renaming.
2. The LSR editor does the renaming (or debates the topic with you), then warns the LSR-to-git person (wanted: better title) about the renaming.
3. LSR-to-git person does his normal job, but then also renames any copies of the snippets in Documentation/snippets/new/, and any instances of the snippet name in the documentation.

   git grep is highly recommended for this task.

7.7 Updating the LSR to a new version

To update the LSR, perform the following steps:

1. Start by emailing the LSR maintainer, Sebastiano, and liaising with him to ensure that updating the snippets is synchronised with updating the binary running the LSR.
2. Download the latest snippet tarball from https://lsr.di.unimi.it/download/ and extract it. The relevant files can be found in the all subdirectory. Make sure your shell is using an English language version, for example LANG=en_US, then run convert-ly on all the files. Use the command-line option --to=version to ensure the snippets are updated to the correct stable version.
3. Make sure that you are using convert-ly from the latest available release to gain best advantage from the latest converting-rules-updates.

   For example:

   • LSR-version: 2.12.2
   • intended LSR-update to 2.14.2
   • latest release 2.15.30

   Use convert-ly from 2.15.30 and the following terminal command for all files:

   convert-ly -e -t2.14.2 *.ly
   

4. There might be no conversion rule for some old commands. To make an initial check for possible problems you can run the script at the end of this list on a copy of the all subdirectory.
5. Copy relevant snippets (i.e., snippets whose version is equal to or less than the new version of LilyPond running on the LSR) from Documentation/snippets/new/ into the set of files to be used to make the tarball. Make sure you only choose snippets which are already present in the LSR, since the LSR software isn’t able to create new snippets this way. If you don’t have a Git repository for LilyPond, you’ll find these snippets in the source-tarball on https://lilypond.org/development.html. Don’t rename any files at this stage.
6. Verify that all files compile with the new version of LilyPond, ideally without any warnings or errors. To ease the process, you may use the shell script that appears after this list.

   Due to the workload involved, we do not require that you verify that all snippets produce the expected output. If you happen to notice any such snippets and can fix them, great; but as long as all snippets compile, don’t delay this step due to some weird output. If a snippet is not compiling, update it manually. If it’s not possible, delete it for now.

7. Remove all headers and version-statements from the files. Phil Holmes has a python script that will do this and which needs testing. Please ask him for a copy if you wish to do this.
8. Create a tarball and send it back to Sebastiano. Don’t forget to tell him about any deletions.
9. Use the LSR web interface to change any descriptions you want to. Changing the titles of snippets is a bit fraught, since this also changes the file names. Only do this as a last resort.
10. Use the LSR web interface to add the other snippets from Documentation/snippets/new/ which compile with the new LilyPond version of the LSR. Ensure that they are correctly tagged, including the tag docs and that they are approved.
11. When LSR has been updated, wait a day for the tarball to update, then download another snippet tarball. Verify that the relevant snippets from Documentation/snippets/new/ are now included, then delete those snippets from Documentation/snippets/new/.
12. Commit all the changes. Don’t forget to add new files to the git repository with git add. Run make, make doc and make test to ensure the changes don’t break the build. Any snippets that have had their file name changed or have been deleted could break the build, and these will need correcting step by step.

Below is a shell script to run LilyPond on all .ly files in a directory. If the script is run with a -s parameter, it runs silently except for reporting failed files. If run with -c it also runs convert-ly prior to running LilyPond.

#!/bin/bash

while getopts sc opt; do
    case $opt in
        s)
            silent=true
            ;;
        c)
            convert=true
            ;;
    esac
done
param=$ if [ $silent ]; then
    param=${param:3}
fi
if [ $convert ]; then
    param=${param:3}
fi
filter=${param:-"*.ly"}

for LILYFILE in $filter
do
    STEM=$(basename "$LILYFILE" .ly)
    if [ $convert ]; then
        if [ $silent ]; then
            $LILYPOND_BUILD_DIR/out/bin/convert-ly -e "$LILYFILE" >& "$STEM".con.txt
        else
            $LILYPOND_BUILD_DIR/out/bin/convert-ly -e "$LILYFILE"
        fi
    fi
    if [ ! $silent ]; then
        echo "running $LILYFILE..."
    fi
    $LILYPOND_BUILD_DIR/out/bin/lilypond --format=png "$LILYFILE" >& "$STEM".txt
    RetVal=$?
    if [ $RetVal -gt 0 ]; then
       echo "$LILYFILE failed"
    fi
done

Output from LilyPond is in filename.txt and convert-ly in filename.con.txt.

8.1 Introduction to issues

The term ‘issues’ refers not just to software bugs but also includes feature requests, documentation additions and corrections as well as any other general code ‘TODOs’ that need to be kept track of. Tasks revolving around issues include:

• Monitoring the LilyPond Bugs mailing list looking for any issues reported by other users ensuring that they are accurate and contain enough information for the developers to work with, preferably with Tiny examples and if applicable, screenshots.
• Adding new issues to the issue tracker or updating existing issues with new information.

To start working on bug triage, follow these steps:

1. Read every section of the Issues chapter in this guide.
2. Subscribe your email account to bug-lilypond. See https://lists.gnu.org/mailman/listinfo/bug-lilypond.
3. Create your own GitLab login (required to manage issues):
   • Go to https://gitlab.com/users/sign_in.
   • Click on the ‘Register’ tab to create a new account.
   • Fill in your details as required and click the Register button to complete the registration.
4. Go to https://gitlab.com/lilypond and ‘Request access’ to the group. Additionally send your GitLab username (not your email address) to bug-lilypond@gnu.org, asking to be given appropriate permissions to manage issues.
5. Configure your email client to use some kind of sorting and filtering as this will significantly reduce and simplify your workload. Suggested email folder names are mentioned below to work when sorted alphabetically.

   Any email sent To: or CC: to bug-lilypond should be configured to go into a bug-current folder.

8.2 Triaging bugs

Emails to you personally

Sometimes a confused user will send a bug report (or an update to a report) to you personally. If that happens, please forward such emails to the bug-lilypond list.

Emails to bug-answers

Some of these emails will be comments on issues that you added to the tracker.

• If they are asking for more information, give the additional information.
• If the email says that the issue was classified in some other manner, read the rationale given and take that into account for the next issue you add.
• Otherwise, move them to your bug-ignore folder.

Some of these emails will be discussions about Bug Squad work; read those.

Emails to bug-current

Dealing with these emails is your main task. Your job is to get rid of these emails in the first method which is applicable:

1. If the email has already been handled by a Bug Squad member (i.e. check to see who else has replied to it), delete it.
2. If the email is a question about how to use LilyPond, reply with this response:

   For questions about how to use LilyPond, please read our
   documentation available from:
     https://lilypond.org/manuals.html
   or ask the lilypond-user mailing list.
   

3. If the email mentions “the latest git”, or any version number that has not yet been officially released, forward it to lilypond-devel.
4. If a bug report is not in the form of a Tiny example, direct the user to resubmit the report with this response:

   I'm sorry, but due to our limited resources for handling bugs, we
   can only accept reports in the form of Tiny examples.  Please see
   step 2 in our bug reporting guidelines:
     https://lilypond.org/bug-reports.html
   

5. If anything is unclear, ask the user for more information.

   How does the graphical output differ from what the user expected? What version of lilypond was used (if not given) and operating system (if this is a suspected cause of the problem)? In short, if you cannot understand what the problem is, ask the user to explain more. It is the user’s responsibility to explain the problem, not your responsibility to understand it.

6. If the behavior is expected, the user should be told to read the documentation:

   I believe that this is the expected behavior -- please read our
   documentation about this topic.  If you think that it really is a
   mistake, please explain in more detail.  If you think that the
   docs are unclear, please suggest an improvement as described by
   “Simple tasks -- Documentation” on:
     https://lilypond.org/help-us.html
   

7. If the issue already exists in the tracker, send an email to that effect:

   This issue has already been reported; you can follow the
   discussion and be notified about fixes here:
   

   (copy+paste the GitLab issue URL)

8. Accept the report as described in Adding issues to the tracker.

All emails should be CC’d to the bug-lilypond list so that other Bug Squad members know that you have processed the email.

8.3 Issue classification

We have several labels:

• Critical: normally a regression against the current stable version or the previous stable version. Alternatively, a regression against a fix developed for the current version. This does not apply where the “regression” occurred because a feature was removed deliberately – this is not a bug.

  Currently, only Critical items will block a stable release.

• Maintainability: hinders future development.
• Crash: any input which produces a crash.
• Ugly: overlapping or other ugly notation in graphical output.
• Defect: a problem in the core program. (the lilypond binary, scm files, fonts, etc).
• Documentation: inaccurate, missing, confusing, or desired additional info. Must be fixable by editing a texinfo, ly, or scm file.
• Build: problem or desired features in the build system. This includes the makefiles and python scripts.
• Scripts: problem or desired feature in the non-build-system scripts. Mostly used for convert-ly, lilypond-book, etc.
• Enhancement: a feature request for the core program. The distinction between enhancement and defect isn’t extremely clear; when in doubt, mark it as enhancement.
• Other: anything else.
• Regression: it used to work intentionally in the current stable release or the previous stable release. If the earlier output was accidental (i.e., we didn’t try to stop a collision, but it just so happened that two grobs didn’t collide), then breaking it does not count as a regression.

  To help decide whether the change is a regression, please adopt the following process:

  1. Are you certain the change is OK? If so, do nothing.
  2. Are you certain that the change is bad? Add it to the tracker as a regression.
  3. If you’re not certain either way, add it to the tracker as a regression but be aware that it may be recategorised or marked invalid.

  In particular, anything that breaks a regression test is a regression.

• Frog: the fix is believed to be suitable for a new contributor (does not require a great deal of knowledge about LilyPond). The issue should also have an estimated time in a comment.
• Bounty: somebody is willing to pay for the fix. Only add this tag if somebody has offered an exact figure in US dollars or euros.
• Warning: graphical output is fine, but lilypond prints a false/misleading warning message. Alternately, a warning should be printed (such as a bar line error), but was not. Also applies to warnings when compiling the source code or generating documentation.
• Performance: performance issue.

In addition, the following labels may be used when closing an issue:

• Invalid: issue should not have been added in the current state.
• Duplicate: issue already exists in the tracker.
• Shelved: issue won’t fix and was abandoned.

Assign an issue to yourself to indicate that you are currently working on it.

8.4 Adding issues to the tracker

1. Check if the issue falls into any previous category given on the relevant checklists in Triaging bugs. If in doubt, add a new issue for a report. We would prefer to have some incorrectly-added issues rather than lose information that should have been added.
2. Add the issue and classify it according to the guidelines in Issue classification. In particular, the item should have Status and type labels.
3. Include output. Usually, the problem can be demonstrated in an image created using lilypond -dcrop bug.ly, which generates bug.cropped.png. However, for spacing bugs, this image may not show the problem; attach the full PDF produced by a normal lilypond invocation in this case.
4. After adding the issue, please send a response email to the same group(s) that the initial patch was sent to. If the initial email was sent to multiple mailing lists (such as both user and bugs), then reply to all those mailing lists as well. The email should contain a link to the issue you just added.

If patches are sent to the bug list, please submit them via GitLab (or help the author to do so). Alternatively, if discussion is needed, forward the patch to lilypond-devel.

9.1 Introduction to regression tests

LilyPond has a complete suite of regression tests that are used to ensure that changes to the code do not break existing behavior. These regression tests comprise small LilyPond snippets that test the functionality of each part of LilyPond.

Regression tests are added when new functionality is added to LilyPond or when bugs are fixed.

The regression tests are compiled using special make targets. There are three primary uses for the regression tests. First, successful completion of the regression tests means that LilyPond has been properly built. Second, the output of the regression tests can be manually checked to ensure that the graphical output matches the description of the intended output. Third, the regression test output from two different versions of LilyPond can be automatically compared to identify any differences. These differences should then be manually checked to ensure that the differences are intended.

Regression tests (“regtests”) are available in precompiled form as part of the documentation. Regtests can also be compiled on any machine that has a properly configured LilyPond build system.

9.2 Precompiled regression tests

Regression test output

As part of the release process, the regression tests are run for every LilyPond release. Full regression test output is available for every stable version and the most recent development version.

Regression test output is available in HTML and PDF format. Links to the regression test output are available at the developer’s resources page for the version of interest.

The latest stable version of the regtests is found at:

https://lilypond.org/doc/stable/input/regression/collated-files.html

The latest development version of the regtests is found at:

https://lilypond.org/doc/latest/input/regression/collated-files.html

9.3 Compiling regression tests

Developers may wish to see the output of the complete regression test suite for the current version of the source repository between releases. Current source code is available; see Working with source code.

For regression testing ../configure should be run with the --disable-optimising option. Then you will need to build the LilyPond binary; see Compiling LilyPond.

Uninstalling the previous LilyPond version is not necessary, nor is running make install, since the tests will automatically be compiled with the LilyPond binary you have just built in your source directory.

From this point, the regtests are compiled with:

make test

If you have a multi-core machine you may want to use the -j option and CPU_COUNT variable, as described in Saving time with CPU_COUNT. For a quad-core processor the complete command would be:

make -j5 CPU_COUNT=5 test

The regtest output will then be available in input/regression/out-test. input/regression/out-test/collated-examples.html contains a listing of all the regression tests that were run, but none of the images are included. Individual images are also available in this directory.

The primary use of ‘make test’ is to verify that the regression tests all run without error. The regression test page that is part of the documentation is created only when the documentation is built, as described in Generating documentation. Note that building the documentation requires more installed components than building the source code, as described in Requirements for building documentation.

9.4 Regtest comparison

Before modified code is committed to master, a regression test comparison must be completed to ensure that the changes have not caused problems with previously working code. The comparison is made automatically upon compiling the regression test suite twice.

1. Before making changes to the code, establish a baseline for the comparison by checking out the current git master, going to the $LILYPOND_GIT/build/ directory and running:

   make clean # whenever any files in mf/ are modified
   make test-baseline
   

2. Make your changes, or apply the patch(es) to consider.
3. Check for unintentional changes to the regtests:

   make check
   

   After this has finished, a regression test comparison will be available (relative to the current build/ directory) at:

   out/test-results/index.html
   

   For each regression test that differs between the baseline and the changed code, a regression test entry will be displayed. Ideally, the only changes would be the changes that you were working on. If regressions are introduced, they must be fixed before committing the code.

4. If you are happy with the results, then skip to the final step.

   If you want to continue programming, then make any additional code changes, and continue.

5. Finally, you should verify that make doc completes successfully.

9.5 Pixel-based regtest comparison

As an alternative to the make test method for regtest checking (which relies upon .signature files created by a LilyPond run and which describe the placing of grobs) there is a script which compares the output of two LilyPond versions pixel-by-pixel. To use this, start by checking out the version of LilyPond you want to use as a baseline, and run make. Then, do the following:

cd $LILYPOND_GIT/scripts/auxiliar/
./make-regtest-pngs.sh -j9 -o

The -j9 option tells the script to use 9 CPUs to create the images - change this to your own CPU count+1. -o means this is the "old" version. This will create images of all the regtests in

$LILYPOND_BUILD_DIR/out-png-check/old-regtest-results/

Now checkout the version you want to compare with the baseline. Run make again to recreate the LilyPond binary. Then, do the following:

cd $LILYPOND_GIT/scripts/auxiliar/
./make-regtest-pngs.sh -j9 -n

The -n option tells the script to make a "new" version of the images. They are created in

$LILYPOND_BUILD_DIR/out-png-check/new-regtest-results/

Once the new images have been created, the script compares the old images with the new ones pixel-by-pixel and prints a list of the different images to the terminal, together with a count of how many differences were found. The results of the checks are in

$LILYPOND_BUILD_DIR/out-png-check/regtest-diffs/

To check for differences, browse that directory with an image viewer. Differences are shown in red. Be aware that some images with complex fonts or spacing annotations always display a few minor differences. These can safely be ignored.

9.6 Finding the cause of a regression

Git has special functionality to help tracking down the exact commit which causes a problem. See the git manual page for git bisect. This is a job that non-programmers can do, although it requires familiarity with git, ability to compile LilyPond, and generally a fair amount of technical knowledge. A brief summary is given below, but you may need to consult other documentation for in-depth explanations.

Even if you are not familiar with git or are not able to compile LilyPond you can still help to narrow down the cause of a regression simply by downloading the binary releases of different LilyPond versions and testing them for the regression. Knowing which version of LilyPond first exhibited the regression is helpful to a developer as it shortens the git bisect procedure.

Once a problematic commit is identified, the programmers’ job is much easier. In fact, for most regression bugs, the majority of the time is spent simply finding the problematic commit.

More information is in Regression tests.

git bisect setup

We need to set up the bisect for each problem we want to investigate.

Suppose we have an input file which compiled in version 2.13.32, but fails in version 2.13.38 and above.

1. Begin the process:

   git bisect start
   

2. Give it the earliest known bad tag:

   git bisect bad release/2.13.38-1
   

   (you can see tags with: git tag )

3. Give it the latest known good tag:

   git bisect good release/2.13.32-1
   

   You should now see something like:

   Bisecting: 195 revisions left to test after this (roughly 8 steps)
   [b17e2f3d7a5853a30f7d5a3cdc6b5079e77a3d2a] Web: Announcement
   update for the new “LilyPond Report”.
   

git bisect actual

1. Compile the source:

   make
   

2. Test your input file:

   out/bin/lilypond test.ly
   

3. Test results?
   • Does it crash, or is the output bad? If so:

     git bisect bad
     

   • Does your input file produce good output? If so:

     git bisect good
     

4. Once the exact problem commit has been identified, git will inform you with a message like:

   6d28aebbaaab1be9961a00bf15a1ef93acb91e30 is the first bad commit
   %%% ... blah blah blah ...
   

   If there is still a range of commits, then git will automatically select a new version for you to test. Go to step #1.

Recommendation: use two terminal windows

• One window is open to the build/ directory, and alternates between these commands:

  make
  out/bin/lilypond test.ly
  

• One window is open to the top source directory, and alternates between these commands:

  git bisect good
  git bisect bad
  

9.7 MusicXML tests

LilyPond comes with a complete set of regtests for the MusicXML language. Originally developed to test ‘musicxml2ly’, these regression tests can be used to test any MusicXML implementation.

The MusicXML regression tests are found at input/regression/musicxml/.

The output resulting from running these tests through ‘musicxml2ly’ followed by ‘lilypond’ is available in the LilyPond documentation:

https://lilypond.org/doc/latest/input/regression/musicxml/collated-files

10.1 Overview of LilyPond architecture

LilyPond processes the input file into graphical and musical output in a number of stages. This process, along with the types of routines that accomplish the various stages of the process, is described in this section. A more complete description of the LilyPond architecture and internal program execution is found in Erik Sandberg’s master’s thesis.

The first stage of LilyPond processing is parsing. In the parsing process, music expressions in LilyPond input format are converted to music expressions in Scheme format. In Scheme format, a music expression is a list in tree form, with nodes that indicate the relationships between various music events. The LilyPond parser is written in Bison.

The second stage of LilyPond processing is iterating. Iterating assigns each music event to a context, which is the environment in which the music will be finally engraved. The context is responsible for all further processing of the music. It is during the iteration stage that contexts are created as necessary to ensure that every note has a Voice type context (e.g. Voice, TabVoice, DrumVoice, CueVoice, MensuralVoice, VaticanaVoice, GregorianTranscriptionVoice), that the Voice type contexts exist in appropriate Staff type contexts, and that parallel Staff type contexts exist in StaffGroup type contexts. In addition, during the iteration stage each music event is assigned a moment, or a time in the music when the event begins.

Each type of music event has an associated iterator. Iterators are defined in *-iterator.cc. During iteration, an event’s iterator is called to deliver that music event to the appropriate context(s).

The final stage of LilyPond processing is translation. During translation, music events are prepared for graphical or midi output. The translation step is accomplished by the polymorphic base class Translator through its two derived classes: Engraver (for graphical output) and Performer (for midi output).

Translators are defined in C++ files named *-engraver.cc and *-performer.cc. Much of the work of translating is handled by Scheme functions, which is one of the keys to LilyPond’s exceptional flexibility.

10.2 LilyPond programming languages

Programming in LilyPond is done in a variety of programming languages. Each language is used for a specific purpose or purposes. This section describes the languages used and provides links to reference manuals and tutorials for the relevant language.

C++

The core functionality of LilyPond is implemented in C++.

C++ is so ubiquitous that it is difficult to identify either a reference manual or a tutorial. Programmers unfamiliar with C++ will need to spend some time to learn the language before attempting to modify the C++ code.

The C++ code calls Scheme/Guile through the Guile interface, which is documented in the Guile Reference Manual.

Flex

The LilyPond lexer is implemented in Flex, an implementation of the Unix lex lexical analyser generator. Resources for Flex can be found here.

GNU Bison

The LilyPond parser is implemented in Bison, a GNU parser generator. The Bison homepage is found at gnu.org. The manual (which includes both a reference and tutorial) is available in a variety of formats.

GNU Make

GNU Make is used to control the compiling process and to build the documentation and the website. GNU Make documentation is available at the GNU website.

Guile or Scheme

Guile is the dialect of Scheme that is used as LilyPond’s extension language. Many extensions to LilyPond are written entirely in Guile. The Guile Reference Manual is available online.

Structure and Interpretation of Computer Programs, a popular textbook used to teach programming in Scheme is available in its entirety online.

An introduction to Guile/Scheme as used in LilyPond can be found in the Scheme tutorial.

MetaFont

MetaFont is used to create the music fonts used by LilyPond. A MetaFont tutorial is available at the METAFONT tutorial page.

PostScript

PostScript is used to generate graphical output. A brief PostScript tutorial is available online. The PostScript Language Reference is available online in PDF format.

Python

Python is used for XML2ly and is used for building the documentation and the website.

Python documentation is available at python.org.

Scalable Vector Graphics (SVG)

Scalable Vector Graphics (SVG) is an XML-based markup language used to generate graphical output. A brief SVG tutorial is available online through W3 Schools. The World Wide Web Consortium’s SVG 1.2 Recommendation is available online in PDF format.

10.3 Programming without compiling

Much of the development work in LilyPond takes place by changing *.ly or *.scm files. These changes can be made without compiling LilyPond. Such changes are described in this section.

Modifying distribution files

Much of LilyPond is written in Scheme or LilyPond input files. These files are interpreted when the program is run, rather than being compiled when the program is built, and are present in all LilyPond distributions. You will find .ly files in the ly/ directory and the Scheme files in the scm/ directory. Both Scheme files and .ly files can be modified and saved with any text editor. It’s probably wise to make a backup copy of your files before you modify them, although you can reinstall if the files become corrupted.

Once you’ve modified the files, you can test the changes just by running LilyPond on some input file. It’s a good idea to create a file that demonstrates the feature you’re trying to add. This file will eventually become a regression test and will be part of the LilyPond distribution.

Desired file formatting

Files that are part of the LilyPond distribution have Unix-style line endings (LF), rather than DOS (CR+LF) or Mac~OS~9 and earlier (CR). Make sure you use the necessary tools to ensure that Unix-style line endings are preserved in the patches you create.

Tab characters should not be included in files for distribution. All indentation should be done with spaces. Most editors have settings to allow the setting of tab stops and ensuring that no tab characters are included in the file.

Scheme files and LilyPond files should be written according to standard style guidelines. Scheme file guidelines can be found at http://community.schemewiki.org/?scheme-style. Following these guidelines will make your code easier to read. Both you and others that work on your code will be glad you followed these guidelines.

For LilyPond files, you should follow the guidelines for LilyPond snippets in the documentation. You can find these guidelines at Texinfo introduction and usage policy.

10.4 Finding functions

When making changes or fixing bugs in LilyPond, one of the initial challenges is finding out where in the code tree the functions to be modified live. With nearly 3000 files in the source tree, trial-and-error searching is generally ineffective. This section describes a process for finding interesting code.

Using the ROADMAP

The file ROADMAP is located in the main directory of the lilypond source. ROADMAP lists all of the directories in the LilyPond source tree, along with a brief description of the kind of files found in each directory. This can be a very helpful tool for deciding which directories to search when looking for a function.

Using grep to search

Having identified a likely subdirectory to search, the grep utility can be used to search for a function name. The format of the grep command is

grep -i functionName subdirectory/*

This command will search all the contents of the directory subdirectory/ and display every line in any of the files that contains functionName. The -i option makes grep ignore case – this can be very useful if you are not yet familiar with our capitalization conventions.

The most likely directories to grep for function names are scm/ for scheme files, ly/ for lilypond input (*.ly) files, and lily/ for C++ files.

Using git grep to search

If you have used git to obtain the source, you have access to a powerful tool to search for functions. The command:

git grep functionName

will search through all of the files that are present in the git repository looking for functionName. It also presents the results of the search using less, so the results are displayed one page at a time.

Using TAGS support

Many programs, including Emacs, ex, vi, and less, provide the ability to jump directly to the definition of an identifier based on precomputed cross-reference data. This data is usually contained in files named TAGS, for Emacs, or tags, for vi and other programs.

To generate these cross-reference data files the source code must be installed, but it is not necessary to compile LilyPond. Follow the instructions found in Getting the source code through ‘Checking build dependencies’. Once the configure command has run successfully, invoke the following command in the build directory.

make TAGS

This will create both TAGS and tags files in the source directory tree. To enable and use tags in a particular program, see the associated program documentation.

Searching on the git repository at GitLab and Savannah

GitLab’s web interface provides a built-in search.

• Go to https://gitlab.com/lilypond/lilypond/
• Type functionName in the search box on the top, and hit enter/return

Alternatively you can also use the equivalent of git grep on the Savannah server.

• Go to https://git.sv.gnu.org/gitweb/?p=lilypond.git
• In the pulldown box that says commit, select grep.
• Type functionName in the search box, and hit enter/return

This will initiate a search of the remote git repository.

10.5 Code style

This section describes style guidelines for LilyPond source code.

   file names

        ".hh"   Include files
             ".cc"      Implementation files
             ".icc"     Inline definition files
             ".tcc"     non inline Template defs

   in emacs:

             (setq auto-mode-alist
                   (append '(("\\.make$" . makefile-mode)
                        ("\\.cc$" . c++-mode)
                        ("\\.icc$" . c++-mode)
                        ("\\.tcc$" . c++-mode)
                        ("\\.hh$" . c++-mode)
                        ("\\.pod$" . text-mode)
                        )
                      auto-mode-alist))

clang-format -i filename

clang-format -i $(git ls-files "*.cc" "*.hh" "*.icc" "*.tcc")

scripts/auxiliar/fixscm.sh filename

scripts/auxiliar/fixscm.sh $(git ls-files "*.scm")

autopep8 -ia --ignore=E402 file.py

setlocal cindent
setlocal cinoptions=>4,n-2,{2,^-2,:2,=2,g0,h2,p5,t0,+2,(0,u0,w1,m1
setlocal shiftwidth=2
setlocal softtabstop=2
setlocal textwidth=79
setlocal fo-=ro fo+=cql
" use spaces instead of tabs
setlocal expandtab
" remove trailing whitespace on write
autocmd BufWritePre * :%s/\s\+$//e

" Additional Guile-specific 'forms'
syn keyword schemeSyntax define-public define*-public
syn keyword schemeSyntax define* lambda* let-keywords*
syn keyword schemeSyntax defmacro defmacro* define-macro
syn keyword schemeSyntax defmacro-public defmacro*-public
syn keyword schemeSyntax use-modules define-module
syn keyword schemeSyntax define-method define-class

" Additional LilyPond-specific 'forms'
syn keyword schemeSyntax define-markup-command define-markup-list-command
syn keyword schemeSyntax define-music-function def-grace-function

" All of the above should influence indenting too
setlocal lw+=define-public,define*-public
setlocal lw+=define*,lambda*,let-keywords*
setlocal lw+=defmacro,defmacro*,define-macro
setlocal lw+=defmacro-public,defmacro*-public
setlocal lw+=use-modules,define-module
setlocal lw+=define-method,define-class
setlocal lw+=define-markup-command,define-markup-list-command
setlocal lw+=define-music-function,def-grace-function

" These forms should not influence indenting
setlocal lw-=if
setlocal lw-=set!

" Try to highlight all ly: procedures
syn match schemeFunc "ly:[^) ]\+"

if exists("did_load_filetypes")
  finish
endif
augroup filetypedetect
  au! BufRead,BufNewFile *.itely setfiletype texinfo
  au! BufRead,BufNewFile *.itexi setfiletype texinfo
  au! BufRead,BufNewFile *.tely  setfiletype texinfo
augroup END

setlocal expandtab
setlocal shiftwidth=2
setlocal textwidth=66

This_is_a_class

Type Class::member_

THIS_IS_A_MACRO

cxx_multiword_variable

scheme-multiword-variable

10.6 Warnings, Errors, Progress and Debug Output

Available log levels

LilyPond has several log levels to specify how verbose the output on the console should be, as shown in the following table. Note that only the first few characters of a given value are checked (as indicated); additionally, case doesn’t matter. In other words, ERR is the same as error or ERRO, for example.

NONE

no output at all, even on failure

ERR[OR]

only error messages

WARN

only error messages and warnings

BASIC

warnings, errors, and basic progress (success, etc.)

PROG[RESS]

warnings, errors, and full progress messages

INFO

warnings, errors, progress, and more detailed information (default)

DEBUG

all messages, including full debug messages (very verbose!)

The log level can either be set with the environment variable LILYPOND_LOGLEVEL or on the command line with the --loglevel=... option.

Functions for debug and log output

LilyPond has two different types of error and log functions:

• If a warning or error is caused by an identified position in the input file, e.g., by a grob or by a music expression, the functions of the Input class provide logging functionality that prints the position of the message in addition to the message.
• If a message can not be associated with a particular position in an input file, e.g., the output file cannot be written, then the functions in the flower/include/warn.hh file will provide logging functionality that only prints out the message, but no location.

There are also Scheme functions to access all of these logging functions from scheme. In addition, the Grob class contains some convenience wrappers for even easier access to these functions.

The message and debug functions in warn.hh also have an optional argument newline, which specifies whether the message should always start on a new line or continue a previous message. By default, progress_indication does NOT start on a new line, but rather continue the previous output. They also do not have a particular input position associated, so there are no progress functions in the Input class. All other functions by default start their output on a new line.

The error functions come in three different flavors: fatal error messages, programming error messages and normal error messages. Errors written by the error () function will cause LilyPond to exit immediately, errors by Input::error () will continue the compilation, but return a non-zero return value of the LilyPond call (i.e., indicate an unsuccessful program execution). All other errors will be printed on the console, but not exit LilyPond or indicate an unsuccessful return code. Their only differences to a warnings are the displayed text and that they will be shown with log level ERROR.

If the Scheme option warning-as-error is set, any warning will be treated as if Input::error was called.

All logging functions at a glance

10.7 Debugging LilyPond

The most commonly used tool for debugging LilyPond is the GNU debugger gdb. The gdb tool is used for investigating and debugging core LilyPond code written in C++. Another tool is available for debugging Scheme code using the Guile debugger. This section describes how to use both gdb and the Guile Debugger.

./configure --disable-optimising
make

out/bin/lilypond

gdb out/bin/lilypond

run test.ly

ddd out/bin/lilypond

file $LILYPOND_GIT/build/out/bin/lilypond
b programming_error
b Grob::programming_error

define ps
   print ly_display_scm($arg0)
end
define pgrob
  print ly_display_scm($arg0->self_scm_)
  print ly_display_scm($arg0->mutable_property_alist_)
  print ly_display_scm($arg0->immutable_property_alist_)
  print ly_display_scm($arg0->object_alist_)
end
define pmusic
  print ly_display_scm($arg0->self_scm_)
  print ly_display_scm($arg0->mutable_property_alist_)
  print ly_display_scm($arg0->immutable_property_alist_)
end

#(module-define! (resolve-module '(guile-user))
                 'lilypond-module (current-module))

#(top-repl)

guile> (set-current-module lilypond-module)

guile> fret-diagram-verbose-markup
#<procedure fret-diagram-verbose-markup (layout props marking-list)>

guile> fret-diagram-verbose-markup
ERROR: Unbound variable: fret-diagram-verbose-markup
ABORT: (unbound-variable)

guile> (quit)

\include "guile-debugger.ly"

guile> (set-break! my-scheme-procedure)

guile> (quit)

debug> help

#(set-break! my-scheme-procedure)

(use-modules (scm guile-debugger))

(define (print-book-with book process-procedure)
  (let* ((paper (ly:parser-lookup '$defaultpaper))
         (layout (ly:parser-lookup '$defaultlayout))
         (outfile-name (get-outfile-name book)))
    (process-procedure book paper layout outfile-name)))

(define-public (print-book-with-defaults book)
  (print-book-with book ly:book-process))

(define-public (print-book-with-defaults-as-systems book)
  (print-book-with book ly:book-process-to-systems))

(set-break! print-book-with)

(set-trace-call! my-scheme-procedure)

(set-trace-subtree! my-scheme-procedure)