Usage example

Usage example#

Before diving into all the details in later sections, we will in the following go through a small usage example. The code in this example will not do anything useful in itself, and is merely meant to illustrate how one might lay out files for a given project or task, use the main simplebuild command sb to configure and build everything, and finally invoke a few of the resulting commands.

Example project files#

The following lists all the files in our little (silly) user project. Technically, these files are spread over three simplebuild packages and these packages are collected in a single simplebuild bundle. Our example directory thus contains a simplebuild.cfg file (exactly one such is always associated with a bundle), as well as three packages, named SomePkgA, SomePkgB, and SomePkgC – each kept in their separate subdirectory of the same name as the package (the subdirectory name actually defines the name of the package). In addition to code files, each package subdirectory contains a pkg.info file, with information about the dependencies of the given package.

  • example_project/simplebuild.cfg (TOML)

    # A simplebuild.cfg file is always needed to define a bundle, but for this super
# simple example it can essentially be empty.

  • example_project/SomePkgA/pkg.info

    # This package has no dependencies. It provides a few python modules, as well as
# a single command-line application written in Python.
package()

  • example_project/SomePkgA/pycpp_bar/mod.cc (C++)

    #include "Core/Python.hh"
#include <iostream>

namespace {
  void somecppfunc()
  {
    std::cout<<"in somecppfunc in a python module"<<std::endl;
  }
}

PYTHON_MODULE( mod )
{
  //The following binding is essentially pybind11 code:
  mod.def("somecppfunc", &somecppfunc );
}

  • example_project/SomePkgA/python/foo.py (Python)

    def somefunc():
    print("in somefunc in a python module")
def mysquarefunc( x ):
    return x * x

  • example_project/SomePkgA/scripts/mycmd (Python)

    #!/usr/bin/env python3
import SomePkgA.foo
import SomePkgA.bar
print("I am calling functions from python modules in my own package:")
SomePkgA.foo.somefunc()
SomePkgA.bar.somecppfunc()

  • example_project/SomePkgB/pkg.info

    # This package provides a commandline application implemented in Python. The
# application uses Python modules implemented in another package (SomePkgA), so
# we must list that package as a dependency here:
package( USEPKG SomePkgA )

  • example_project/SomePkgB/scripts/mycmd (Python)

    #!/usr/bin/env python3
import SomePkgA.foo
import SomePkgA.bar
print("I am calling functions from python modules in another package:")
SomePkgA.foo.somefunc()
SomePkgA.bar.somecppfunc()

  • example_project/SomePkgC/pkg.info

    # This package provides a single commandline application implemented in C++. To
# make it less trivial, the application builds against an external library
# (zlib), so we must list that as dependency here:
package( USEEXT ZLib )

  • example_project/SomePkgC/app_foobar/main.cc (C++)

    #include <iostream>
#include "zlib.h"

int main()
{
  std::cout << "I am a silly little C++ application" << std::endl;
  std::cout << "I am built against zlib in version: "
            << ZLIB_VERSION << std::endl;
}

Building the project#

To build our example project, we simply step into the directory where we have the simplebuild.cfg file by issuing a command like cd /some/where/example_project (it is also OK to step into any sub-directory under that directory). Technically this directory is the package root of our package bundle. Then we simply invoke sb with no arguments to perform the build (this includes both configuration, build, and installation steps of more traditional setups):

$> sb
sbld:  Inspecting environment via CMake
-- Using CMAKE_BUILD_TYPE=Release
-- The C compiler identification is GNU 15.3.0
-- The CXX compiler identification is GNU 15.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /conda/envs/sbenv/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /conda/envs/sbenv/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Python executable: /conda/envs/sbenv/bin/python3
-- Python version: 3.14.10
-- Found pybind11 version 3.1.0
-- Checking for ZLib installation
-- Checking for ZLib installation -- yes
-- Found ZLib version: 1.3
-- Skipped unused dependencies: ASE DL Fortran Garfield Geant4 Gemmi
--                              HDF5 MCPL Mantidpython NCrystal Numpy OSG
--                              Pandas PyToml Pymatgen ROOT Scipy Spglib
--                              Threads matplotlib mpmath
-- Configuring done (6.6s)
-- Generating done (0.0s)
-- Build files have been written to: /some/where/example_project/simplebuild_cache/bld/cmake
sbld:  Environment inspection done (6.6 seconds)
sbld:  Configuration completed => Launching build with 4 parallel processes
make[1]: Entering directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
Build started
  Updating symlinks SomePkgB/scripts
  Updating symlinks SomePkgB/testlogs
  Installing Core headers
  Updating symlinks SomePkgC/testlogs
  Updating symlinks SomePkgA/python
  Updating symlinks SomePkgA/scripts
  Generating SomePkgA/__init__.py
  Updating symlinks SomePkgA/testlogs
  Updating symlinks Core/python
  Building Core/app_cmakebuildtype/main.o
Installing sbldpy
  Updating symlinks Core/scripts
  Generating Core/__init__.py
  Updating symlinks Core/testlogs
Installing global system modules
Package SomePkgB done
  Building SomePkgC/app_foobar/main.o
  Building Core/libsrc/FPE.o
  Building Core/libsrc/File.o
  Building Core/libsrc/FindData.o
  Building Core/libsrc/String.o
  Building Core/libsrc/static_asserts.o
  Building SomePkgA/pycpp_bar/mod.o
  Building Core/pymod_misc/mod.o
  Building Core/pymod__fpe/mod.o
  Creating shared library for package Core
  Creating application sb_somepkgc_foobar
  Creating application sb_core_cmakebuildtype
Package SomePkgC done
  Creating python module Core.misc (sbldpy)
  Creating python module Core._fpe (sbldpy)
Package Core done
  Creating python module SomePkgA.bar
Package SomePkgA done
All done
make[1]: Leaving directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
sbld:  Summary:
sbld: 
sbld:    Main bundle pkg root             : /some/where/example_project
sbld:    Installation directory           : /some/where/example_project/simplebuild_cache/install
sbld:    Build directory                  : /some/where/example_project/simplebuild_cache/bld
sbld:    Package search path              : /some/where/example_project (3 pkgs)
sbld:                                       ${CONDA_PREFIX}/lib/python3.14/site-packages/_simple_build_system/data/pkgs-core (1 pkgs)
sbld:    System                           : Linux-7.0.0-1017-azure
sbld:    Required dependencies            : C[GNU/15.3.0] CMake[4.4.5] CXX[GNU/15.3.0]
sbld:                                       Python[3.14.10] pybind11[3.1.0]
sbld:    Optional dependencies present    : ZLib[1.3]
sbld:    Optional dependencies missing[*] : ASE DL Fortran Garfield Geant4 Gemmi HDF5
sbld:                                       MCPL Mantidpython NCrystal Numpy OSG Pandas
sbld:                                       PyToml Pymatgen ROOT Scipy Spglib Threads
sbld:                                       matplotlib mpmath
sbld:    Package filter[*]                : <none>
sbld:    Build mode                       : release
sbld:  
sbld:    4 packages built successfully    : Core SomePkgA SomePkgB SomePkgC
sbld:    0 packages skipped due to [*]    : <none>
sbld: 
sbld:  Build done. You are all set to begin using the software!
sbld: 
sbld:  To see available applications, type "sb_" and hit the TAB key twice.
sbld: 
[last command took 13.12 seconds to complete] $>

Several things happened above:

  1. simplebuild searched the directory tree under /some/where/example_project for pkg.info files to determine which simplebuild packages to consider. It also added the Core package (defined elsewhere in the core bundle), since that particular package is always required (for instance because it provides the "Core/Python.hh" header file).

  2. It investigated each package to determine what kind of code and data files it provided, and established dependency relationships and other configuration details concerning the various pieces.

  3. Behind the scenes it created a cache directory in /some/where/example_project/simplebuild_cache, to use for any sort of temporary files needed for the following steps. Note that if you ever wish to clean up the cache files, you can simply invoke sb -c.

  4. It launched CMake to inspect the environment, to learn about compilers, configuration flags, etc. It also specifically looked in the environment for ZLib since one of our packages (SomePkgC) had listed that as an external dependency. It took several seconds, but the result are cached for future reusage.

  5. It went ahead and actually built our various code files into their final product. For C++ code that means actual compilation into binary code, copied into appropriate “installation folders” in the cache directory. On the other hand, pure data files and scripts (like our pure Python files) are merely symlinked into their installation folders. The advantage of symlinking scripts and data files, is that any future editing will be instantly applied without any need for another sb invocation. The build step was actually a bit slow, taking many seconds. This is mostly due to compilation of the (pybind11-based) C++-Python bindings. One side-effect of symlinking, is that all script files must be executable (e.g. chmod +x <pathtofile>).

  6. It produced a little summary of all the operations.

Environment setup for non-conda installations.

If you did not install simplebuild via the conda package, the output above might have included a warning about needing to invoke eval "$(sb --env-setup)" to setup your environment. One solution is then to simply run that command, or you can refer to the relevant documentation for how to address this in general.

Although the entire build process took a bit of time, the cache usage means that if we go ahead and invoke sb once again, it will this time more or less skip the expensive steps 4 and 5 above, and finish almost instantaneously:

$> sb
sbld:  Configuration completed => Launching build with 4 parallel processes
make[1]: Entering directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
Build started
All done
make[1]: Leaving directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
sbld:  Summary:
sbld: 
sbld:    Main bundle pkg root             : /some/where/example_project
sbld:    Installation directory           : /some/where/example_project/simplebuild_cache/install
sbld:    Build directory                  : /some/where/example_project/simplebuild_cache/bld
sbld:    Package search path              : /some/where/example_project (3 pkgs)
sbld:                                       ${CONDA_PREFIX}/lib/python3.14/site-packages/_simple_build_system/data/pkgs-core (1 pkgs)
sbld:    System                           : Linux-7.0.0-1017-azure
sbld:    Required dependencies            : C[GNU/15.3.0] CMake[4.4.5] CXX[GNU/15.3.0]
sbld:                                       Python[3.14.10] pybind11[3.1.0]
sbld:    Optional dependencies present    : ZLib[1.3]
sbld:    Optional dependencies missing[*] : ASE DL Fortran Garfield Geant4 Gemmi HDF5
sbld:                                       MCPL Mantidpython NCrystal Numpy OSG Pandas
sbld:                                       PyToml Pymatgen ROOT Scipy Spglib Threads
sbld:                                       matplotlib mpmath
sbld:    Package filter[*]                : <none>
sbld:    Build mode                       : release
sbld:  
sbld:    4 packages built successfully    : Core SomePkgA SomePkgB SomePkgC
sbld:    0 packages skipped due to [*]    : <none>
sbld: 
sbld:  Build done. You are all set to begin using the software!
sbld: 
sbld:  To see available applications, type "sb_" and hit the TAB key twice.
sbld: 
[last command took 0.37 seconds to complete] $>

If we go ahead and edit a C++ file, we must then invoke sb again, in order to get the corresponding binary output files rebuilt. Likewise, if we are adding or removing files to a package (or adding/removing packages), we should also invoke sb again, to make sure the content of the installation folders are once again up to date and consistent with the source code files. However, if we are merely editing existing scripts or data files, this is not needed due to the symlinking mentioned above. Now, let us see this in practice. So let us pretend we have edited the file example_project/SomePkgC/app_foobar/main.cc (e.g. open it, add a blank line, close it - or merely use the unix touch command). We have also added a new file, example_project/SomePkgB/scripts/newcmd with the content:

#!/usr/bin/env python3
print("I am a new command")

Since it is a script, we have made the new file executable (chmod +x SomePkgB/scripts/newcmd). If we had forgotten, sb would tell us later.

With these changes, we now invoke sb once again:

$> sb
sbld:  Configuration completed => Launching build with 4 parallel processes
make[1]: Entering directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
Build started
  Updating symlinks SomePkgB/scripts
  Building SomePkgC/app_foobar/main.o
Installing global system modules
Package SomePkgB done
  Creating application sb_somepkgc_foobar
Package SomePkgC done
All done
make[1]: Leaving directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
sbld:  Summary:
sbld: 
sbld:    Main bundle pkg root             : /some/where/example_project
sbld:    Installation directory           : /some/where/example_project/simplebuild_cache/install
sbld:    Build directory                  : /some/where/example_project/simplebuild_cache/bld
sbld:    Package search path              : /some/where/example_project (3 pkgs)
sbld:                                       ${CONDA_PREFIX}/lib/python3.14/site-packages/_simple_build_system/data/pkgs-core (1 pkgs)
sbld:    System                           : Linux-7.0.0-1017-azure
sbld:    Required dependencies            : C[GNU/15.3.0] CMake[4.4.5] CXX[GNU/15.3.0]
sbld:                                       Python[3.14.10] pybind11[3.1.0]
sbld:    Optional dependencies present    : ZLib[1.3]
sbld:    Optional dependencies missing[*] : ASE DL Fortran Garfield Geant4 Gemmi HDF5
sbld:                                       MCPL Mantidpython NCrystal Numpy OSG Pandas
sbld:                                       PyToml Pymatgen ROOT Scipy Spglib Threads
sbld:                                       matplotlib mpmath
sbld:    Package filter[*]                : <none>
sbld:    Build mode                       : release
sbld:  
sbld:    4 packages built successfully    : Core SomePkgA SomePkgB SomePkgC
sbld:    0 packages skipped due to [*]    : <none>
sbld: 
sbld:  Build done. You are all set to begin using the software!
sbld: 
sbld:  To see available applications, type "sb_" and hit the TAB key twice.
sbld: 
[last command took 0.66 seconds to complete] $>

As we can see, only parts of the build related to our changes actually had to be redone, and accordingly it was blazingly fast.

To find out where the time goes when building, sb --buildtimes shows the slowest compilation and linking steps (and sb --buildtimes=N the N slowest), along with the slowest packages. The timings are recorded whenever files are built, so they reflect the most recent build of each file, and no special build is needed. Note that the timings come from builds with several steps running in parallel, which might inflate them a bit. Additionally, sb --includecosts (which requires the Python module numpy) estimates which files are the most costly to compile, including header files. This is based on the build timings and the include statements in the code: the compilation time of each file is modelled as the sum of the costs of the files it needs (its source file, the header files of the project it includes directly or indirectly, and the system headers included by those), and the costs of the files are estimated from all the timings. This can of course only give rough estimates, but is useful for finding the worst offenders.

Using the project#

Having built our example project, let us now try to actually use it! As a first example, let us try to invoke the commandline application defined in our file SomePkgC/app_foobar/main.cc. If we look carefully in the output above, we can notice a line saying Creating application sb_somepkgc_foobar. In general this is how it works, all commandline applications will end up with a name sb_<pkgname>_<appname> (lowercased) where in this case <pkgname> lowercased is somepkgc and <appname> is foobar. For scripts, <appname> instead becomes the actual filename. The reason for this naming policy is one of name-spacing: by prefixing all commands are with sb_<pkgname>_, there should be almost no chance of any name-clashes between applications in different packages, or even with other commands on your unix system. We can try to run it:

$> sb_somepkgc_foobar
I am a silly little C++ application
I am built against zlib in version: 1.3.2

Environment setup for non-conda installations.

if you got a sb_somepkgc_foobar: command not found you most likely did not install simplebuild via conda. Refer to the relevant documentation for how to deal with this.

For completeness, here are some more examples of us using our project:

$> sb_somepkga_mycmd
I am calling functions from python modules in my own package:
in somefunc in a python module
in somecppfunc in a python module

$> python3 -c 'import SomePkgA.foo; SomePkgA.foo.somefunc()'
in somefunc in a python module

$> python3 -c 'import SomePkgA.bar; SomePkgA.bar.somecppfunc()'
in somecppfunc in a python module

$> sb_somepkgb_mycmd
I am calling functions from python modules in another package:
in somefunc in a python module
in somecppfunc in a python module

$> sb_somepkgb_newcmd
I am a new command

Adding tests#

As is discussed in more details in a dedicated section, it is possible to mark certain applications or scripts as being a test. Launching sb --tests (or sb -t for short) will then not only build the project, but will also launch all the tests. Any command that fails (i.e. exits with non-zero exit code) will be marked as a test failure. Additionally, it is possible to add text files containing the expected output of a given command. If the actual output fails to match such a reference log file, it will also result in a test failure.

Having such tests to perform a quick validation that everything still works is tremendously useful, and here we will simply show a quick example of this in practice. Specifically, we would like a unit test that verifies the output of the mysquarefunc that is already a part of the SomePkgA.foo module of our example’s bundle. Thus, we add a new file example_project/SomePkgA/scripts/testfoo with the content:

#!/usr/bin/env python3
import SomePkgA.foo
for x in [ -1, 0, 1, 3, 5 ]:
    if SomePkgA.foo.mysquarefunc( x ) != x**2:
        raise SystemExit(f'SomePkgA.foo.mysquarefunc({x}) != { x**2 }')

Again, since it is a script, we have made the new file executable (chmod +x SomePkgA/scripts/testfoo). If we had forgotten, sb would tell us later. Having added this new script, we now launch sb --tests:

$> sb --tests
sbld:  Configuration completed => Launching build with 4 parallel processes
make[1]: Entering directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
Build started
  Updating symlinks SomePkgA/scripts
Installing global system modules
Package SomePkgA done
All done
make[1]: Leaving directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
sbld:  Summary:
sbld: 
sbld:    Main bundle pkg root             : /some/where/example_project
sbld:    Installation directory           : /some/where/example_project/simplebuild_cache/install
sbld:    Build directory                  : /some/where/example_project/simplebuild_cache/bld
sbld:    Package search path              : /some/where/example_project (3 pkgs)
sbld:                                       ${CONDA_PREFIX}/lib/python3.14/site-packages/_simple_build_system/data/pkgs-core (1 pkgs)
sbld:    System                           : Linux-7.0.0-1017-azure
sbld:    Required dependencies            : C[GNU/15.3.0] CMake[4.4.5] CXX[GNU/15.3.0]
sbld:                                       Python[3.14.10] pybind11[3.1.0]
sbld:    Optional dependencies present    : ZLib[1.3]
sbld:    Optional dependencies missing[*] : ASE DL Fortran Garfield Geant4 Gemmi HDF5
sbld:                                       MCPL Mantidpython NCrystal Numpy OSG Pandas
sbld:                                       PyToml Pymatgen ROOT Scipy Spglib Threads
sbld:                                       matplotlib mpmath
sbld:    Package filter[*]                : <none>
sbld:    Build mode                       : release
sbld:  
sbld:    4 packages built successfully    : Core SomePkgA SomePkgB SomePkgC
sbld:    0 packages skipped due to [*]    : <none>
sbld: 
sbld:  Running tests in /some/where/example_project/simplebuild_cache/bld/testresults:
sbld: 
make[1]: Entering directory '/some/where/example_project'
make[1]: Leaving directory '/some/where/example_project'
sbld:   ---------------------------------------+-----------+--------+----------+------------------
sbld:    Test job results                      | Time [ms] | Job EC | Log-diff | Trouble info
sbld:   ---------------------------------------+-----------+--------+----------+------------------
sbld:    sb_somepkga_testfoo                   |     165   |   OK   |    --    | --
sbld:   ---------------------------------------+-----------+--------+----------+------------------
sbld: 
sbld:    Test results are also summarised in: simplebuild_test_results_junitformat.xml
sbld: 
sbld:    All tests completed without failures!
sbld: 
sbld:  Build done. You are all set to begin using the software!
sbld: 
sbld:  To see available applications, type "sb_" and hit the TAB key twice.
sbld: 
[last command took 0.88 seconds to complete] $>

All went well in this case! If you had issues, you could go and look in the directory listed in the output (the Trouble info column) for clues as to what went wrong. Adding for instance --testexcerpts=20 will additionally print the first and last 20 lines of the output of failed tests, or of the differences between the output and the reference log.

The report of the test results can also be printed again later with sb --testreport, without building anything or running any tests (as usual, the build variant can be selected with e.g. -d, and the results of each build variant are kept separately). The report can be customised, e.g. sb --testreport=failed,time shows only the failed tests, with the slowest first (other sort orders are name, and relevance, the default, which in printed reports puts failed tests at the bottom, where they remain visible in the terminal, with slower tests towards the bottom within each group), and --testfilter and --testexcerpts can be used to select tests and show excerpts (if --testfilter does not select any tests when running tests, sb stops with an error already before building anything). With sb --testreport=interactive (which can also be combined with -t), the results can instead be browsed interactively: the output of tests, or the differences between their output and their reference logs, can be inspected, and tests can be run again, optionally after rebuilding (press ? for a list of keys). By default, the interactive view shows running tests first, then failed tests, tests waiting to run, and finally successful tests. When running tests, the slowest tests (as measured in previous runs) are started first, to avoid ending up waiting for a single slow test at the end (with --testorder=fastest, the fastest tests are instead started first, to get results as quickly as possible). To only run the tests which can complete within a given time, use e.g. sb -t --testtime=10, which skips the slowest tests (based on previous runs) as needed to finish within approximately 10 seconds (tests without timings from previous runs are always run). To stop as soon as a test fails (for instance in CI), use sb -t --failearly: tests which are still running or not yet started are then aborted, and reported as such (and as skipped in the JUnit XML file). When running tests in a terminal, the progress is shown while they run, including an estimate of the remaining time (based on previous runs), and pressing i opens the interactive view, in which results appear as tests finish (finished tests can be inspected while other tests are still running). The interactive view can also run the shown tests again in the background (t).

Reference logs must match the output exactly, including whitespace. If the output differs from a reference log only in whitespace (e.g. due to an editor removing trailing spaces from the reference log), this is pointed out explicitly. If there was a problem with a reference log (the Log-diff column), you can use sb --reflogupdate to review the differences in an interactive viewer, which highlights changed words and whitespace (press ? in the viewer for a list of keys). From the viewer, you can also run the test again (r), or rebuild and then run the test again (b), for instance after editing the code producing the output (if the build fails, its output is shown, and it can be shown again later with e). Finally, the viewer can update the reference log in the package with the new output, but you should not actually do so unless you are the developer maintaining it. To make you think before updating, a randomly generated code must be typed in reverse order for each update (for large scale maintenance, this requirement can be disabled for the rest of the session).