Usage example

Before diving into all the details in later sections, we will in the following go through a small usage example. The code in this example will not do anything useful in itself, and is merely meant to illustrate how one might lay out files for a given project or task, use the main simplebuild command sb to configure and build everything, and finally invoke a few of the resulting commands.

Example project files

The following lists all the files in our little (silly) user project. Technically, these files are spread over three simplebuild packages and these packages are collected in a single simplebuild bundle. Our example directory thus contains a simplebuild.cfg file (exactly one such is always associated with a bundle), as well as three packages, named SomePkgA, SomePkgB, and SomePkgC – each kept in their separate subdirectory of the same name as the package (the subdirectory name actually defines the name of the package). In addition to code files, each package subdirectory contains a pkg.info file, with information about the dependencies of the given package.

• example_project/simplebuild.cfg (TOML)

  # A simplebuild.cfg file is always needed to define a bundle, but for this super
  # simple example it can essentially be empty.
  

• example_project/SomePkgA/pkg.info

  # This package has no dependencies. It provides a few python modules, as well as
  # a single command-line application written in Python.
  package()
  

• example_project/SomePkgA/pycpp_bar/mod.cc (C++)

  #include "Core/Python.hh"
  #include <iostream>
  
  namespace {
    void somecppfunc()
    {
      std::cout<<"in somecppfunc in a python module"<<std::endl;
    }
  }
  
  PYTHON_MODULE( mod )
  {
    //The following binding is essentially pybind11 code:
    mod.def("somecppfunc", &somecppfunc );
  }
  

• example_project/SomePkgA/python/foo.py (Python)

  def somefunc():
      print("in somefunc in a python module")
  def mysquarefunc( x ):
      return x * x
  

• example_project/SomePkgA/scripts/mycmd (Python)

  #!/usr/bin/env python3
  import SomePkgA.foo
  import SomePkgA.bar
  print("I am calling functions from python modules in my own package:")
  SomePkgA.foo.somefunc()
  SomePkgA.bar.somecppfunc()
  

• example_project/SomePkgB/pkg.info

  # This package provides a commandline application implemented in Python. The
  # application uses Python modules implemented in another package (SomePkgA), so
  # we must list that package as a dependency here:
  package( USEPKG SomePkgA )
  

• example_project/SomePkgB/scripts/mycmd (Python)

  #!/usr/bin/env python3
  import SomePkgA.foo
  import SomePkgA.bar
  print("I am calling functions from python modules in another package:")
  SomePkgA.foo.somefunc()
  SomePkgA.bar.somecppfunc()
  

• example_project/SomePkgC/pkg.info

  # This package provides a single commandline application implemented in C++. To
  # make it less trivial, the application builds against an external library
  # (zlib), so we must list that as dependency here:
  package( USEEXT ZLib )
  

• example_project/SomePkgC/app_foobar/main.cc (C++)

  #include <iostream>
  #include "zlib.h"
  
  int main()
  {
    std::cout << "I am a silly little C++ application" << std::endl;
    std::cout << "I am built against zlib in version: "
              << ZLIB_VERSION << std::endl;
  }
  

Building the project

To build our example project, we simply step into the directory where we have the simplebuild.cfg file by issuing a command like cd /some/where/example_project (it is also OK to step into any sub-directory under that directory). Technically this directory is the package root of our package bundle. Then we simply invoke sb with no arguments to perform the build (this includes both configuration, build, and installation steps of more traditional setups):

$> sb
sbld:  Inspecting environment via CMake
-- Using CMAKE_BUILD_TYPE=Release
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /conda/envs/sbenv/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /conda/envs/sbenv/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Python executable: /conda/envs/sbenv/bin/python3
-- Python version: 3.12.6
-- Performing Test compiler_supports_Wno-array-bounds
-- Performing Test compiler_supports_Wno-array-bounds - Success
-- Performing Test compiler_supports_Wno-stringop-overread
-- Performing Test compiler_supports_Wno-stringop-overread - Success
-- Found pybind11 version 2.13.6
-- Checking for ZLib installation
-- Checking for ZLib installation -- yes
-- Found ZLib version: 1.3.1
-- Skipped unused dependencies: ASE DL Fortran Garfield Geant4 Gemmi
--                              HDF5 Mantidpython NCrystal Numpy OSG Pandas
--                              Pymatgen ROOT Scipy Spglib Threads matplotlib
--                              mpmath
-- Configuring done (3.9s)
-- Generating done (0.0s)
-- Build files have been written to: /some/where/example_project/simplebuild_cache/bld/cmake
sbld:  Environment inspection done (3.9 seconds)
sbld:  Configuration completed => Launching build with 4 parallel processes
make[1]: Entering directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
Build started
  Updating symlinks SomePkgB/scripts
  Updating symlinks SomePkgB/testlogs
  Installing Core headers
  Updating symlinks SomePkgC/testlogs
  Updating symlinks SomePkgA/python
  Updating symlinks SomePkgA/scripts
  Generating SomePkgA/__init__.py
  Updating symlinks SomePkgA/testlogs
  Updating symlinks Core/python
  Building Core/pycpp_misc/mod.o
  Building Core/pycpp_FPE/mod.o
  Updating symlinks Core/scripts
  Building Core/app_cmakebuildtype/main.o
  Generating Core/__init__.py
  Updating symlinks Core/testlogs
Installing global system modules
Package SomePkgB done
  Building SomePkgC/app_foobar/main.o
  Building Core/libsrc/FPE.o
  Building Core/libsrc/File.o
  Building Core/libsrc/FindData.o
  Building Core/libsrc/String.o
  Building Core/libsrc/static_asserts.o
  Building SomePkgA/pycpp_bar/mod.o
  Creating shared library for package Core
  Creating application sb_somepkgc_foobar
  Creating application sb_core_cmakebuildtype
Package SomePkgC done
  Creating python module Core.misc
  Creating python module SomePkgA.bar
Package SomePkgA done
  Creating python module Core.FPE
Package Core done
All done
make[1]: Leaving directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
sbld:  Summary:
sbld: 
sbld:    Main bundle pkg root             : /some/where/example_project
sbld:    Installation directory           : /some/where/example_project/simplebuild_cache/install
sbld:    Build directory                  : /some/where/example_project/simplebuild_cache/bld
sbld:    Package search path              : /some/where/example_project (3 pkgs)
sbld:                                       ${CONDA_PREFIX}/lib/python3.12/site-packages/_simple_build_system/data/pkgs-core (1 pkgs)
sbld:    System                           : Linux-6.8.0-1014-azure
sbld:    Required dependencies            : C[GNU/13.3.0] CMake[3.30.4] CXX[GNU/13.3.0]
sbld:                                       Python[3.12.6] pybind11[2.13.6]
sbld:    Optional dependencies present    : ZLib[1.3.1]
sbld:    Optional dependencies missing[*] : ASE DL Fortran Garfield Geant4 Gemmi HDF5
sbld:                                       Mantidpython NCrystal Numpy OSG Pandas Pymatgen
sbld:                                       ROOT Scipy Spglib Threads matplotlib mpmath
sbld:    Package filter[*]                : <none>
sbld:    Build mode                       : release
sbld:  
sbld:    4 packages built successfully    : Core SomePkgA SomePkgB SomePkgC
sbld:    0 packages skipped due to [*]    : <none>
sbld: 
sbld:  Build done. You are all set to begin using the software!
sbld: 
sbld:  To see available applications, type "sb_" and hit the TAB key twice.
sbld: 
[last command took 11.11 seconds to complete] $>

Several things happened above:

1. simplebuild searched the directory tree under /some/where/example_project for pkg.info files to determine which simplebuild packages to consider. It also added the Core package (defined elsewhere in the core bundle), since that particular package is always required (for instance because it provides the "Core/Python.hh" header file).

2. It investigated each package to determine what kind of code and data files it provided, and established dependency relationships and other configuration details concerning the various pieces.

3. Behind the scenes it created a cache directory in /some/where/example_project/simplebuild_cache, to use for any sort of temporary files needed for the following steps. Note that if you ever wish to clean up the cache files, you can simply invoke sb -c.

4. It launched CMake to inspect the environment, to learn about compilers, configuration flags, etc. It also specifically looked in the environment for ZLib since one of our packages (SomePkgC) had listed that as an external dependency. It took several seconds, but the result are cached for future reusage.

5. It went ahead and actually built our various code files into their final product. For C++ code that means actual compilation into binary code, copied into appropriate “installation folders” in the cache directory. On the other hand, pure data files and scripts (like our pure Python files) are merely symlinked into their installation folders. The advantage of symlinking scripts and data files, is that any future editing will be instantly applied without any need for another sb invocation. The build step was actually a bit slow, taking many seconds. This is mostly due to compilation of the (pybind11-based) C++-Python bindings. One side-effect of symlinking, is that all script files must be executable (e.g. chmod +x <pathtofile>).

6. It produced a little summary of all the operations.

Environment setup for non-conda installations.

If you did not install simplebuild via the conda package, the output above might have included a warning about needing to invoke eval "$(sb --env-setup)" to setup your environment. One solution is then to simply run that command, or you can refer to the relevant documentation for how to address this in general.

Although the entire build process took a bit of time, the cache usage means that if we go ahead and invoke sb once again, it will this time more or less skip the expensive steps 4 and 5 above, and finish almost instantaneously:

$> sb
sbld:  Configuration completed => Launching build with 4 parallel processes
make[1]: Entering directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
Build started
All done
make[1]: Leaving directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
sbld:  Summary:
sbld: 
sbld:    Main bundle pkg root             : /some/where/example_project
sbld:    Installation directory           : /some/where/example_project/simplebuild_cache/install
sbld:    Build directory                  : /some/where/example_project/simplebuild_cache/bld
sbld:    Package search path              : /some/where/example_project (3 pkgs)
sbld:                                       ${CONDA_PREFIX}/lib/python3.12/site-packages/_simple_build_system/data/pkgs-core (1 pkgs)
sbld:    System                           : Linux-6.8.0-1014-azure
sbld:    Required dependencies            : C[GNU/13.3.0] CMake[3.30.4] CXX[GNU/13.3.0]
sbld:                                       Python[3.12.6] pybind11[2.13.6]
sbld:    Optional dependencies present    : ZLib[1.3.1]
sbld:    Optional dependencies missing[*] : ASE DL Fortran Garfield Geant4 Gemmi HDF5
sbld:                                       Mantidpython NCrystal Numpy OSG Pandas Pymatgen
sbld:                                       ROOT Scipy Spglib Threads matplotlib mpmath
sbld:    Package filter[*]                : <none>
sbld:    Build mode                       : release
sbld:  
sbld:    4 packages built successfully    : Core SomePkgA SomePkgB SomePkgC
sbld:    0 packages skipped due to [*]    : <none>
sbld: 
sbld:  Build done. You are all set to begin using the software!
sbld: 
sbld:  To see available applications, type "sb_" and hit the TAB key twice.
sbld: 
[last command took 0.17 seconds to complete] $>

If we go ahead and edit a C++ file, we must then invoke sb again, in order to get the corresponding binary output files rebuilt. Likewise, if we are adding or removing files to a package (or adding/removing packages), we should also invoke sb again, to make sure the content of the installation folders are once again up to date and consistent with the source code files. However, if we are merely editing existing scripts or data files, this is not needed due to the symlinking mentioned above. Now, let us see this in practice. So let us pretend we have edited the file example_project/SomePkgC/app_foobar/main.cc (e.g. open it, add a blank line, close it - or merely use the unix touch command). We have also added a new file, example_project/SomePkgB/scripts/newcmd with the content:

#!/usr/bin/env python3
print("I am a new command")

Since it is a script, we have made the new file executable (chmod +x SomePkgB/scripts/newcmd). If we had forgotten, sb would tell us later.

With these changes, we now invoke sb once again:

$> sb
sbld:  Configuration completed => Launching build with 4 parallel processes
make[1]: Entering directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
Build started
  Updating symlinks SomePkgB/scripts
  Building SomePkgC/app_foobar/main.o
Installing global system modules
Package SomePkgB done
  Creating application sb_somepkgc_foobar
Package SomePkgC done
All done
make[1]: Leaving directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
sbld:  Summary:
sbld: 
sbld:    Main bundle pkg root             : /some/where/example_project
sbld:    Installation directory           : /some/where/example_project/simplebuild_cache/install
sbld:    Build directory                  : /some/where/example_project/simplebuild_cache/bld
sbld:    Package search path              : /some/where/example_project (3 pkgs)
sbld:                                       ${CONDA_PREFIX}/lib/python3.12/site-packages/_simple_build_system/data/pkgs-core (1 pkgs)
sbld:    System                           : Linux-6.8.0-1014-azure
sbld:    Required dependencies            : C[GNU/13.3.0] CMake[3.30.4] CXX[GNU/13.3.0]
sbld:                                       Python[3.12.6] pybind11[2.13.6]
sbld:    Optional dependencies present    : ZLib[1.3.1]
sbld:    Optional dependencies missing[*] : ASE DL Fortran Garfield Geant4 Gemmi HDF5
sbld:                                       Mantidpython NCrystal Numpy OSG Pandas Pymatgen
sbld:                                       ROOT Scipy Spglib Threads matplotlib mpmath
sbld:    Package filter[*]                : <none>
sbld:    Build mode                       : release
sbld:  
sbld:    4 packages built successfully    : Core SomePkgA SomePkgB SomePkgC
sbld:    0 packages skipped due to [*]    : <none>
sbld: 
sbld:  Build done. You are all set to begin using the software!
sbld: 
sbld:  To see available applications, type "sb_" and hit the TAB key twice.
sbld: 
[last command took 0.51 seconds to complete] $>

As we can see, only parts of the build related to our changes actually had to be redone, and accordingly it was blazingly fast.

Using the project

Having built our example project, let us now try to actually use it! As a first example, let us try to invoke the commandline application defined in our file SomePkgC/app_foobar/main.cc. If we look carefully in the output above, we can notice a line saying Creating application sb_somepkgc_foobar. In general this is how it works, all commandline applications will end up with a name sb_<pkgname>_<appname> (lowercased) where in this case <pkgname> lowercased is somepkgc and <appname> is foobar. For scripts, <appname> instead becomes the actual filename. The reason for this naming policy is one of name-spacing: by prefixing all commands are with sb_<pkgname>_, there should be almost no chance of any name-clashes between applications in different packages, or even with other commands on your unix system. We can try to run it:

$> sb_somepkgc_foobar
I am a silly little C++ application
I am built against zlib in version: 1.3.1

Environment setup for non-conda installations.

if you got a sb_somepkgc_foobar: command not found you most likely did not install simplebuild via conda. Refer to the relevant documentation for how to deal with this.

For completeness, here are some more examples of us using our project:

$> sb_somepkga_mycmd
I am calling functions from python modules in my own package:
in somefunc in a python module
in somecppfunc in a python module

$> python3 -c 'import SomePkgA.foo; SomePkgA.foo.somefunc()'
in somefunc in a python module

$> python3 -c 'import SomePkgA.bar; SomePkgA.bar.somecppfunc()'
in somecppfunc in a python module

$> sb_somepkgb_mycmd
I am calling functions from python modules in another package:
in somefunc in a python module
in somecppfunc in a python module

$> sb_somepkgb_newcmd
I am a new command

Adding tests

As is discussed in more details in a dedicated section, it is possible to mark certain applications or scripts as being a test. Launching sb --tests (or sb -t for short) will then not only build the project, but will also launch all the tests. Any command that fails (i.e. exits with non-zero exit code) will be marked as a test failure. Additionally, it is possible to add text files containing the expected output of a given command. If the actual output fails to match such a reference log file, it will also result in a test failure.

Having such tests to perform a quick validation that everything still works is tremendously useful, and here we will simply show a quick example of this in practice. Specifically, we would like a unit test that verifies the output of the mysquarefunc that is already a part of the SomePkgA.foo module of our example’s bundle. Thus, we add a new file example_project/SomePkgA/scripts/testfoo with the content:

#!/usr/bin/env python3
import SomePkgA.foo
for x in [ -1, 0, 1, 3, 5 ]:
    if SomePkgA.foo.mysquarefunc( x ) != x**2:
        raise SystemExit(f'SomePkgA.foo.mysquarefunc({x}) != { x**2 }')

Again, since it is a script, we have made the new file executable (chmod +x SomePkgA/scripts/testfoo). If we had forgotten, sb would tell us later. Having added this new script, we now launch sb --tests:

$> sb --tests
sbld:  Configuration completed => Launching build with 4 parallel processes
make[1]: Entering directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
Build started
  Updating symlinks SomePkgA/scripts
Installing global system modules
Package SomePkgA done
All done
make[1]: Leaving directory '/some/where/example_project/simplebuild_cache/bld/makefiles'
sbld:  Summary:
sbld: 
sbld:    Main bundle pkg root             : /some/where/example_project
sbld:    Installation directory           : /some/where/example_project/simplebuild_cache/install
sbld:    Build directory                  : /some/where/example_project/simplebuild_cache/bld
sbld:    Package search path              : /some/where/example_project (3 pkgs)
sbld:                                       ${CONDA_PREFIX}/lib/python3.12/site-packages/_simple_build_system/data/pkgs-core (1 pkgs)
sbld:    System                           : Linux-6.8.0-1014-azure
sbld:    Required dependencies            : C[GNU/13.3.0] CMake[3.30.4] CXX[GNU/13.3.0]
sbld:                                       Python[3.12.6] pybind11[2.13.6]
sbld:    Optional dependencies present    : ZLib[1.3.1]
sbld:    Optional dependencies missing[*] : ASE DL Fortran Garfield Geant4 Gemmi HDF5
sbld:                                       Mantidpython NCrystal Numpy OSG Pandas Pymatgen
sbld:                                       ROOT Scipy Spglib Threads matplotlib mpmath
sbld:    Package filter[*]                : <none>
sbld:    Build mode                       : release
sbld:  
sbld:    4 packages built successfully    : Core SomePkgA SomePkgB SomePkgC
sbld:    0 packages skipped due to [*]    : <none>
sbld: 
sbld:  Running tests in /some/where/example_project/simplebuild_cache/bld/testresults:
sbld: 
make[1]: Entering directory '/some/where/example_project'
make[1]: Leaving directory '/some/where/example_project'
sbld:   ---------------------------------------+-----------+--------+----------+------------------
sbld:    Test job results                      | Time [ms] | Job EC | Log-diff | Trouble info
sbld:   ---------------------------------------+-----------+--------+----------+------------------
sbld:    sb_somepkga_testfoo                   |      64   |   OK   |    --    | --
sbld:   ---------------------------------------+-----------+--------+----------+------------------
sbld: 
sbld:    Test results are also summarised in: simplebuild_test_results_junitformat.xml
sbld: 
sbld:    All tests completed without failures!
sbld: 
sbld:  Build done. You are all set to begin using the software!
sbld: 
sbld:  To see available applications, type "sb_" and hit the TAB key twice.
sbld: 
[last command took 0.41 seconds to complete] $>

All went well in this case! If you had issues, you could go and look in the directory listed in the output (the Trouble info column) for clues as to what went wrong. If there was a problem with a reference log (the Log-diff column), you could also use the sb_core_reflogupdate command to check what was wrong (just don’t actually update the log files unless you are the developer maintaining them).