Python modules with sbldpy (and how to migrate)

The compiled Python modules in dgcode are being moved from pycpp_* subdirectories (based on pybind11) to pymod_* subdirectories, which are based on the much simpler sbldpy approach of simplebuild. Modules based on sbldpy are faster to compile and do not depend on the version of Python. New code should therefore use pymod_* subdirectories.

Existing pycpp_* modules in your own projects will keep working for now, and the two kinds of modules can be used together (for instance, a geometry from a pycpp_* module can be used together with a generator from a pymod_* module in the same simulation). However, support for pycpp_* modules will be removed in a future release, so you are encouraged to migrate your code as described below.

Geometry, generator and filter modules

Most projects only have compiled Python modules which provide a geometry, a particle generator or a step filter, like the GeoSkeletonSP geometry used in new projects created with dgcode_newsimproject. Such a module can be migrated with a few simple steps, here shown for a geometry module GeoMyStuff in a package G4GeoMyStuff:

  1. Rename the directory G4GeoMyStuff/pycpp_GeoMyStuff/ to G4GeoMyStuff/pymod__GeoMyStuff/ (note the two underscores, which makes it a private module named _GeoMyStuff).

  2. In the code, include G4Interfaces/GeoConstructToPy.hh instead of G4Interfaces/GeoConstructPyExport.hh, and replace:

    PYTHON_MODULE( mod )
{
  GeoConstructPyExport::exportGeo<GeoMyStuff>(mod, "GeoMyStuff");
}

    with:

    SBLDPY_MODULE( _GeoMyStuff ) { G4Interfaces::geoToPy<GeoMyStuff>( m ); }

  3. Add a file G4GeoMyStuff/python/GeoMyStuff.py, providing the Python module G4GeoMyStuff.GeoMyStuff with the usual create() function:

    from ._GeoMyStuff import create_handle as _create_handle
from G4Interfaces.ToPy import component_creator as _component_creator
create = _component_creator( _create_handle )

That is all. Python code using the module (e.g. geo = G4GeoMyStuff.GeoMyStuff.create(), setting parameters like geo.thickness_mm = 2.0, or calling methods like geo.dump()) works as before, and the C++ code of the geometry class itself is unchanged.

Generators and step filters are migrated in exactly the same way, using G4Interfaces/ParticleGenToPy.hh and G4Interfaces::genToPy (instead of G4Interfaces/ParticleGenPyExport.hh and ParticleGenPyExport::exportGen), or G4Interfaces/StepFilterToPy.hh and G4Interfaces::filterToPy (instead of G4Interfaces/StepFilterPyExport.hh and StepFilterPyExport::exportFilter).

If the pybind11 based module added extra methods to the Python objects, like:

PYTHON_MODULE( mod )
{
  GeoConstructPyExport::exportGeo<GeoMyStuff>(mod, "GeoMyStuff")
    .def("detectorArea",&GeoMyStuff::detectorArea);
}

the same can be achieved with:

SBLDPY_MODULE( _GeoMyStuff )
{
  G4Interfaces::geoToPy<GeoMyStuff>( m )
    .def( "detectorArea", &GeoMyStuff::detectorArea );
}

The methods must be member functions of the class, taking and returning values of the types supported by sbldpy (except that any integer type can be used, and that parameters can be of type float).

The objects created by such modules are instances of the Python class G4Interfaces.ToPy.Component, which provides the same methods for accessing, tying and exposing parameters as the pybind11 based objects (and they can be used together with those). One difference is that they are not instances of the pybind11 based classes, so code which checks the type of these objects (e.g. with isinstance(geo,G4Interfaces.GeoConstructBase)) must be updated. In particular, pybind11 based code of your own (in pycpp_* folders) which receives such objects from Python must use G4Interfaces::componentFromPy<G4Interfaces::GeoConstructBase>(o) from G4Interfaces/ComponentFromPy.hh (which supports both kinds of objects) rather than e.g. py::cast<G4Interfaces::GeoConstructBase*>(o).

Other modules

Other modules can be migrated by following the sbldpy documentation. Note in particular the following differences compared to pybind11:

  • Integer parameters must have the type std::int64_t.

  • Functions can not have default values of arguments or several overloads. If needed, put the compiled code in a private module (e.g. pymod__impl/, providing the module _impl) and add the missing features in Python code using it (e.g. python/mymodule.py). For examples of this, see GriffFormat.dumpfile or G4DataCollect.

  • A pycpp__init/ module was actually a module named _init, imported automatically by the __init__.py file of the package. A pymod__init/ module directly provides the __init__.py file of the package, so if you also have a python/__init__.py file, use a private module (e.g. pymod__impl/) instead.

  • Objects of C++ classes can not be passed between sbldpy modules. For Geant4 user actions (G4UserSteppingAction and G4UserEventAction) and hooks (G4Interfaces::PreGenCallBack, G4Interfaces::PostGenCallBack and G4Interfaces::HookFct), which are typically passed to the launcher (e.g. with launcher.setUserSteppingAction(obj)), register the objects with G4Interfaces::registerObject from G4Interfaces/CppObjectHandles.hh, and wrap the resulting handles in G4Interfaces.ToPy.CppObject objects on the Python side. For an example, see G4Tests.CppObjTest.

  • Similarly, SimpleHists histograms and collections are shared between modules with handles (see SimpleHists/Handles.hh). The Python histograms and collections have a handle() method, and a C++ function receiving the handle (as a std::int64_t parameter) can access the very same object with SimpleHists::getHist(handle) or SimpleHists::getCollection(handle), which return a std::shared_ptr. Conversely, C++ code can register a histogram or collection with SimpleHists::registerHist or SimpleHists::registerCollection and return the handle, which Python code can turn into a Python object with SimpleHists.histFromHandle(handle) or SimpleHists.HistCollection.fromHandle(handle). For an example, see SimpleHistsTests._handletest.

  • The launcher is an sbldpy based module, and Python’s global interpreter lock (GIL) is released while it runs a simulation. Therefore, if C++ code of your own in pycpp_* modules calls Python code (or otherwise uses Python objects) during a simulation, for instance from a particle generator, it must first acquire the GIL by adding the line py::gil_scoped_acquire gil; at the beginning of the functions in question. Python code called via sbldpy callbacks (like launcher hooks) does not need this.