Development Workflow¶

CCL development proceeds via pull requests on GitHub. In short, the code base is forked on GitHub, a new change is pushed to a new branch, and then a pull request is issued by the developer. Once a pull request is issued, CCL uses Travis-CI to automatically run the benchmark/unit tests and the linter. Please read the sections below for guidance on how to modify the source code, test, and then document your modifications.

Working on the Python library¶

After completing the Developer Installation, you can change the Python library in your local copy of CCL. Any changes you make will be visible in Python after you restart your interpreter. A typical workflow is to make some change, rerun a failing benchmark or unit test, etc. until you’ve completed your new feature and/or fixed a bug. See Writing and Running Benchmarks and Writing and Running Unit Tests for instructions on how to run these test suites. Please add unit tests and benchmarks as appropriate for any new functionality. You should also make sure to run the linter by following the instructions in Linting with flake8. Finally, after you are satisfied with your new code, make sure to follow Updating the Changelog and Documentation in order to the properly document your new additions.

Adding New C-level Features to CCL¶

Before adding new features to the CCL C code, please read Understanding the Python-C Interface in order to understand how the Python and C codes interact.

The following steps are needed to add a new source file ccl_newfile.c and header ccl_newfile.h to the library.

Put ccl_newfile.c in the src directory.
Put ccl_newfile.h in the include directory.
Make sure to add a C-level include of your new header to include/ccl.h.
Add the new source file ccl_newfile.c to the CCL_SRC variable in CMakeLists.txt.
Write a ccl_newfile.i SWIG interface file for the new functions. This file gets placed in the pyccl directory. You may need to consult another CCL developer for help with the SWIG interface files.
Add your new SWIG interface file ccl_newfile.i to the list of includes in the pyccl/ccl.i file.
Write the public Python APIs for your new function in a file called pyccl/newfile.py. This Python package should access the C code you wrote from the new functions in pyccl.ccllib.
Import your new functions into the pyccl/__init__.py file as needed.
Add unit tests and benchmarks as appropriate for the new code. See Writing and Running Benchmarks and Writing and Running Unit Tests for details.
Make sure to run the linter by following the instructions in Linting with flake8.
Follow Updating the Changelog and Documentation in order to the properly document your new additions.

Occasionally, modifications made correctly as described above will still not function properly. This might be due to multiple pyccl installation files not being properly overwritten, and the Python interpreter getting confused. At this point it is often best to resort to the “nuclear option” of deleting all Python files related to pyccl and starting from scratch. The procedure is as follows:

Execute pip uninstall pyccl (possibly multiple times) to remove all installed versions of pyccl.
Delete the compiled C files.
- Execute rm -r build/ to delete the build directory.
- Execute rm pyccl/ccl_wrap.c pyccl/ccllib.py to delete the autogenerated SWIG files.
- Check various locations for the Python site-packages directory for any pyccl files and delete them.
Reinstall CCL according to the Developer Installation instructions.

Updating the Changelog and Documentation¶

Please follow the following guidelines for documenting changes to CCL:

New features and bug fixes should be documented in the CHANGELOG.md with a brief description and the GitHub pull request number.
Any new derivations/math essential to understanding a new CCL feature should be documented in the CCL note. See :ref:cclnote for instructions on how to modify and compile it.
All additions to the Python public API should have Python docstrings. These are written in Sphinx compatible format so that they can be incorporated into the CCL Read the Docs pages. See the current Python docstrings in the Python source for examples.
Additions to the C code should be documented/commented so that other CCL developers can understand the code.

Linting with `flake8`¶

CCL uses flake8 to ensure that the Python code has a consistent style. flake8 is available via pip or conda via [pip|conda] install flake8. You can run this tool locally by executing

$ flake8 pyccl

Any problems will be printed to STDOUT. No output indicates that flake8 has succeeded.

Debug mode in Python¶

Because of the way the Python wrapper handles exceptions that occur inside the C code, by default users will only see error messages for the most recent error that occurs. If multiple errors occurred during a CCL function call, all but the most recent error message will be overwritten. This convention can make it difficult to debug the root cause of a problem.

To help with debugging this kind of issue, you can enable debug mode in the Python wrapper. To do so, simply call pyccl.debug_mode(True). This will cause the Python wrapper to print all C error messages to STDERR whenever they occur. Python exceptions will only be raised for the most recent error, as before. (Note that Jupyter notebooks do not print C STDERR messages by default.)

Continuous Integration with Travis-CI¶

Travis-CI is a continuous integration service that reads the file .travis.yml file in the repository and then runs the benchmarks/unit tests. More details on Travis-CI can be found here: https://docs.travis-ci.com/user/getting-started/.

Every time you push a commit, Travis-CI will automatically try to build the libraries with your new changes and run the benchmark/unit tests. You can check the status of your builds by following the links from the pull request page. If your build errors or fails, you can scroll through the log to find out what went wrong. Warnings from flake8 will result in the tests not passing. If your additions require new dependencies, make sure that you include them in the conda environments defined in .travis/install.sh.

Deploying a New Release¶

When cutting a new release, the procedure is as follows:

Make sure any API changes are documented in CHANGELOG.md
Commit to master
Create a new release from the GitHub interface here: https://github.com/LSSTDESC/CCL/releases/new
Manually create a source distribution from the root CCL folder:
```
$ python setup.py sdist
```
This command will create a .tar.gz file in the dist folder.
Upload source distribution to PyPi using twine (can be installed using pip or conda):
```
$ twine upload  dist/pyccl-x.x.x.tar.gz
```
Make sure your twine and setuptools packages are up to date, otherwise the markdown formatting of the README.md will not be correctly processed on the CCL PyPi page.
The conda-forge automated release bots will detect the new PyPi release and automatically make a pull request on the CCL feedstock. Once this pull request is merged, the new CCL release will be available on conda after a few hours.
Rebuild and redeploy the Read the Docs pages per the instructions in Building the Read the Docs Documentation. Note that you may need to adjust the major version number in readthedocs/conf.py if the new version has a major version number bump.

A CCL administrators will need to complete the steps above.

Building the `Read the Docs` Documentation¶

To build the Read the Docs documentation, follow the following steps:

$ cd readthedocs
$ make clean
$ make html

You can then inspect the outputs in readthedocs/_build/index.html to make sure the formatting is correct. Finally, contact the CCL administrators to redeploy the live documentation.

Building the CCL Note¶

The CCL note is a latex’ed documented located in doc/0000-ccl_note. It is used to document the scientific content of the CCL library. Note that documentation of the actual APIs and code should reside in the Python doc strings and other code comments.

To compile the CCL note, type make in the doc/0000-ccl_note directory.

If you need to modify the note, the files to modify are:

authors.csv: To document your contribution.

main.tex: To detail the changes to the library.

main.bib: To add new references.