********************
Development Workflow
********************

CCL development proceeds via pull requests on GitHub. In short, the code base
is forked on GitHub, a new change is pushed to a new branch, and then a pull request
is issued by the developer. Once a pull request is issued, CCL uses Github Actions to
automatically run the benchmark/unit tests. Note that we also require any new ``Python``
code to be PEP-8-compliant (see below). Please read the sections below for guidance on
how to modify the source code, test, and then document your modifications.


Working on the Python library
=============================

After completing the :ref:`devinstall`, you can change the ``Python`` library
in your local copy of CCL. Any changes you make will be visible in ``Python``
after you restart your interpreter. A typical workflow is to make some change,
feature and/or fixed a bug. See :ref:`benchmarks` and :ref:`unittests` for instructions
on how to run these test suites. Please add unit tests and benchmarks as appropriate for
any new functionality. You should also make sure to run the linter by following
the instructions in :ref:`flake8`. Finally, after you are satisfied with your new
code, make sure to follow :ref:`updocs` in order to the properly document your
new additions.

For a quick development cycle:

.. code-block:: bash

   # 1. Active environment
   $ conda activate test

   # 2. Make changes to Python or C files

   # 3. If C files changed, rebuild
   $ pip install -v -e .

   # 4. Run relevant tests
   $ pytest -vv pyccl/tests/test_yourmodule.py

   # 5. Run linting
   $ flake8 pyccl/

   # 6. Full test suite before commit
   $ OMP_NUM_THREADS=2 pytest -vv pyccl benchmarks


Adding New Features to CCL
==========================

Before adding new features to the CCL ``C`` code, please read :ref:`pycint` in
order to understand how the ``Python`` and ``C`` codes interact. Most often you will
**not** need to use ``C`` at all, in which case you can skip to step 7 below.

The following steps are needed to add a new source file ``ccl_newfile.c`` and
header ``ccl_newfile.h`` to the library.

#. Put ``ccl_newfile.c`` in the ``src`` directory.
#. Put ``ccl_newfile.h`` in the ``include`` directory.
#. Make sure to add a C-level include of your new header to ``include/ccl.h``.
#. Add the new source file ``ccl_newfile.c`` to the ``CCL_SRC`` variable in
   ``CMakeLists.txt``.
#. Write a ``ccl_newfile.i`` ``SWIG`` interface file for the new functions. This
   file gets placed in the ``pyccl`` directory. You may need to consult another
   CCL developer for help with the ``SWIG`` interface files.
#. Add your new ``SWIG`` interface file ``ccl_newfile.i`` to the list of includes
   in the ``pyccl/ccl.i`` file.
#. Write the public ``Python`` APIs for your new function in a file called
   ``pyccl/newfile.py``. This ``Python`` package should be able to access any
   ``C`` code you wrote (if you did) from the new functions in ``pyccl.ccllib``.
#. Import your new functions into the ``pyccl/__init__.py`` file as needed.
#. Add unit tests and benchmarks as appropriate for the new code. See :ref:`benchmarks`
   and :ref:`unittests` for details.
#. Make sure to run the linter by following the instructions in :ref:`flake8`.
#. Follow :ref:`updocs` in order to the properly document your new additions.


Occasionally, modifications made correctly as described above will still not
function properly. This might be due to multiple ``pyccl`` installation files
not being properly overwritten, and the ``Python`` interpreter getting confused.
At this point it is often best to resort to the "nuclear option" of
deleting all ``Python`` files related to ``pyccl`` and starting from scratch. The
procedure is as follows:

#. Execute ``pip uninstall pyccl`` (possibly multiple times) to remove
   all installed versions of ``pyccl``.

#. Delete the compiled ``C`` files.

   - Execute ``rm -r build/`` to delete the build directory.
   - Execute ``rm pyccl/ccl_wrap.c pyccl/ccllib.py`` to delete the autogenerated
     ``SWIG`` files.
   - Check various locations for the ``Python`` ``site-packages`` directory
     for any ``pyccl`` files and delete them.

#. Reinstall CCL according to the :ref:`devinstall` instructions.


.. _updocs:

Updating the Changelog and Documentation
========================================

Please follow the following guidelines for documenting changes to CCL:

- New features and bug fixes should be documented in the ``CHANGELOG.md``
  with a brief description and the GitHub pull request number.
- Any new derivations/math essential to understanding a new CCL feature should be
  documented in the CCL note. See :ref:``cclnote`` for instructions on how to
  modify and compile it.
- All additions to the ``Python`` public API should have ``Python`` docstrings.
  These are written in ``Sphinx`` compatible format so that they can be
  incorporated into the CCL ``Read the Docs`` pages. See the current ``Python``
  docstrings in the ``Python`` source for examples.
- Additions to the ``C`` code should be documented/commented so that other
  CCL developers can understand the code.


.. _flake8:

Linting with ``flake8``
=======================

CCL uses ``flake8`` to ensure that the ``Python`` code has a consistent style.
``flake8`` is available via ``pip`` or ``conda`` via ``[pip|conda] install flake8``.
You can run this tool locally by executing

.. code-block:: bash

   $ flake8 pyccl

Any problems will be printed to ``STDOUT``. No output indicates that ``flake8``
has succeeded.


Debug mode in Python
====================

Because of the way the ``Python`` wrapper handles exceptions that occur inside
the ``C`` code, by default users will only see error messages for the most recent
error that occurs. If multiple errors occurred during a CCL function call, all but
the most recent error message will be overwritten. This convention can make it
difficult to debug the root cause of a problem.

To help with debugging this kind of issue, you can enable debug mode in the ``Python`` wrapper. To do
so, simply call ``pyccl.debug_mode(True)``. This will cause the ``Python`` wrapper to print all ``C``
error messages to ``STDERR`` whenever they occur. ``Python`` exceptions will only be raised for the most
recent error, as before. (Note that Jupyter notebooks do not print ``C`` ``STDERR`` messages by default.)


Continuous Integration with Github Actions
==========================================

Github Actions (GHA) is a software workflow service used by CCL for continuous integration. Every time
you push a commit, GHA will automatically try to build the libraries with your new changes and run the
benchmark and unit tests. You can check the status of your builds by following the links from the pull
request page. If your build errors or fails, you can scroll through the log to find out what went wrong.
Warnings from ``flake8`` will result in the tests not passing.

What GHA does is dictated by the contents of the ``.github/`` folder in the CCL repo.
Inside that folder you will find an ``environment.yml`` file, which contains a list of all packages
needed by GHA to run the tests. You will need to edit it if any new code you've implemented introduces a
new dependency. The file ``.github/workflows/ci.yml`` then contains the series of steps that GHA will
take to run all tests.


Deploying a New Release
=======================

New releases are now automatically deployed after tagging them on github. Administrators can do so
here https://github.com/LSSTDESC/CCL/releases/new once the new code is in the ``master`` branch.


.. _rtd:

Building the ``Read the Docs`` Documentation
============================================

To build the ``Read the Docs`` documentation, follow the following steps:

.. code-block:: bash

   $ cd readthedocs
   $ make clean
   $ make html

You can then inspect the outputs in ``readthedocs/_build/index.html`` to make
sure the formatting is correct. Finally, contact the CCL administrators to redeploy
the live documentation.


.. _cclnote:

Building the CCL Note
=====================

The CCL note is a latex'ed documented located in ``doc/0000-ccl_note``. It is
used to document the scientific content of the CCL library. Note that documentation
of the actual APIs and code should reside in the ``Python`` doc strings and
other code comments.

To compile the CCL note, type ``make`` in the ``doc/0000-ccl_note`` directory.

If you need to modify the note, the files to modify are:

  - ``authors.csv``: To document your contribution.
  - ``main.tex``: To detail the changes to the library.
  - ``main.bib``: To add new references.