Camlp5 sources
Information for developpers of the Camlp5 program.
Kernel
The sources are composed of:
- the OCaml stuff, copied from the OCaml compiler
- the kernel composed of the directories:
- odyl : the dynamic loading system
- lib : the library
- main : the main program camlp5
- meta : the parsers for revised syntax, ast quotations, EXTEND statement, etc/
- the rest: directories etc, compile, ocpp
Some other directories contain configuration files, tools, documentation and manual pages.
The kernel is sufficient to make the core system work: it is possible to compile and bootstrap only it. All sources being in revised syntax, the first compilation of Camlp5 is done by a version of this kernel in pure OCaml syntax, located in the directory ocaml_src.
These sources in pure OCaml syntax are not modified by hand. When changes are made to the kernel, and a check is done that it correctly compiles and bootstraps, the kernel in pure OCaml syntax is rebuilt using Camlp5 pretty print. This is done by the command "make bootstrap_sources".
Compatibility
This distribution of Camlp5 is compatible with several versions of OCaml. The definition of OCaml syntax trees may change from OCaml version to version, which can be a problem. Since OCaml does not install the sources nor the compiled versions of its syntax tree, a copy of the necessary source files, borrowed from the source of the OCaml compiler is in the directory 'ocaml_stuff', in subdirectories with the OCaml version number.
If the present distribution of Camlp5 is not compatible with the version of OCaml you have (the command 'configure' tells you), it is possible to add it. For that, you need the sources to your specific OCaml distribution. If you have them then a 'configure' telling you that camlp5 is not compatible, do:
make steal OCAML_SRC=<path-to-OCaml-sources>
This creates a new directory in 'ocaml_stuff' with sources of the syntax tree of your OCaml compiler.
If you want to check that the sources of the syntax tree of OCaml are up-to-date (e.g. if this is the current OCaml developpement), do:
make compare_stolen OCAML_SRC=<path-to-OCaml-sources>
The compatibility is also done with the file 'main/ast2pt.ml', which is the module converting Camlp5 syntax tree into OCaml syntax tree.
In the directory 'ocaml_src' which contains the pure OCaml sources of the Camlp5 core (see chapter TREE STRUCTURE below), there are as many versions of this files as versions of OCaml. They are named 'ast2pt.ml_<version>'. If you are adding a new version of OCaml, you need this file. As a first step, make a copy from a close version:
cd ocaml_src/main cp ast2pt.ml_<close_version> ast2pt.ml_<version>
Then, you can rerun "configure" and do "make core". If the file 'ocaml_src/main/ast2pt.ml' has a compilation problems, fix them and to 'make core' again.
Later, the same file 'main/ast2pt.ml' in Camlp5 syntax may have similar compilation problems. There is only a single version of this file, thanks to IFDEF constructs used here or there.
While compiling with some specific version of OCaml, this file is compiled with 'OCAML_vers' defined where 'vers' is the version number form the beginning to the first space or charcter '+' with all dots converted into underscores. For example, if your OCaml version is 7.04.2+dev35, you can see in the compilation process of ast2pt.ml that OCAML_7_04_2 is defined, and you can add statements defined by the syntax extension 'pa_macro.cmo', for example IFDEF OCAML_7_04_2. Add statements like that in 'main/ast2pt.ml' to make it compile successfully.
Tree structure
The directory 'ocaml_src' contains images in pure OCaml syntax of the directories odyl lib main and meta. This allows the creation of a core version of Camlp5 with only the OCaml compiler installed.
You can decompose the building of the Camlp5 core into:
- 1. make library_cold
- just makes the directory 'ocaml_src/lib' and copy the cmo and cmi files into the directory 'boot'
- 2. make compile_cold
- makes the other directories of ocaml_src
- 3. make promote_cold
- copies the executables "camlp5", "camlp5r" and the syntax extensions (cmo files) into the directory 'boot'
From this point, the core Camlp5 is in directory 'boot'. The real sources in the top directories odyl, lib, main and meta, which are written in revised syntax with some syntax extensions (grammars, quotations) can be compiled. To achieve their compilation, you can do:
make core
Or to compile everything do:
make all
or just:
make
Notice that doing "make core" or "make all" from scratch (after a make clean), automatically starts by making the core files from their pure OCaml versions.
Fast compilation from scratch
./configure make clean core compare make coreboot make all opt opt.opt
Testing changes
1. do your changes
2. do:
make core compare
if it says that the bootstrap is ok, you can do:
make all make opt make opt.opt
otherwise, to make sure everything is ok, first do:
make coreboot
sometimes two bootstraps ('make coreboot' twice) are necessary, in particular if you change things in the directory 'lib'. It is even possible that three bootstraps are necessary.
If things go wrong, it is possible to return to the previous version by typing:
make restore clean_hot
then you can change what is necessary and continue by typing:
make core
and test the bootstrap again:
make coreboot
After several bootstraps (by 'make coreboot' or 'make bootstrap'), many versions are pushed in the directory 'boot' (you can type 'find boot -type d -print' to see that). If your system correctly bootstraps, you can clean that by typing:
make cleanboot
which keeps only two versions. (The command 'make clean' also removes these stack of versions.)
Before committing your changes
Make sure that the cold start with pure OCaml sources work. For that, do:
make compare_sources | less
This shows you the changes that would be done in the OCaml pure sources of the directory ocaml_src.
To make the new versions, do:
make new_sources make promote_sources
Notice that these pure OCaml sources are not supposed to be modified by hand, but only created by the above commands. Although their source is pretty printed they are usually not easy to read, particularly for expanded grammars (of the statement 'EXTEND').
If these sources do not compile, due to changes in the OCaml compiler, it is possible however to edit them. In this case, similar changes may need to be performed in the normal sources in revised syntax.
After doing 'make new_sources' above, and before doing 'make promote_sources' below, it is possible to do 'make untouch_sources' which changes the dates of the newly created files with the dates of the old files if they are not modified. This way, the "svn commit" will not need to compare these files, which may be important if your network is not fast.
The 'make new_sources' builds a directory named 'ocaml_src.new'. If this directory still exists, due to a previous 'make new_sources', the command fails. In this case, just delete it (rm -rf ocaml_src.new) without problem: this directory is not part of the distribution, it is just temporary.
The 'make clean_sources' deletes old versions of ocaml_src, keeping only the last and the before last ones.
The command:
make bootstrap_sources
is a shortcut for:
make new_sources make untouch_sources make promote_sources make clean_sources
If there are changes in the specific file 'main/ast2pt.ml', do also:
make compare_all_ast2pt
and possibly:
make bootstrap_all_ast2pt
because this file, in 'ocaml_src/main' directory has different versions according to the OCaml version.
After having rebuilt the pure OCaml sources, check that they work by rebuilding everything from scratch, starting with "configure".
If you change the main parser
If you change the main parser 'meta/pa_r.ml', you should check that the quotations expanders of syntax tree 'meta/q_MLast.ml' match the new version. For that, do:
cd meta make compare_q_MLast
If no differences are displayed, it means that 'q_MLast.ml' is ok, relatively to 'pa_r.ml'.
Otherwise, if the displayed differences seem reasonable, update the version by typing:
make bootstrap_q_MLast
Then returning to the top directory, do 'make core compare' and possibly 'make coreboot' (one of several times) to check the correctness of the file.
And don't forget, if you want to commit, to re-create the pure OCaml sources like indicated above.
Switching between transitional and strict mode
If Camlp5 is compiled in some mode, it is possible to change its mode in two boostrapping steps. Type:
make MODE=T coreboot
to switch to transitional mode, or:
make MODE=S coreboot
to switch to strict mode.
After two (necessary) bootstraps, the kernel is compiled in the new mode. Complete the compilation by:
make MODE=T all opt opt.opt
or:
make MODE=S all opt opt.opt
according to the new mode you want to use.
Another solution is, of course, recompile everything from scratch:
make clean ./configure -transitional make world.opt
or:
make clean ./configure -strict make world.opt↑
