Directory structure for manuals is broken

[frankluebeck]

I did not find information how the Manuals directory (wherever it is located physically) is created.

Currently, package documentation seems to be in locations like:

Manuals/pkg/GAPDoc/doc/....

But when package documentation is built for a release then the relative links to documentation in other packages contains the directory names from the distribution, say

Manuals/pkg/GAPDoc-1.6.4/doc/...

such that corresponding links are broken on the GAP website.

To make sure that links between manuals work the directory structure for all manuals should be the same as in a GAP installation (or it should include the structure from an installation, say, by adding appropriate symbolic links).

[wilfwilson]

Thank you for the report and the suggestions.

For example, in this manual entry for the Digraphs package, the link to FileString in the GAPDoc manual is https://www.gap-system.org/Manuals/pkg/GAPDoc-1.6.1/doc/chap6_mj.html#X7E14D32181FBC3C3, which no longer resolves.

The Manuals directory is currently constructed with the script etc/extract_manuals.py of this repository: https://github.com/gap-system/GapWWW/blob/master/etc/extract_manuals.py

This script extracts the manuals from an installation of the appropriate GAP release. It deliberately removes the version number from the end of the directory name of each package. That is how the problem is being introduced.

[frankluebeck]

Thanks for the hint to etc/extract_manuals.py.

Links between help books were quite robust, even over several package versions or GAP releases, when packages were located in directories pkg/[standardized lower case package name]. But now most package are put in directories whose name changes between versions. This makes cross links invalid, as in the mentioned example in the digraphs package whose documentation was build with an older GAP version.

Should we maybe introduce some symbolic links which provide such outdated paths? I attach a python script which creates such links. It seems to work inside the Manuals directory which is used on the GAP website. But it also works in the GAP root of the GAP distribution (the same problem with the discussed link exists, e.g., in GAP 4.11.1 which contains a newer GAPDoc version).

REMOVED old version of pkglinks.py, see below for updated version.

[ThomasBreuer]

The broken HTML links between manuals on the GAP website are one problem. It can be fixed (after each release of new package versions) by inserting symbolic links to package directories, as suggested by Frank.

(One might argue whether such a solution is logically correct, but in practice an HTML link to something in an old package version is usually valid also in newer versions of this package.)

The same problem is present also in any local GAP installation, and here the situation is worse because it is likely that individual packages get updated or newly installed, for example via the PackageManager package.
Then also the following may happen:

• I install a new version of a package A which contains an HTML link to a version of package B that is newer than the installed version of B.
• Some mechanism fixes this link problem by creating a symbolic link that points to the old version of B, but with the new version number in the directory name.
• Later I try to install also the new version of B. The directory for this version is already available via the symbolic link, but the contents of this directory belongs to the old version. What will happen: Will the files of the new version overwrite the old ones?

In this sense, deliberately adding symbolic links with "wrong" directory names may cause new problems.
Besides this, I would find it confusing if directory names claim to belong to package versions that are in fact not present.
Note that the InstallationPath values listed in entries of GAPInfo.PackagesInfo can be symbolic links.

[fingolfin]

We actually discussed hosting multiple versions (ideally: all past distributed versions) of all package manuals (like Python, Julia, and many others do), but this is currently a problem due to the required storage space -- the PDFs in particular sum up to many gigabytes of data. If we restricted this to the HTML manuals alone, though, it would be doable. Similar for the GAP manual itself: i.e., instead of only hosting the current version, we could also host older versions (either all; or for each 4.x series the latest).

We can discuss this some more during the upcoming GAP Days. If hosting is moved from St Andrews to Kaiserslautern, storage won't be an issue either.

[frankluebeck]

Of course, I agree with @ThomasBreuer that introducing links with misleading names is no optimal solution.

Nevertheless, here is an updated version of a Python script that scans all html-files in a GAP root directory, or in the main directory of the documentation on the GAP website, and adds symbolic links in the pkg/ subdirectory that make almost all cross references between manuals work as expected. This version of the script also adds a file RemoveTemporarySymLinks in the pkg/ directory which can be called to remove all the added symbolic links.

pkglinks.py.gz

PS: I hope that the two meanings of "link" in this context are not confusing...

[ThomasBreuer]

As was mentioned today, the problem with cross-references refers to the HTML files; the PDF files do not contain hyperlinks to other documents.
Thus we could decide to keep older versions of HTML documentation but provide only the latest PDF version.
One disadvantage would be that users will likely be guided to older versions when they just follow the hyperlinks:
Starting from a recent version of package A, one gets to an old version of package B, and a link from B to A will point to an older version of A than where one has started.

Concerning the alternative solution to regenerate the documentation of all packages centrally after each release, this causes problems whenever the build process for a package documentation involves external programs. For example, building the full documentation for the CTblLib package involves the creation of some pictures for the HTML pages.
Perhaps it would eventually be desirable to require that one can build the package documentation "centrally", but currently there is a problem.

Another problem (perhaps another issue?):
Currently the package documentation that is accessible from https://www.gap-system.org/Packages/packages.html seems to contain only one book per package; GAPDoc allows for several books.
(This problem was solved with /pull/266.)

[fingolfin]

The problem should be fully fixed now: the server now automatically redirects
these URLs by replacing PackageName-VERSION by packagename using a regex and Apache's mod_rewrite module. So e.g.

https://www.gap-system.org/Manuals/pkg/GAPDoc-1.6.3/doc/chap0.html
https://docs.gap-system.org/pkg/GAPDoc-1.6.3/doc/chap0.html

both are automatically redirected to

https://docs.gap-system.org/pkg/gapdoc/doc/chap0.html

(The use of a subdomain is also new, see issue #255, but mostly tangential for this issue).