Altering callmaps

Intro

One of the first things most contributors start with is proposing changes to callmaps.

Callmap is a data file (formatted as a PHP file returning an array) that tells Psalm what arguments function/method takes and what it returns. Psalm loads dictionaries/CallMap_<PHP major version><PHP minor version>.php for the analyzed PHP version (CallMap_84.php for PHP 8.4).

These files are generated: never edit them by hand. bin/stubs/gen_callmap.php builds them from two sources:

  • dictionaries/autogen/CallMap_*.php: the signatures read by reflection from PHP and a set of extensions, one file per PHP version;
  • dictionaries/override/: hand written corrections. CallMap.php holds the signatures of the newest PHP version, and the delta files CallMap_*_delta.php hold how they changed in each version.

An override entry wins over the reflected one. Overrides equal to the reflected signature in every version are dropped.

Changing a signature

  1. Edit the entry in dictionaries/override/CallMap.php (to add a missing one, copy it from dictionaries/autogen/CallMap_$latest.php).
  2. Run php -d memory_limit=-1 bin/stubs/gen_callmap.php. It regenerates dictionaries/CallMap_*.php and rewrites the delta files.

The edit applies to every older version up to the first delta that changes the function. Narrowing the parameter of array_key_first this way changes CallMap_80.php to CallMap_86.php and the new side of its changed entry in CallMap_80_delta.php, while CallMap_74.php keeps the old side. To change an older signature, edit the old side of the entry in that delta.

Regenerating the reflected callmaps

bin/stubs/gen_callmap.sh (composer gen_callmap, part of composer build) runs bin/stubs/gen_base_callmap.php inside one Docker image per PHP version (ghcr.io/<owner>/psalm:internal_stubs_<version>), then gen_callmap.php. bin/stubs/build_docker.php builds those images and generates their Dockerfile on the fly.

To add the types of an extension, add it to EXTENSIONS in bin/stubs/build_docker.php (or to INCLUDES, for some PHP versions only) with its version source in SOURCES. PIE_PACKAGES lists the extensions installed with PIE, and EXCLUDES the ones that don't build on a PHP version yet.

Every push runs the "Build docker image for stubs" workflow: it builds the images, runs composer build, and when a committed file is out of date, commits the difference to the pushed branch and uploads it as the composer-build-patch artifact. Pull before pushing again. If the commit can't be pushed, bin/ci/apply-composer-build-patch.sh applies the artifact to your checkout (set PSALM_REPO=<you>/psalm for a run on your fork).

Checking a callmap against reflection

composer verify-callmap runs tests/Internal/Codebase/InternalCallMapHandlerTest.php, which compares the callmap with what reflection of the installed PHP and its extensions says.

Functions that don't pass yet are listed in that test ($ignoredFunctions, and for return types only $ignoredReturnTypeOnlyFunctions). To fix one:

  1. Remove its entry from the ignore list and run composer verify-callmap.
  2. Fix the errors it reports in the callmap (and in the deltas, if the signature changed in an older version).
  3. Run composer verify-callmap again, then open the PR.

An entry that is already right must not be on the list: the test fails with Remove 'name' from InternalCallMapHandlerTest::$ignoredFunctions.

Full callmap format

Each entry keeps its analysis (or "doc") return type at index 0. The optional index -1 records the native or tentative return type extracted from PHP:

'strlen' => [
    0 => 'int<0, max>',
    -1 => 'int',
    'string' => 'string',
],
'amqpvalue::toamqpvalue' => [
    0 => 'mixed',
],
'jsonserializable::jsonserialize' => [
    -1 => 'mixed',
    -2 => 'tentative',
],
'limititerator::getposition' => [ // PHP 8.0: no native or tentative return signature
    0 => 'int',
    -2 => 'none',
],

Only the signature return type imposes a native return-type requirement on an implementation. -1 => 'mixed' is an explicit signature, not an absent type. Optional index -2 records its provenance:

-1 -2 Meaning
present absent Native return signature
present tentative Tentative return signature, available since PHP 8.1
absent none Reflection confirmed there is no return signature in this PHP version
absent absent Not reflected by the generator; the stub or runtime may provide a signature

#[ReturnTypeWillChange] suppresses tentative return-signature checks, including an explicitly incompatible return type. It does not suppress a native signature. Stubs and index 0 still provide the types used to analyse calls; PHPDoc compatibility checks are separate from native signature checks.

Generated files omit index 0 when its type equals index -1; the signature then also supplies the analysis return type. Read it with $entry[0] ?? $entry[-1]; getCallMap() preserves the compact entries without recreating index 0.

Edit the analysis return type in the overrides, then regenerate. Signature return types come from reflection and are retained independently of overrides and alternate signatures; the generator does not copy analysis refinements into them.

Reflection coverage

Each image must load every extension in the version-specific manifest returned by php bin/stubs/build_docker.php --extensions-plain <version>. Generation fails before writing a callmap if its PHP version or a required extension is wrong. dictionaries/autogen/CallMapExtensions_*.php records the extension of each reflected API. These manifests describe coverage; they do not list every PHP extension. For example, the current images do not load bz2, calendar or dba. Their handwritten entries can therefore lack reflected signature metadata.

The merger preserves overrides for APIs absent from an image. An untyped downloaded PHPStorm stub does not establish that an extension has no native return type. For an API without generated metadata, Psalm first uses its stub signature. Runtime reflection can supply a missing signature only when the analysed PHP major/minor matches the PHP running Psalm; it never imports a newer runtime signature into an older target. Results for these uncovered APIs can consequently depend on the installed extensions and their versions. A different target version needs a stub or generated metadata for that version.

A callmap (dictionaries/CallMap_*.php, and dictionaries/override/CallMap.php) has function/method names as keys and an array representing the corresponding signature as a value.

Entries with string keys represent function/method parameters. Parameter name for an optional parameter is postfixed with =, references with &/&r_/&w_/&rw_ (depending on the read/write meaning of the reference param) and variadic args are prefixed with ....

Callmaps also support function aliases: aliases are very useful to specify that a certain function behaves differently according to the parameter types.

For example, the following aliases are currently in use in the callmap to specify that the version_compare function can be called with an operator parameter, in which case it will return a boolean; otherwise it will return an integer.

[
    'version_compare' => [
        0 => 'bool',
        'version1' => 'string',
        'version2' => 'string',
        'operator' => '\'!=\'|\'<\'|\'<=\'|\'<>\'|\'=\'|\'==\'|\'>\'|\'>=\'|\'eq\'|\'ge\'|\'gt\'|\'le\'|\'lt\'|\'ne\'|null',
    ],
    'version_compare\'1' => [
        0 => 'int',
        'version1' => 'string',
        'version2' => 'string',
    ],
]

Note: the above example doesn't provide almost any useful information for type inference; in fact, the real logic for return type inference is contained in the VersionCompareReturnTypeProvider.

The callmap is mainly useful when treating functions and methods as callable values, for example:

<?php
function naive_version_compare(string $a, string $b, ?string $operator = null): int|bool {
    return 0;
}

$a = ["1.0", "2.0"];

// OK
usort($a, "version_compare");

// InvalidArgument: Argument 2 of usort expects callable(string, string):int, but impure-callable(string, string, null|string=):(bool|int) provided
usort($a, "naive_version_compare");

The first usort succeeds, because psalm chooses the correct alias to use between the two provided in the callmap.
The second usort fails (equivalent to the non-split return type of version_compare inferred by reflection), because the return type is a union of the two possible signatures of version_compare.

When you have multifaceted functions like these, it's a very good idea to at least define a templated stub in stubs/ for them, or a custom return type provider for even more complex logic, not representable with templates/conditional types/etc in a stub.

Also note that bin/stubs/gen_callmap.php has some validation logic which will re-add back removed parameters in overridden aliased callmaps: to avoid this, explicitly whitelist aliased functions by editing assertParameter in bin/stubs/gen_callmap_utils.php, and eventually bin/stubs/gen_callmap.php as needed.

Delta file format

Delta files (named CallMap_<PHP major version><PHP minor version>_delta.php) list changes that happened in the corresponding PHP version. There are three section with self-explanatory names: added (for functions/methods that were added in that PHP version), removed (for those that were removed) and changed.

Entry format for removed and added section matches that of a full callmap, while changed entries list old and new signatures.

How the generator uses delta files

bin/stubs/gen_callmap.php starts from dictionaries/override/CallMap.php (the newest PHP version) and applies each delta in reverse to get the overrides of the version before it: entries in added are removed, those in removed are added back, and changed.new signatures are replaced with changed.old. It merges the overrides of each version over the reflected callmap of that version, then rewrites the delta files from the result, so they always match override/CallMap.php.

The "Build docker image for stubs" workflow fails when a committed callmap differs from what the generator produces.

Adding a function

The functions and methods of PHP and of the extensions in the stub images come from reflection, so a new one reaches the callmaps without an override: array_is_list() (PHP 8.1) has no override entry. Add one only to refine its types, as described in Changing a signature.