Writing a data source

This guide is for a developer connecting the platform to a federation, a booking system or another league tool. You write one class and its tests; nothing in the core changes. It follows src/Source/Example/ExampleFederationSource.php, a complete connector for a sample federation's API, which you can run before writing a line: in development, System, Data sources lists "Example federation (sample API)", and "Run now" brings its futsal league in.

You'll write:

• The source: src/Source/<Name>/<Name>Source.php, one class.
• Its HTTP client: a few lines in config/packages/framework.yaml.
• Its contract test: tests/Contract/Source/<Name>SourceContractTest.php, a few lines, which runs the checks every source passes.
• Its own tests: how it maps the source's answers, against answers recorded from the real API.

1. Start from the example

Copy src/Source/Example/ExampleFederationSource.php to, say, src/Source/Frf/FrfSource.php, and rename the class and its namespace. Then:

• Remove #[WhenNot('prod')]. It keeps the example out of production; yours is meant for it.
• name() is stored with every record the source brings: short (2 to 40 lowercase letters, digits and dashes), and never changed once a run has used it. frf is a good name.
• label() is what staff see in the admin.

The class is found by itself: implementing App\Source\DataSource (or SendsResults, below) is enough.

2. Connect to the API

Give the source its own HTTP client, with the API's address and credentials, in config/packages/framework.yaml:

framework:
    http_client:
        scoped_clients:
            frf.client:
                base_uri: '%env(FRF_API_URL)%'
                auth_bearer: '%env(FRF_API_TOKEN)%'
                timeout: 30

Add placeholders to .env (FRF_API_URL=https://api.frf.example/v1/ and an empty FRF_API_TOKEN=), and a made-up token to .env.test (FRF_API_TOKEN=test-token) so the source is switched on in tests. The real token goes in the server's environment or .env.local, never in the repository. An API that wants a key in a header takes headers: { X-Api-Key: '%env(FRF_API_KEY)%' } instead of auth_bearer.

The source asks for its client by name, and switches itself on once it's configured:

public function __construct(
    #[Target('frf.client')]
    private readonly HttpClientInterface $api,
    #[Autowire(env: 'FRF_API_TOKEN')]
    private readonly string $token,
) {
}

public function isEnabled(): bool
{
    return $this->token !== '';
}

3. Map what the source has

The import asks the source for six kinds of record, in this order, each from its own method. A method yields the records it has, and returns [] for a kind the source doesn't have (a booking system may have no results; an iCal feed has no players).

Every record is a small read-only class in src/Source, built with its fields in this order.

Ids

• Each record carries the source's own id for it. The pair of the source's name and that id is how a second run finds the record it brought before, so it updates instead of duplicating.
• Ids are stable. Never make one up (a random value, a position in a list): a new id every run is a new record every run. A source without ids builds them from what doesn't change; the spreadsheet source uses a fixture's competition, sides and day.
• Unique within each kind. A team and a fixture may both be "17"; two teams can't. Up to 120 characters.
• Records refer to each other by these ids: a fixture names its competition, its sides and its venue; a result names its fixture. Every record a fixture names comes in the same run.
• A team in several competitions is one team, with every competition's id in competitions.

Sports

A competition's and a team's sport is one of the platform's sport keys (the list is at the end). Map the source's own names with a constant, as the example's SPORTS does. An install can add sports of its own: Settings, Sports in the admin shows each one's key.

Times

A fixture's startsAt is in its own time zone, as the venue's clock reads it. An API that gives UTC is converted:

(new \DateTimeImmutable($match['kickoff']))->setTimezone(new \DateTimeZone('Europe/Bucharest'))

with timeZone set to the same 'Europe/Bucharest'. Don't pass a UTC time with a local time zone: the match would move by hours. The contract test catches it.

Statuses and results

• A fixture's status is scheduled, to_arrange (no date yet), postponed or cancelled. A finished match is scheduled, and its result makes it played. Map the source's states with a constant, as the example's STATUSES does.
• A result's score is the final score: goals like 2-1; sets like 6-4, 7-5; a set's tiebreak like 7-6 (7-5); a match tiebreak like [10-8]; cricket like 245/8 (40) - 198 (36.2). A source's results count as confirmed: they move tables and levels at once. Only send a finished match's score.

Players and squads

A player the source brings for the first time, with a team, joins that team's squad. After that, squads are the platform's: a run never moves a player between teams.

Reading

• Read the API once a run. The import calls the six methods in turn. Keep what a run read, keyed by the run's SourceContext: the example keeps it in a WeakMap, so it's forgotten with the run.
• Read every page. The example follows page until pages.
• Use $context->now, never the clock, for anything that depends on the time ("this season", "the next four weeks"), so tests can fix the time.
• $context->options holds what a run was given, for a source that's given something (below).
• Only read. The import writes what the source returns; a source never saves the platform's records.

4. Problems and failures

• A record you can't map (a state you don't know, a match without a side): note it with $context->problem('Match 90009: ...') and leave it out. The rest still comes in, and the problem shows in the run's log.
• When the source can't be read (unreachable, the token refused, an answer that isn't what the API promises): throw. The HTTP client throws by itself for an error status once you read the answer. The run fails, its log says why, and nothing half-read comes in.

5. Who owns which field

Each run brings the source's version of the fields it owns, over what's there. The platform owns everything else, and no run touches it.

• What a first run creates: a competition comes in as a running league with one division and stage, its teams entered and accepted; fixtures come in published.
• A played fixture stays played, even when the source still lists it as scheduled.
• Detaching: staff can detach a record from its source in the admin (a fixture moved by hand, say). Runs then leave it as edited, until staff hand it back.

6. Sending results back

A source that keeps its own tables, like a federation, can take the results confirmed on the platform. Implement App\Source\SendsResults instead of DataSource, and its sendResult():

public function sendResult(Fixture $fixture, Result $result): void
{
    $score = $result->getScore()->parts[0];
    $this->api->request('POST', sprintf('matches/%s/result', $fixture->getSourceId()), [
        'json' => ['home' => $score->home, 'away' => $score->away],
    ])->getContent();
}

• When it's called: once a result for one of the source's fixtures is confirmed on the platform (both captains agreeing, an organiser confirming, or the confirmation deadline passing), in the background.
• Never for a result the source sent.
• A failure throws: the sending is retried three times, a little later each time, and each failure is logged.
• $fixture->getSourceId() is the source's id for the match. $result->getScore()->parts holds each part's home and away (a set each, for sports with sets), and $result->getScoreLine() the score as text.

7. A source that's given something

A source that reads what staff give it each time (a spreadsheet uploaded in the admin) implements App\Source\NeedsInput. Its inputRoute() names the admin page where staff give it, and what they gave reaches the source in $context->options. Such a source doesn't run on a schedule or from "Run now". src/Source/Csv/CsvSource.php is one.

8. Test it

Record the API's answers

Save real answers from the API (its sandbox, or a request with your token) as JSON files, one per request, and take out personal data you don't need. In tests, the source's client answers from them:

• A class answering from your recordings, like src/Source/Example/SampleFederationApi.php: copy it to tests/Support/FrfApi.php, put your answers in a folder beside it (named in recorded()), set its BASE_PATH to the path of your API's address (/v1/) and its TOKEN to the test token, register it for tests in config/services.yaml, and point the client at it:

  # config/services.yaml
  when@test:
      services:
          App\Tests\Support\FrfApi: ~
  
  # config/packages/framework.yaml
  when@test:
      framework:
          http_client:
              scoped_clients:
                  frf.client:
                      base_uri: 'https://api.frf.example/v1/'
                      auth_bearer: 'test-token'
                      mock_response_factory: App\Tests\Support\FrfApi

  Add these under the when@test sections the two files already have.

• Or, in one test, replace the client before the source is first used, with the API's address: self::getContainer()->set('frf.client', new MockHttpClient(new MockResponse($json), 'https://api.frf.example/v1/')).

The contract

Every source passes the same checks. Add tests/Contract/Source/FrfSourceContractTest.php, named after the source's class:

<?php

namespace App\Tests\Contract\Source;

use App\Source\DataSource;
use App\Source\Frf\FrfSource;
use App\Tests\Support\DataSourceContractTestCase;

final class FrfSourceContractTest extends DataSourceContractTestCase
{
    protected function source(): DataSource
    {
        return $this->service(FrfSource::class);
    }
}

A source that's given something overrides context() to give it options, and setUp() (calling parent::setUp() first) to prepare them; CsvSourceContractTest and IcalSourceContractTest do. EverySourceHasAContractTest fails until a source has its contract test.

The contract checks that the source:

1. Is found by its name, and no other source has that name.
2. Has a lasting name and a label; a source that's given something names a real admin page.
3. Is switched on when it has what it reads.
4. Sends complete records: ids present, unique and up to 120 characters; names; known sports; real time zones; kick-offs in the fixture's time zone; known statuses; scores that read as scores.
5. Refers only to records it sends in the same run, and never has a team play itself.
6. Reads the same records twice when nothing changed.
7. Only reads: reading changes nothing on the platform.
8. Brings everything in on a first run, without problems, each record carrying the source's name and id.
9. Changes nothing when run again with the same records.
10. Takes a result back without failing, if it implements SendsResults.

Each failure says what to fix. Run it with:

php vendor/bin/phpunit tests/Contract

Its own tests

Test what's particular to your source the way tests/Integration/Source/ExampleFederationSourceTest.php does: import with SourceImport and a fixed time, find records by the source's ids, and check what they became (names, local kick-offs on both sides of a clock change, statuses, scores). Change a recorded answer to see a problem logged, make the API fail to see a failed run, and, for SendsResults, confirm a result and check what the API received. Then run the whole suite with php vendor/bin/phpunit.

9. Run it

• From the command line: php bin/console app:source:import lists the sources and whether they're on; php bin/console app:source:import frf runs one now and says what it created, updated, left and skipped.
• In the admin: System, Data sources sets each source's schedule (every so many hours, or only when asked), has "Run now", and keeps the log of runs: what each did, its problems, and why a run failed.
• On a schedule: php bin/console app:matchday:run, run by cron every few minutes, starts the runs whose schedule says so. Runs and results sent back are background jobs: keep a worker running with php bin/console messenger:consume async.

Checklist

• name() is short and lasting; label() reads well in the admin.
• isEnabled() is true only once the source is configured.
• Ids are the source's own, stable, and unique within each kind.
• Sports, statuses and scores are mapped; kick-offs are in the fixture's time zone.
• The API is read once a run, every page, with $context->now for anything about time.
• A record that can't be mapped is a problem; a source that can't be read throws.
• Credentials are in the environment, with placeholders in .env.
• The contract test and the source's own tests pass, and so does the whole suite.

Sport keys

The sports every install starts with: