What an end-to-end test proves
An end-to-end test uses Orbit the way a customer does. It writes data through the GitLab API, then queries Orbit through GitLab.
flowchart LR
accTitle: The path of one end-to-end test
accDescr: A test writes data in GitLab. Siphon copies the change into the data lake. The indexer writes the graph. The test reads the graph with a query through GitLab.
Write[Write in GitLab] --> Siphon --> Lake[(Data lake)] --> Indexer --> Graph[(Graph)] --> Query[Query in GitLab]Nothing on that path is mocked, and the answer passes through the same authorization and redaction as in production. A test passes only when every piece on that path works. No other layer sees all the pieces at once.
Each run deploys its own copy of GitLab and Orbit, in namespaces named after the commit, on a shared Kubernetes cluster. Two runs never share a database or a queue. A run removes its stack at the end, and each new run first removes stacks that are a few hours old.
When the tests run
| Pipeline | Main job | Version lanes and the ClickHouse HA variant |
|---|---|---|
| After each merge to main | Runs, and a failure turns main red | Run, and a failure also turns main red |
| A merge request | Manual, and does not block | Manual, and do not block |
| The rolling pin-bump merge request | Runs, and blocks the merge | The lanes run and do not block. The HA variant is manual. |
| A schedule | Has no schedule of its own | Run on their own schedules |
On a merge request, the run builds Orbit from the branch. After a merge, it uses the image that the main pipeline built.
Pages
- The test cluster covers the shared cluster, the harness repository, and how runs stay apart.
- How a run works follows one run from the trigger to the result.
- Suites covers what the suites check and the rules for a new suite.
- Versions and pins covers the GitLab version lanes, the pins, and the ClickHouse HA variant.
The commands to run a stack yourself are in the end-to-end testing harness.
Design rationale
We test through GitLab, because customers reach Orbit only through GitLab. We deploy a fresh stack for each run, because a shared environment keeps state from earlier runs. A failure then depends on what ran before.