IME these categories obscure the actual mechanics of testing APIs. This is how I would actually describe different categories of tests

Health Check Tests

Setup a /health endpoint to test as much of the service logic as possible. The endpoint should be queried periodically and the container should be taken out of a ready state or restarted if the request fails.
Smoke tests go here.

Test Suite Tests

You should try to put as much as possible in here. Mock external dependencies so you test as much e2e logic in the test suite as possible. The test suite should never have to make an external network call.
Functional tests, regression tests, security tests, UI tests, fuzz testing should go here

Pre-Production Tests

Deploy service to a pre-production environment (either canary or shadow if possible). Pre production service should get a small % of production traffic so service behavior can be visualized on a dashboard.
Integration tests, load tests, and stress tests go here
This is the hardest category to setup automated acceptance criteria for. Most teams I've seen just visualize service behavior on a dashboard and verify manually. However in theory it should be possible to setup alerts on the pre production environment for this. As long as no alerts fire, the tests are considered to have passed.

-1 points spline_reticulator
IME these categories obscure the actual mechanics of testing APIs. This is how I would actually describe different categories of tests

Health Check Tests

Setup a /health endpoint to test as much of the service logic as possible. The endpoint should be queried periodically and the container should be taken out of a ready state or restarted if the request fails.

Smoke tests go here.

Test Suite Tests

You should try to put as much as possible in here. Mock external dependencies so you test as much e2e logic in the test suite as possible. The test suite should never have to make an external network call.

Functional tests, regression tests, security tests, UI tests, fuzz testing should go here

Pre-Production Tests

Deploy service to a pre-production environment (either canary or shadow if possible). Pre production service should get a small % of production traffic so service behavior can be visualized on a dashboard.

Integration tests, load tests, and stress tests go here

This is the hardest category to setup automated acceptance criteria for. Most teams I've seen just visualize service behavior on a dashboard and verify manually. However in theory it should be possible to setup alerts on the pre production environment for this. As long as no alerts fire, the tests are considered to have passed.

1 points nekokattt

health doesnt check service logic. It is an inexpensive check to assess if something is ready to use or not.

E.g. is the database connection pool open... is kafka responding... etc.

You are confusing it with end to end testing and integration testing which focus on functionality.

Preproduction and canary are also not the same thing... that is blurring the line between UAT and pilot. Preproduction should be testable without affecting the existing product.

parent root