Skip to content

Decide how project-only and shared models coexist when they advertise the same pipeline or algorithm key #1427

Description

@mihow

Summary

Processing services can be added to a single project, by design, so a team can run its own models. Pipelines and algorithms, however, live in one platform-wide namespace and carry no record of which service or project introduced them. When two services advertise the same pipeline slug or algorithm key, they silently share the same rows, and whichever registered first decides what those rows say. This ticket asks how shared and project-only models should coexist. Found while working on #1424, where each algorithm gets a species list whose visibility follows the algorithm, which makes the question concrete.

Observation (from reading the registration code)

  • Algorithm.key is unique across the platform, and an algorithm is looked up by key and version at registration.
  • A pipeline is looked up by slug, or by name and version, across the platform.
  • Registration only adds. An existing algorithm keeps its name, description and category map whatever a later service reports; an existing pipeline can have further algorithms attached by any service that advertises the same slug.
  • Neither model has a link to a project or a visibility setting of its own. They are reached through processing services and per-project pipeline settings.

Why it matters

  • A project-only service that reuses a well-known key (deliberately, or because it is a fork of the same model code with different weights or classes) attaches to the shared algorithm. Its predictions are then recorded against an algorithm whose category map may not match what it actually runs.
  • The reverse order is worse: if a project-only service registers a key first, the shared service that registers later inherits that category map.
  • There is no way to answer "is this algorithm public?" from the algorithm itself. Classifier species lists as taxa lists and support for public lists #1424 needs an answer, because "which models can predict this species" must not reveal or count a model that is private to another project. The working definition there is: an algorithm is public if at least one public processing service offers it (Let taxa lists and processing services be shared with every project, with safe rules for who can edit them #1425 adds the public setting to services), otherwise it is visible only to the projects of the services that offer it.
  • Creating and registering services is limited to staff today, so exposure is limited to trusted users for now. It widens as soon as project managers can add their own services, which is the intended direction (Review M2M permissions for things that belong to multiple projects #1120).

None of this has been reproduced in a test yet.

Directions to discuss

  1. Record provenance. Note which processing service first registered an algorithm and a pipeline. Cheap, changes no behaviour, and makes conflicts visible.
  2. Detect conflicts instead of absorbing them. When a service advertises an existing key with a different category map (the stored labels hash makes the comparison cheap), refuse or warn rather than silently keeping the first one.
  3. Namespace project-only models. Scope keys and slugs registered through a project-only service to that service or project, so they cannot collide with shared ones. The largest change, since keys are used in the contract with processing services and in stored results.
  4. Give algorithms and pipelines the same public setting that taxa lists and services get in Let taxa lists and processing services be shared with every project, with safe rules for who can edit them #1425, set from the service at registration, rather than deriving it through joins each time.

What still needs verifying

  • A test with two services in different projects advertising the same algorithm key with different category maps: which map wins, and what a classification saved from the second service points at.
  • The same for a shared pipeline slug with a different set of algorithms.
  • How often keys already collide in practice: compare the keys advertised by each registered service.
  • Whether the add-only rule is relied on anywhere (for example to keep historical results pointing at stable rows) before changing it.

Related: #1424, #1425, #1120.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions