Operational definition
What the concept covers — and what it does not.
The protocol starts from the decisions to support and separates collection, modelling, validation, scenarios, governance, export and exit cost. A genuine comparison requires dated tests applied to every solution with the same grid.
Authorise a shortlist only after a documented test whose inputs, expected results, deviations and exit conditions are kept.
Functional coverage, transparency, tests, export, interoperability, total cost and dependency.
A convincing interface can hide transformations, priors or allocation rules that cannot be audited.
Decision framework
A method that keeps the assumptions auditable.
Write three representative test cases and their expected results before any vendor demonstration.
Run the same input through each solution and keep parameters, transformations, diagnostics and exports.
Compute licence, integration, operation, skills, migration and exit cost over the same period.
Separate observed, declared and unassessed capabilities, with a review date and an owner.
Non-negotiable parameters
The unit, the horizon and the comparator must be fixed before the calculation.
These five parameters determine what the result actually means. If they change during the analysis, the scenarios are no longer comparable and the recommendation has to be recomputed.
Observation or resource identified by its source, its population, its unit, its period, its version and its usage rights.
State the collection date, the freshness that matters for the decision and the review cadence; a recent date does not fix an inadequate definition.
Compare distributions and methods on a consistent perimeter; never treat an external benchmark as an automatic local target.
Publish sample size, dispersion, exclusions, transformations, selection bias and the limits on reproducibility.
A resource enters a decision only when its provenance, its applicability and its transformations can be audited.
Decision dossier
What the committee must receive — beyond a central result.
The deliverable must make it possible to redo the reasoning, to challenge an assumption and to know when the conclusion stops holding. An isolated figure, however precise, does not do that.
Prepare a verifiable evaluation. Write down the options genuinely open, the status quo and the person responsible for the choice.
Functional coverage, transparency, tests, export, interoperability, total cost and dependency. State the unit, the numerator, the denominator, the population and the period.
Compute licence, integration, operation, skills, migration and exit cost over the same period. Keep the same definition of cost and outcome for every option.
Separate observed, declared and unassessed capabilities, with a review date and an owner. Report the checks that fail as clearly as those that pass.
A convincing interface can hide transformations, priors or allocation rules that cannot be audited. Quantify the impact of an over-optimistic assumption and of a delayed decision.
Quarterly review. Define the signal, the threshold and the date that will trigger keeping, correcting or stopping.
Worked example
Educational example of full cost
Situation. Two fictional solutions are compared solely to illustrate the formula. A costs €40k and requires 80 days of integration; B costs €70k and 20 days. An internal day is valued at €700.
Calculation. Educational first-year cost: A = 40,000 + 80 × 700 = €96,000; B = 70,000 + 20 × 700 = €84,000.
Reading. This example ranks no real software. Auditability, maintenance and exit cost remain to be measured with a dated protocol.
Decision rules
When to treat the analysis as solid enough.
- Authorise a shortlist only after a documented test whose inputs, expected results, deviations and exit conditions are kept.
- Measure at minimum: functional coverage, transparency, tests, export, interoperability, total cost and dependency.
- Validation condition: separate observed, declared and unassessed capabilities, with a review date and an owner.
Limits
What the data and the method do not allow you to assert.
- A convincing interface can hide transformations, priors or allocation rules that cannot be audited.
- A sector benchmark is not a normative target.
- Public data can be well documented and still be unfit for a local decision.
Bibliographic references
The works this analysis rests on.
- Peng, R. D. (2011), Science.
- Sculley, D. et al. (2015), “Hidden Technical Debt in Machine Learning Systems”.
- Wilkinson, M. D. et al. (2016), “The FAIR Guiding Principles for Scientific Data Management”, Scientific Data.
- Peng, R. D. (2011), “Reproducible Research in Computational Science”, Science.
Putting it into practice
The canvas to produce in order to decide
Decision · target · objective · horizon · options · constraints · interactions · economic unit · evidence · review threshold. The accompanying document also keeps the assumptions, limits, version and review date.
See the tool or the accompanying document →