Plectis: What a Stranger Can Check

The problem

I have a larger private software system that I cannot responsibly publish: it contains personal records, credentials, live browser and account information, my working notes, and unfinished tools. I would still like strangers to be able to test some claims about it instead of accepting my say-so. I use in a deliberately local sense: a claim is answerable when a stranger can run a public test that bears on it, see how the result is judged, and locate a disagreement in a named input, rule, or output. This is not a new synonym for truth, reproducibility, verification, or independent review. For the rest of the paper I use the plainer phrase . The question is what conclusions such a test can support when the larger system and the unpublished pool from which the test was selected remain invisible.

Plectis is my working answer: a public software repository, stored at github.com/wcook04/plectis. A is an organised project folder, usually kept with its revision history; to a repository is to copy that folder and history to another computer. Everything this paper says a stranger can check is in that repository. This paper describes how its claims are exposed and where the evidence stops.

The main result is a boundary, not a score. Public code, inputs, rules, and run records can make a selected claim easier to challenge. They cannot by themselves establish three missing premises: that the public code came from the private system, that the project’s expected answers and pass rules are correct, or that the selected cases represent the unseen whole. The audio example in Section  shows all three limits in one elementary calculation. The later inventory records what was published; it is not evidence of breadth, quality, or independence. Runeson and Höst likewise treat configuration-managed project documents as archival data while warning that records created for another purpose may omit needed information and be difficult to assess without complementary sources [1]. The repository therefore lets a reader test claims about the public objects it actually exposes, within the limits stated for each test.

One public test before the general design

Consider four decimal audio samples: $$0,\quad 0.05,\quad -0.05,\quad 0.1.$$ The published procedure squares them, averages the squares, takes the square root, and multiplies by eight. Their squared average is $0.00375$, so the stated rule gives $8\sqrt{0.00375}=0.489897\ldots$. The public program produces $0.489897949$. Anyone with a calculator can reproduce the expected number without trusting the program.

That agreement establishes something real and small: this public program agrees with the stated formula on this test case. It does not establish that the formula is the right account of audio level, that the public code came from the private system, that other inputs behave similarly, or that important risks were selected for testing. Those conclusions require additional premises that the observed match does not contain.

The paper asks the same six questions of every published item:

1. What sentence is offered for belief?

2. What public object or procedure was checked?

3. Who supplied the interpretation or expected result?

4. What does the observed match establish?

5. What remains unestablished?

6. Which independently controlled evidence would strengthen it?

Section  gives the full command, all five audio cases, and the saved run record. We now generalise from this example to the project’s common record format.

The component contract

A in Plectis is one registered, testable part of the project. Its is a fixed sample input: in ordinary testing language, a test case. A is the saved record of a run. The repository supplies one from an earlier run; rerunning can write another. A is the program that applies a component’s pass and refusal rules. In testing terminology, the rule and expected result together act as the test oracle: they say what should count as correct for that case. A is a named, saved version of the repository. These are Plectis’s internal names for familiar objects—test case, test oracle, run record, and saved software version—and the paper uses the familiar terms whenever the internal filename is irrelevant.

Here means the public input, procedure, output, and judgement rule offered in support of a claim. Calling them evidence does not establish their adequacy. means only that the claim, those materials, and a limit are stated together so readers can check their fit. Nothing in these labels grades the importance of what a component does.

At commit 57ffb7f5830c the registry, the component list read by the programs, contains 88 entries. Every entry is required to connect the same things: a claim stated in prose; a fixture; a command; the output files the command writes; a stored receipt from a prior run; an explicit rule for what counts as a pass and what must be refused; and a stated limit on what a pass establishes. Figure  lays the required parts out, together with the fact about them that the rest of this paper keeps returning to: every part is supplied by me, and every part is exposed to a stranger.

The repository cannot by itself support my assertions about the unseen private system. A component does something narrower: it gives a related public claim a procedure that can fail in front of a stranger. The procedure does not make the claim true or establish the published fragment’s origin. It changes the form disagreement can take: a reader who doubts a component’s claim can point at a particular input, a particular line of the validator, or a particular number in the output, instead of arguing with my description of software they cannot see. A component that fails is still answerable in this sense, and a component can be answerable and trivial; answerability concerns the form a claim takes, not its truth and not its importance.

A component may also end in a declared refusal rather than an answer. The worked example below refuses an input format it does not support. Components that depend on an external tool, such as the Lean proof checker (a program that verifies mathematical proofs), must record a refusal with a stated limit when the tool is absent, rather than reporting success. therefore means that the operation and its declared failure boundary are present in public form; it does not mean that every optional dependency exists on every machine. A refusal is informative only when it is declared in advance and distinguishable from a breakdown: a predeclared refusal marks the edge of what the component claims, while an unexpected crash is a defect. The two must not be read as the same kind of event. One honesty condition on refusals cannot be discharged from inside the repository at all: the public record shows each boundary as declared at the named commit, and it cannot show whether the boundary was drawn before or after an awkward case was first met. A boundary drawn afterwards turns a failure into a designed refusal in the telling. Saving the version before an outside evaluation begins, as Section  proposes, is what would make that ordering checkable.

The idea does not depend on software. The contract needs a claim in words; a determinate public case; a procedure a stranger can cause to run; an observable result; a rule connecting result to claim; a declared boundary of application; and a stated ceiling on inference. That the case here is a file of numbers, the procedure a command, and the record a text file is a convenience of this example. The same contract could be satisfied in another field by a laboratory protocol with declared rejection conditions, or by a sealed evaluation service with published scoring rules. I make no claim about those settings; the point is only that the contract is a shape for exposing claims to strangers, and this repository is one example. Read simply as software, it is a test suite with a registry and unusually complete paperwork. That is a fair description. The paper asks what such a test suite lets a stranger challenge, and which conclusions still lie beyond it.

This local notion overlaps with reproducibility and transparency but is not a synonym for either. A run can repeat perfectly with no stated claim attached, and a claim can be answerable while still awaiting a good test. The design also begins from partial disclosure: it offers one limited point of contact, not a view of the hidden system. A component exposes the published fragment to challenge. It does not make the private whole answerable.

Answerability is not the same property as computational reproducibility, although the component contract supplies materials needed to test the latter. Under the convention used by the National Academies, computational reproducibility means obtaining consistent results using the same input data, computational steps, methods, code, and conditions of analysis [2]. The report separately cautions that exact reproducibility does not guarantee computational correctness: undetected erroneous code can reproduce the same erroneous result . In software testing, a is a procedure used to distinguish correct from incorrect behaviour. The challenge of making that distinction is called the test oracle problem [3]. Barr et al. distinguish a partial test oracle $D$ from a conceptual ground-truth oracle that always gives the right answer, modelled as a total oracle $G$, while observing that such ground truth is unknowable except in trivial cases [3]. Soundness requires $D(a)\Rightarrow G(a)$, and completeness requires $G(a)\Rightarrow D(a)$. A passing validator therefore applies its published rule; it does not by itself establish that the rule is sound. The risk that a curated collection disproportionately exposes favourable cases is analogous to selective publication and the file-drawer problem [4]. An links system claims to arguments and evidence. The Object Management Group’s Structured Assurance Case Metamodel (SACM), version 2.3, defines such a case as a collection of auditable claims, arguments, and evidence. It separately represents an artifact citation and the relationship said to connect that artifact to a claim; the relationship is itself an assertion by the case’s author. Representing an argument is therefore not the same as establishing its validity [5]. The GSN Community Standard states the boundary more directly: GSN documents an asserted chain of reasoning and its evidential references, but the notation itself establishes neither the truth of the argument nor the sufficiency of its support [6]. Plectis is not an implementation of that standard or of GSN, and makes no claim that the private system is safe. Its validator applies a published rule and records a result; it neither models nor independently justifies the evidential link from that result to the prose claim. The Association for Computing Machinery (ACM) calls digital research materials such as code, scripts, and data . Its badge means those materials passed an independent audit for documentation, consistency, completeness, ability to run, and evidence of verification and validation. The NISO recommended practice separates object review, reproduction with the author’s objects, and replication by a new study [7]. ACM’s harmonised current names are and ; its badges are independent, and results validation requires a peer-reviewed report . Plectis claims neither status: its receipts are author-generated, not independent review or validation. These standards are comparison points, not statuses awarded here. This paper claims no new theory of assurance, testing, or reproducibility; it reads each public procedure against its claim and stated limit.

One run, examined

The component batch8_audio_level_rms_port computes a loudness level from short arrays of audio samples. I chose it because it is ordinary. It needs only a square root and no knowledge of the private system; its claim is small enough that every part of the evidence can be seen at once.

The reported origin is this. The private system contains a small macOS recording application, and one of that application’s functions reduces a short stretch of audio samples held in memory to a level between zero and one. The public component re-expresses that calculation in Python. The component’s code and its stored receipts name the private file it claims to re-express (AudioLevelMonitor.swift); a stranger cannot open that file, so the name is a report, not evidence. What the stranger can check is everything else.

The declared calculation begins with root-mean-square (the rms in the component’s name): square each sample, average the squares, and take the square root. It then applies the project’s gain of eight and caps the result at one. Signed sixteen-bit integer samples are first divided by 32,767, the largest positive value in that representation, to bring them onto the same scale. An empty input must produce zero. An unrecognised format name must be refused rather than guessed at.

The component’s public command runs it against its fixture and writes fresh output outside the repository (the backslashes split one long command across lines):

PYTHONPATH=src python3 -m plectis \
  batch8-audio-level-rms-port run \
  --input fixtures/first_wave/batch8_audio_level_rms_port/input \
  --out /tmp/plectis-audio-rms \
  --acceptance-out /tmp/plectis-audio-rms-acceptance.json

At commit 57ffb7f5830c, the run produced:

Here means “refused as expected”; pcm24 was not accepted. Because the inputs and the rule are both printed above, the first row can be checked without the repository or a computer: the squares of the four samples average to 0.00375, whose square root is $0.061237\ldots$, and eight times that is $0.489897\ldots$. A reader with a calculator can recompute this expected value from the stated formula alone. For this one case the reader can supply the test oracle: elementary arithmetic gives an expected number derived independently of the program’s output, but not of the project’s formula.

Almost nothing else in the collection is like that, which is why the example matters. Elsewhere the project usually supplies the implementation, the input, the expected values, and the validator through the same author-controlled process. A test assembled that way can pass for four different reasons: the implementation and the expectation are both correct; the two share a mistake; the chosen cases happen to sit where the implementation behaves; or the validator checks a weaker property than the prose claim suggests. A matching result therefore establishes repeatability under that public test. Independent evidence bearing on correctness needs an expected result justified apart from the implementation. Arithmetic supplies that for the first row only after accepting the project’s formula. Fresh inputs alone do not supply one.

The example suggests a question for any software test: where did the expected value come from, and who supplied it? The answer changes what a match can support:

Most components in the collection sit in the first position, except where a reader promotes a case to the second by recomputing it, as the worked example’s decimal-samples row invites. Components that invoke a named external tool occupy a hybrid of the first and third positions: the project still chooses the case and the rule, while the tool supplies a result that inherits its own limits. The evaluations in Section  describe ways to move consequential choices out of the project’s control.

batch8_audio_level_rms_port_validation_receipt.json is the stored reference for the table. It sits under receipts/first_wave/batch8_audio_level_rms_port/, so a fresh run is compared against a fixed record rather than against memory. A stored record by itself does not prove that the stated command produced it; its use is that anyone can generate a fresh one and compare the two.

The limit is part of the component. This run checks one public Python calculation over synthetic sample arrays (arrays composed for the test, not captured from a device). It does not run the original application, start an audio session, request microphone permission, or capture sound, and it cannot establish that the named private file is where the calculation came from. Nor does the table show that the task was difficult (it was not), that the component is useful elsewhere, or that the remaining entries resemble it. What it shows is that one assertion now has a form in which a stranger can rerun it, dispute its rule, perturb its input, or replace my expected value with a better-derived one.

The calculator check derives one expected number without the implementation. It shows agreement with the project-supplied formula on that case, not that the formula is the right audio rule. It does not establish origin, reach, or risks beyond the declared scans. Its correctness evidence is limited to that calculation. The next section treats these five gaps separately.

Five distinctions

Five labels organise the gaps in Figure : , where the material came from; , whether an answer is right; , whether a rule tests its claim; , whether shown cases stand for unshown ones; and , what the declared checks may miss. These short labels correspond to familiar questions about provenance, test-oracle validity, claim–test fidelity, representativeness, and residual risk. They are not five new kinds of evidence. Each subsection separates what the public run directly shows from the additional premise needed for a stronger conclusion.

What the collection contains

This section is a dated inventory, not a result. The categories are useful for finding a test and for seeing which kind of limitation accompanies it. Their names and counts do not measure importance, correctness, independence, or coverage of the private system.

At commit 57ffb7f5830c the 88 components fall into 7 groups:

The groups do not share a subject. Their only unity is the contract of Section : each item became a small public test. The collection is organised by how claims are checked, not by a common scientific question.

The registry also classifies what kind of evidence each component supplies. It calls this classification a . A route says whether the public component preserves source text, re-creates behaviour on fixed cases, records a run, or checks a declared rule. The four routes below cover all 88 entries. Each leaves a different question open:

Copied source preserves text but demonstrates little behaviour; a reimplementation does the converse; an external-tool run inherits the tool’s limits; and a contract checker is worth exactly the rule its author chose. To inspect one item, use the public component map, ORGANS.md (an inherited filename). Follow it rather than guessing, because some components share a page. The audio row points to paper_modules/batch8_audio_level_rms_port.md, which in turn names the fixture, receipts, route, and limit.

Two cautions govern the tables. First, components are not independent witnesses: they share code, fixture style, authorship, and judgement. 88 entries are registered claims, not 88 confirmations. Pages, entries, and receipts generated from the same underlying records repeat one witness rather than supply several. Machine checks can catch their accidental disagreement. They cannot create an independent witness. The four routes impose four different limits on what may be concluded; they are not four independent methods confirming one claim.

Second, names such as computer-checked mathematics and AI reliability and safety tests borrow gravity from serious fields. Read each row as the weakest property its tests actually check—often that a fixed case replays or a record has a stated shape. The names help find components; they do not weigh them.

The author’s hand

All five distinctions return to control. At the commit this paper describes, I hold every consequential choice on both sides of every test. I chose which private mechanisms were candidates, which were selected, and how finely each was divided into components, and so what the count counts. I chose which of the four routes each took, which inputs went into the fixtures, what the expected values are, what each validator checks, and where each refusal boundary sits. And I chose which group each entry is filed under, which failures this paper explains and in what tone, which single component the worked example honours, and where the publication boundary lies. Somebody had to make each of these choices, and the privacy constraint meant it was me. None of that is misconduct.

But that concentration of choice permits a failure mode, and it involves no false statement anywhere. Select the mechanisms that show well. Divide the favourable ones finely. Narrow each claim until its fixed cases satisfy it. Give awkward material the least demanding check. Declare as refusals whatever would otherwise fail. File small replays under weighty names. Then let the counts, the categories, and the machinery of records accumulate into an impression of scale and maturity that no individual sentence asserts. A document can be cautious sentence by sentence while its arrangement still overstates, because a reader’s impression forms on what is counted, named, worked through, and left out, as much as on what is claimed. This paper is inside its own arrangement, so declaring the risk does not discharge it; nothing in the text can establish that the text is not doing what this paragraph describes.

Making disagreement local can also displace the larger question. A reader can settle an argument about one input or one rule while leaving untouched whether the selection is fair, whether the whole is sound, or what any of it is for. An objection about selection is not answered by the successful execution of any number of selected items. The registry hosts the small arguments; it does not settle the large ones.

Two corrections need no cooperation from me. First, read any component at the weakest property its validator checks, as Section  proposes. Second, treat every count as an inventory, and every agreement among my records as one witness. A third point concerns what no reader can inspect: a private mechanism is easiest to publish when it can be separated into a small task, gives the same output from the same input, and runs in isolation. A collection like this therefore over-represents pure calculation, replay, and record-checking, and under-represents stateful, entangled, long-running behaviour. The published picture tilts towards what publishes well before any question of honesty arises.

The National Aeronautics and Space Administration (NASA) defines independence for independent verification and validation (IV&V), a formal software check, through three separations: technical (the evaluator did not develop the system); managerial (a separate organisation chooses scope, methods, and schedule); and financial (an independent group controls the budget without adverse financial pressure) [11]. These are criteria for NASA IV&V, not a universal test. Plectis has undergone no IV&V. No other outside evaluation report is cited or included, and I know of none; an unreported private rerun would remain invisible. I use the NASA criteria only for the paper’s five gaps: evidence about a choice depends less on me once the choice is no longer mine. Section  lists evaluations that address those dependencies.

What stronger evidence would look like

A passing run shows that named public code produced a stated result from a supplied input in a stated environment. It does not establish private provenance, representativeness, correctness beyond self-supplied criteria, usefulness, security, originality, or the private system’s reliability. The computer-checked mathematics label covers different checks: proof-work records, small public statements checked by Lean, and one exact finite calculation. The label itself supplies no theorem; each entry supports only its own public claim and stated limit. At the named commit, the repository’s evidence still came from the project. It includes no outside evaluator’s report; I know of none, although a private rerun could be unreported.

Five forms of independent evaluation could add evidence that the public repository cannot supply by itself. These are not the four publication routes in Section . A publication route classifies evidence already in the registry; the evaluations below describe new work by someone outside the project. The order is only for exposition. It ranks nothing, and no evaluation is a prerequisite for another. Figure  shows which choice each evaluation transfers and which claim it still leaves open. The common question is who makes the choice that matters for that claim.

means a person with no involvement in the project reruns the supplied commands in their own environment and publishes what happened, including failures. That would reduce dependence on my machine and test whether the published instructions suffice elsewhere. It would not by itself establish correctness.

means the evaluator chooses components from the full registry, without substitution when one proves awkward, and original failures stay on the record even if I later repair them. The units themselves are open to the same challenge: an evaluator who finds that two entries are one dependent mechanism, or that a boundary excludes the difficult part of a task, is disputing the registry, not misusing it. That addresses curation within the public collection. It cannot address whether the collection represents the private system, because the private pool stays invisible.

can move repeatability towards evidence bearing on correctness: inputs chosen after the version is saved, and expected results worked out by a method that does not pass through my code, whether independent calculation, a second implementation, or an external reference. The result would support claims about those cases only to the extent that the derivation is credible and independent and the cases are adequate; general correctness still would not follow.

is what a claim of advantage over ordinary alternatives would need: someone bringing a task that was not designed around the project’s own categories and comparing the component against an ordinary alternative, which for several components would be a short conventional script. Nothing in this paper makes that comparison.

, among the five evaluations shown here, bears most directly on the origin story: a trusted person examining selected private material against the public collection under confidentiality. Prior public commitments could provide another kind of provenance evidence. Even completed, this review would transfer trust rather than remove it: public readers would be relying on the reviewer’s access, competence, and report instead of on mine, and the review itself could not be published. The privacy cost may reasonably never be paid. If it is not, the origin story remains testimony, and this paper is written so that its argument does not depend on it.

plectis hypothesis-handoff --input packet.json --format text

Conclusion

Plectis makes selected public claims testable; it does not verify the private system as a whole. Its fixed cases, procedures, outputs, rules, and stated limits let a stranger repeat a result and identify exactly what they dispute. A pass is no stronger than the test oracle—the rule and expected answer—or than the basis for believing that oracle.

The worked audio calculation demonstrates the hierarchy. Repeating the stored result supports computational reproducibility for that public case. Deriving one expected number by arithmetic adds evidence that the implementation agrees with the stated formula on that case. Neither step establishes that the formula is the right audio rule, that the code came from the private system, or that the selected case represents it.

More author-controlled components enlarge the inventory without changing who chose the cases or rules. Moving a relevant choice out of the author’s hands can strengthen evidence about that choice; it does not repair every gap.

Dated reproduction record

Except where a paragraph gives a later date, this appendix records one run on 17 July 2026 under macOS 15.6.1 on Apple silicon (arm64) and Python 3.13.7. The repository version was commit 57ffb7f5830cc446a2cbd8083d5d80bcab2af563, abbreviated 57ffb7f5830c elsewhere. Later commits or machines may differ; the body does not depend on them.

A typed hypothesis handoff makes the next outside contribution more specific: it exposes the project’s tentative answer, alternatives, and update rule before the outcome is known. That is a better invitation to disagree, not stronger evidence that the project is right.

cp -R fixtures/first_wave/batch8_audio_level_rms_port/input/. /tmp/plectis-audio-rms-input

PYTHONPATH=src python3 -m plectis batch8-audio-level-rms-port run \
  --input /tmp/plectis-audio-rms-input --out /tmp/plectis-audio-rms-probe \
  --acceptance-out /tmp/plectis-audio-rms-check.json; echo $?

git clone https://github.com/wcook04/plectis
cd plectis
git checkout 57ffb7f5830c
PYTHONPATH=src python3 -m plectis tour --format text .

python3 -m pip install .
plectis tour --format text .