# WP3-003b -- Discovery Access Audit

**Status:** Draft for implementation\
**Version:** 0.1\
**Date:** 2026-08-07\
**Depends on:** WP3-001, WP3-002, WP3-003, WP3-003a, ED-04-010\
**Scope:** Structured operational audit of external Candidate Discovery
accesses

------------------------------------------------------------------------

## 1. Purpose

Reconcilix increasingly uses external Candidate Discovery services. The
first implemented services are xTree and lobid/GND; additional external
services may follow.

External services introduce failure modes outside Reconcilix itself, for
example:

-   invalid or malformed responses,
-   temporary service failures,
-   HTTP errors,
-   timeouts,
-   connection errors.

WP3-003b introduces a small, structured **Discovery Access Audit** so
that Reconcilix can answer:

> Which external Discovery access occurred when, through which adapter
> and service, for which query and vocabulary, with what result and, in
> case of failure, with what technical cause?

The audit primarily supports engineering and operational support.

It is not a fachliche evaluation mechanism.

------------------------------------------------------------------------

## 2. Design principles

### 2.1 KISS

The first implementation uses:

``` text
structured audit model
        ↓
JSONL
        ↓
filesystem
```

No database, monitoring platform or GUI is introduced.

### 2.2 Observe, do not change behavior

WP3-003b records Discovery behavior.

It does not introduce:

-   retries,
-   rate limiting,
-   queues,
-   caching,
-   fallback behavior,
-   altered Candidate Discovery semantics.

### 2.3 Infrastructure concern

Discovery Access Audit belongs to the infrastructure layer.

It does not change the Reconcilix Domain Model and does not introduce a
new aggregate.

### 2.4 Adapter-independent

OpenRefine is the first delivery client, not the generic model.

The audit therefore uses generic Reconcilix identifiers and
source-system metadata and must remain usable for future adapters and
source systems.

------------------------------------------------------------------------

## 3. Scope

WP3-003b audits **external Candidate Discovery accesses**.

Initial scope:

``` text
xTree adapter
→ xTree API

lobid/GND adapter
→ lobid Reconciliation API
```

The `LocalReconciliationStoreAdapter` is outside the Discovery Access
Audit because it performs no external service access.

Existing application/error logging remains unchanged except that an
audit access identifier may additionally be referenced in a technical
error log.

------------------------------------------------------------------------

## 4. Correlation model

Every audit entry shall contain sufficient information to correlate an
external Discovery access with the Reconcilix request context.

Minimum correlation fields:

``` text
sourceDeliveryId
externalDeliveryId       optional
tenantId
sourceSystemId
deliveryChannel
discoveryAccessId
```

### 4.1 sourceDeliveryId

`sourceDeliveryId` is the internal Reconcilix identifier.

For the current OpenRefine channel, ED-04-010 established the technical
behavior:

``` text
1 OpenRefine HTTP POST
=
1 SourceDelivery
```

This remains unchanged.

The SourceDelivery is therefore a reliable technical correlation point
for the current request batch.

### 4.2 externalDeliveryId

`externalDeliveryId` is optional and source-system-specific.

Its semantics are determined by the external source system or adapter.

For OpenRefine it may remain stable across multiple reconciliation runs
because it is part of the stored service configuration.

Therefore:

> `externalDeliveryId` must not be interpreted globally as a
> reconciliation-run identifier.

Future adapters may obtain `sourceSystemId` and `externalDeliveryId`
automatically from their external systems.

### 4.3 No ReconciliationRun concept

WP3-003b introduces no `ReconciliationRun` object or identifier.

The audit must not label a SourceDelivery or audit file as an
"OpenRefine reconciliation run".

------------------------------------------------------------------------

## 5. DiscoveryAccessId

Every external Discovery access receives a new UUID:

``` text
discoveryAccessId
```

Example:

``` text
3b8f56a1-28af-48a7-84d2-74c346abc123
```

Purpose:

``` text
SourceDelivery
        │
        ├── DiscoveryAccess A
        ├── DiscoveryAccess B
        ├── DiscoveryAccess C
        └── ...
```

`discoveryAccessId` identifies one concrete access to an external
Discovery service.

### Storage in v0.1

The identifier is stored:

1.  in the JSONL Discovery Audit entry;
2.  optionally in the existing technical application/error log when the
    corresponding Discovery access fails.

It is **not** persisted in `source_delivery`, `source_value`,
`reconciliation_result` or other domain/persistence tables.

It is not returned to OpenRefine in v0.1.

------------------------------------------------------------------------

## 6. Audit model

WP3-003b introduces one small audit data structure:

``` text
DiscoveryAccessAuditEntry
```

No Session, Run or additional audit object hierarchy is required in
v0.1.

Recommended fields:

``` text
discoveryAccessId

timestamp

sourceDeliveryId
externalDeliveryId?
tenantId
sourceSystemId
deliveryChannel

queryId?
query?

adapterKey
service
operation

targetVocabularyUri
scopeVocabularyUri?

durationMs
status
candidateCount?

httpStatus?
responseBytes?

errorType?
errorMessage?
responsePreview?
```

Optional fields are written only when the information is available.

------------------------------------------------------------------------

## 7. Status model

The operational status model remains deliberately small:

``` text
SUCCESS
EMPTY
ERROR
```

### SUCCESS

The external service responded successfully and Candidate Discovery
returned one or more candidates.

``` text
status = SUCCESS
candidateCount > 0
```

### EMPTY

The external service responded successfully, but no candidate was found.

``` text
status = EMPTY
candidateCount = 0
```

`EMPTY` is not an error.

### ERROR

The external Discovery access failed technically.

``` text
status = ERROR
errorType = ...
```

The technical cause is expressed separately through `errorType`.

------------------------------------------------------------------------

## 8. Error types

Initial error classification:

``` text
TIMEOUT
HTTP_ERROR
INVALID_RESPONSE
CONNECTION_ERROR
UNKNOWN
```

The classification may be extended later if operational evidence
requires it.

Examples:

``` text
HTTP 503
→ status = ERROR
→ errorType = HTTP_ERROR

invalid JSON
→ status = ERROR
→ errorType = INVALID_RESPONSE

connection timeout
→ status = ERROR
→ errorType = TIMEOUT
```

The separation between `status` and `errorType` keeps the operational
model stable while allowing technical diagnostics to evolve.

------------------------------------------------------------------------

## 9. JSONL persistence

### 9.1 Directory structure

Audit files are partitioned by month:

``` text
logs/
└── discovery/
    ├── 2026-08/
    ├── 2026-09/
    └── ...
```

### 9.2 File name

One audit file is used per Reconcilix SourceDelivery:

``` text
{sourceDeliveryId}.jsonl
```

Example:

``` text
logs/discovery/2026-08/
001bef3e-bdf1-4db1-bd5a-e12a4646f3b0.jsonl
```

`externalDeliveryId` is deliberately **not** included in the file name
because:

-   it is optional,
-   its semantics are source-system-specific,
-   it may contain characters requiring filesystem normalization,
-   `sourceDeliveryId` is already unique and controlled by Reconcilix.

`externalDeliveryId` remains available inside every relevant audit
entry.

### 9.3 JSONL semantics

Each line represents exactly one `DiscoveryAccessAuditEntry`.

Example:

``` json
{
  "discoveryAccessId": "3b8f56a1-28af-48a7-84d2-74c346abc123",
  "timestamp": "2026-08-07T15:42:13+02:00",

  "sourceDeliveryId": "001bef3e-bdf1-4db1-bd5a-e12a4646f3b0",
  "externalDeliveryId": "OR-TEST-20260725-001",
  "tenantId": "digicult-prototype",
  "sourceSystemId": "digiCULT.web",
  "deliveryChannel": "OPENREFINE_RECONCILIATION_API",

  "queryId": "q7",
  "query": "Alexianerkrankenhaus und -kloster",

  "adapterKey": "xtree-json:lvr_vocnet_org_wnk",
  "service": "xtree",
  "operation": "getSearchVocItemsByTerm",

  "targetVocabularyUri": "http://lvr.vocnet.org/wnk",

  "durationMs": 487,
  "status": "ERROR",

  "httpStatus": 200,
  "responseBytes": 812,

  "errorType": "INVALID_RESPONSE",
  "errorMessage": "xTree API returned invalid JSON.",
  "responsePreview": "<html>..."
}
```

------------------------------------------------------------------------

## 10. Privacy and security

The Discovery Audit must never expose credentials or authentication
material.

Never log:

-   API keys,
-   passwords,
-   Authorization headers,
-   session cookies,
-   authentication tokens,
-   URLs containing credentials.

Recommended configuration:

``` php
'discovery_audit' => [
    'enabled' => true,
    'log_query_values' => true,
    'response_preview_max_length' => 1000,
],
```

### Query values

`log_query_values` must be configurable.

If:

``` text
log_query_values = false
```

the raw query value is omitted.

This allows Reconcilix installations with potentially sensitive query
data to reduce audit exposure without changing adapter code.

### Response preview

`responsePreview`:

-   is written only for `ERROR`,
-   is optional,
-   is hard-limited by configuration,
-   must be sanitized so that credentials cannot leak into the audit.

The initial maximum length is:

``` text
1000 characters
```

------------------------------------------------------------------------

## 11. Infrastructure design

Suggested package structure:

``` text
src/
└── Infrastructure/
    └── Discovery/
        └── Audit/
            ├── DiscoveryAccessAuditEntry.php
            ├── DiscoveryAccessAuditLogger.php
            └── JsonlDiscoveryAccessAuditLogger.php
```

### Logger interface

Conceptually:

``` php
interface DiscoveryAccessAuditLogger
{
    public function log(
        DiscoveryAccessAuditEntry $entry
    ): void;
}
```

External Candidate Discovery adapters depend on the interface, not on
the JSONL implementation.

This keeps future persistence options open, for example:

``` text
DatabaseDiscoveryAccessAuditLogger
OpenTelemetryDiscoveryAccessAuditLogger
```

without changing the Candidate Discovery adapters.

These future implementations are outside WP3-003b.

------------------------------------------------------------------------

## 12. Adapter integration

Initial integration targets:

``` text
Xtree Candidate Discovery Adapter
Lobid/GND Candidate Discovery Adapter
```

For every external access the adapter or its supporting infrastructure
captures:

1.  start time;
2.  generated `discoveryAccessId`;
3.  request context;
4.  service/adapter metadata;
5.  response status;
6.  duration;
7.  candidate count or error diagnostics;
8.  one final audit entry.

Conceptual flow:

``` text
CandidateDiscoveryAdapter
        │
        ├── generate discoveryAccessId
        ├── start timer
        │
        ▼
external service
        │
        ├── success
        ├── empty
        └── error
        │
        ▼
DiscoveryAccessAuditEntry
        │
        ▼
DiscoveryAccessAuditLogger
        │
        ▼
JSONL
```

------------------------------------------------------------------------

## 13. Audit failure behavior

A critical invariant:

> Discovery Access Audit must never break Candidate Discovery.

Example:

``` text
external xTree access = SUCCESS
audit filesystem write = ERROR
```

The reconciliation result must remain successful.

Audit write failures are diagnostic infrastructure failures and must not
alter:

-   Candidate Discovery results,
-   reconciliation results,
-   persistence of SourceValue,
-   OpenRefine responses.

Where possible, an audit write failure is reported through the existing
technical application/error logging.

No recursive audit logging is attempted.

------------------------------------------------------------------------

## 14. Existing application/error logging

WP3-003b does not replace the existing technical logging.

The two mechanisms serve different purposes:

``` text
Application/Error Log
→ Reconcilix technical runtime problems

Discovery Access Audit
→ structured trace of external Discovery accesses
```

For failed external Discovery accesses, the existing error log may
additionally contain:

``` text
discoveryAccessId
```

This creates a bridge:

``` text
Application Error Log
        │
        │ discoveryAccessId
        ▼
Discovery Access Audit
```

No database correlation is required in v0.1.

------------------------------------------------------------------------

## 15. Out of scope

WP3-003b explicitly does **not** implement:

-   database persistence for audit data,
-   GUI,
-   reporting dashboard,
-   Grafana,
-   Prometheus,
-   OpenTelemetry,
-   retries,
-   rate limiting,
-   concurrency control,
-   queues,
-   external-service health monitoring,
-   Discovery caching,
-   ReconciliationRun model,
-   SourceDelivery lifecycle changes,
-   changes to `externalDeliveryId` semantics,
-   support-reference responses to OpenRefine.

These may be addressed by later work packages based on operational
evidence.

------------------------------------------------------------------------

## 16. Tests

Minimum test coverage:

### T-01 -- Audit entry serialization

A complete `DiscoveryAccessAuditEntry` serializes to one valid JSON
line.

### T-02 -- Optional fields

Unavailable optional values are handled without invalid JSON or
fabricated values.

### T-03 -- JSONL file creation

The logger creates:

``` text
logs/discovery/YYYY-MM/{sourceDeliveryId}.jsonl
```

and appends entries correctly.

### T-04 -- SUCCESS

Successful external Discovery with candidates produces:

``` text
status = SUCCESS
candidateCount > 0
```

### T-05 -- EMPTY

Successful external Discovery without candidates produces:

``` text
status = EMPTY
candidateCount = 0
```

### T-06 -- INVALID_RESPONSE

An invalid external response produces:

``` text
status = ERROR
errorType = INVALID_RESPONSE
```

with available HTTP and response diagnostics.

### T-07 -- Audit write failure

A filesystem/audit logger failure does not alter the Discovery result.

### T-08 -- Query logging disabled

With:

``` text
log_query_values = false
```

the raw query is not written.

### T-09 -- xTree integration

An xTree Discovery access creates a valid audit entry.

### T-10 -- lobid integration

A lobid/GND Discovery access creates a valid audit entry.

### T-11 -- LocalStore

LocalStore reconciliation remains unaffected and creates no external
Discovery Access Audit entry.

### T-12 -- Regression

Existing OpenRefine, xTree, lobid/GND and LocalStore tests continue to
pass.

------------------------------------------------------------------------

## 17. Acceptance criteria

WP3-003b is complete when:

1.  every external xTree Discovery access creates one structured audit
    entry;
2.  every external lobid/GND Discovery access creates one structured
    audit entry;
3.  `SUCCESS`, `EMPTY` and `ERROR` are distinguishable;
4.  errors contain the available HTTP/response diagnostics;
5.  every entry contains a unique `discoveryAccessId`;
6.  every entry can be correlated to `sourceDeliveryId`;
7.  `externalDeliveryId` is recorded when available without assigning
    global run semantics;
8.  audit files follow
    `logs/discovery/YYYY-MM/{sourceDeliveryId}.jsonl`;
9.  query logging can be disabled;
10. secrets and authentication data are never written;
11. audit write failures cannot break Candidate Discovery;
12. LocalStore behavior remains unchanged;
13. existing reconciliation tests remain successful.

------------------------------------------------------------------------

## 18. Planned acceptance scenario

The xTree error observed during the 100-record test provides a concrete
acceptance scenario.

Test query:

``` text
Alexianerkrankenhaus und -kloster
```

If the corresponding xTree fix is temporarily removed, Reconcilix should
record an audit entry similar to:

``` text
service       = xtree
status        = ERROR
errorType     = INVALID_RESPONSE
httpStatus    = available HTTP status
query         = Alexianerkrankenhaus und -kloster
responsePreview = truncated invalid response
```

The `discoveryAccessId` must make the individual failed external access
identifiable.

This scenario verifies that a failure which previously required
reconstruction from missing `SourceValue` records can be diagnosed
directly from the Discovery Access Audit.

------------------------------------------------------------------------

## 19. Future evolution

WP3-003b intentionally creates a structured audit model rather than an
unstructured log format.

This leaves a clean evolution path:

``` text
JSONL
  ↓
optional persistent audit store
  ↓
operational queries / statistics
  ↓
GUI
```

Possible future capabilities include:

-   service reliability statistics,
-   latency analysis,
-   error-rate analysis,
-   support lookup by `discoveryAccessId`,
-   delivery-oriented views,
-   service health views,
-   rate-limiting and resilience decisions based on measured evidence.

None of these capabilities is required for WP3-003b.

------------------------------------------------------------------------

## 20. Result

WP3-003b adds a small operational boundary around external Candidate
Discovery:

``` text
Reconcilix
    │
    ▼
Candidate Discovery Adapter
    │
    ├──── Discovery Access Audit
    │
    ▼
External Discovery Service
```

The implementation remains deliberately small:

``` text
one audit entry model
+
one logger interface
+
one JSONL implementation
+
xTree/lobid integration
```

It changes neither Candidate Discovery semantics nor the Domain Model.

The result is a reproducible and supportable record of external
Discovery behavior while preserving the KISS principle and leaving
future operational tooling open.