# ED-04-010 – SourceDelivery Lifetime Analysis

- **Status:** Completed
- **Date:** 2026-08-07
- **Scope:** SourceDelivery lifecycle in the OpenRefine reconciliation channel
- **Basis:** empirical tests with 100 and 10,000 records, OpenRefine POST/response logs, `source_delivery` and `source_value`

## 1. Objective

The analysis clarifies the relationship between an OpenRefine reconciliation run, OpenRefine HTTP request batches, `externalDeliveryId`, Reconcilix `SourceDelivery`, and persisted `SourceValue` records.

Central question:

> What is the actual lifetime and semantic boundary of a `SourceDelivery` when OpenRefine is used as delivery channel?

No implementation change is part of ED-04-010.

## 2. Relevant concepts

### 2.1 External delivery

`externalDeliveryId` is supplied externally via the Reconcilix service URL, for example:

```text
...?api_key=...&delivery_id=OR-TEST-20260725-001
```

In OpenRefine this service configuration can remain stored and be reused for multiple reconciliation runs.

Therefore `externalDeliveryId` identifies an external/fachliche delivery context, but it does **not** identify one individual OpenRefine reconciliation run.

### 2.2 Reconciliation run

A reconciliation run is the user action in OpenRefine that starts reconciliation for a project or selection.

Reconcilix currently receives no reliable run identifier from OpenRefine that correlates all HTTP requests belonging to exactly this one run.

### 2.3 HTTP request batch

OpenRefine transports reconciliation queries in batches via multiple HTTP POST requests.

A batch is therefore a transport-level unit and must not be interpreted as a stable fachliche delivery unit.

## 3. Test A – 100 records

### 3.1 OpenRefine transport behavior

For the 100-record test, OpenRefine sent:

```text
1 untyped preliminary POST with 10 queries
+
10 typed POSTs with 10 queries each
```

The 100 actual typed reconciliation queries were therefore transported in ten batches.

### 3.2 Reconcilix persistence behavior

Reconcilix created 11 records in `source_delivery`.

The deliveries shared the same contextual values, including:

```text
tenant_id            = digicult-prototype
source_system_id      = digiCULT.web
delivery_channel      = OPENREFINE_RECONCILIATION_API
external_delivery_id  = OR-TEST-20260725-001
```

The grouped `source_value` data showed:

```text
10 SourceDelivery IDs × 10 SourceValues = 100 SourceValues
```

The SourceDelivery corresponding to the untyped preliminary request contained no SourceValues.

Current implementation:

```text
1 HTTP POST
=
1 SourceDelivery
```

This includes an empty SourceDelivery for the untyped preliminary OpenRefine request.

## 4. OpenRefine batch-size behavior

OpenRefine Issue #5603 and the related implementation describe adaptive batch sizing. Relevant for the present analysis:

- if no usable larger batch size results, OpenRefine retains the default batch size of 10;
- otherwise OpenRefine can use approximately 1% of the dataset size;
- the reconciliation service manifest limits the permitted batch size.

This behavior was verified empirically in the Reconcilix tests.

### 4.1 100-record test

With 100 records, 1% of the dataset is 1. This is below the default batch size of 10.

Observed:

```text
100 records
→ typed requests with 10 queries each
```

### 4.2 10,000-record test

Before the test, the Reconcilix manifest was configured with:

```text
batchSize = 100
```

For 10,000 records:

```text
1% of 10,000 = 100
```

Observed:

```text
10,000 records
batchSize = 100
→ OpenRefine sent batches of 100 queries
```

The adaptive batch behavior is therefore confirmed for the tested environment.

## 5. Additional observation – reuse of reconciliation results

A first 10,000-row experiment was created by repeating the same 100-record dataset.

OpenRefine queried Reconcilix only for the first occurrence of the repeated values and reused the already available reconciliation results for subsequent duplicate rows. The repeated rows nevertheless displayed the correct results in OpenRefine.

Therefore:

> The number of project rows is not necessarily equal to the number of queries sent to the reconciliation service.

Project size, number of unique reconciliation queries, number of HTTP batches and number of `SourceValue` records are separate metrics.

For realistic load tests, sufficiently distinct query values are required.

## 6. Findings

### F-01 – OpenRefine batches are transport units

The OpenRefine batch is adaptive and depends on factors including dataset size and the manifest's `batchSize`.

```text
OpenRefine batch
≠
stable fachliche delivery unit
```

### F-02 – Current SourceDelivery lifetime follows the HTTP request

The current Reconcilix implementation creates one `SourceDelivery` per OpenRefine POST.

```text
HTTP request lifecycle
=
current SourceDelivery lifecycle
```

This is an implementation property of the current OpenRefine channel. It does not imply:

```text
HTTP POST = fachliche external delivery
```

or:

```text
HTTP POST = OpenRefine reconciliation run
```

### F-03 – externalDeliveryId is not a run identifier

Because `delivery_id` is part of the service configuration stored in OpenRefine, it can remain stable across many reconciliation runs.

Therefore this correlation would be incorrect:

```text
tenant_id
+
delivery_channel
+
external_delivery_id
=
one reconciliation run
```

A SourceDelivery must **not** be reused solely by `externalDeliveryId`.

### F-04 – No usable run correlation was observed

Within the analyzed request flow, no reliable identifier was observed that Reconcilix could use to correlate all POST batches belonging to exactly one user-triggered reconciliation run.

Reconcilix must therefore not invent such a correlation from timing, batch size or `externalDeliveryId`.

### F-05 – Batch size must not influence domain semantics

The tested values:

```text
100 records    → batches of 10
10,000 records → batches of 100
```

demonstrate that the HTTP batch boundary is variable.

`SourceDelivery` semantics must therefore not be defined by a specific batch size such as 10, 50 or 100.

## 7. Assessment against ADR002

ADR002 models conceptually:

```text
SourceSystem
    ↓
SourceDelivery
    ↓
SourceValue*
```

The current implementation technically satisfies the `SourceValue*` cardinality, but in the OpenRefine channel the lifetime of `SourceDelivery` is currently aligned with a transport request.

ED-04-010 does **not** redefine ADR002. It identifies an unresolved distinction between:

```text
external/fachliche delivery
OpenRefine reconciliation run
HTTP request batch
Reconcilix SourceDelivery
```

A future architectural decision may introduce an explicit concept such as `ReconciliationRun` or `RequestBatch`, or refine the SourceDelivery semantics. That decision is outside ED-04-010.

## 8. Decision

### D-01 – No SourceDelivery lifecycle change now

The current implementation:

```text
1 OpenRefine POST
→
1 SourceDelivery
```

remains unchanged for now.

This is accepted as current technical behavior, not established as final fachliche semantics.

### D-02 – Do not reuse SourceDelivery by externalDeliveryId

No implementation shall merge/reuse SourceDeliveries solely on the basis of:

```text
tenant_id
+
delivery_channel
+
external_delivery_id
```

because the same external delivery ID can span multiple OpenRefine reconciliation runs.

### D-03 – Keep the unresolved lifecycle question explicit

The relationship between fachliche delivery, reconciliation run and transport batch remains an architectural follow-up topic.

It should be revisited when OpenRefine or another client provides a reliable run identifier, Reconcilix requires explicit run-level functionality, or operational/evaluation requirements make a dedicated run concept necessary.

## 9. Consequence for WP3-003b – Discovery Access Audit

WP3-003b can proceed without resolving the run-level semantics.

For the first KISS implementation, an audit entry can reliably reference:

```text
sourceDeliveryId
externalDeliveryId (optional)
tenantId
deliveryChannel
timestamp
query/access identifier
adapter/service
status
duration
```

A `source_delivery.id` is therefore a valid technical correlation identifier for the current request batch.

However:

> WP3-003b must not label a SourceDelivery audit file or record as an “OpenRefine reconciliation run”.

A later GUI or reporting layer may group multiple audit records, but such grouping must not imply a run identity unless Reconcilix has an explicit, reliable correlation mechanism.

## 10. Parking-lot item

### SourceDelivery / ReconciliationRun Semantics

Future questions:

- Does Reconcilix need an explicit `ReconciliationRun` concept?
- Should transport requests/batches become explicit infrastructure objects?
- How should non-OpenRefine delivery channels map to the same model?
- Can future clients supply an explicit run/correlation ID?
- What is the precise semantic boundary between `externalDeliveryId` and `SourceDelivery`?
- Does a future operational GUI require run-level grouping?

No action is required for WP3-003b.

## 11. Final conclusion

Observed integration model:

```text
External Delivery Context
(externalDeliveryId; potentially long-lived)
        │
        ├── OpenRefine Reconciliation Run A
        │       ├── HTTP Batch 1 → SourceDelivery
        │       ├── HTTP Batch 2 → SourceDelivery
        │       └── ...
        │
        └── OpenRefine Reconciliation Run B
                ├── HTTP Batch 1 → SourceDelivery
                └── ...
```

The `OpenRefine Reconciliation Run` level is conceptually observable from the user perspective but is **not currently identifiable by Reconcilix through a stable run ID**.

The HTTP batch size is adaptive:

```text
100 records    → observed batch size 10
10,000 records → observed batch size 100
                 with manifest batchSize = 100
```

OpenRefine can additionally reuse reconciliation results for duplicate queries, so dataset size alone does not determine request volume.

Therefore the current technical mapping:

```text
1 HTTP POST
=
1 SourceDelivery
```

remains unchanged until a reliable run-level correlation mechanism or a refined domain decision exists.

**ED-04-010 is completed. WP3-003b may proceed on this basis.**