Data integration

Status: current technical guide. The purpose of connectivity is not to accumulate more points, but to make one real run useful for engineering judgment.

This document defines the integration contract for equipment and business data entering Ingot, including the identity semantics of SiteId, EdgeId, and ExecutionId, plus provenance, mapping, time, quality-result linkage, and analysis-admission requirements. Every real integration must verify data ownership, traceable provenance, and quality-result linkage.

Definition of a complete connection

A qualified data chain answers:

• Which equipment, product, and run is this?
• What did the engineer plan, and what did the equipment actually use?
• How did the process evolve through its stages?
• Which material, tooling, lot, calibration, and maintenance state applied?
• What were the final quality and safety outcomes?
• Where did each value come from, in which unit and configuration version?

Data enters process diagnosis and recipe optimization only when these facts can be linked reliably.

Data sources

Ingot does not treat a PLC as the only source. One run may combine:

• process specifications, state, stages, and process signals from a control system;
• measurements from instruments, sensors, vision, or edge gateways;
• laboratory, inline, or manual inspection results;
• MES, QMS, work orders, barcodes, and lot traceability;
• tooling, material, calibration, and maintenance records.

Adapters map raw values to stable business codes, standard units, quality state, time, and provenance. R&D projects reference those semantics rather than vendor addresses or protocols.

Identity and linkage

Stable identifiers have distinct, non-interchangeable responsibilities:

• SiteId: owning production cell and the boundary for site isolation, capacity, and failure domains;
• EdgeId: stable installed-node identity bound to SiteId and independent credentials;
• ExecutionId: generated by Edge and used as the unified identity for a real process execution and its field events;
• ExecutionKey: association between an R&D experiment plan and real execution.

Platform must record the ExecutionKey–ExecutionId relationship explicitly. Production events and typed process samples both preserve SiteId and EdgeId; object summaries are site-scoped and cannot merge same-named equipment from different factories. MES work orders, barcodes, instrument sample IDs, or equipment registers may carry the ExecutionKey as an external reference, but they do not replace the Edge-generated ExecutionId. Inspection records use the same relationship.

If a controller stores numbers only, maintain a deterministic mapping between short numbers and ExecutionKeys. Never guess execution-to-inspection linkage after the fact from time proximity alone.

Configuration lifecycle

Acquisition configuration progresses through:

draft → target-Edge probe → real-value validation → publish → local validation → safe application → state confirmation

Before publishing, verify:

• address, port, authentication, and network reachability;
• real point readability;
• data type, byte order, scale, and offset;
• raw and converted values are physically plausible;
• required points, run boundaries, and context sources exist;
• standard units match the process model.

Edge retains the last successful version. A candidate passes startup health only after producing a decodable snapshot with every required value; a connection or OPC UA subscription alone is not success. A failed version leaves the old one running and reports failure rather than silently switching to unversioned local configuration.

Runtime status reports read attempts, successful device reads, valid snapshots, emitted events, inactive empty snapshots, duplicate suppression, stalled source identity, and the latest error separately. running is based on valid data rather than one overloaded success counter. Delivery status also reports pending event count, oldest pending age, queue capacity usage, storage size, recent shipping rate, and estimated drain time; high queue occupancy degrades health. Platform retains seven days of heartbeat health snapshots, timestamped on receipt, so operators can identify when acquisition or backlog failed and recovered without trusting a drifting field clock for online status.

HTTP and MQTT document payloads use bounded parsing. An HTTP body or decompressed MQTT payload over 16 MiB, or JSON deeper than 64 levels, is rejected so a malformed source cannot exhaust Edge memory. HTTP supports GET/POST, content type, request body, fixed headers, and secret-reference headers. Authorization, Cookie, and common API-key headers must use secret references. Fixed bodies and ordinary headers are stored and exported verbatim, so credentials must not be placed in them. An HTTP data path must remain relative to the configured device base address, and device requests do not follow redirects, preventing secret headers from reaching another host. Connection and single-operation timeouts have a 1000ms minimum; a timeout degrades and retries the task instead of silently terminating its worker.

Driver capabilities

Protocols support different configuration. One capability matrix drives both the UI and publication validation so a field cannot be accepted and then ignored by a driver.

Modbus and MELSEC never blind-scan addresses that the user did not declare.

Point selectors

Modbus TCP

<register-area>:<address>:<type>[:<byte-order>:<word-order>]
<register-area>:<address>.<bit>:boolean

Register areas are holding-register, input-register, coil, and discrete-input. Coils and discrete inputs can only be boolean. A bit inside a holding or input register requires an explicit offset.

The fourth string segment is byte length. Connection configuration defines address origin; when manuals use one-based addresses, Edge converts them before the protocol request.

Mitsubishi MC A-compatible 1E

<device>:<number>:<type>[:<byte-length>]
<word-device>:<number>.<bit>:boolean

• X / Y numbers are octal.
• B / W numbers are hexadecimal.
• Other common devices are decimal.
• M / X / Y / B / S / L use bit-unit reads for boolean.
• Reading a bit device as int16 returns a packed word of consecutive bits.

Adjacent points may be merged into contiguous reads to reduce network round trips. Both Modbus and MELSEC limit the maximum merge gap and do not cross large undeclared address ranges. Set merge distance to zero when diagnosing an individual address.

Register-based device time explicitly declares unix-s, unix-ms, or iso-8601 encoding. Publication validation checks selector width so a 32-bit Unix-seconds value is not interpreted as milliseconds.

MQTT with multiple topics

A gateway may split one equipment snapshot across topics:

• each subscription declares a stable channel code, actual topic, optional payload root, and topic variables;
• a point may bind to its source topic;
• values merge into an equivalent sample snapshot;
• a sample is emitted only after every required point has appeared;
• context-only topics update the snapshot without emitting samples alone;
• configuration referencing an unsubscribed topic is rejected.
• overlapping subscription filters that can receive the same message are rejected because payload roots, channels, and topic variables would be ambiguous;
• a topic-variable level index must be within the filter's level range.

Set a reasonable maximum age for cross-topic values. When a topic stops, stale values must become missing rather than remaining current indefinitely.

Payloads may declare UTF-8, GBK, GB 18030, or Big5 encoding and no compression, gzip, deflate, or Brotli compression. Probe and runtime acquisition share the same bounded decode, decompress, and JSON-parse path.

Equipment templates and instances

Bulk integration of similar equipment uses four objects with separate responsibilities:

A data source and a task binding are therefore not the same configuration. The former answers “which device is connected”; the latter answers “which ingestion template this device uses.” Templates do not support arbitrary inheritance or point overrides; mapping changes create a new template version.

After the first device passes a real probe through its target Edge, an operator can extract a published template, first-device source, and binding from the published task. Extraction is a configuration migration: it creates a new task version with provenance and retires the prior runtime version instead of rewriting an already-published version. MQTT templates retain stable channel codes that materialization resolves to each source's actual topics.

The four CSV actions are import and export for two different objects, not duplicate features:

1. Export the data-source CSV to obtain its header and existing device rows.
2. Add or change the Edge, identity, address, and secret references for similar devices, then import the data-source CSV.
3. Export the task-binding CSV to obtain its header and existing template–source combinations.
4. Select a template version for each new source, then import the task-binding CSV.

Both files are headed UTF-8 CSV, support up to 500 rows per atomic batch, and reject the entire batch if any row is invalid. Import data sources before bindings that reference them. Credential fields store only secret references. Fixed request bodies and ordinary headers are exported with data sources and must not contain credentials. Exports neutralize spreadsheet-formula prefixes so names and other text are not executed by spreadsheet software. Batch materialization creates drafts only; every device must pass its own real probe before publication.

The template, data source, binding, and final task all retain version provenance. A published version is immutable and changes require a new version. This makes the mapping and connection versions behind each field event auditable instead of losing provenance through copied configuration.

Value conversion and quality policy

Each mapping can declare source type and unit, scale, offset, quality path and accepted quality values, valid range, out-of-range behavior, missing-value behavior, and a default value. Publication validation ensures that:

• the target item exists in the referenced process data model;
• every non-null item is mapped and cannot omit missing or out-of-range values;
• units match the process model; scale and offset do not stand in for an undeclared unit conversion;
• defaults convert to the target type, and integer targets reject fractional conversion results;
• booleans accept explicit true/false forms or 0/1 rather than silently treating every nonzero number as true;
• OPC UA rejects every non-Good status unless explicitly accepted;
• the publication probe must observe every configured value, quality, context, device-time, device-sequence, and process-specification path at least once, including runtime-optional points. Leave the sequence path empty when the device does not provide one; a configured but unverified path blocks publication. Probe intermittent fields while the device can emit them.
• device timestamps may not lead Edge receipt time by more than 300000ms by default. The threshold is configurable and violating samples are rejected, preventing a bad device clock or encoding from corrupting the event timeline.

OPC UA subscription startup checks each MonitoredItem's server status. Runtime acquisition enforces maximum value age and timestamp skew across required points, and prolonged notification silence during an active run degrades health. The current OPC UA driver acquires variable nodes, not events or alarms; this is an explicit driver boundary.

Business mappings

Controlled variables

control-parameter:<control-parameter-code>
signal:<signal-code>:<feature-code>
signal:<signal-code>:<feature-code>:<stage-code>

For example:

control-parameter:holding-temperature
signal:temperature:mean:holding

Objectives and outcome constraints

inspection:<characteristic-code>
inspection-outcome:<inspection-definition-code>

For example:

inspection:form-error
inspection:crack-rate
inspection-outcome:final-inspection

inspection-outcome uses the dimensionless unit 1 and maps a formal inspection record's PASS outcome to 1 and FAIL outcome to 0. INCONCLUSIVE is never guessed and excludes the run. After an explicit mapping exists, a missing actual value also excludes the run. The system never fills model training with planned values.

Runs and stages

Run boundaries and stages have different responsibilities:

• start and completion events determine run completeness;
• stage numbers align trajectories and calculate stage features;
• stages may vary in length, repeat, or jump and do not alone determine completeness.

Process features may include mean, extrema, standard deviation, slope, integral, peak, overshoot, arrival time, dwell time, and coverage. Feature definitions are versioned; changing a definition changes its version and content hash.

Manufacturing context

Context resolves from field signals, business systems, or time-effective registries. At run start, the system freezes a snapshot and capture status:

• supplied and resolved by the source;
• resolved from versioned manufacturing context;
• missing configuration or unresolved.

If a scenario-required field cannot resolve, the run does not enter that analysis. Other fields remain available for traceability and coverage assessment.

Data quality and analysis admission

Before analysis, verify at least:

• the run is complete;
• actual settings exist;
• process data and required features are available;
• objective and outcome-constraint inspections are complete;
• required context has resolved;
• units, time, and configuration versions agree;
• numbers are finite;
• run-to-inspection linkage is unique.

Excluded runs retain raw records, source hashes, and reasons. The data-quality view should answer what is missing, why, which analysis is affected, and how to repair it.

New-scenario connection order

1. Define the engineering problem, run boundary, and stable equipment identity.
2. Select or implement a protocol driver.
3. Build equipment templates and point mappings.
4. Define actual process specifications, signals, stages, and units.
5. Configure manufacturing context and inspection characteristics.
6. Validate real values through the target Edge and publish.
7. Complete one run–trajectory–inspection loop.
8. Review data quality and analysis admission.
9. Enter comparison, diagnosis, and optimization only after engineer review.