Building Heritage Digital Twins with Open-Source Tools

Human-in-the-Loop Semantic Knowledge Production from Documentary Evidence

Daniel Antal

Anna Márta Mester

Ieva Pigozne

What is a Heritage Digital Twin

A very small collection: Delini artefacts

Our starting point

Three heritage artefacts and two documentary records.

For now, we simply observe what we have.

From observations to knowledge

Informative data and usable knowledge

Atomic semantic assertions

An elementary semantic assertion can be represented as

\[(s,p,v)\]

where \(s\) is a subject, \(p\) a predicate, and \(v\) an asserted value (measured coordinates here: 56°59'35.38"N, 24°15'23.25"E) or related entity (photo001.jpg → Photograph 1).

subject	predicate	value
P7101561.jpg	instantiation_of	Photograph 1
Photograph 1	depicts	Delini farmhouse
Delini farmhouse	instance_of	farmhouse
Delini farmhouse	coordinates	56°59'35"N, 24°15'23"E

Betwixt: Review candidates

Betwixt: Review draft

Betwixt: Review draft

Betwixt: create a wide tabular dataset

row_number	plane	subject	label	instance_of	heritage_of
1	candidate	fuds:Q7328	Delini farmstead	farmhouse	Livonians
1	reviewed	fuds:Q7328	Delini farmstead	farmhouse	Livonians
1	status	corroborated	corroborated	corroborated	corroborated
5	candidate	bed (PLM 7201)	bed (PLM 7201)	bed	Livonians
5	reviewed	bed (PLM 7201)	bed (PLM 7201)	bed	Do not know
5	status	corroborated	corroborated	corroborated	deferred

Betwixt: create a long tabular dataset

A#	R#	plane	subject	predicate	value	status
1	1	candidate	fuds:Q7328	label	Delini farmstead	corroborated
2	1	candidate	fuds:Q7328	description	the farmhouse…	corroborated
3	1	reviewed	fuds:Q7328	label	Delini farmstead	corroborated
4	1	reviewed	fuds:Q7328	description	the farmhouse…	corroborated
5	1	candidate	fuds:Q7328	instance_of	farmhouse	corroborated
6	1	candidate	fuds:Q7328	heritage_of	Livonians	corroborated
7	1	reviewed	fuds:Q7328	instance_of	farmhouse	corroborated
8	1	reviewed	fuds:Q7328	heritage_of	Livonians	corroborated

First Review: Read back with the app

Betwixt app

You can use Betwixt directly in your browser:

https://dataobservatory.shinyapps.io/betwixt/

Or download the Betwixt Windows App

https://usebetwixt.com/download/dist/betwixt_Setup_009.exe

Follow this presentation on your computer:
https://downloads.reprex.nl/2026/eadh2026/EADH_HDT_workshop.htm

Two ways to work

• Browser app — no installation required.
• Windows app — works locally on your computer.

Both provide the same review workflow used in the exercises.

Creating candidate assertions

@prefix btx: <https://usebetwixt.com/ns/> .
@prefix ex: <https://example.org/> .

ex:assertion12
    a btx:Assertion ;
    btx:rowNumber 3 ;
    btx:subject "bed (PLM 7201)" ;
    btx:predicate "heritage_of" ;
    btx:value "Do not know" ;
    btx:status btx:Deferred .

• This is standard RDF serialisation.
• See the Betwixt vignette.
• This format can be imported into graph databases and knowledge graphs such as Apache Jena Fuseki, Wikibase and Wikidata.

Exercises

Exercise One

Taxonomic classification

Review the proposed classification of each entity.

1. Open the exercise
EADH Workshop — Exercise 1

2. Examine the evidence
Use the image, description and available definition links to decide whether the proposed classification is appropriate.

3. Review the classification
Keep the proposed class when it is correct, select another available class when necessary, or choose Other… to enter a different value. You may also Defer or Reject an assertion.

4. Save your work
Save a draft when you have reviewed the candidate values.

Desktop app
Download Betwixt for Windows

This first exercise focuses on one semantic operation only: taxonomic classification.

The participants see a small subset of the Dēliņi dataset. Each subject has a proposed instance_of value. Most proposals are plausible, but one deliberate error has been introduced: the Dēliņi farmhouse is classified as a shirt rather than as a building.

Do not reveal the error initially. Ask participants to use the evidence and definition links rather than simply scanning for an obviously unusual value.

Point out that the dropdown represents a candidate range, not necessarily a closed vocabulary. Other… allows the reviewer to propose a value outside the supplied range.

This is also the first opportunity to distinguish changing a candidate value from Defer and Reject. A correction says that another assertion should replace the proposal; rejection says that the proposed assertion should not be retained; deferral leaves the decision unresolved.

Ask participants to save a draft rather than finalising the review. We will return to review states and finalisation later.

Exercise Two

Parts and collections

Review the proposed relationships between entities.

1. Open the exercise
EADH Workshop — Exercise 2

2. Examine the evidence
Use the image, description and available definition links to understand how each entity relates to the others.

3. Review the relationships
Decide whether an entity is physically or structurally part of another entity, or whether it belongs to a curated collection. Correct, defer or reject assertions where necessary.

4. Save your work
Save a draft when you have reviewed the proposed relationships.

This exercise introduces mereology and distinguishes part-whole relationships from curated collection membership.

The selected examples include a farmhouse that is part of the Dēliņi farmstead, an archival record that is part of a record set, and a sash whose relationship to the Dēliņi material is curated rather than mereological.

A deliberate error has been introduced for the sash: it is proposed as part_of the Dēliņi farmstead. Do not reveal the error initially. Ask participants to consider whether the relationship expresses physical or structural composition, or institutional curation and collection membership.

Emphasise that two entities can be strongly associated without one being part of the other. part_of expresses a different semantic relationship from curated_member_of.

The exercise also shows why choosing the correct predicate matters as much as choosing the correct value. A plausible object-value pair can still produce an incorrect assertion if the relationship itself is wrong.

Ask participants to save a draft after reviewing the relationships. We will later use these examples when discussing atomic assertions and the distinction between domain semantics and contextual information.

Exercise Three

Multiple assertions and context

Review several proposed assertions about the same entities.

1. Open the exercise
EADH Workshop — Exercise 3

2. Examine the evidence and context
Use the image, description, definition links and contextual information to interpret each proposed assertion.

3. Review the assertions
Review both the proposed instance_of classifications and heritage_of relationships. Some definitions now refer to an external knowledge source.

4. Save your work
Save a draft when you have reviewed the proposed assertions.

This exercise makes the candidate dataset wider. Instead of reviewing a single semantic predicate, participants now review both instance_of and heritage_of assertions for the selected entities.

The heritage_of candidates introduce Livonians, Latvians and Estonians. Their definitions link to Wikidata, demonstrating that the semantic definition of a candidate value does not have to come from the same knowledge graph or vocabulary as the source dataset.

Some deliberately questionable heritage assignments have been included. The purpose is not to teach an ethnic classification of the objects, but to make participants examine the evidence and decide whether the proposed relationship is warranted.

The exercise also introduces context_held_by. Emphasise that this information is visible because it can help the reviewer understand the object and its institutional context, but it is not itself an assertion under review.

This distinction is important for understanding the review data model. A wide table can contain evidence, descriptive information, contextual fields and several candidate semantic assertions. Their presence in the same row does not give them the same role.

After this exercise we can contrast the wide representation with a long representation of atomic subject, predicate, value assertions.

Exercise Four

Long-form assertions

Review two proposed assertions about the Dēliņi farmhouse.

1. Open the exercise
EADH Workshop — Exercise 4

2. Examine the structure
Unlike the previous exercise, the predicates are no longer separate columns. Each row contains a predicate and a value.

3. Review the assertions
Review the proposed predicate and value in each row. Together with the subject, they form an atomic semantic assertion.

4. Save your work
Save a draft when you have reviewed the proposed assertions.

This exercise presents the same basic semantic information in long rather than wide form.

There is only one subject, the Dēliņi farmhouse, but it appears in two rows. One row proposes instance_of with the value building; the other proposes heritage_of with the value Livonians.

In the previous exercise, instance_of and heritage_of were separate columns. Here they become values of the predicate column, while building and Livonians become values of the value column.

This makes the atomic structure explicit: each row can be read as a subject, predicate, value triple.

Point out that long and wide representations do not necessarily express different knowledge. They can be different tabular projections of the same candidate assertions, chosen for different processing and review purposes.

The contextual field context_held_by remains available to support interpretation, but it is not part of the subject, predicate, value assertion being reviewed.

This prepares the transition from human-readable tabular review to the assertion-oriented representation used later for review states, provenance and RDF serialisation.

Exercise Five

Translation review

Create and review alternative labels and descriptions in your own language.

1. Open the exercise
EADH Workshop — Exercise 5

2. Translate the descriptions
Use Google Translate to translate the supplied English descriptions into your mother tongue and paste the results into the alternative descriptions.

3. Add alternative labels
Give each object a meaningful short label in the same language.

4. Review the translations
Read the machine-generated translations critically and correct them where necessary.

5. Save your work
Save a draft when you are satisfied with your alternative labels and descriptions.

This exercise moves from semantic assertions to natural-language expressions. Participants work with six different objects and create alternative labels and descriptions in their own language.

Ask participants first to use Google Translate rather than translating the descriptions themselves. The machine-generated text is deliberately treated as a proposal that requires human review.

The existing English label and description provide the source material. Participants paste the machine translation into alternative_description and create a concise alternative_label in the same language.

Unlike the semantic candidate columns in the previous exercises, alternative_label and alternative_description are descriptive review fields. They are not controlled selections from a candidate range and they do not require vocabulary definitions.

The important point is that human-in-the-loop review is not limited to ontology or knowledge-graph assertions. Natural-language expressions generated by an automated agent can also enter the workflow as proposals that a competent human reviews and corrects.

Encourage participants to look for errors that are easy for a fluent speaker to recognise but difficult to detect mechanically: awkward terminology, incorrect interpretation of compound object names, inappropriate word choice and descriptions that are grammatically acceptable but misleading in the heritage context.

This exercise prepares the discussion of agency: Google Translate acts as a simple external AI agent producing candidate expressions, while responsibility for accepting or correcting those expressions remains with the human reviewer.

Translation Review: Read back with the app

Betwixt app

You can use Betwixt directly in your browser:

https://dataobservatory.shinyapps.io/betwixt/

Or download the Betwixt Windows App

https://usebetwixt.com/download/dist/betwixt_Setup_009.exe

from the workshop repository and run it locally.

Two ways to work

• Browser app — no installation required.
• Windows app — works locally on your computer.

Both provide the same review workflow used in the exercises.

Generative AI

Discussion of agency and human control

Programmatic generation of review candidates

review_input <- tibble::tribble(
  ~page_id, ~label_en, ~description_en, ~label_hu, ~description_hu,
  ~page_url, ~thumbnail_url,
  ~subject, ~predicate, ~value, ~value_definition,
  "635780",
  "sweater, women's",
  "MuIS museum record 635780.",
  "női kardigán",
  "A MuIS 635780 számú múzeumi rekordja.",
  "https://www.muis.ee/museaalview/635780",
  paste0(
    "https://www.muis.ee/digitaalhoidla/api/meedia/pisipilt?",
    "id=ebc07930-f719-44f2-a108-6698bcecc20b"
  ),
  paste0(
    "https://www.muis.ee/digitaalhoidla/api/meedia/pisipilt?",
    "id=ebc07930-f719-44f2-a108-6698bcecc20b"
  ),
  "depicts",
  "300209900",
  "http://vocab.getty.edu/page/aat/300209900",
  "633053",
  "gloves",
  "MuIS museum record 633053.",
  "kesztyű",
  "A MuIS 633053 számú múzeumi rekordja.",
  "https://www.muis.ee/museaalview/633053",
  paste0(
    "https://www.muis.ee/digitaalhoidla/api/meedia/pisipilt?",
    "id=6440f24f-eaad-4cd4-84d7-1ae7a9d44d5a"
  ),
  paste0(
    "https://www.muis.ee/digitaalhoidla/api/meedia/pisipilt?",
    "id=6440f24f-eaad-4cd4-84d7-1ae7a9d44d5a"
  ),
  "depicts",
  "300148821",
  "http://vocab.getty.edu/page/aat/300148821",
  "635778",
  "shirt, women's",
  "MuIS museum record 635778.",
  "női ing",
  "A MuIS 635778 számú múzeumi rekordja.",
  "https://www.muis.ee/museaalview/635778",
  paste0(
    "https://www.muis.ee/digitaalhoidla/api/meedia/pisipilt?",
    "id=826c402e-c130-4860-b11e-9538bd403ecf"
  ),
  paste0(
    "https://www.muis.ee/digitaalhoidla/api/meedia/pisipilt?",
    "id=826c402e-c130-4860-b11e-9538bd403ecf"
  ),
  "depicts",
  "300212499",
  "http://vocab.getty.edu/page/aat/300212499"
)

—>

Provenance

Provenance

Review again the data

From review to status

The candidate and reviewed values are compared only after the human review.

What happened?	Review status
The row has not yet been finalised	pending
The reviewer kept the proposed value	corroborated
The reviewer replaced it with another value	corrected
The reviewer deleted the proposed value	rejected
No optional descriptive assertion was made	deferred
The reviewer explicitly chose Defer	deferred
The reviewer explicitly chose Reject	rejected

Explicit review decisions take precedence over the comparison of values.

Three views of the review

The same review can be projected as three aligned tables:

• candidate — what was proposed;
• reviewed — what the human reviewer retained or changed;
• status — what happened during review.

The same coordinate refers to the proposed assertion, its reviewed state, and the review decision.

From the review matrix to assertions

The reviewed matrix can be projected back into atomic assertions.

subject	predicate	value	status
Delini farmhouse	instance_of	farmhouse	corroborated
Delini farmhouse	coordinates	…	corrected

This is the bridge from a human review interface to machine-readable semantic knowledge.

BREAK

Representation and represented entity

What is evidence?

Exercise Four

Long-form assertions

Review two proposed assertions about the Dēliņi farmhouse.

1. Open the exercise
EADH Workshop — Exercise 4

2. Examine the structure
Unlike the previous exercise, the predicates are no longer separate columns. Each row contains a predicate and a value.

3. Review the assertions
Review the proposed predicate and value in each row. Together with the subject, they form an atomic semantic assertion.

4. Save your work
Save a draft when you have reviewed the proposed assertions.

This exercise presents the same basic semantic information in long rather than wide form.

There is only one subject, the Dēliņi farmhouse, but it appears in two rows. One row proposes instance_of with the value building; the other proposes heritage_of with the value Livonians.

In the previous exercise, instance_of and heritage_of were separate columns. Here they become values of the predicate column, while building and Livonians become values of the value column.

This makes the atomic structure explicit: each row can be read as a subject, predicate, value triple.

Point out that long and wide representations do not necessarily express different knowledge. They can be different tabular projections of the same candidate assertions, chosen for different processing and review purposes.

The contextual field context_held_by remains available to support interpretation, but it is not part of the subject, predicate, value assertion being reviewed.

This prepares the transition from human-readable tabular review to the assertion-oriented representation used later for review states, provenance and RDF serialisation.

Evidence and the subject / provenance

Filesystem observations

A digital file is also an observed entity.

fscontext records the filesystem context in which a file was observed:

• file name and path
• size and modification time
• media type
• checksum / content identity

The observation does not claim what the file means.

It establishes which digital object was available to the knowledge-production process at a particular point in time.

fscontext

A preserved web resource

Preserving an observation

A URL tells us where we found something.

It does not preserve what we saw.

A WACZ capture preserves the web resource as it was observed, including its content and capture context.

The preserved resource can therefore become part of the evidence trail.

This is a preserved web resource opened from a WACZ archive in ArchiveWeb.page.

The distinction is important: a URL is a reference to a location on the web. The resource available at that location can subsequently change or disappear.

For semantic knowledge production, we may need to preserve the resource that was actually available when an assertion was created or reviewed.

The WACZ capture gives us a preserved observation of that resource. It can include the retrieved content and embedded resources together with information about the web capture.

This does not make the preserved page true, nor does it automatically make everything on the page evidence. It gives us a stable digital object that can be referred to when documenting what a reviewer was able to inspect.

This extends the same principle introduced with filesystem observations: preserve the observation first; interpretation and evidential use come later.

What evidence was observed with what kind of competenece?

Normative Plane

needs to change

Modelling governance

Authorship & other personal rights

A representation is not only about something. It is also made by someone, under particular conditions.

Once a representation may be published or reused, new questions appear:

Who created it? → authorship and attribution: in Europe, authors have rights concerning attribution, remuneration and the publication of their works.

Under what conditions may it be reused? → copyright and licensing.

Who or what is represented in it? → personality, privacy and consent; a CC BY licence on the representation may not be enough.

These are different assertions. They may require different evidence and may lead to different review decisions. Copyright, and related rights for sound and audiovisual recordings, concern rights in the representation. But the rights or legitimate interests associated with represented persons—and, in some jurisdictions, restrictions concerning represented works or objects, such as freedom-of-panorama rules—may also need to be considered together with the rights governing the representing medium.

Up to this point, the exercises have concentrated mainly on the semantic plane: what an entity is, how entities relate to one another, and how a representation relates to what it represents.

We now move into the normative plane.

The important transition is that a digital representation does not become reusable merely because we have described it correctly. It was created by someone, it may be governed by copyright or licensing conditions, and it may depict people whose interests are independent of the copyright in the photograph.

These questions should not be collapsed into a single field such as “rights cleared” or “open”.

The creator of a photograph is one assertion. The licence under which the photograph is made available is another. Whether recognisable people are depicted is another. Their identity and whether they may still be alive introduce further assertions.

The represented entity can matter as well. A photograph may itself be openly licensed while the work, person, building or other subject represented in it introduces additional rights or restrictions. This is why the rights governing the representation and those associated with what is represented need to remain distinguishable.

These assertions can also depend on different evidence. A museum record may support authorship or licensing information. A photograph itself may show that people are present. Identification may depend on catalogue documentation or contextual knowledge. Consent may require yet another document.

The next two exercises introduce this incrementally. Exercise Seven starts with the comparatively familiar questions of authorship and reuse rights. Exercise Eight then shows why resolving copyright is not necessarily the end of the normative review: the content of the representation itself may introduce questions concerning identifiable persons.

Exercise Seven

Authorship and reuse rights

Review authorship and reuse rights for four photographs.

1. Open the exercise
EADH Workshop — Exercise 7

2. Examine the evidence
Use the photographs, descriptions and source records to determine their provenance.

3. Review the assertions
Review creator and use_rights separately. The creator of a photograph and the licence governing its reuse are different assertions.

4. Save your work
Save a draft when you have reviewed the proposed assertions.

This exercise introduces the normative plane through authorship and reuse rights.

The four photographs come from two different sources. Two are photographs from the Dēliņi dataset created by Daniel Antal and made available under CC BY-SA 4.0. Two come from MuIS, with institutional authorship and CC BY reuse rights.

The evidence links lead back to the corresponding source records: the Dēliņi records in our knowledge base and the MuIS museum records for the Estonian objects. Participants can therefore inspect the source information rather than relying only on the candidate values presented in the review table.

Emphasise that creator and use_rights are separate assertions. Knowing who created a digital representation does not by itself establish the conditions under which it may be reused. Conversely, seeing a reuse licence does not establish who should receive attribution.

This is the first exercise concerned explicitly with the normative plane. The semantic description of an object may be perfectly adequate while information governing the use of its representation remains incomplete or incorrect.

The exercise also prepares the next step. Even when both authorship and copyright licensing are known, this does not necessarily settle whether a photograph can be published or reused. Photographs containing people can introduce additional questions concerning identification, living persons and personality rights.

Exercise Eight therefore adds another layer of normative review rather than replacing the copyright information established here.

Exercise Eight

Persons and personality rights

Review information about people depicted in photographs.

1. Open the exercise
EADH Workshop — Exercise 8

2. Examine the evidence and context
Use the photograph, description, evidence links and contextual names to understand who, if anyone, is depicted.

3. Review the assertions
Review whether recognisable people are present, whether they are identified, and whether any of them may still be alive.

4. Save your work
Save a draft when you have reviewed the proposed assertions.

This exercise moves from copyright and authorship to personality and privacy considerations concerning people depicted in photographs.

The first two Dēliņi photographs provide simple cases in which no people are present. These establish that questions concerning depicted persons are not applicable to every photographic representation.

The inspection photograph provides a contemporary case. Ieva Pigozne is recognisable and identified, and the evidence link points to supporting documentation. The photograph therefore illustrates the distinction between recognising that a person is present, identifying that person, and considering their living status.

The first Garamantas photograph provides a different case. People are visible, but the available contextual information does not individually identify everyone depicted. Their living status is also not established by the available information.

The second Garamantas photograph identifies Arvīds Knapše by name, but the available information does not by itself establish whether he is still alive. This demonstrates that identification and living status are separate assertions and may require different evidence.

Emphasise that the three reviewed fields answer different questions: humans_present concerns what can be observed in the image; persons_identified concerns whether identities have been established; and persons_alive concerns information that may require evidence beyond the photograph itself.

This is why a clear copyright or reuse licence is not necessarily sufficient for deciding whether a photograph should be published or reused. A digital representation can have known authorship and copyright status while questions concerning identifiable depicted persons remain unresolved.

The normative plane therefore contains several distinct kinds of claims. They should be made explicit and reviewed independently rather than collapsed into a single statement that an image is simply “open” or “restricted”.

Summary: Semantic Knowledge Production

What exactly are we trying to build?

Heritage Digital Twin

A maintained, inspectable state of warranted knowledge about a heritage entity, explicitly connected to its evidence and to the processes through which that knowledge was established.

Linguistic warrant
Connects terms across languages and collections.

Evidential warrant
Connects evidence to claims about the heritage entity.

Competency warrant
Records the professional judgement needed to establish claims.

Four planes of knowledge production

Knowledge sharing: bounded federation

Bounded federation

Federation is defined relative to a pragmatic purpose and a set of competency questions.

Only candidate assertions within the federation boundary are reviewed for exchange.

Sharing knowledge without sharing an ontology

Two knowledge graphs may express the same proposition using different identifiers and vocabularies.

FUDSS

Q5613 P14 Q6

Wikidata

Q83326 P31 Q5

Both express:

Béla Bartók → instance of → human

But all three semantic components differ.

Mappings are also claims

To exchange the assertion we need three correspondences:

	FUDSS	Wikidata
subject	Q5613	Q83326
predicate	P14	P31
value	Q6	Q5

These mappings should not simply be hidden in transformation code.

They can themselves be proposed, reviewed, corrected, rejected or deferred.

References

References

Antal, Daniel. 2026a. A Review Algebra. Version v20260825. Reprex. https://doi.org/10.5281/zenodo.22091484.

Antal, Daniel. 2026b. Betwixt. Version v20260825. Reprex. https://doi.org/10.5281/zenodo.22091536.

Antal, Daniel. 2026c. Bounded Federation: Reviewable Semantic Exchange Between Independently Governed Knowledge Graphs. Version v20260825. Reprex. https://doi.org/10.5281/zenodo.22091553.

Antal, Daniel. 2026d. fscontext: Reconstructing Filesystem Context for Provenance-Aware Digital Collections. https://doi.org/10.32614/CRAN.package.fscontext.

Antal, Daniel, Kata Gábor, Ieva Pigozne, and Bogáta Timár. 2026. “Federating Open Knowledge Through Wikibase: The Case of the Finno-Ugric Data Sharing Space.” Digital Humanities in the Nordic and Baltic Countries Publications 7 (4). https://doi.org/10.5617/dhnbpub.12939.

Antal, Daniel, and Ieva Pigozne. 2025. “Linking Garments to Knowledge: TextileBase as an Interdisciplinary Graph for Dress and Textile Research.” Culture Crossroads, ahead of print, December. https://doi.org/10.55877/cc.vol31.571.

Contact