Harvard Art Museums · generative AI · restoration

Fragments Made Whole

Nine broken works, each restored twice by an image model: once knowing nothing, once knowing what the museum knows. The question is what the machine makes up.

A course project by ruminate18 · built with the Harvard Art Museums API and Prodia

The experiment

Same model, same photograph, two prompts. Every fragment's museum photograph went to the same image model (Gemini 3 Pro, through Prodia) twice.

Uninformed: Restore this damaged artwork to show how it originally looked when it was complete and new., the same words for every object.

Evidence-led: a prompt built only from the museum's record and published scholarship, which tells the model what survives, what is known and what not to add.

Then each result was read against the record: what the evidence supports, and what the model invented. 18 images, $2.70 of course credit.

Almost everything in an archaeology gallery is a piece of something larger. A head without its body, a band without its tunic, a painting cut out of its book. Museums answer the question "what was the rest?" carefully, because the answer becomes what visitors believe. The 1964 Venice Charter, still the reference point for conservators, says restoration "must stop at the point where conjecture begins." Cesare Brandi's theory of restoration asked that any new work be recognizable as new when you look closely, so no one mistakes a repair for the original.

Generative image models don't stop where conjecture begins. Ask one to restore a fragment and it answers at once, in seamless high resolution, with no visible join between what survives and what it imagined. That is the Ecce Homo problem at machine speed: in 2012 an amateur's well-meant repainting of a fresco in Borja, Spain, replaced a damaged face with a new one. The difference is that the model's version looks right.

The models also carry assumptions. Ancient sculpture and relief were painted, but centuries of white marble in museums trained the modern eye, and probably the training data, to expect bare stone. Reconstructions like Vinzenz Brinkmann's Gods in Color still surprise visitors. So the experiment asks two things of each fragment. What does a model assume when it knows nothing? And when the prompt carries the museum's evidence (pigment traces, the slab next door in another museum, the tunic in Boston), does the guess get better, or only more confident?

The nine works come from the Harvard Art Museums and were chosen for the kind of loss they show: lost color, lost body, lost room, lost garment, lost floor, lost page, and in the last case, almost everything. Each case below shows the fragment, both reconstructions, the evidence, and a reading of what the evidence supports and what the model invented.

01
Lost color, lost tomb

Relief fragment from the tomb of Niankhnesut: Man holding a cow by a rope

Egyptian · 2323-2150 BCE · Harvard Art Museums 1934.17

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • Carved in shallow raised relief and originally painted; some color is still visible (museum record).
  • From the mastaba tomb of the official Niankhnesut at Saqqara, Dynasty 6. Found in 1917 already disturbed; at least 60 fragments were sold and are now in at least fifteen museums in eight countries.
  • A raised line along the top marks the edge of a register, and a hieroglyphic caption sits at upper center.
  • Old Kingdom painters used flat mineral colors with set conventions: red-brown skin for men, black hair, outlines in black or red, and individually painted hieroglyphs.
Read the evidence-led prompt

Repaint this ancient Egyptian limestone relief exactly as it looked when newly finished in an Old Kingdom tomb at Saqqara, about 2300 BCE. Keep the carving, figures, composition, and the block's edges exactly the same; do not add anything beyond them. Apply flat, matte mineral paint following Old Kingdom conventions: the herdsman's skin red-brown ochre, his hair black, his short loincloth white; the cow's coat painted in solid color with dark details; the rope dark; crisp black and red outlines; each hieroglyph at upper center painted separately in blue, green, red and black; a pale grey-white background; the raised register line along the top painted as a colored band. No shading, no gloss, no modern lighting effects.

The uninformed guess

  • Cleaned and sharpened the carving into crisp, bright white limestone.
  • Added no paint at all, although the museum records that the relief was painted and some color survives.
  • Redrew the herdsman's body and pose; he now holds a staff he does not have in the carving.

What the evidence supports

  • Flat, unshaded paint in Old Kingdom conventions: red-brown skin, black hair, a white loincloth.
  • The register line along the top painted as a colored band.
  • The man, the cow and the rope kept in their carved positions.

What the machine invented

  • Large horns on the cow that the surviving carving does not show.
  • New hieroglyphs. The carved signs at upper center became different signs, so the painted caption no longer says what the stone says.
  • Fresh paint laid over ancient pitting and chips, mixing 'new' and 'damaged' in one image.

Verdict. Evidence brought back the color the record mentions, but the model rewrote the one piece of text on the block. A visitor would have no way to know the caption is invented.

02
Lost body

Fragment of a Wall Relief: Head of a Winged Protective Spirit

Neo-Assyrian · 883-859 BCE · Harvard Art Museums 1940.13

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • Part of slab B-22 in the throne room (Room B) of Ashurnasirpal II's Northwest Palace at Nimrud, placed to the right of the throne base (museum record).
  • The genie wears the horned cap of a deity and was probably performing a ritual. The king's 'standard inscription' ran across the figure's body.
  • The legs below the knees stayed in place in the palace; the middle of the figure does not survive. The neighbouring slab B-23 is in the British Museum.
  • The museum notes the reliefs were originally enhanced by paint.
Read the evidence-led prompt

Show the complete Neo-Assyrian alabaster wall relief this carved head came from, as it looked when new in King Ashurnasirpal II's throne room at Nimrud, about 870 BCE. Keep the head, the horned cap, the beard, and the carving style exactly as in the photograph, and continue the slab downward to show the whole standing figure in profile facing left: a bearded winged protective spirit with two large feathered wings, wearing a fringed robe, armlets and bracelets, performing a ritual gesture. Carve a band of cuneiform inscription in horizontal lines straight across the middle of the figure's body and robe, in the same low relief. Keep the stone mostly its natural pale alabaster color with restrained paint: black hair and beard, black outlines of the eyes, and touches of red on details. Plain uncarved background. Show the full slab head to feet.

The uninformed guess

  • Cleaned the head and kept the slab's broken outline, as if the fragment were the whole work.
  • Added a tasselled necklace that is not in the carving.
  • No color, though the museum notes these reliefs were painted.

What the evidence supports

  • A full standing winged figure, as the record says the slab showed.
  • A band of cuneiform carved straight across the body, matching the record's 'standard inscription' running across the genie.
  • Restrained paint: black hair and beard, red touches on the wings and robe.

What the machine invented

  • It removed the horned cap, the detail the record singles out as marking him as divine, and gave him a plain domed cap.
  • It turned the figure around: the surviving head faces left, the reconstruction faces right.
  • The face and beard were redrawn rather than kept, and the cuneiform band is almost certainly decorative marks rather than Ashurnasirpal's actual text.

Verdict. The most convincing image of the nine, and it is wrong in two ways a specialist would notice in seconds: the god has lost his horns and faces the wrong way.

03
Lost color, trimmed edges

Relief Fragment: Ahuramazda in the Winged Disk

Achaemenid · reigns of Xerxes I and Artaxerxes I, 486-460 BCE · Harvard Art Museums 1943.1062

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • From the west side of the southeast doorway of the Hall of 100 Columns at Persepolis, reign of Xerxes I and Artaxerxes I, 486–460 BCE (museum record).
  • The figure hovered high inside the doorway above an image of the enthroned king, who was carried by representatives of the empire's subject peoples. Most of the doorway is still at Persepolis.
  • The block's top and bottom edges are smooth where it joined other blocks; its sides are ragged, where it was trimmed to make a collector's item.
  • Traces of blue, green and red paint remain on the limestone (museum record).
Read the evidence-led prompt

Show this Achaemenid Persian limestone relief block as it looked when new, high in a doorway at Persepolis around 470 BCE. Keep the god's figure, his crown, beard, raised hand, the ring around his waist and the carving exactly as in the photograph. Restore what was trimmed off the sides: complete the winged disk with two long, straight, horizontal wings of layered feathers spreading symmetrically to left and right, and the feathered tail below the ring, in the same carved style. Paint the relief with the colors whose traces survive: blue, green and red, applied to the feathers, the crown, the robe and the ring, leaving the background as smooth pale grey limestone. Show the whole wide block, flat and frontal.

The uninformed guess

  • Completed the wings, but folded them upward around the figure inside a huge ring.
  • Kept the grey stone unpainted, though the record lists traces of blue, green and red.
  • Treated the trimmed collector's block as the original shape.

What the evidence supports

  • Long horizontal wings and a tail, the familiar form of the Achaemenid winged disk.
  • Paint in exactly the three colors whose traces survive: blue, green and red.
  • Curling streamers below the ring, as in other Achaemenid versions of the motif.

What the machine invented

  • A whole doorway, with the god mounted on the lintel facing out. The record places him high inside the doorway, above a carved scene of the enthroned king carried by the empire's peoples, which the model left out.
  • Saturated, glossy color that reads as modern enamel rather than paint on limestone.
  • The god's carved details are simplified at this scale.

Verdict. The evidence-led version gets the motif and the palette right and the architecture wrong. It restored the sculpture and lost the setting that gave it meaning: the god blessing the king.

04
Lost room

Wall Painting Fragment from the Villa at Boscotrecase

Roman · 10 BCE-1 BCE · Harvard Art Museums 1921.37

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • From the 'Mythological Room' (room 19) of the villa at Boscotrecase near Pompeii, about 10–1 BCE; an inscription suggests the owner was Agrippa Postumus, Augustus's grandson (museum record).
  • It comes from the top register of red-paneled walls: a golden frieze between narrow white bands with red fleur-de-lis, above a bright red panel.
  • Two vignettes survive in the gold band: a kneeling figure in a blue headdress and red garment, and two birds at a puddle of water. The secco paint has worn.
  • A fine example of the Pompeian Third Style, with its thin, delicate architecture and Egyptianizing details favored after the victory over Cleopatra.
Read the evidence-led prompt

Zoom out from this Roman wall painting fragment to show the whole painted wall of the room it came from, as it looked when new around 10 BCE in a villa near Pompeii. Keep this exact strip, with its golden frieze, small vignettes, white bands with red fleur-de-lis and the red below, as the upper zone of the wall. Below it, show the large main zone in the same bright red, in the refined Pompeian Third Style: very thin, delicate painted columns and candelabra dividing the red into panels, and at the center a framed mythological landscape picture with small figures. At the bottom, a dark dado. Flat frontal view of the wall, fresh unfaded fresco colors, no furniture and no people in the room.

The uninformed guess

  • Filled the cracks and the missing corner of red, and repainted the worn vignettes, giving the kneeling figure a full blue robe.
  • Restored the modern wooden museum frame as if it were part of the painting.
  • Kept to the fragment's edges: a cleaner strip, not a wall.

What the evidence supports

  • A red wall with a gold frieze above and a dark dado below, the scheme the record describes.
  • Thin painted columns and candelabra dividing the red into panels, characteristic of the Third Style.
  • A framed picture at the center of the main zone.

What the machine invented

  • The frieze itself. The fragment's white bands with red fleur-de-lis disappear, replaced by garlands and extra vignettes.
  • The central picture, a cupid scene in a much later style. The large pictures from this room are known: those now in the Metropolitan Museum show Polyphemus and Galatea, and Perseus and Andromeda.
  • The wall's proportions and the number of panels.

Verdict. A good guess at the type of wall and a poor guess at this wall. The evidence that would fix it exists in another museum, which is the case for reconstructions made by people who can go and look.

05
Lost body

Acrolithic Head from a Roman Cult Statue of a Goddess

Graeco-Roman · late 2nd century CE · Harvard Art Museums 1905.6

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • An acrolithic head from a Roman cult statue of a goddess, late 2nd century CE, carved from Greek island marble (museum record).
  • In an acrolith, only the exposed parts (head, hands, feet) were marble. The body was made of cheaper materials such as wood, then covered with drapery, gilding or paint.
  • The goddess's identity is not recorded; the head was acquired together with another acrolithic head (1905.7).
  • Ancient marble statues were usually painted, at least in the hair, eyes and lips.
Read the evidence-led prompt

Show the complete Roman cult statue this marble head came from, as it stood when new in a temple around 180 CE. It is an acrolith: keep this exact marble head, with its face, hairstyle and expression unchanged, and set it on an over-life-size seated goddess whose body is made of wood covered in richly painted and gilded drapery, a long chiton and a mantle, with marble hands and marble feet. Paint the marble lightly: warm tinted hair, colored eyes and lips. Do not identify her with attributes of any specific goddess. Show the whole statue on a stone base inside a dim temple room.

The uninformed guess

  • Polished the head into flawless white marble and kept it a bust.
  • Invented a new nose and filled every break, which modern conservators no longer do.
  • No paint, the white-marble ideal in its purest form.

What the evidence supports

  • An acrolith: marble face, hands and feet on a body of painted and gilded drapery.
  • Light paint on the marble: tinted hair, colored eyes and lips.
  • No attributes naming a specific goddess, as the prompt asked, because the record does not identify her.

What the machine invented

  • The face and hair. The head is no longer this head: the hairstyle changed and the face aged.
  • The seated pose came from my prompt, not from the evidence. The record does not say whether she sat or stood.
  • Cinematic oil-lamp lighting and a temple interior with reliefs on every wall.

Verdict. The idea of an acrolith comes through well, and it is the one thing a visitor looking at a white head would never guess. But the model would not keep the actual face, which is the only part that survives.

06
Lost garment

Clavus (tunic band): Dionysos (Bacchus)

Egyptian · c. 400-500 CE · Harvard Art Museums 1919.52

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • A tapestry-woven shoulder band (clavus), Egypt, about 400–500 CE: Dionysos, nude with a blue halo, a cape, sandals and a thyrsus, with a charging lion below and a griffin above, on a pink panel inside a floral border (museum record).
  • An identical band is the right band on a tunic front at the Museum of Fine Arts, Boston (96.355); both came from Denman Ross's collection and likely belonged to the same garment.
  • Late Roman tunics from Egypt were woven of undyed linen with pairs of clavi running from the shoulders down the front and back.
Read the evidence-led prompt

Show the complete late Roman tunic from Egypt (about 450 CE) that this woven band was cut from, laid flat and seen from the front, as it looked when new. It is a wide, T-shaped tunic of plain undyed cream linen with sleeves. Two identical vertical bands like this one run from the shoulders straight down the front, one on each side; keep this band's design exactly: the pink panel, the floral border, Dionysos with his halo and staff, the lion and the griffin, and make the second band its twin. Fresh, unfaded wool colors. Plain neutral background, museum flat-lay photograph.

The uninformed guess

  • Rewove the band as new: crisp edges, bright pink, the frayed border gone.
  • Turned the small hanging tassel at the bottom into a long cord.
  • Kept Dionysos, the lion and the griffin close to the original design.

What the evidence supports

  • A plain, undyed linen tunic with two matching bands from the shoulders.
  • The pink panel, floral border and figures repeated in the bands.
  • Fresh wool colors, plausible for a new garment.

What the machine invented

  • Bands running all the way to the hem. The Harvard band ends in a rounded tip with a hanging ornament, a common way for clavi to stop partway down; the model ignored that and repeated the pattern to the floor.
  • The tunic's cut, length and neckline.
  • The real comparison, the tunic front in Boston that probably wore the twin band, could not be given to the model at all.

Verdict. The evidence answered 'what kind of thing was this?' well. The fragment itself held the answer to 'how long was the band?', and the model did not read it.

07
Lost floor

Fragment of a Mosaic Pavement with Maenad, from the Constantinian Villa at Daphne-Harbie

Roman · 325-350 CE · Harvard Art Museums 1939.147

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • From room 1 of the Constantinian Villa at Daphne, a suburb of Antioch, 325–350 CE; excavated in 1935 (museum record).
  • One of several medallions with Dionysiac themes in a large pavement, including Dionysos, a satyr and Silenus; the best-preserved, a dancing satyr, is in the Baltimore Museum of Art.
  • What survives: a female head tilted to the right with brown hair, a bundle of deep purple and pink flowers, traces of green glass, inside a pointed frame that would have formed an ellipse.
  • The right half was cleaned and conserved; the left half is still in the condition it was lifted in.
Read the evidence-led prompt

Show this Roman floor mosaic panel as it looked when new, around 330 CE in a villa at Antioch. Keep the stone and glass tesserae technique, the pointed frame and its border pattern, and the woman's head, hair and flowers exactly as preserved. Complete the frame into a full pointed ellipse and complete the figure inside it: a maenad, a female follower of Dionysos, shown to the waist or in a dancing pose, carrying the bundle of purple and pink flowers, her garment in colored tesserae with green glass accents, on a white tessera ground. Clean the whole surface to the bright, even colors of the conserved right half. Seen from directly above.

The uninformed guess

  • Completed the woman as a bust and cleaned the whole panel, including the uncleaned left half.
  • Turned her head upright and frontal; in the mosaic it tilts to the right.
  • Kept the 1935 wooden shipping crate as part of the work.

What the evidence supports

  • A dancing female figure carrying purple and pink flowers, fitting the Dionysiac medallions of this floor.
  • Green glass accents and colored stone tesserae.
  • The tilted head and the pointed frame kept.

What the machine invented

  • Her clothing, colors and pose, all of which are lost.
  • The frame was not completed into an ellipse; it still runs off the bottom of the panel.
  • The shipping crate, again, in both versions.

Verdict. The two versions show the model's habit of treating everything in the photograph as the artwork. A crate from the 1935 excavation became part of a 4th-century floor.

08
Lost page

Bathing Maidens Observed by the Master of the Garden, illustrated folio from a manuscript of the Haft Paykar of Nizami

Persian · c. 1410 · Harvard Art Museums 2011.539

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • Cut from a folio of Nizami's Haft Paykar (The Seven Beauties), Iran, about 1410; the painting was trimmed to keep the parts that extended beyond the gold ruling (museum record).
  • The back of the folio preserves four columns of text in nineteen lines of nasta'liq script.
  • The painting conflates two moments of a tale told to Bahram Gur: a young man enters his own garden through a breach in the wall and finds maidens bathing in his pool.
  • Only 8.5 × 5.2 cm survives.
Read the evidence-led prompt

Show the complete manuscript page this Persian painting was cut from, as it looked when new around 1410: a folio from a manuscript of Nizami's poem Haft Paykar. Keep the painting exactly as it is, set within the page so that parts of it, like the building and tree, extend past the gold ruled frame into the margin. Above and below the painting, add panels of Persian verse in four columns of black nasta'liq calligraphy, separated by thin gold and colored rulings, on cream paper with wide plain margins. Flat, frontal view of the single page.

The uninformed guess

  • Covered the whole background in gold leaf, which the painting never had.
  • Repainted abraded areas and smoothed the figures' faces.
  • Kept the trimmed outline: a cutout, not a page.

What the evidence supports

  • A full manuscript page: text panels above and below, ruled frames, cream paper and margins.
  • Four columns of text, as on the back of the surviving folio.
  • Parts of the painting, like the tree, extending beyond the ruled frame, as the museum describes.

What the machine invented

  • The text. It looks like Persian nasta'liq, but it is not the poem: several lines simply repeat the poet's name, 'Nizami'.
  • The painting was shrunk and simplified to fit the new page.
  • The layout and proportions of text to image.

Verdict. The most dangerous image of the nine, because fake writing looks like evidence. A reader who cannot read Persian would take it as the lost page.

09
Almost everything

Fragment of Egyptian blue on mud from the Palace of Sargon the Great at Khorsabad, Assyria

Assyrian · 8th-7th century BCE · Harvard Art Museums 1942.219.1

What survives. Museum photograph.
Uninformed. “Restore this damaged artwork…”
Evidence-led. Prompt built from the record.

The evidence

  • From the Palace of Sargon at Khorsabad, found by the University of Chicago's Oriental Institute expedition (museum record).
  • Laboratory study found Egyptian blue pigment on a white priming coat of impure lime whitewash over mud plaster (museum record).
  • That is all the fragment can tell us: there is no surviving image, edge or pattern.
Read the evidence-led prompt

Show the section of wall this small fragment came from, as it looked when new around 710 BCE in the palace of the Assyrian king Sargon II at Khorsabad: a mud-plaster wall with a white lime-wash priming coat, painted in part with Egyptian blue. Use only what the fragment proves: mud plaster, white lime wash, and areas of bright Egyptian blue paint. Do not invent figures, scenes or text.

The uninformed guess

  • Replaced the lump of plaster with a blue faience scarab inscribed with hieroglyphs.
  • It seems to have read 'Egyptian blue' as 'Egyptian object'.
  • Nothing of the fragment survives in the result.

What the evidence supports

  • Mud plaster, a white lime-wash coat and flat Egyptian blue, the three things the laboratory found.
  • No figures, scenes or text, as the prompt asked.

What the machine invented

  • The rectangular panel layout, which the fragment cannot show.
  • How large the blue areas were, and where.

Verdict. The Venice Charter in one pair of images. The honest reconstruction is almost an abstract painting, because conjecture began almost immediately. The confident one is a different object.

Across nine cases

What the experiment shows

  1. With no evidence, the model mostly cleans. In six of nine cases the uninformed prompt produced a tidied version of the fragment (sharper carving, filled cracks, rewoven threads) rather than the missing whole. 'Restore' meant 'make it look new', not 'show what is lost'.
  2. It assumes bare stone. Asked to restore with no evidence, it painted none of the four stone pieces, though the museum records paint on three of them. Given the evidence, it painted all four. The white-marble bias that reconstructions like Gods in Color argue against is the model's default.
  3. It can't tell the artwork from the museum. It restored a modern picture frame, kept a 1935 shipping crate in both versions of the mosaic, and gilded a background that was never gold. Everything in the photograph counts as the object.
  4. Evidence fixes what you ask about and nothing else. The evidence-led images got the asked-for things right: pigment colors, the inscription band, the acrolith's materials, the page layout. Unasked details moved without warning. The genie lost his horns and turned around, the cow grew horns, a hieroglyphic caption changed, and Persian verse turned into the poet's name repeated.
  5. The prompt is a reconstruction too. The goddess is seated because my prompt said so, not because anything shows it. Evidence-led prompting doesn't remove conjecture; it moves it from the model to the person writing the prompt, who now carries the conservator's responsibility to mark it.
  6. The best image is the least impressive one. For the Khorsabad plaster, the uninformed model invented a scarab and the evidence-led model painted a nearly blank wall. Brandi and the Venice Charter would choose the blank wall. A generative model has to be told to stop where conjecture begins, and even then it does not show you where that was.