Video Virtualisation architecture
How ION transforms rendered video into an addressable, orchestratable data layer for intelligent systems.
Audio walkthrough10 min12 sections
Video, rewritten as a programmable data layer
ION’s video virtualisation architecture transforms rendered video from static, file-bound media into an addressable, time-indexed, orchestratable data layer that intelligent systems can assemble, personalise, and deliver progressively in real time, without re-encoding or duplicating source media.
Traditional video infrastructure treats compressed media as monolithic assets. Even within ISO Base Media containers where samples are technically indexed, the assembly map is static, generated once at encode time, binding structure, sequencing, and timing permanently to a single playback experience.
ION generalises and externalises this assembly layer.
Rather than treating the MP4 container as the final authority of structure, ION virtualises encoded video samples into a programmable reference space. Each compressed frame or segment becomes independently addressable, with timing, ordering, and inclusion controlled dynamically by an orchestration engine rather than a fixed sample table.
How the architecture works
Encoded sources
- Video filemp4480 MB
- Video filemp4820 MB
- Video filemp4580 MB
- Video filemp4340 MB
- Video filemp4320 MB
ION
Virtual Video EngineExtracts samples and sets their orderSample blocks
Viewer
- The viewer plays a virtual video.
- The virtual video calls ION.
- ION extracts samples from external video files and sets their order for delivery.
- ION delivers the organised sample blocks.
- The sample blocks are assembled into the virtual video.
- The video is complete.
Programmable assembly
This enables intelligent systems to construct video experiences on demand. AI models can select, sequence, and time video components in real time based on user context, intent, behavioural signals, or inference outputs. Instead of rendering new video files, the orchestration layer computes a personalised assembly graph that maps virtualised references to underlying encoded media.
Progressive delivery
Progressive delivery occurs as the assembly is resolved. The system can begin streaming immediately while subsequent segments are dynamically selected and scheduled, enabling real-time personalisation, adaptive narratives, contextual highlights, summaries, and AI-driven content composition without re-encoding or duplicating media.
Decoupled infrastructure layers
From a systems perspective, ION decouples the following into independent, programmable layers:
- Encoded media storage
- Structural assembly
- Temporal orchestration
- Delivery sequencing
This collapses rendering pipelines, eliminates exponential storage growth, and enables AI-native interaction with video as a composable data substrate rather than a static asset.
The approach remains compatible with existing codec ecosystems and streaming infrastructure. Underlying media remains standard compressed samples. What changes is the control plane that governs how those samples are referenced, assembled, and delivered.
How ISO Base Media functions as an assembly framework
Understanding ION requires understanding what MP4 and ISO Base Media Format are. They are not simply file formats. They are time-indexed assembly frameworks for compressed media.
The encoding pipeline
When a modern video encoder runs, whether H.264, HEVC, AV1, or similar, it does not create a movie file. It produces a continuous stream of compressed data units. Each unit represents the information needed to reconstruct a single moment in time. These units are called samples.
In almost all modern video workflows, one sample corresponds directly to one frame of video. Frame one becomes sample one, frame two becomes sample two, and so on. Each sample is simply a block of compressed bytes: no timing, no structure, just encoded data. The movie file itself comes later.
Container architecture
MP4, MOV, and the wider ISO Base Media File Format are not codecs. They are structured containers built from a hierarchy of boxes, sometimes called atoms, that organise raw encoded data into a navigable, time-aware system. Two are foundational:
mdat- The media data box. All encoded video data lives here. Inside mdat, samples are written sequentially, back to back: sample one, sample two, sample three. There is no inherent meaning inside mdat. No timing. No frame boundaries. No playback logic. It is simply a warehouse of compressed data chunks stored in order.
moov- The intelligence of the file lives here. This is where the sample tables live. The sample tables describe exactly how to interpret the raw bytes stored in mdat.
mdatWarehouseraw bytes
Compressed samples stored back to back, with no timing, boundaries or playback logic.
moovAssembly mapsample tables
stcowherestszhow bigsttshow longstsskeyframes
The sample tables answer the critical questions every player needs:
- Where does each sample start in the data stream?
- How large is each sample in bytes?
- How long should each sample be displayed?
- Which samples are keyframes?
Several tables work together to provide this map. One records the size of each sample, another records the byte offset of each sample inside mdat, another defines how long each sample lasts in time, and another flags random access frames. Together, these tables define how encoded samples are located, timed, and presented.
| Table | Role |
|---|---|
stco / co64 | Byte offset of each sample inside mdat |
stsz | Size of each sample in bytes |
stts | Duration of each sample in timescale units |
stss | Flags identifying random access keyframes |
ctts | Composition offset for decode versus presentation reordering |
The timescale clock system
MP4 does not store time in frames per second; instead, it uses a clock system based on time units called a timescale. The media declares how many ticks equal one second.
- timescale=25,000 ticks = 1 second
- 25,000 ÷ 25 fps=1,000 ticks per sample
- 25 samples × 1,000=25,000 ticks = 1 second
Example: 25 fps at timescale 25,000
Each video sample is then assigned a duration in those units. At 25 frames per second, there are 25 samples per second, so each sample must last:
25,000 ÷ 25 = 1,000 time units per sample
The timing table simply records that each sample has a duration of 1,000 ticks. After 25 samples, the accumulated duration reaches 25,000 ticks, exactly one second of playback.
Example: 25 fps at timescale 90,000
Many professional systems use a timescale of 90,000. The maths works the same way.
90,000 ÷ 25 = 3,600 ticks per sample
Each frame lasts 3,600 time units. Accumulate 25 of them and you have one second.
The file never stores 25 fps. It stores durations. Playback time is calculated by summing sample durations.
Frame rate to sample rate
In standard video, one sample equals one frame:
| Frame rate | Samples per second |
|---|---|
| 24 fps | 24 |
| 25 fps | 25 |
| 30 fps | 30 |
| 60 fps | 60 |
Audio works differently, but video is almost always one compressed sample per frame.
Putting the pipeline together
- Raw frames go into the encoder.
- The encoder compresses each frame into a sample.
- The muxer writes every sample sequentially into mdat.
- At the same time, it builds the sample tables inside moov, recording for each sample its byte position, byte size, duration in time units, and whether it is a keyframe or B-frame.
- Once complete, the MP4 file becomes a fully indexed, time-aware assembly of media.
Grouped runs, decode and presentation time
Grouped runs
Timing inside the file is often stored in grouped runs. Instead of listing every frame individually, the timing table might specify eight samples, each lasting 3,600 ticks. In a 90,000 timescale system, that equals:
3,600 ÷ 90,000
= 0.04 seconds per sample
= 25 fps
Eight samples together
= 0.32 seconds of timeline
Again, the clock emerges purely from accumulated durations.
Decoding time versus presentation time
The format includes two related timing concepts: decoding time and presentation time. Because modern codecs reorder frames internally, the order frames are decoded is not always the order they are displayed. The container handles this using an additional composition offset table. But even here, everything is still expressed as time deltas in ticks. The same clock model applies.
The components, and their role in ION
| Component | Role in ION architecture |
|---|---|
| mdat box | Warehouse of compressed frame data. Raw, sequential, untouched by virtualisation. |
| moov and sample tables | Assembly map. Defines byte positions, sizes, durations, and keyframe flags for every sample. |
| Timescale system | Clock mechanism. Playback time is computed by summing sample durations, never stored as fps. |
| ION reference layer | Externalises and programs the assembly map. Samples become independently addressable across any source file. |
| Orchestration engine | Computes personalised assembly graphs dynamically. Delivers progressive streams without re-encoding. |
| Virtual Video File | The resulting primitive: a programmable, AI-addressable video data structure compatible with standard players and CDNs. |
The static sample table, made programmable
ION treats the static sample table not as a fixed output but as a programmable reference space.
Rather than treating the MP4 container as the final authority on structure, ION virtualises encoded video samples so their timing, ordering, and inclusion can be dynamically controlled by an orchestration engine rather than a fixed sample table generated at encode time.
Each compressed frame or segment becomes independently addressable. The orchestration layer computes a personalised assembly graph that maps virtualised references to underlying encoded media across one or more source files, resolving SMPTE timecodes (H:M:S:F) to sample locations, managing the container structure, format, sample mapping, and reference file locations simultaneously.
This is the technical nucleus of ION’s architecture: assembling content from distributed encoded sources into a single virtual, orchestratable stream without re-encoding or modifying the underlying media.
The virtualisation principle
ION does for video what virtual memory did for compute and what object storage did for cloud: it introduces an abstraction layer that separates the logical structure from the physical storage. Rendered media becomes addressable, composable, and dynamically orchestrated by intelligent systems.
Four decoupled layers
The architectural consequence of ION’s virtualisation is that four previously bound operations become independently programmable:
- Encoded media storage
- Underlying compressed samples remain standard. No re-encoding required. Existing codec ecosystems and CDN infrastructure are compatible.
- Structural assembly
- The sample tables governing which frames appear and in what order are computed dynamically by the ION orchestration engine at request time.
- Temporal orchestration
- Timing, sequencing, and narrative structure are controlled by AI inference outputs, user context, or behavioural signals rather than fixed timecodes.
- Delivery sequencing
- Progressive delivery begins immediately while subsequent segments are dynamically selected and scheduled, enabling real-time personalised streaming.
Video as a real-time computable medium
For AI platforms, ION’s architecture creates a new primitive: video as an orchestratable, real-time computable medium. Intelligent systems do not generate new video files. They generate assembly instructions.
AI models can select, sequence, and time video components based on user context, intent, behavioural signals, or inference outputs. The orchestration layer resolves these instructions into a valid virtual assembly graph referencing existing encoded media. Progressive delivery of the assembled stream begins immediately.
This architecture enables capabilities that are not practically achievable with traditional rendering models:
- Scalable personalisation at infrastructure cost profiles, with no per-variation encode cost
- Contextual highlights, summaries, and adaptive narratives assembled in real time
- AI-driven content composition across distributed source media without duplication
- Continuous adaptive video experiences that respond to evolving inference outputs
- Agent-driven content creation where AI orchestration replaces traditional editorial pipelines
A programmable layer above existing media
ION’s approach remains fully compatible with existing codec ecosystems and streaming infrastructure. Underlying media is standard compressed samples stored in standard containers. What changes is the control plane that governs how those samples are referenced, assembled, and delivered.
Existing CDNs, players, and delivery infrastructure operate without modification. ION sits upstream as the data layer: it produces valid, standards-compliant virtual assembly outputs that downstream infrastructure handles identically to conventionally encoded streams.
For organisations operating at scale across content, AI, media delivery, or personalised digital experiences, this integration profile is critical. ION does not require infrastructure replacement. It adds a programmable assembly layer above existing encoded media stores, extending the capability of the entire downstream stack without disrupting it.
A foundational granted patent estate
The video virtualisation architecture described in this document is protected by four foundational granted patents, with an additional 2026 patent filing extending protection into the runtime control and governance layer. The foundational patent estate covers the core method of virtualising encoded video samples into an independently addressable reference space, enabling dynamic assembly without re-encoding.
| Patent | Coverage |
|---|---|
| US 8,893,203 B2 | Core video virtualisation method. Generalisation and externalisation of the ISO Base Media assembly layer. |
| US 10,721,507 B2 | Continuation. Extended claims covering orchestration and dynamic assembly workflows. |
| US 9,516,392 B2 | Continuation. Additional claims covering progressive delivery and reference-based streaming. |
| US 9,544,657 B2 | Content blockchain. Distributed ledger integration for virtualised media asset provenance and rights management. |
In 2026, ION filed a further patent application in Australia and the United States, extending this protection into the control layer of Virtual Video. The filing covers token-governed runtime resolution, including authorisation, licensing, territorial conditions, consent parameters, transaction rules, and device or session validation at the moment of assembly.
This extends ION’s protection beyond video virtualisation and dynamic assembly into the governance and commercialisation layer required for AI-native and agentic media environments.
A virtualisation layer for video
ISO Base Media is not just a file format. It is a time-indexed assembly framework for compressed media. The separation between the mdat warehouse and the moov assembly map is a foundational architectural design choice. ION extends this design by making the assembly map programmable, dynamic, and AI-driven.
The result is a virtualisation layer for video that follows the same architectural logic as virtual memory for compute or object storage for cloud: an abstraction that decouples logical structure from physical storage, enabling a new class of infrastructure capabilities that the original system was not designed to provide.
For any organisation building AI systems, cloud infrastructure, or intelligent media workflows at scale, ION represents a foundational infrastructure primitive: the layer that makes rendered video programmable by intelligent systems.