Rendering optimizations

[rganchev]

Investigation

• Safari on iOS crashes in most tapestries after basic navigation gestures.
• The crashes are due to Safari's memory and CPU restrictions - not only in terms of quantity but also in terms of rates at which these resources can be allocated by the page.
• Displaying tapestry items as DOM nodes causes Safari to allocate its own internal rendering layers to paint the DOM tree. Applying CSS transformation upon navigation inside the tapestry (to move the tapestry items around) causes Safari to restructure its rendering layers - destroying some of them and creating new ones. After some CSS optimizations I was able to significantly reduce the number and size of these rendering layers but even then moving around the tapestry often triggered Jetsam (iOS' internal "hitman" process) to kill our app.
• It seems that using DOM nodes for tapestry items and letting Safari do the painting doesn't work. The alternative is to generate image data for each tapestry item and draw it manually. Then display the DOM nodes only if the user needs to interact with the content of an item.
• Initially I tried to generate this image data on-the-fly, when the tapestry is loading. However, this didn't work, since Jetsam started killing the app due to increased CPU usage.
• Finally, I resorted to generating image representations (thumbnails) of each item at multiple levels of detail (LOD) on the backend.

Solution

• The backend now generates thumbnails for all tapestry items.
• The thumbnails are then converted to different resolutions (levels of detail / LOD) using the sharp library. These are called "renditions", i.e. a single image asset can have multiple renditions. Renditions can vary in size, resolution, or format, but they are always different versions of the same image.
• The frontend initially loads the lowest-quality renditions of the thumbnails for all items. Then, based on the current viewport transform, higher-quality renditions are loaded for separate items as necessary. This is controlled in ItemThumbnailController.
• The loaded thumbnails are rendered in the new ItemRenderer. This renderer also optionally renders an icon overlay on top of the thumbnail.
• The DOM nodes for tapestry items are not added to the DOM tree at all until an item is activated. When an item is active, the thumbnail is hidden, and the DOM node is displayed instead.
• Some items (pdfs, webpages, text boxes, video, audio) have persistent progress (scroll/playback position). In order to preserve this progress, the corresponding DOM nodes are not removed from the DOM tree if they have been activated at least once. Instead, they are positioned far away from the viewport in a manner which maximally simplifies and constrains their layout for efficiency. display: none is not used, because it leads to undesired flashes in some cases.
• I also optimized TapestryElementRenderer. Now each renderer extracts a "render state" from the view model on each frame and only re-renders if the render state differs.

Further improvements

• iPhone still heats up while viewing a tapestry. The main reason for this is probably the RAF loop caused by the Pixi app's ticker. We don't actually need a ticker since we don't do animations. One improvement would be to remove the ticker and only render the Pixi app on demand (i.e. when the model changes).
• Apparently we cannot afford to have arbitrarily large items. We previously didn't have any restriction on the dimensions of text items. However, if we have a 10,000x1000 pixels text item (example from an actual tapestry), mobile browsers (Safari in particular) cannot render it efficiently and if we try to "screenshot" it, it would produce 10 million pixels of image data. In all cases, we don't have an efficient way to render such items. My suggestion is to forbid the creation of such items at the format level, i.e. the Zod schema, and treat tapestry JSONs having such items as invalid.
• Since now all items have thumbnails, including text frames, some items can look very distorted, for example if resized and the new thumbnail has not been generated yet. We need to display some kind of indication in such cases.
• It is probably a good idea to allow users to manually request thumbnail re-generation for specific items or and/or for the whole tapestry.
• Imported/cloned tapestries don't always have proper thumbnails. We need to make sure thumbnail processing is scheduled for them.