When using reference maps to generate new images, the most common problem is not “models don't understand”, but “models look too much”. All you wanted to do was use the position and position of the person on the reference map, but the model brought the color, the costume, the material, the light and even the wind. The more words are written, the more the requirements may compete for control。
The key to solving such problems is to decompose reference material into different responsibilities: role maps for identity and clothing, style maps for colour and quality, depth maps for proximity, contours and space relations, and movements. In-depth videos can also continue to provide clues for the movement of people, the blocking of change and the motion of lenses. Each package controls only one major variable and the model is easier to understand “what to keep and what to redesign”。
However, the Depth Map is often referred to in an oral sense as the “A gray model”. It's convenient, but it's easy to assume that the same control can be achieved by turning a picture into black and white or producing a grey three-dimensional effect picture. This section first clarifies the concept and then gives a direct picture and video message。
I. WHAT THE Depth Chart IS
The depth map is a two-dimensional map that records the distance of the picture. Each of its pixels no longer primarily depicts the colour, but uses different values to indicate the distance of the position relative to the lens. For ease of view, these values are often shown as black, white, grey: a common agreement is that it is whiteer in the vicinity, darker in the distance, and there are tools to use the opposite rule. Therefore, it is important to confirm the black and white direction of the current tool before it is used, and not to rely on mere physical speculation。
Example
It's called "single depth estimation" from a single common photograph. "Single" is just one shot. Because of the absence of real distance data for input, models typically rely on vision, shielding, object size and available visual experience to speculate back and forth. The result is mostly “relative depth”: it can judge who is closer and who is further away, without necessarily knowing whether the person is 1.5 m or 2 m from the lens。
This is why the depth map is suitable for creative control and cannot be used as a precision measurement. It can compress images into a cleaner structural hint that helps generate model attention:
– The location and size of the subject in the picture
- Levels of perspective, perspective, context
– The covert relationship between the person and the props
– The sense of space that comes from high and low lenses, proximity and insight
– These relationships change over time。
Depth maps do not normally provide reliable retention of identity, clothing, material, colour and light atmosphere. It happens to be its value: it only transfers the structure and does not allow the original style to take full control。
It's not the same as a grey map, a gray model and a depth map
Greyscale Chart
Just take off the color, the shadow of the original picture still exists. White clothes may be bright and black hair may be dark, but this darkness is not the same as being close. You can't get a Depth map of the colored photos。
Greyscale Chart
Grey Moot
Usually, neutral material previews are made in three-dimensional productions: models, scenes and lights are still in place, except for the use of end-posters and materials for the time being. It is appropriate to check shapes, blocks and lights, but it is still a retrofitted image, not pixel-by-pixel data。
Depth
It's visualization of space distance. The face, clothing and body of the same person, as long as they are at a close distance, usually show a close greyscale; the wall behind them falls into another greyscale. The exact term should be “to generate a single-column relative depth map” or “to convert the video into a series of continuous depths”, rather than simply “to make it gray” or “to generate a grey model”。
Three, decide what you want to control
Not all references need to be turned into depth maps. Before I start, ask myself: What do I really want to keep from this reference
If you want to keep colour, light, material or overall atmosphere, it would be more appropriate to use original maps or style references. The posture skeleton is usually more direct if it is to retain the person ' s joint posture but allows for a change of scene and lens. If only to retain a clear outline, the edge map may be more flexible than the depth map. A deep map is the appropriate option if it is necessary to maintain the structure, the front and the rear, the main scale and the cover relationship。
So is the video. In-depth video is appropriate to transmit changes in space, such as the person ' s proximity or distance from the camera, the person ' s massive motion, the camera ' s thrust or retreat, and the subject and the future. It is not very good at describing how the fingers hold small objects, how the hands of two people cross and how the ropes entangle。
Picture workflow: available depth maps from reference maps
Direct Reference Generation
Before using the Depth Chart, a direct replacement can be made for: At the same time, the person attitude reference, person and scene maps were uploaded, requiring the model to replace the original person in the reference posture with the designated person and then place it in the new scene。
I use the following hints:
Replace the original person in figure 1 with the person in figure 2 and place the person in the scene in figure 3。
The physical posture, head orientation, limb angle, position of hands and feet, centre of gravity, movement range, position of the person, main size, lens view and image of the person in figure 1 are strictly preserved and the action is not re-engineered。
The face, hair, body and clothing of the new person is based on figure 2 and the face, hair, clothing and identity of the original figure 1 is completely removed and no replacement marks are retained。
The content, colour, light and atmosphere of the scene are shown in figure 3, but the scene is in the lens of map 1 as required. The person ' s foot is naturally exposed to the ground, the body is in the right proportion and the light, shadow and environment colour are naturally integrated with the scenery。
The face of the person is: [insert specific expression]. Do not add new characters, do not change movement, do not cause physical misalignment, excess fingers, suspension of the person, marginal remnant or background deformation。
The result is a full-colour picture of the identity of the person, his or her attitude, his or her scenes and nature。
The effects are extremely unstable, and there are also elements of the reference
Either you don't act, or you're replaced
Depth Chart Reference Generation
initial step
Select a shot and action map that you want to reset. Focusing on its position, position of the person in the picture, physical action and the relationship between the back and the back of space, it is not necessary to break down multiple reference materials。
second step
Uploads the reference to Image 2.0 and converts it to depth. The model is to be told to “convert the original map” rather than free to produce a so-called grey model, otherwise the person's position and image could be redesigned。
The following phrase is used directly:
Converts the input images into a single-column relative depth map to be used as space structure control for subsequent image generation. It must be maintained: the size of the original drawing, the image, the lens perspective, the main position and scale, the position of the person, the contours of the object, the pre- and post-occupying relationship and the background space structure. Depth rule: Displays white in close proximity, black in long distances, with continuous grey steps in the middle distance; smooth depth transitions on the same continuous surface and clear boundaries of objects. Prohibition: Re-engineering of the drawings, moving, adding or removing persons and objects, changing positions, replicating new details, retaining original drawings of colour, text, text, material and photostyle. Output: Only one greyscale depth map, corresponding to the size of the original map, is exported, and no border, description text, colour drawings or decorations are added。
third step
Confirms that the picture is available and goes directly to the soluble link. In turn, it uploads person charts, scene maps and depth maps, and then uses a hint to figure out the role of three maps: who is responsible for person charts, where is it, and depth maps for "how to stand, where to film, how to sequence " 。
You can use the following:
The person ' s face features, hair, body and clothing are preserved by the person in the first image. The person is naturally placed in the scene of Photo 2, the scene environment, the space content, the colour and the light are based on Photo 2. Strict reference is made to the structure, lens view, position of the person, physical action, size of the body, front, back and back, and cover relationship in the depth map of picture 3, but no black and white images are generated or the grey-scale appearance of the depth map is inherited. The result is a full, natural colour image. The identity of the person is based on image 1, the scene is based on picture 2, and the lens and action are based on picture 3. Do not add new characters, do not change their clothing, do not change their position and movement。
generate with nanobanan 2
V. Video workflow
Depth Video Generation
Deep video is not simply black and white from frame to frame, but makes each frame a depth map while maintaining time stability。
it is recommended to use codex for conversion. i've got a dance video here. it's from @poetry morning
The following hints can be used:
Please convert the video that I have provided to a single-direction relative depth video, which is read-only and is not covered. Confirm the only input video; if multiple candidate files are found, name them and ask me, do not speculate. The full frame is treated using the available mono-visual depth estimation model, using a uniform model, pre-treatment and depth direction. Near white, far black. Keep the scale, frame rate, total duration, frame order and lens movement of the original video. Use a uniform depth integration range for the whole video, without automatic stretch contrasts by frame; moderate time stabilization to reduce scintillation. Outputs an MP4 720p deep video coded by H264 and splits in 15s, without the original audio and without overwhelming the original file。
The following video was converted:
when you get a deep video, you can then hand it over to a role-setting, costume or style chart to a soetable 2.0. the hint may be:
Based on three types of reference material I provided, a full colour video was generated。
Personal Reference Chart: used to identify the lead actor. The person ' s face features, hair, shape, clothing and overall appearance are strictly preserved and not replaced by the image of the person in the in-depth reference video。
Site reference map: used to determine the context of the story. The space layout, the main objects, the colour, the light and the overall atmosphere of the scene are preserved and the person is naturally integrated into the scene。
In-depth reference video: as the main template for actions and mirrors. Strictly imitating the initial position, the sequence of actions, the direction of movement, the speed, the range, the change of gravity, the pause time and the end of the position of the person in question, as well as the altitude of the camera, the angle of the photograph, the change of view, the direction of movement, the speed of movement, the advance, the retreat, the shaking, the following and the orbit。
Keeps the action rhythm of the reference video, the camera rhythm, the person's movement, the main size, the back and the back, the blocking of change and the focus of the image. Do not add any action, change the direction of the movement, change positions or redesign the mirror。
Only action, movement, spatial relations and lens movement in the deep reference video, and not its black-and-white grey-scale appearance, character identity, dress or original scene. The final figure is based on the character reference map and the final environment is based on the scene reference map and the action and mirrors are based on the deep reference video。
The physical structure of the person is natural, his/her movements are consistent, his/her footsteps are stable to the ground and his/her hands and props are reasonable. Do not add new characters, do not change the number of main props, do not cause a physical misalignment, figure drift, background beat, or suddenly the camera reverse。
Output a full colour video that is as stable as possible, consistent in action, mirrors and depth reference videos。
i created two 15-s videos and put them together
Depth map doesn't control anything
The Depth Chart mainly records the proximity of the object to the lens and can only help the model to understand the contours, the position of the person, the contours of the body and the context. It is not a personal profile, so it cannot be expected to accurately preserve all movements and expressions。
First of all, it's hard to record an expression and an eye. Smiles, frowning, blinking, opening mouths and the direction of the light do not usually result in a sufficiently significant change in distance. The person ' s face may be a grey area in the depth map, so that the depth map can only determine the general position and direction of the face, and the specific expression needs to be further refined by a hint。
For example, it can be added that “persons frown slightly, their lips are tight, they look to the left side of the picture, they look nervous, but they restrain.” The description must be written in terms of abrow, eye, mouth and emotional state, not just “natural expression”。
Second, fingers and little moves can easily go wrong. The distance between the fingers is close, often shielded from each other, and the conversion may be adhesive or even missing. Actions such as gripping, handing, pushing, holding the wheel need not only to know the back and forth, but also to understand how the finger bends, where the hand is touched, and this information cannot be presented in its entirety on a Depth map alone。