Imagine you want to edit an image, but you don’t have the skills or the software to do it. You wish you could just tell the computer what you want to change, and it would do it for you. Sounds like science fiction, right? Well, not anymore. Meet Emu Edit, a new model introduced in the paper “Emu Edit: Precise Image Editing via Recognition and Generation Tasks” authored by a team of experts and published on 16 Nov 2023. This model can perform any image editing operation using natural language instructions. 

Emu Edit is a breakthrough in the field of instruction-based image editing, which means editing images by giving commands in words, rather than using a mouse or a keyboard. Emu Edit can understand and execute a wide range of instructions, such as “Dress the emu in a fireman outfit”, “let’s see it graduating”, or “Mark the drinks”. It can also do more complex tasks, such as removing objects, enhancing the resolution, or combining multiple edits.

How does Emu Edit do all this? It uses a powerful technique called multi-task learning, which means it learns to do many different tasks at the same time. Emu Edit is trained on a large and diverse set of tasks, such as region-based editing, free-form editing, and computer vision tasks. All these tasks are formulated as generative tasks, which means Emu Edit has to produce a new image as the output. By learning from different tasks, Emu Edit can improve its generalization and robustness, and handle a variety of editing scenarios.

But how does Emu Edit know which task to perform, given an instruction? Emu Edit uses a clever trick called learned task embeddings, which are vectors that represent the type of the task. For example, one vector might represent the task of changing the color of an object, while another might represent the task of adding text to the image. Emu Edit learns these vectors from the data and uses them to guide the generation process toward the correct edit type. This way, Emu Edit can avoid confusion and ambiguity, and produce more accurate and realistic results.

Emu Edit is not only good at doing the tasks it was trained on but also at learning new tasks with minimal examples. For example, Emu Edit can learn to do image inpainting, which means filling in missing parts of an image, or super-resolution, which means increasing the quality of a low-resolution image, with just a few labeled examples. This capability is very useful when high-quality data is limited and expensive. Emu Edit can also do compositions of editing tasks, which means combining multiple edits in one instruction, such as “crop the image and make it brighter”.

Emu Edit is a remarkable model that demonstrates the potential of instruction-based image editing. It can perform a wide range of editing operations with natural language instructions, and generalize to new tasks with minimal examples. It can also handle complex and detailed tasks that require more reasoning and understanding. Emu Edit is a step towards making image editing more accessible and intuitive for everyone.

Reference:

https://arxiv.org/pdf/2311.10089.pdf