The prompts and the post-processing of Florence-2, as Florence2Processor of Python transformers 4.57. Transmuter.Vision.image_to_text/2 uses it for a repository whose encoder takes inputs_embeds.
A task is a task token, such as "<CAPTION>" or "<OD>". The tables of the tasks come from preprocessor_config.json, as transformers.js reads them: task_prompts_without_inputs and task_prompts_with_input give the prompt of each task, and tasks_answer_post_processing_type gives the post-processing of its answer. A task with input puts the text of the input in the place of {input} of its prompt.
The answer of a location task has location tokens <loc_N>, with N from 0 to 999. A location token is the center of a bin of 1/1000 of the width or the height of the image, so the coordinate is (N + 0.5) * width / 1000 for x and (N + 0.5) * height / 1000 for y. The coordinates are floats, in pixels of the input image. Python transformers 4.57 truncates them to integers, and its remote code of microsoft/Florence-2-base-ft gives floats.
Summary
Functions
Loads the tables of the tasks from preprocessor_config.json of repo. Raises when the configuration does not have them.
Returns the result of task from the generated text, which has the special tokens, for an image of size {height, width}.
Returns the prompt of task with the input text, or nil for no input.
Functions
Loads the tables of the tasks from preprocessor_config.json of repo. Raises when the configuration does not have them.
Returns the result of task from the generated text, which has the special tokens, for an image of size {height, width}.
The result always has :text: the text without <s> and </s>, and without spaces at the ends. The post-processing type of the task adds keys:
"pure_text"- no other keys."description_with_bboxes","bboxes"and"phrase_grounding"-:boxes, a list of%{xmin: x, ymin: y, xmax: x, ymax: y}, and:labels, the label of each box."ocr"-:quad_boxes, a list of[x1, y1, x2, y2, x3, y3, x4, y4], and:labels, the text of each quad box."polygons"-:polygons, a list with the polygons of each label, and:labels. A polygon is[x1, y1, x2, y2, ...]."description_with_bboxes_or_polygons"-:boxes,:box_labels,:polygonsand:polygon_labels.
Returns the prompt of task with the input text, or nil for no input.