Skip to content

Latest commit

 

History

History
861 lines (788 loc) · 44.2 KB

README.md

File metadata and controls

861 lines (788 loc) · 44.2 KB
Raw

Awesome-Sketch-Synthesis

Awesome

A collection of papers about Sketch Synthesis (Generation). Mainly focus on stroke-level vector sketch synthesis.

Feel free to create a PR or an issue.

examples

Outlines

0. Survey

Paper Source Code/Project Link
Deep Learning for Free-Hand Sketch: A Survey TPAMI 2022 [code]

1. Datasets

Here Vector strokes means having svg data. With photos means having the photo-sketch paired data.

Level Dataset Source Vector strokes With photos Notes
Characters Omniglot ✔️ ❌ Alphabets characters
KanjiVG ✔️ ❌ Chinese characters
Kuzushiji ❌ ❌ Japanese characters
Icon FIGR-8-SVG ✔️ ❌ Icons with text descriptions
Systematic Symbol SketchIME ACM MM 2023 ✔️ ❌ Systematic sketches with semantic annotations
Instance-level TU-Berlin SIGGRAPH 2012 ✔️ ❌ Multi-category hand sketches
Sketchy SIGGRAPH 2016 ✔️ ✔️ Multi-category photo-sketch paired
QuickDraw ICLR 2018 ✔️ ❌ Multi-category hand sketches
QMUL-Shoe-Chair-V2 CVPR 2016 ✔️ ✔️ Only two categories
Sketch Perceptual Grouping (SPG) ECCV 2018 ✔️ ❌ With part-level semantic segmentation information
FaceX AAAI 2019 ✔️ ❌ Labeled facial sketches
Creative Sketch ICLR 2021 ✔️ ❌ With annotated part segmentation
ImageNet-Sketch NeurIPS 2019 ❌ ❌ 50 images for each of the 1000 ImageNet classes
SEVA NeurIPS 2023 ✔️ ✔️ 90K human-generated sketches that vary in detail
Scene-level SketchyScene ECCV 2018 ❌ ✔️ With semantic/instance segmentation information
CMPlaces TPAMI 2018 ❌ ✔️ Cross-modal scene dataset
Context-Skecth Expressive 2018 ❌ ✔️ Context-based scene sketches for co-classification
SketchyCOCO CVPR 2020 ❌ ✔️ Scene sketch, segmentation and normal images
FS-COCO ECCV 2022 ✔️ ✔️ Scene sketches with text description
SFSD VC 2022 ✔️ ✔️ Completely hand-drawn scene sketches with label annotation
Drawing from photos Photo-Sketching WACV 2019 ✔️ ✔️ ScenePhoto-sketch paired
Tracing-vs-Freehand SIGGRAPH 2021 ✔️ ✔️ Tracings and freehand drawings of images
Drawing from 3D models DifferSketching SIGGRAPH Asia 2022 ✔️ ❌ 3D model-sketch paired, with novice and professional ones
Portrait CUFS TPAMI 2009 ❌ ✔️ Face-sketch pairs
APDrawing CVPR 2019 ❌ ✔️ Portrait-sketch paired
SKSF-A EG 2024 ❌ ✔️ Face-sketch pairs of seven styles
Children's Drawing Amateur Drawings TOG 2023 ❌ ✔️ With character bounding boxes, segmentation masks, and joint location annotations
Rough sketch Da Vinci CGI 2018 ❌ ✔️ Line drawing restoration dataset
Rough Sketch Benchmark SIGGRAPH Asia 2020 ✔️ ✔️ Rough and clean sketch pairs (only for evaluation)
CAD ld3d SIGGRAPH 2008 ❌ ❌ Line Drawings of 3D Shapes
OpenSketch SIGGRAPH Asia 2019 ✔️ ❌ Product Design Sketches
SketchGraphs ICML 2020 Workshop ✔️ ❌ Sketches extracted from real-world CAD models
Fusion 360 Gallery SIGGRAPH 2021 ✔️ ❌ For 'sketch and extrude' designs
FloorPlanCAD ICCV 2021 ✔️ ❌ With instance and semantic annotations
Anime Danbooru2021 / ❌ ✔️ Anime images annotated with tags
DanbooRegion ECCV 2020 ❌ ❌ Anime images with region annotations
Danbooru-Parsing TOG 2023 ❌ ✔️ For anime portrait parsing and anime translation
CreativeFlow+ CVPR 2019 ❌ ✔️ Large densely annotated artistic video dataset
ATD-12K CVPR 2021 ❌ ✔️ Animation frames with flow annotations
AnimeRun NeurIPS 2022 ❌ ✔️ Correspondence dataset for 2D-styled cartoons
AnimeCeleb ECCV 2022 ❌ ❌ Animation head images with pose annotations
PaintBucket-Character CVPR 2024 ❌ ✔️ Animation frames with region annotations
Sakuga-42M arxiv 24.05 ❌ ✔️ Cartoon videos with text descriptions and tags
Anita online 2024 ❌ ✔️ Professional hand-drawn cartoon keyframes, with 1080P sketch and color images

2. Sketch-Synthesis Approaches

1) Semantic Concept-to-sketch

Level Paper Source Code/Project Link
Instance-level A Neural Representation of Sketch Drawings (sketch-rnn) ICLR 2018 [Code] [Project] [Demo]
Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories [Code]
AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches AAAI 2019 [Project]
Stroke-based sketched symbol reconstruction and segmentation (stroke-rnn) CGA 2019
BézierSketch: A generative model for scalable vector sketches ECCV 2020 [Code]
Pixelor: A Competitive Sketching AI Agent. So you think you can beat me? SIGGRAPH Asia 2020 [Project] [Code]
Creative Sketch Generation ICLR 2021 [Project] [Code]
Computer-Aided Design as Language arxiv 2105
DoodleFormer: Creative Sketch Drawing with Transformers ECCV 2022 [Project] [Code]
SketchKnitter: Vectorized Sketch Generation with Diffusion Models ICLR 2023 [Code]
Self-Organizing a Latent Hierarchy of Sketch Patterns for Controllable Sketch Synthesis TNNLS 2023 [Code]
Scale-Adaptive Diffusion Model for Complex Sketch Synthesis ICLR 2024

2) Photo-to-sketch

  • vector image generation
Data type Paper Source Code/Project Link
Facial Style and abstraction in portrait sketching TOG 2013
Instance-level Free-Hand Sketch Synthesis with Deformable Stroke Models IJCV 2017 [Project] [code]
Learning to Sketch with Shortcut Cycle Consistency CVPR 2018 [Code1] [Code2]
Learning Deep Sketch Abstraction CVPR 2018
CLIPasso: Semantically-Aware Object Sketching SIGGRAPH 2022 [Project] [Code]
Scene-level Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale AAAI 2021 [Code]
CLIPascene: Scene Sketching with Different Types and Levels of Abstraction ICCV 2023 [Project] [Code]
Learning Realistic Sketching: A Dual-agent Reinforcement Learning Approach ACM MM 2024
Technical Drawings Deep Vectorization of Technical Drawings ECCV 2020 [Project] [code]
  • raster image generation
Type Paper Source Code/Project Link
Facial ArtLine Online demo [Code]
APDrawingGAN: Generating Artistic Portrait Drawings from Face Photos with Hierarchical GANs CVPR 2019 [Code] [Demo]
Unpaired Portrait Drawing Generation via Asymmetric Cycle Mapping CVPR 2020 [Code]
Line Drawings for Face Portraits From Photos Using Global and Local Structure Based GANs TPAMI 2020 [Code]
Quality Metric Guided Portrait Line Drawing Generation from Unpaired Training Data TPAMI 2022 [Code]
Human-Inspired Facial Sketch Synthesis with Dynamic Adaptation ICCV 2023 [Code]
Stylized Face Sketch Extraction via Generative Prior with Limited Data EG 2024 [Code] [Project]
Instance-level Deep Factorised Inverse-Sketching ECCV 2018
Synthesizing human-like sketches from natural images using a conditional convolutional decoder WACV 2020 [Code]
Anime sketchKeras online demo [Code]
LineDistiller online demo [Code]
Anime2Sketch online demo [Code]
Reference Based Sketch Extraction via Attention Mechanism SIGGRAPH Asia 2022 [Code]
Scene-level Photo-Sketching: Inferring Contour Drawings from Images WACV 2019 [Code] [Project]
Learning to generate line drawings that convey geometry and semantics CVPR 2022 [Code] [Project]
Arbitrary Semi-supervised reference-based sketch extraction using a contrastive learning SIGGRAPH 2023 [Code] [Project]

3) Text/Attribute-to-sketch

  • raster image generation
Type Paper Source Code/Project Link
Facial Text2Sketch: Learning Face Sketch from Facial Attribute Text ICIP 2018
Scene-level Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications UIST 2019
Scene-level Scones: Towards Conversational Authoring of Sketches IUI 2020
  • vector image generation
Type Paper Source Code/Project Link
Arbitrary Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts arxiv 21.09 [code] [project]
Arbitrary CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders NeurIPS 2022 [code]
Arbitrary StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation IJCAI 2022 [code]
SVG VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models CVPR 2023 [project]
Arbitrary SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation BMVC 2023 [code]
Arbitrary DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models NeurIPS 2023 [project] [code]
Icon IconShop: Text-Based Vector Icon Synthesis with Autoregressive Transformers SIGGRAPH Asia 2023 [project]
SVG Text-Guided Vector Graphics Customization SIGGRAPH Asia 2023 [project]
Arbitrary Text-based Vector Sketch Editing with Image Editing Diffusion Prior ICME 2024 [code]
SVG SVGDreamer: Text Guided SVG Generation with Diffusion Model CVPR 2024 [project] [code]
SVG NIVeL: Neural Implicit Vector Layers for Text-to-Vector Generation CVPR 2024 [project]
SVG Text-to-Vector Generation with Neural Path Representation SIGGRAPH 2024 [project]
Arbitrary SVGCraft: Beyond Single Object Text-to-SVG Synthesis with Comprehensive Canvas Layout arxiv 24.04 [code]
SVG VectorPainter: A Novel Approach to Stylized Vector Graphics Synthesis with Vectorized Strokes arxiv 24.05

4) 3D shape-to-sketch

Paper Source Code/Project Link
DeepShapeSketch : Generating hand drawing sketches from 3D objects IJCNN 2019
Neural Contours: Learning to Draw Lines from 3D Shapes CVPR 2020 [project] [code]
Cloud2Curve: Generation and Vectorization of Parametric Sketches CVPR 2021 [project]
Neural Strokes: Stylized Line Drawing of 3D Shapes ICCV 2021 [code]
Learning a Style Space for Interactive Line Drawing Synthesis from Animated 3D Models PG 2022
CAD2Sketch: Generating Concept Sketches from CAD Sequences SIGGRAPH Asia 2022 [project]

5) Art-to-sketch

Here we list sketch synthesis based on other image types, like Manga, line art, rough sketch, etc.

a) Line art

  • Vector-to-Vector
Paper Source Code/Project Link Deep learning?
Closure-aware Sketch Simplification SIGGRAPH Asia 2015 [Project] No
StrokeAggregator: Consolidating Raw Sketches into Artist-Intended Curve Drawings SIGGRAPH 2018 [Project] No
StrokeStrip: Joint Parameterization and Fitting of Stroke Clusters SIGGRAPH 2021 [Project] [code] No
StripMaker: Perception-driven Learned Vector Sketch Consolidation SIGGRAPH 2023 No
Region-Aware Simplification and Stylization of 3D Line Drawings EG 2024 No
  • Raster-to-Vector (a.k.a. Vectorization)
Paper Source Code/Project Link Deep learning?
Topology-Driven Vectorization of Clean Line Drawings TOG 2013 No
Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization SIGGRAPH 2016 [Project] No
A Delaunay triangulation based approach for cleaning rough sketches C&G 2018 [Code] No
Semantic Segmentation for Line Drawing Vectorization Using Neural Networks EG 2018 [project] [code] Yes
Deep Line Drawing Vectorization via Line Subdivision and Topology Reconstruction PG 2019 Yes
Inertia-based Fast Vectorization of Line Drawings PG 2019 No
Vectorization of Line Drawings via Polyvector Fields TOG 2019 [Code] No
Integer-Grid Sketch Simplification and Vectorization SGP 2020 [Project] [Code] No
Deep Vectorization of Technical Drawings ECCV 2020 [project] [code] Yes
General Virtual Sketching Framework for Vector Line Art SIGGRAPH 2021 [project] [code] Yes
Keypoint-Driven Line Drawing Vectorization via PolyVector Flow SIGGRAPH Asia 2021 [project] Hybrid
End-to-end Line Drawing Vectorization AAAI 2022 Yes
Vectorizing Line Drawings of Arbitrary Thickness via Boundary-based Topology Reconstruction CGF 2022 No
Singularity-Free Frame Fields for Line Drawing Vectorization SGP 2023 [code] No
Deep Sketch Vectorization via Implicit Surface Extraction SIGGRAPH 2024 [project] [code] Hybrid

b) Rough sketch simplification / cleanup

  • Datasets and benchmark
Paper Source Code/Project Link
A Benchmark for Rough Sketch Cleanup SIGGRAPH Asia 2020 [Project] [Code]
  • Methods
Paper Source Code/Project Link
Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup SIGGRAPH 2016 [Code] [Project]
Mastering Sketching: Adversarial Augmentation for Structured Prediction SIGGRAPH 2018 [Code] [Project]
Real-Time Data-Driven Interactive Rough Sketch Inking SIGGRAPH 2018 [Code] [Project]
Perceptual-aware Sketch Simplification Based on Integrated VGG Layers TVCG 2019

c) Manga (Comics)

Paper Source Code/Project Link
Deep extraction of manga structural lines SIGGRAPH 2017 [Code]
Manga Filling Style Conversion with Screentone Variational Autoencoder SIGGRAPH Asia 2020 [Project] [Code]
Generating Manga from Illustrations via Mimicking Manga Workflow CVPR 2021 [Project] [Code]
MangaGAN: Unpaired Photo-to-Manga Translation Based on The Methodology of Manga Drawing AAAI 2021
MARVEL: Raster Gray-level Manga Vectorization via Primitive-wise Deep Reinforcement Learning TCSVT 2023

3. Vector Graphics Generation (2D)

Here we focus on learning-based vector graphics generation without depending on vector training data, and traditional vectorization algorithms.

  • Learning with external black-box (non-differentiable) rendering simulator
Paper Source Code/Project Link
Synthesizing Programs for Images using Reinforced Adversarial Learning ICML 2018 [Code]
Unsupervised Doodling and Painting with Improved SPIRAL arxiv 1910 [Project]
  • Learning with built-in differentiable rendering module
Paper Source Code/Project Link
Layered Image Vectorization via Semantic Simplification arxiv 24.06 [webpage] [code]
ProcessPainter: Learning to draw from sequence data SIGGRAPH Asia 2024 [code]
Segmentation-guided Layer-wise Image Vectorization with Gradient Fills ECCV 2024
Towards High-fidelity Artistic Image Vectorization via Texture-Encapsulated Shape Parameterization CVPR 2024
SuperSVG: Superpixel-based Scalable Vector Graphics Synthesis CVPR 2024 [code]
Vector Graphics Generation via Mutually Impulsed Dual-domain Diffusion CVPR 2024
Optimize and Reduce: A Top-Down Approach for Image Vectorization AAAI 2024 [code]
Segmentation-Based Parametric Painting arxiv 23.11 [code] [project]
Editable Image Geometric Abstraction via Neural Primitive Assembly ICCV 2023
Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region ACM MM 2023 [code]
Intelli-Paint: Towards Developing More Human-Intelligible Painting Agents ECCV 2022 [project]
Towards Layer-wise Image Vectorization CVPR 2022 [code] [project]
Paint Transformer: Feed Forward Neural Painting with Stroke Prediction ICCV 2021 [code]
Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings CVPR 2021 [code]
Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes CVPR 2021 [code]
Im2Vec: Synthesizing Vector Graphics without Vector Supervision CVPR 2021 [Project] [code]
Stylized Neural Painting CVPR 2021 [Code] [project]
Learning to Paint With Model-based Deep Reinforcement Learning ICCV 2019 [code]
Strokenet: A neural painting environment ICLR 2019 [Code]
Neural Painters: A learned differentiable constraint for generating brushstroke paintings arxiv 1904 [Code]
Learning to Sketch with Deep Q Networks and Demonstrated Strokes arxiv 1810
Unsupervised Image to Sequence Translation with Canvas-Drawer Networks arxiv 1809
  • Vectorization
Paper Source Code/Project Link
Depixelizing pixel art SIGGRAPH 2011
Perception-Driven Semi-Structured Boundary Vectorization SIGGRAPH 2018 [Webpage]
PolyFit: Perception-aligned Vectorization of Raster Clip-art via Intermediate Polygonal Fitting SIGGRAPH 2020 [Webpage] [Code]
ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis TVCG 2021
TCB-Spline-Based Image Vectorization TOG 2022
Image vectorization and editing via linear gradient layer decomposition SIGGRAPH 2023

4. Vector Graphics Generation (3D)

Paper Source Code/Project Link
3Doodle: Compact Abstraction of Objects with 3D Strokes SIGGRAPH 2024 [code]
Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering arxiv 24.05 [webpage]
  • Wire Art
Paper Source Code/Project Link
Wired Perspectives: Multi-View Wire Art Embraces Generative AI CVPR 2024 [code] [webpage]
Fabricable 3D Wire Art SIGGRAPH 2024