-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
281 lines (264 loc) · 13.2 KB
/
index.html
File metadata and controls
281 lines (264 loc) · 13.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta content="Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence" property="og:title">
<meta content="Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence" property="twitter:title">
<meta content="Diffusion Hyperfeatures is a framework for consolidating the multi-scale and multi-timestep internal representations of a diffusion model for tasks such as semantic correspondence." name="description">
<meta content="Diffusion Hyperfeatures is a framework for consolidating the multi-scale and multi-timestep internal representations of a diffusion model for tasks such as semantic correspondence." property="og:description">
<meta content="Diffusion Hyperfeatures is a framework for consolidating the multi-scale and multi-timestep internal representations of a diffusion model for tasks such as semantic correspondence." property="twitter:description">
<meta property="og:type" content="website">
<title>Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence</title>
<meta name="description" content="Diffusion Hyperfeatures is a framework for consolidating the multi-scale and multi-timestep internal representations of a diffusion model for tasks such as semantic correspondence.">
<meta name="keywords" content="Diffusion Hyperfeatures, diffusion hyperfeatures, hypercolumns, semantic correspondence, representation learning, diffusion models">
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-TPWJGFG8CS"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-TPWJGFG8CS');
</script>
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">
<link rel="stylesheet" href="./static/css/bulma.min.css">
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="./static/css/bulma-slider.min.css">
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="./static/css/index.css">
<link rel="stylesheet" href="./static/css/custom.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script defer src="./static/js/fontawesome.all.min.js"></script>
<script src="./static/js/bulma-carousel.min.js"></script>
<script src="./static/js/bulma-slider.min.js"></script>
</head>
<body>
<section class="hero space-background has-text-white">
<div class="hero-body space-background-overlay">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-2 publication-title has-text-white">Diffusion Hyperfeatures: <br> <span class="is-size-4">Searching Through Time and Space for Semantic Correspondence</span></h1>
<div class="is-size-5 publication-authors">
<span class="author-block">
<a href="https://people.eecs.berkeley.edu/~graceluo" target="_blank">Grace Luo</a><sup>1</sup>,</span>
<span class="author-block">
<a href="https://www.lisabdunlap.com" target="_blank">Lisa Dunlap</a><sup>1</sup>,</span>
<span class="author-block">
<a href="https://scholar.google.com/citations?user=_kJ-zUYAAAAJ&hl=en" target="_blank">Dong Huk Park</a><sup>1*</sup>,
</span>
<span class="author-block">
<a href="https://holynski.org" target="_blank">Aleksander Holynski</a><sup>1, 2*</sup>,
</span>
<span class="author-block">
<a href="http://people.eecs.berkeley.edu/~trevor" target="_blank">Trevor Darrell</a><sup>1*</sup>
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1</sup>UC Berkeley,</span>
<span class="author-block"><sup>2</sup>Google Research</span>
<br>
<span class="is-size-7">*Equal advising contribution.</span>
<br><br>
<span class="author-block">NeurIPS 2023</span>
</div>
<div class="column has-text-centered">
<div class="publication-links">
<span class="link-block">
<a href="https://arxiv.org/abs/2305.14334" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span>
</a>
</span>
<!-- Code Link. -->
<span class="link-block">
<a href="https://github.com/diffusion-hyperfeatures/diffusion_hyperfeatures" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code <span class="has-text-success">(New!)</span></span>
</a>
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- Abstract. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-4">Abstract</h2>
<div class="content has-text-justified">
<p>
Diffusion models have been shown to be capable of generating high-quality images,
suggesting that they could contain meaningful internal representations.
Unfortunately, the feature maps that encode a diffusion model's internal information are spread not only over layers of the network,
but also over diffusion timesteps, making it challenging to extract useful descriptors.
</p>
<p>
We propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and
multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks.
These descriptors can be extracted for both synthetic and real images using the generation and inversion processes.
We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence:
our method achieves superior performance on the SPair-71k real image benchmark.
We also demonstrate that our method is flexible and transferable:
our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features
of synthetic image pairs with unseen objects and compositions.
</p>
</div>
</div>
</div>
<!--/ Abstract. -->
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- A -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-4">Diffusion Hyperfeatures</h2>
<div class="content has-text-justified">
<p>
We extract feature maps varying across timesteps and layers from the diffusion process and
consolidate them with our lightweight aggregation network to create our Diffusion Hyperfeatures, in
contrast to prior methods that select a subset of raw diffusion features. For real images, we extract
these features from the inversion process, and for synthetic images we extract these features from the
generation process. Given a pair of images, we find semantic correspondences by performing nearest
neighbors over their Diffusion Hyperfeatures.
</p>
</div>
<figure class="image approach-figure">
<img src="./assets/approach.png">
</figure>
</div>
</div>
<!--/ A -->
<br>
<!-- B -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-4">Semantic Keypoint Matching</h2>
<div class="content has-text-justified">
<p>
<b>Tuning on Real Images.</b> We distill Diffusion Hyperfeatures for semantic correspondence by tuning our aggregation network on real images from SPair-71k. The dataset is comprised of image pairs with annotated common semantic keypoints for a variety of object categories spanning vehicles, animals, and household objects.
</p>
</div>
<div class="columns">
<figure class="image column">
<img src="./assets/spair_subset_1.png">
</figure>
<figure class="image column">
<img src="./assets/spair_subset_2.png">
</figure>
</div>
</div>
</div>
<!--/ B -->
<!-- C -->
<div class="columns is-centered">
<div class="column is-full-width">
<!-- <h2 class="title is-3">Transfer to Synthetic Images</h2> -->
<div class="content has-text-justified">
<p>
<b>Transfer to Synthetic Images.</b> We can take this same aggregation network tuned on these real images representing a limited set of object categories and apply it to synthetic images containing completely unseen and out-of-domain objects.
</p>
</div>
<div class="columns">
<figure class="image column">
<img src="./assets/synthetic_subset_1.png">
</figure>
<figure class="image column">
<img src="./assets/synthetic_subset_2.png">
</figure>
</div>
</div>
</div>
<!--/ C -->
<!-- D -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-4">Dense Warping</h2>
<div class="content has-text-justified">
<p>
Our aggregation network, which was trained on the semantic keypoint matching task, can also be used for dense warping. We follow the visualization format from <a href="https://sd-complements-dino.github.io" target="_blank">Zhang et. al. (arXiv 2023)</a>. Here, we show examples of warps on both real and synthetic images for cat, dog, and bird (where the synthetic images where synthesized from the prompt "Full body photo of {category}."). When warping we compute nearest neighbors matches for all pixels within an object mask, with no visual postprocessing.
</p>
</div>
<div class="columns">
<figure class="image column">
<img src="./assets/real_warp_1.png">
</figure>
<figure class="image column">
<img src="./assets/real_warp_2.png">
</figure>
</div>
<div class="columns">
<figure class="image column">
<img src="./assets/synthetic_warp_1.png">
</figure>
<figure class="image column">
<img src="./assets/synthetic_warp_2.png">
</figure>
</div>
</div>
</div>
<!--/ D -->
<!-- F -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-4">Concurrent Work</h2>
<div class="content has-text-justified">
We would also like to acknowledge the following concurrent works, which use raw diffusion features for the task of semantic correspondence:
<ul>
<li><a href="https://sd-complements-dino.github.io" target="_blank">A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence</a></li>
<li><a href="https://diffusionfeatures.github.io" target="_blank">Emergent Correspondence from Image Diffusion</a></li>
<li><a href="https://arxiv.org/abs/2305.15581" target="_blank"> Unsupervised Semantic Correspondence Using Stable Diffusion</a></li>
</ul>
</div>
</div>
</div>
<!--/ F -->
</div>
</section>
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>
@inproceedings{luo2023dhf,
title={Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence},
author={Luo, Grace and Dunlap, Lisa and Park, Dong Huk and Holynski, Aleksander and Darrell, Trevor},
booktitle={Advances in Neural Information Processing Systems},
year={2023}
}
</code></pre>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This website is licensed under a
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">
Creative Commons Attribution-ShareAlike 4.0 International License
</a>.
The website template is from the
<a href="https://github.com/nerfies/nerfies.github.io">
Nerfies
</a>
project page.
</p>
</div>
</div>
</div>
</div>
</footer>
</body>
</html>