ํ‹ฐ์Šคํ† ๋ฆฌ ๋ทฐ

๋ฐ˜์‘ํ˜•
CVPR 2024.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, Xinggang Wang
School of CS | Huazhong University of Science and Technology 2School of EIC | Huazhong University of Science and Technology | Huawei Inc.
15 Jul 2024

 

Introduction

๋ณธ ๋…ผ๋ฌธ์€ ์›€์ง์ด๋Š” ์˜์ƒ์— ๋Œ€ํ•ด scene์„ ๋ Œ๋”๋งํ•˜๋Š” ์—ฐ๊ตฌ์ด๋‹ค.

Gaussian Splatting ๊ธฐ๋ฒ•์„ ์‚ฌ์šฉํ•œ Dynamic scene ๋ชจ๋ธ๋ง ์—ฐ๊ตฌ๋ฅผ ํ†ตํ•ด ํ”ผ์‚ฌ์ฒด๊ฐ€ ์›€์ง์—ฌ๋„ ์‹œ๊ฐ„์˜ ๋ณ€ํ™”์— ๋”ฐ๋ผ ์ž์—ฐ์Šค๋Ÿฝ๊ฒŒ ๋ Œ๋”๋งํ•  ์ˆ˜ ์žˆ๊ฒŒ ํ•œ๋‹ค.

๊ธฐ์กด์˜ 3D-GS๋ฅผ ์‚ฌ์šฉํ•œ ๋ฐฉ๋ฒ•๋“ค์€ ์ž…๋ ฅ ์ด๋ฏธ์ง€๋ฅผ ์š”๊ตฌ or ๋ฉ”๋ชจ๋ฆฌ์™€ ํŠธ๋ ˆ์ด๋‹์˜ ๋ฌธ์ œ๊ฐ€ ์กด์žฌ
•3D-GS: ์ •์ ์ธ ์žฅ๋ฉด ์ค‘์ 
•Dynamic 3D-GS: ์ž…๋ ฅ ์ด๋ฏธ์ง€ ์š”๊ตฌ & ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰ ์ฆ๊ฐ€
•Deformable 3DGS: training ๋น„ํšจ์œจ์ 

 

๋ณธ ๋…ผ๋ฌธ์—์„œ ์ œ์•ˆํ•˜๋Š” ๋ฐฉ๋ฒ•๋ก ์—์„œ๋Š” spatial-temporal structure encoder๋ฅผ ์‚ฌ์šฉํ•ด ์ธ์ ‘ํ•œ ์„œ๋กœ ๋‹ค๋ฅธ 3D Gaussian๋“ค์„ ์—ฐ๊ฒฐํ•˜์—ฌ ๋ณด๋‹ค ์ •ํ™•ํ•œ ์›€์ง์ž„๊ณผ ๋ชจ์–‘ ๋ณ€ํ˜•์„ ์˜ˆ์ธกํ•  ์ˆ˜ ์žˆ๋‹ค.

๊ถ๊ทน์ ์œผ๋กœ 4D Gaussian Splatting ํ†ตํ•ด ํšจ์œจ์ ์ธ ํ•™์Šต ํšจ์œจ์„ฑ๊ณผ ์‹ค์‹œ๊ฐ„ ๋ Œ๋”๋ง์„ ์–ป์„ ์ˆ˜ ์žˆ๋‹ค.

 

Method

4D Gaussian Splatting Framework

์ „์ฒด์ ์ธ ํ”„๋ ˆ์ž„์›Œํฌ๋ฅผ ๋ณด์ž.

4D Gaussian์€ Staticํ•œ 3D Gaussian์„ ๋งŒ๋“  ํ›„,

์‹œ๊ฐ„์— ๋”ฐ๋ฅธ ๊ฐ 3D Gaussian๋“ค์˜ Position, Rotation, Scaling ๋ณ€ํ™”๋Ÿ‰์„ ๋ชจ๋ธ๋งํ•œ๋‹ค.

์ด ๋ณ€ํ™”๋Ÿ‰์„ Deformation Field๋กœ ํ‘œํ˜„ํ•˜๊ณ , ( 3D Gaussian์„ ์ž…๋ ฅ์œผ๋กœ) ์–ผ๋งˆ๋‚˜ ๋ณ€ํ˜•์‹œ์ผฐ๋Š”์ง€ ๋Œ€ํ•ด ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ถœ๋ ฅํ•˜๋Š” ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค.

Encoder๋ฅผ ๋ณด๋ฉด ๋จผ์ € 6๊ฐ€์ง€์˜ matrix์œผ๋กœ ๋ณ€ํ˜•๋˜๊ณ , ๊ทธ๋‹ค์Œ feature vector๋กœ ํ•ฉ์ณ์ง€๋ฉฐ, MLP๋ฅผ ํ†ต๊ณผํ•˜์—ฌ ์ตœ์ข… ๊ฒฐ๊ณผ ๊ฐ’์„ ํš๋“ํ•˜๊ฒŒ ๋œ๋‹ค.

 

4D-GS rendering process

NeRF๊ธฐ๋ฐ˜์˜ Dynamic Model์€ ray์œ„์— point๋“ค์„ deformationํ–ˆ๊ธฐ ๋•Œ๋ฌธ์—, ๊ฐ point์˜ ์„œ๋กœ ๋‹ค๋ฅธ ์†๋„๋ฅผ ์ž˜ ๋ชจ๋ธ๋งํ•˜์ง€ ๋ชปํ•˜์—ฌ ํ€„๋ฆฌํ‹ฐ ํ•˜๋ฝ์ด ์žˆ๋‹ค.

4D Gaussian Splatting์—์„œ๋Š” ๊ฐ Gaussian์ด ray์— ์˜์กดํ•˜์ง€ ์•Š๊ณ  ์„œ๋กœ ๋‹ค๋ฅธ ์†๋„๋กœ ์ด๋™์ด ๊ฐ€๋Šฅํ•˜๊ธฐ ๋•Œ๋ฌธ์—, ์‹œ๊ฐ„ t์— ๋”ฐ๋ผ Gaussian์˜ ์œ„์น˜๊ฐ€ ์ด๋™ํ•˜๋ฉด ๋‹ค๋ฅธ ray๋ฅผ ํ†ตํ•ด ์ด๋™๋œ Gaussian์„ rendering ํ•  ์ˆ˜ ์žˆ๊ฒŒ ๋œ๋‹ค.

 

Gaussian Deformation Field Network

Multi-resolution HexPlane์œผ๋กœ 3D Gaussian์˜ spatial, temporal ๊ฐ’์„ encoding ํ•œ๋‹ค.

์—ฌ๋Ÿฌ๊ฐœ(multi)์˜ Resolution์œผ๋กœ Rank๋ฅผ ๊ตฌ์„ฑํ•˜๊ณ  ์ด๋ฅผ MLP์˜ input์˜ feature๋กœ ์‚ฌ์šฉํ•œ ๊ฒƒ์ด๋‹ค.

i, j์— ๋Œ€ํ•œ ๊ฒƒ์€ ๊ฐ ํ‰๋ฉด์˜ ์ฐจ์›์„ ์˜๋ฏธํ•˜๊ณ , R์€ ๊ทธ ์ฐจ์›์œผ๋กœ ๊ตฌ์„ฑ๋œ Rank๋ฅผ ์˜๋ฏธํ•œ๋‹ค.

interpolation์€ ํƒ€๊ฒŸ ์ขŒํ‘œ์˜ ์ฃผ๋ณ€์˜ Tensor ๊ฐ’๋“ค๋กœ ๋ณด๊ฐ„(interpolation)ํ•œ๋‹ค๋Š” ๊ฒƒ์„ ๋‚˜ํƒ€๋‚ธ๋‹ค.

๊ฐ ์ฐจ์›์— ๋Œ€ํ•ด interpolationํ•œ ๊ฐ’์„ concatํ•˜์—ฌ voxel์— ๋Œ€ํ•œ feature๊ฐ’์œผ๋กœ ๋งŒ๋“ค๊ฒŒ ๋œ๋‹ค. (fx๊ฐ€ ๋‰ด๋Ÿด ๋ณต์…€์— ๋Œ€ํ•œ ํ”ฝ์…€์ž„)

์œ„์™€ ๊ฐ™์ด ๋ชจ๋ธ๋งํ•˜๊ฒŒ ๋˜๋ฉด, ๊ณต๊ฐ„์ƒ๊ณผ ์‹œ๊ฐ„์ƒ์˜ (x,yํ‰๋ฉด) ๊ฐ๊ฐ ์ธ์ ‘ํ•œ voxel์€ ์œ ์‚ฌํ•œ feature๋“ค์„ ๋‚˜ํƒ€๋‚ด๊ณ , ์‹œ๊ฐ„์ƒ(xtํ‰๋ฉด)์œผ๋กœ ์ธ์ ‘ํ•œ voxel๋“ค์€ ์œ ์‚ฌํ•œ feature๋“ค์„ ๋‚˜ํƒ€๋‚ธ๋‹ค.

3D Gaussian๋“ค์˜ ๋ชจ๋“  feature๋“ค์ด ์ธ์ฝ”๋”ฉ๋˜๋ฉด, decoder๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์›ํ•˜๋Š” ๋ณ€์ˆ˜๋ฅผ ๊ณ„์‚ฐํ•  ์ˆ˜ ์žˆ๋‹ค.

๋ณ„๋„์˜ MLP๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ position, rotation, scaling์˜ ๋ณ€ํ˜•์„ ๊ณ„์‚ฐํ•˜๋ฉด, ์ด์™€ ๊ฐ™์ด ์ฒ˜๋ฆฌ๋ฉ๋‹ˆ๋‹ค.

์ตœ์ข…์ ์œผ๋กœ ๋ณ€ํ˜•๋œ 3D Gaussian์„ ์–ป์„ ์ˆ˜ ์žˆ๋‹ค.

 

Optimization

3D Gaussian ์Šคํ”Œ๋ž˜ํ„ฐ ๋ฐฉ๋ฒ•๊ณผ ๋™์ผํ•˜๊ฒŒ Structure from Motion(SfM) ํฌ์ธํŠธ ์ดˆ๊ธฐํ™”๋ฅผ ํ†ตํ•ด ํ•™์Šต์‹œ์ผœ์„œ ํ€„๋ฆฌํ‹ฐ๋ฅผ ํ–ฅ์ƒํ•œ ๋’ค,

๊ทธ ํ›„์— dynamic scene์„ fine-tuningํ˜•ํƒœ๋กœ ํ•™์Šต์‹œ์ผฐ๋‹ค๊ณ  ํ•œ๋‹ค.

Loss function์€ L1 color loss๋ฅผ ์‚ฌ์šฉํ•˜๊ณ , ๊ทธ๋ฆฌ๋“œ ๊ธฐ๋ฐ˜ tv loss ๋„  ์ถ”๊ฐ€๋กœ ์ ์šฉํ•˜์˜€๋‹ค.

 

Experiments

[D-NeRF Synthetic Dataset]
RTX 3090 GPU 800 x 800

dynamic scene ๋ฐ์ดํ„ฐ์…‹ ์ด๊ธฐ ๋•Œ๋ฌธ์— 3DGS(3D Gaussian Splatting)์˜ PSNR์€ ๋‚ฎ์€ ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ์œผ๋ฉฐ, ์ด์ „ dynamic scene ์—ฐ๊ตฌ์— ๋น„ํ•ด ํ€„๋ฆฌํ‹ฐ๊ฐ€ ๋†’๊ณ , ๋žœ๋”๋ง ์†๋„๊ฐ€ ๋น ๋ฅธ ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ๋‹ค.

 

rendering speed and numbers of 3D Gaussians

 

๋ฐ˜์‘ํ˜•
๊ณต์ง€์‚ฌํ•ญ
์ตœ๊ทผ์— ์˜ฌ๋ผ์˜จ ๊ธ€
์ตœ๊ทผ์— ๋‹ฌ๋ฆฐ ๋Œ“๊ธ€
Total
Today
Yesterday
๋งํฌ
ยซ   2025/05   ยป
์ผ ์›” ํ™” ์ˆ˜ ๋ชฉ ๊ธˆ ํ† 
1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31
๊ธ€ ๋ณด๊ด€ํ•จ
๋ฐ˜์‘ํ˜•