TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions
Jeya Maria Jose
Rajeev Yasarla
Vishal M. Patel
Johns Hopkins University
[Paper]
[GitHub]
Transweather uses a single encoder-decoder architecture and learns weather type queries to solve weather removal.

Abstract

Removing adverse weather conditions like rain, fog, and snow from images is an important problem in many applications. Most methods proposed in the literature have been designed to deal with just removing one type of degradation. Recently, a CNN-based method using neural architecture search (All-in-One) was proposed to remove all the weather conditions at once. However, it has a large number of parameters as it uses multiple encoders to cater to each weather removal task and still has scope for improvement in its performance. In this work, we focus on developing an efficient solution for the all adverse weather removal problem. To this end, we propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder that can restore an image degraded by any weather condition. Specifically, we utilize a novel transformer encoder using intra-patch transformer blocks to enhance attention inside the patches to effectively remove smaller weather degradations. We also introduce a transformer decoder with learnable weather type embeddings to adjust to the weather degradation at hand. TransWeather achieves significant improvements across multiple test datasets over both All-in-One network as well as methods fine-tuned for specific tasks. In particular, TransWeather pushes the current state-of-the-art by +6.34 PSNR on the Test1 (rain+fog) dataset, +4.93 PSNR on the SnowTest100K-L dataset and +3.11 PSNR on the RainDrop test dataset. TransWeather is also validated on real world test images and found to be more effective than previous methods.

Method


A degraded image is forwarded to transformer encoder to extract hierarchical features. The encoder has intra-patch transformer blocks to extract features from smaller sub-patches created from the main patch. The transformer decoder has learnable weather type queries to obtain the task feature. Then, the hierarchical features from encoder as well as the task feature from decoder are forwarded to a convolutional projection block to obtain the clean image.


Results

Quantitative comparison with previous methods across different weather removal tasks.


Some sample results on real-world weather degraded images.


Paper and Supplementary Material

TransWeather
Tech Report
(hosted on ArXiv)


[Bibtex]


Acknowledgements

This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.