Skip to content

GitLab

Commit a75d2bda authored by mashun1's avatar mashun1
Browse files

evtexture

parents
Pipeline #1325 canceled with stages
Hide whitespace changes
Inline Side-by-side
Showing with 1343 additions and 0 deletions
others/rpg_vid2e/viz_events.py 0 → 100644
View file @ a75d2bda
import argparse
import glob
import numpy as np
import os
import matplotlib.pyplot as plt
def render(x, y, t, p, shape):
img = np.full(shape=shape + [3], fill_value=255, dtype="uint8")
img[y, x, :] = 0
img[y, x, p] = 255
return img
if __name__ == "__main__":
parser = argparse.ArgumentParser("""Generate events from a high frequency video stream""")
parser.add_argument("--input_dir", default="")
parser.add_argument("--shape", nargs=2, default=[256, 320])
args = parser.parse_args()
event_files = sorted(glob.glob(os.path.join(args.input_dir, "*.npz")))
fig, ax = plt.subplots()
events = np.load(event_files[0])
img = render(shape=args.shape, **events)
handle = plt.imshow(img)
plt.show(block=False)
plt.pause(0.002)
for f in event_files[1:]:
events = np.load(f)
img = render(shape=args.shape, **events)
handle.set_data(img)
plt.pause(0.002)
others/rpg_vid2e/web_app/README.md 0 → 100644
View file @ a75d2bda
<div id="top"></div>
<!--
*** Thanks for checking out the Best-README-Template. If you have a suggestion
*** that would make this better, please fork the repo and create a pull request
*** or simply open an issue with the tag "enhancement".
*** Don't forget to give the project a star!
*** Thanks again! Now go create something AMAZING! :D
-->
<!-- PROJECT SHIELDS -->
<!--
*** I'm using markdown "reference style" links for readability.
*** Reference links are enclosed in brackets [ ] instead of parentheses ( ).
*** See the bottom of this document for the declaration of the reference variables
*** for contributors-url, forks-url, etc. This is an optional, concise syntax you may use.
*** https://www.markdownguide.org/basic-syntax/#reference-style-links
-->
<!-- [![Contributors][contributors-shield]][contributors-url]
[![Forks][forks-shield]][forks-url]
[![Stargazers][stars-shield]][stars-url]
[![Issues][issues-shield]][issues-url]
[![MIT License][license-shield]][license-url]
[![LinkedIn][linkedin-shield]][linkedin-url] -->
<!-- PROJECT LOGO -->
<br />
<div align="center">
<!-- <a href="https://github.com/othneildrew/Best-README-Template">
<img src="images/logo.png" alt="Logo" width="80" height="80">
</a> -->
<h2 align="center">ESIM Web App Playground</h3>
<p align="center">
A Web App for Generating Events from Upsampled Video Frames & Visualization of Events from Live Video of a Webcam!
<!-- <br />
<a href="https://github.com/othneildrew/Best-README-Template"><strong>Explore the docs »</strong></a>
<br />
<br />
<a href="https://github.com/othneildrew/Best-README-Template">View Demo</a>
·
<a href="https://github.com/othneildrew/Best-README-Template/issues">Report Bug</a>
·
<a href="https://github.com/othneildrew/Best-README-Template/issues">Request Feature</a> -->
<br />
<br />
<a href="https://github.com/iamsiddhantsahu">Siddhant Sahu</a> | <a href="https://github.com/danielgehrig18">Daniel Gehrig</a> | <a href="">Nico Messikommer</a>
</p>
</div>
![Product Name Screen Shot](screencast.gif)
<!-- TABLE OF CONTENTS -->
<details>
<summary>Table of Contents</summary>
<ol>
<li>
<a href="#about-the-project">About The Project</a>
<!-- <ul>
<li><a href="#built-with">Built With</a></li>
</ul> -->
</li>
<li>
<a href="#getting-started">Getting Started</a>
<ul>
<li><a href="#prerequisites">Prerequisites & Installation</a></li>
<li><a href="#Running the Web App Locally">Running the Web App Locally</a></li>
<li><a href="#Running the Web App on Google Colab">Running the Web App on Google Colab</a></li>
</ul>
</li>
<!-- <li><a href="#usage">Usage</a></li> -->
<li><a href="#roadmap">Roadmap</a></li>
<!-- <li><a href="#contributing">Contributing</a></li>
<li><a href="#license">License</a></li>
<li><a href="#contact">Contact</a></li>
<li><a href="#acknowledgments">Acknowledgments</a></li> -->
</ol>
</details>
<!-- ABOUT THE PROJECT -->
## About The Project
<!-- ![Product Name Screen Shot](screencast.gif) -->
The goal of this project is to make ESIM and VID2E available to researchers who do not possess a real event camera. For this we want to deploy VID2E and ESIM as an interactive web app. The app should be easy to use and have the following functional requirements:
* Generation of events from video through dragging and dropping a video into the browser. The resulting events should be downloadable as raw events and rendered video.
* Incorporate video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.
* Visualization and inspection of the event stream in the browser.
* The event generation should be configurable by changing ESIM parameters.
<p align="right">(<a href="#top">back to top</a>)</p>
<!-- ### Built With
This section should list any major frameworks/libraries used to bootstrap your project. Leave any add-ons/plugins for the acknowledgements section. Here are a few examples.
* [Next.js](https://nextjs.org/)
* [React.js](https://reactjs.org/)
* [Vue.js](https://vuejs.org/)
* [Angular](https://angular.io/)
* [Svelte](https://svelte.dev/)
* [Laravel](https://laravel.com)
* [Bootstrap](https://getbootstrap.com)
* [JQuery](https://jquery.com)
<p align="right">(<a href="#top">back to top</a>)</p> -->
<!-- GETTING STARTED -->
## Getting Started
Follow these steps in order to make this app run in your local system.
_This code base assumes you have access to a NVIDIA GPU in your system with proper drivers installed along with the `cuda-toolkit` version `10.1`._
### Prerequisites & Installation
Make sure that you have followed the [Instalation with Anaconda Instruction](https://github.com/uzh-rpg/rpg_vid2e) and have created the `vid2e` Conda environment with the `esim_torch` package installed in this environment.
1. Activate the Conda environment
```sh
conda activate vid2e
conda list | grep 'esim'
```
The above command should output the following
```sh
esim-cuda 0.0.0 pypi_0 pypi
esim-py 0.0.1 pypi_0 pypi
```
2. Install the `streamlit` package in this Conda environment, along with a couple of additional packages.
```sh
pip install streamlit stqdm numba h5py
```
### Running the Web App Locally
If you want to run this app locally on your system then follow these steps.
1. Run the App
```sh
streamlit run web_app.py
```
2. Then you can access the app in Local URL: http://localhost:8501
_When using the 'video upload' feature, the uploaded file is saved in the 'data/original/video_upload' directory. Subsequently, upon clicking the 'Generate Events' button, two new directory, namely, 'data/upsampled/video_upload' and 'data/events/video_upload' are created. Additionally, after the process has finished, you will see a 'Download Events' button, you can click this button to download the events. Events can be downloaded in three different formats: as .h5 file, as .npz or as a rendered video._
<p align="right">(<a href="#top">back to top</a>)</p>
### Running the Web App on Google Colab
To run this web app on Google Colab follow these steps in order.
_Make sure you select the `Run time type` as `GPU` in `Google Colab`_
1. Register and create an account in the [remote.it](https://remote.it/).
2. Install remote.it service in your Google Colab instance.
```sh
!curl -LkO https://raw.githubusercontent.com/remoteit/installer/master/scripts/auto-install.sh
! chmod +x ./auto-install.sh
! sudo ./auto-install.sh
```
3. Install Miniconda in your Google Colab Instance
```sh
! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.8.2-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local
```
4. Update Python Path
```sh
import sys
sys.path.insert(0,'/usr/local/lib/python3.7/site-packages/')
```
5. Install Streamlit package
```sh
!pip install streamlit
```
6. Run the remote.it service
```sh
!sudo connectd_installer
```
- In the menu selection choose `1`
- Enter your remote.it `email address` and `password`
- Enter the name of your device `esim`
- Choose `1` `Attach/reinstall a remote.it Service to an application`
- Protocol Selection Menu, Choose `2`, `Web (HTTP) on port 80`
- Enter a name for this remote.it service, `esim`
- Main menu, choose, `5` `Exit`
7. Mount your Google Drive to this Google Colab Instance
8. Change directory to where to want this project to reside.
9. Clone the repo
```sh
git clone https://github.com/uzh-rpg/rpg_vid2e.git
```
10. Check if the `esim_torch` package is installed
```sh
import esim
print(esim_torch.__path__)
```
11. Built the `esim_torch` package with `pybind11`
```sh
pip install esim_torch/
```
12. Change directory inside `web_app` and run the app on port number `80` as a webservice which can then be accessed for the remote.it service.
```sh
!streamlit run --server.port 80 web_app.py&>/dev/null&
```
Finaly, you can navigate your remote.it dashboard and open these service and use the app!
_Note: Remeber when running this app on Google Colab you need to change the paths for input and output directories for the upsampling to the specific directory in Google Drive and also other paths and esim import._
<p align="right">(<a href="#top">back to top</a>)</p>
<!-- ROADMAP -->
### Roadmap
- [x] Add video upload dragging and dropping a video into the browser.
- [x] Add video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.
- [x] Add events download button (output format)
- [x] HDF5
- [x] NPZ
- [x] Rendered Video
- [x] Add webcam functionality
- [x] Run this app on Google Colab
<!-- CONTRIBUTING
## Contributing
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!
1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
<p align="right">(<a href="#top">back to top</a>)</p> -->
<!-- LICENSE
## License
Distributed under the MIT License. See `LICENSE.txt` for more information.
<p align="right">(<a href="#top">back to top</a>)</p> -->
<!-- CONTACT
## Contact
Siddhant Sahu
Daniel Gehrig
Nico Messikommer
<!-- Siddhant Sahu - [@your_twitter](https://twitter.com/your_username) - email@example.com -->
<!-- Project Link: [https://github.com/your_username/repo_name](https://github.com/your_username/repo_name) -->
<!-- <p align="right">(<a href="#top">back to top</a>)</p> -->
<!-- ACKNOWLEDGMENTS
## Acknowledgments
Use this space to list resources you find helpful and would like to give credit to. I've included a few of my favorites to kick things off!
* [Choose an Open Source License](https://choosealicense.com)
* [GitHub Emoji Cheat Sheet](https://www.webpagefx.com/tools/emoji-cheat-sheet)
* [Malven's Flexbox Cheatsheet](https://flexbox.malven.co/)
* [Malven's Grid Cheatsheet](https://grid.malven.co/)
* [Img Shields](https://shields.io)
* [GitHub Pages](https://pages.github.com)
* [Font Awesome](https://fontawesome.com)
* [React Icons](https://react-icons.github.io/react-icons/search)
<p align="right">(<a href="#top">back to top</a>)</p> -->
<!-- MARKDOWN LINKS & IMAGES -->
<!-- https://www.markdownguide.org/basic-syntax/#reference-style-links -->
[contributors-shield]: https://img.shields.io/github/contributors/othneildrew/Best-README-Template.svg?style=for-the-badge
[contributors-url]: https://github.com/othneildrew/Best-README-Template/graphs/contributors
[forks-shield]: https://img.shields.io/github/forks/othneildrew/Best-README-Template.svg?style=for-the-badge
[forks-url]: https://github.com/othneildrew/Best-README-Template/network/members
[stars-shield]: https://img.shields.io/github/stars/othneildrew/Best-README-Template.svg?style=for-the-badge
[stars-url]: https://github.com/othneildrew/Best-README-Template/stargazers
[issues-shield]: https://img.shields.io/github/issues/othneildrew/Best-README-Template.svg?style=for-the-badge
[issues-url]: https://github.com/othneildrew/Best-README-Template/issues
[license-shield]: https://img.shields.io/github/license/othneildrew/Best-README-Template.svg?style=for-the-badge
[license-url]: https://github.com/othneildrew/Best-README-Template/blob/master/LICENSE.txt
[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555
[linkedin-url]: https://linkedin.com/in/othneildrew
[product-screenshot]: images/screenshot.png
others/rpg_vid2e/web_app/utils/events.py 0 → 100644
View file @ a75d2bda
from os.path import join
from numba.cuda.simulator.api import Event
import numpy as np
import glob
import numba
import h5py
import matplotlib.pyplot as plt
from utils.viz import Visualizer
from utils.utils import EventRenderingType
def load_events(f):
if f.endswith(".npy"):
return np.load(f).astype("int64")
elif f.endswith(".npz"):
fh = np.load(f)
return np.stack([fh['x'], fh['y'], fh['t'], fh['p']], -1)
elif f.endswith(".h5"):
fh = h5py.File(f, "r")
return np.stack([np.array(fh['x']), np.array(fh['y']), np.array(fh['t']), np.array(fh['p'])], -1)
else:
raise NotImplementedErrort(f"Could not read {f}")
class Events:
def __init__(self, shape=None, events=None):
self.shape = shape
self.events = np.array(events)
def __len__(self):
return len(self.events)
@classmethod
def from_folder(cls, folder, shape):
event_files = sorted(glob.glob(join(folder, "*")))
events = np.concatenate([load_events(f) for f in event_files], 0)
return cls(shape=shape, events=events)
@classmethod
def from_file(cls, file, shape):
events = load_events(file)
print(f"Loaded events from {file}, found {len(events)} events with shape {events.shape}")
return cls(shape=shape, events=events)
@property
def p(self):
return self.events[:,3]
@property
def x(self):
return self.events[:, 0]
@property
def y(self):
return self.events[:, 1]
@property
def t(self):
return self.events[:, 2]
def downsample(self, n):
return Events(self.shape, self.events[::n])
def slice_between_t(self, t0, t1):
return self.slice_before_t(t1).slice_after_t(t0)
def slice_before_t(self, t, num_events=-1):
events = self.events[self.events[:, 2] < t]
if num_events > 0:
events = events[-num_events:]
return Events(shape=self.shape, events=events)
def slice_after_t(self, t):
return Events(shape=self.shape, events=self.events[self.t>t])
def slice_num_events(self, num_events):
return Events(shape=self.shape, events=self.events[-num_events:])
def chunk(self, i, j):
return Events(shape=self.shape, events=self.events[i:j])
def render(self, rendering=None, rendering_type=EventRenderingType.RED_BLUE_NO_OVERLAP):
if rendering_type == EventRenderingType.RED_BLUE_OVERLAP:
return _render_overlap(self, rendering, color="red_blue")
elif rendering_type == EventRenderingType.RED_BLUE_NO_OVERLAP:
return _render_no_overlap(self, rendering, color="red_blue")
elif rendering_type == EventRenderingType.BLACK_WHITE_NO_OVERLAP:
return _render_no_overlap(self, rendering, color="black_white")
elif rendering_type == EventRenderingType.TIME_SURFACE:
return _render_timesurface(self)
elif rendering_type == EventRenderingType.EVENT_FRAME:
return _render_event_frame(self)
def __repr__(self):
return self.events.__repr__()
def mask(self, mask):
return Events(shape=self.shape, events=self.events[mask])
def interactive_visualization_loop(self, window_size_ms, framerate, rendering_type=EventRenderingType.RED_BLUE_OVERLAP):
visualizer = Visualizer(self,
window_size_ms=window_size_ms,
framerate=framerate,
rendering_type=rendering_type)
visualizer.visualizationLoop()
def compute_index(self, t):
return np.searchsorted(self.t, t)-1
def _render_overlap(events, rendering, color="red_blue"):
white_canvas = np.full(shape=(events.shape[0], events.shape[1], 3), fill_value=255, dtype="uint8")
rendering = rendering.copy() if rendering is not None else white_canvas
mask = _is_in_rectangle(events.x, events.y, rendering.shape[:2])
x, y, p = events.x.astype("int"), events.y.astype("int"), events.p.astype("p")
if color == "red_blue":
# map p 0, 1 to -1,0
rendering[y[mask], x[mask], :] = 0
rendering[y[mask], x[mask], p[mask]-1] = 255
return rendering
def _render_no_overlap(events, rendering, color="red_blue"):
fill_value = 128 if color == "black_white" else 255
canvas = np.full(shape=(events.shape[0], events.shape[1], 3), fill_value=fill_value, dtype="uint8")
rendering = rendering.copy() if rendering is not None else canvas
H, W = rendering.shape[:2]
mask = (events.x >= 0) & (events.y >= 0) & (events.x <= W - 1) & (events.y <= H - 1)
red = np.array([255, 0, 0])
blue = np.array([0,0,255])
black = np.array([0,0,0])
white = np.array([255,255,255])
if color == "red_blue":
pos = blue
neg = red
elif color == "black_white":
pos = white
neg = black
visited_mask = np.ones(shape=events.shape) == 0
return _render_no_overlap_numba(rendering,
visited_mask,
events.x[mask][::-1].astype("int"),
events.y[mask][::-1].astype("int"),
events.p[mask][::-1].astype("int"),
pos, neg)
@numba.jit(nopython=True)
def _render_no_overlap_numba(rendering, mask, x, y, p, pos_color, neg_color):
for x_, y_, p_ in zip(x, y, p):
if not mask[y_, x_]:
rendering[y_, x_] = pos_color if p_ > 0 else neg_color
mask[y_, x_] = True
return rendering
def _render_timesurface(events):
image = np.zeros(events.shape, dtype="float32")
cm = plt.get_cmap("jet")
tau = 3e4
t = events.t.astype("int")
if len(events) > 2:
value = np.exp(-(t[-1]-t)/float(tau))
_aggregate(image, events.x, events.y, value)
image = cm(image)
else:
image = image.astype("uint8")
return image
def _render_event_frame(events):
img = np.zeros(shape=events.shape, dtype="float32")
img = _aggregate(img, events.x, events.y, 2*events.p-1)
img_rendered = 10 * img + 128
return np.clip(img_rendered, 0, 255).astype("uint8")
def _aggregate_int(img, x, y, v):
np.add.at(img, (y, x), v)
def _aggregate(img, x, y, v):
if x.dtype == np.float32 or x.dtype == np.float64:
_aggregate_float(img, x, y, v)
else:
_aggregate_int(img, x, y, v)
return img
def _aggregate_float(img, x, y, v):
H, W = img.shape
x_ = x.astype("int32")
y_ = y.astype("int32")
for xlim in [x_, x_+1]:
for ylim in [y_, y_+1]:
mask = (xlim >= 0) & (ylim >= 0) & (xlim < W) & (ylim < H)
weight = (1 - np.abs(xlim - x)) * (1 - np.abs(ylim - y))
_aggregate_int(img, xlim[mask], ylim[mask], v[mask] * weight[mask])
def _is_in_rectangle(x, y, shape):
return (x >= 0) & (y >= 0) & (x <= shape[1]-1) & (y <= shape[0]-1)
\ No newline at end of file
others/rpg_vid2e/web_app/utils/utils.py 0 → 100644
View file @ a75d2bda
import enum
class EventRenderingType(enum.IntEnum):
RED_BLUE_OVERLAP = enum.auto()
RED_BLUE_NO_OVERLAP = enum.auto()
BLACK_WHITE_NO_OVERLAP = enum.auto()
TIME_SURFACE = enum.auto()
EVENT_FRAME = enum.auto()
others/rpg_vid2e/web_app/utils/viz.py 0 → 100644
View file @ a75d2bda
import cv2
import numpy as np
from utils.utils import EventRenderingType
class Visualizer:
def __init__(self, events, window_size_ms=10, framerate=100, rendering_type=EventRenderingType.RED_BLUE_NO_OVERLAP):
self.events = events
self.rendering_type = rendering_type
self.window_size_ms = window_size_ms
self.framerate = framerate
self.time_between_screen_refresh_ms = 5
self.is_paused = False
self.is_looped = False
self.update_indices(events, window_size_ms, framerate)
self.index = 0
self.cv2_window_name = 'Events'
self.annotations = {}
cv2.namedWindow(self.cv2_window_name, cv2.WINDOW_NORMAL)
self.print_help()
def update_indices(self, events, window_size_ms, framerate):
self.t0_us, self.t1_us = self.compute_event_window_limits(events, window_size_ms, framerate)
self.t0_index = events.compute_index(self.t0_us)
self.t1_index = events.compute_index(self.t1_us)
def compute_event_window_limits(self, events, window_size_ms, framerate):
t_min_us = events.t[0]
t_max_us = events.t[-1]
t1_us = np.arange(t_min_us, t_max_us, 1e6 / framerate)
t0_us = np.clip(t1_us - window_size_ms * 1e3, t_min_us, t_max_us)
return t0_us, t1_us
def pause(self):
self.is_paused = True
def unpause(self):
self.is_paused= False
def togglePause(self):
self.is_paused = not self.is_paused
def toggleLoop(self):
self.is_looped = not self.is_looped
def forward(self, num_timesteps = 1):
if self.is_looped:
self.index = (self.index + 1) % len(self.t0_index)
else:
self.index = min(self.index + num_timesteps, len(self.t0_index) - 1)
def backward(self, num_timesteps = 1):
self.index = max(self.index - num_timesteps, 0)
def goToBegin(self):
self.index = 0
def goToEnd(self):
self.index = len(self.t0_index) - 1
def render_annotation(self, image, annotation):
refPt = annotation["refPt"]
t = annotation["t"]
cv2.rectangle(image, refPt[0], refPt[1], (0, 255, 0), 2)
cv2.putText(image, f"t={t}", refPt[0], cv2.FONT_HERSHEY_SIMPLEX, fontScale=1, thickness=3, color=(0,0,0))
return image
def cycle_colors(self):
self.rendering_type = (self.rendering_type % len(EventRenderingType))+1
print("New Rendering Type: ", self.rendering_type)
def visualizationLoop(self):
self.refPt = []
self.cropping = False
while True:
self.image = self.update(self.index)
cv2.imshow(self.cv2_window_name, self.image)
if not self.is_paused:
self.forward(1)
c = cv2.waitKey(self.time_between_screen_refresh_ms)
key = chr(c & 255)
if c == 27: # 'q' or 'Esc': Quit
break
elif key == 'r': # 'r': Reset
self.goToBegin()
self.unpause()
elif key == 'p' or c == 32: # 'p' or 'Space': Toggle play/pause
self.togglePause()
elif key == "a": # 'Left arrow': Go backward
self.backward(1)
self.pause()
elif key == "d": # 'Right arrow': Go forward
self.forward(1)
self.pause()
elif key == "s": # 'Down arrow': Go to beginning
self.goToBegin()
self.pause()
elif key == "w": # 'Up arrow': Go to end
self.goToEnd()
self.pause()
elif key == 'l': # 'l': Toggle looping
self.toggleLoop()
elif key == "e":
self.update_window(1.2)
elif key == "q":
self.update_window(1/1.2)
elif key == "c":
self.cycle_colors()
elif key == "h":
self.print_help()
cv2.destroyAllWindows()
def print_help(self):
print("##################################")
print("# interactive visualizer #")
print("# #")
print("# a: backward #")
print("# d: forward #")
print("# w: jump to end #")
print("# s: jump to front #")
print("# e: lengthen time window #")
print("# q: shorten time window #")
print("# c: cycle color scheme #")
print("# esc: quit #")
print("# space: pause #")
print("# h: print help #")
print("##################################")
def update_window(self, factor):
self.window_size_ms *= factor
self.update_indices(self.events, self.window_size_ms, self.framerate)
def update(self, index):
index0 = self.t0_index[index]
index1 = self.t1_index[index]
image = self.events.chunk(index0, index1).render(rendering_type=self.rendering_type)
t = self.t1_us[self.index]
cv2.putText(image, f"t={t}", (10,30), cv2.FONT_HERSHEY_SIMPLEX, fontScale=1, thickness=3, color=(0,0,0))
return image
\ No newline at end of file
others/rpg_vid2e/web_app/web_app.py 0 → 100644
View file @ a75d2bda
import streamlit as st
import numpy as np
import torch
from stqdm import stqdm
import skvideo.io
#from esim_torch import esim_torch
#import esim_torch
#from esim-cuda
import os
import numba
import cv2
import tqdm
from fractions import Fraction
from typing import Union
from pathlib import Path
from io import StringIO
#from .esim_torch import EventSimulator_torch
#from esim_torch import esim_torch
# from esim_torch.esim_torch import EventSimulator_torch
#import esim_torch
import glob
import h5py
from utils.events import Events
import sys
sys.path.append(os.path.join(os.path.abspath(os.getcwd()), "../esim_torch"))
from esim_torch import EventSimulator_torch
st.sidebar.title("Menu")
add_selectbox = st.sidebar.selectbox(
"ESIM Playground Options",
("Offline Video Generator", "Live Webcam")
)
if add_selectbox == "Offline Video Generator":
st.title("ESIM Web App Playground")
st.markdown('The goal of this project is to make ESIM and VID2E available to researchers who do not possess a real event camera. For this we want to deploy VID2E and ESIM as an interactive web app. The app should be easy to use and have the following functional requirements:')
st.markdown('- Generation of events from video through dragging and dropping a video into thebrowser. The resulting events should be downloadable as raw events and rendered video.')
st.markdown('- Incorporate video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.')
st.markdown('- Visualization and inspection of the event stream in the browser.')
st.markdown('- The event generation should be configurable by changing ESIM parameters.')
st.subheader("Upload")
video_file = st.file_uploader("Upload the video file for which you would like the events generation", type=([".webm", ".mp4", ".m4p", ".m4v", ".avi", ".avchd", ".ogg", ".mov", ".ogv", ".vob", ".f4v", ".mkv", ".svi", ".m2v", ".mpg", ".mp2", ".mpeg", ".mpe", ".mpv", ".amv", ".wmv", ".flv", ".mts", ".m2ts", ".ts", ".qt", ".3gp", ".3g2", ".f4p", ".f4a", ".f4b"]))
st.subheader('ESIM Settings')
ct_options = ['0.1', '0.2', '0.3', '0.4', '0.5', '0.6', '0.7', '0.8', '0.9', '1.0', '1.1', '1.2', '1.3', '1.4', '1.5', '1.6', '1.7', '1.8', '1.9', '2.0']
st.subheader('Positive Contrast Threshold')
ct_p = st.select_slider("Choose the +ve contrast threshold", options=ct_options)
st.subheader('Negative Contrast Threshold')
ct_n = st.select_slider("Choose the -ve contrast threshold", options=ct_options)
window_size_options = [24, 30, 60, 80, 100, 120]
st.subheader('Window Size (Number of Frames)')
window_size = st.select_slider("Choose the window size or number of frames for event generation", options=window_size_options)
st.subheader('Upsampling')
upsampling = st.checkbox('Yes', True)
st.subheader('Output Format')
format_name = ['HDF5', 'NPZ', 'Rendered Video']
format = st.radio('Select the output format for the generated events', format_name)
class Sequence:
#def __init__(self):
#normalize = transforms.Normalize(mean=mean, std=std)
#self.transform = transforms.Compose([transforms.ToTensor(), normalize])
def __iter__(self):
return self
def __next__(self):
raise NotImplementedError
def __len__(self):
raise NotImplementedError
class VideoSequence(Sequence):
def __init__(self, video_filepath: str, fps: float=None):
#super().__init__()
self.metadata = skvideo.io.ffprobe(os.path.abspath(video_filepath))
st.write(video_filepath)
#st.write(type(metadata))
#st.write(metadata.keys)
self.fps = fps
if self.fps is None:
self.fps = float(Fraction(self.metadata['video']['@avg_frame_rate']))
assert self.fps > 0, 'Could not retrieve fps from video metadata. fps: {}'.format(self.fps)
#print('Using video metadata: Got fps of {} frames/sec'.format(self.fps))
# Length is number of frames - 1 (because we return pairs).
self.len = int(self.metadata['video']['@nb_frames']) - 1
self.videogen = skvideo.io.vreader(os.path.abspath(video_filepath))
self.last_frame = None
def __next__(self):
for idx, frame in enumerate(self.videogen):
#h_orig, w_orig, _ = frame.shape
#w, h = w_orig//32*32, h_orig//32*32
#left = (w_orig - w)//2
#upper = (h_orig - h)//2
#right = left + w
#lower = upper + h
#frame = frame[upper:lower, left:right]
#assert frame.shape[:2] == (h, w)
#frame = self.transform(frame)
if self.last_frame is None:
self.last_frame = frame
continue
#last_frame_copy = self.last_frame.detach().clone()
self.last_frame = frame
#imgs = [last_frame_copy, frame]
#times_sec = [(idx - 1)/self.fps, idx/self.fps]
img = frame
time_sec = idx/self.fps
#yield imgs, times_sec
yield img, time_sec
def __len__(self):
return self.len
#writer = skvideo.io.FFmpegWriter("outputvideo.mp4")
def is_video_file(filepath: str) -> bool:
return Path(filepath).suffix.lower() in {'.webm', '.mp4', '.m4p', '.m4v', '.avi', '.avchd', '.ogg', '.mov', '.ogv', '.vob', '.f4v', '.mkv', '.svi', '.m2v', '.mpg', '.mp2', '.mpeg', '.mpe', '.mpv', '.amv', '.wmv', '.flv', '.mts', '.m2ts', '.ts', '.qt', '.3gp', '.3g2', '.f4p', '.f4a', '.f4b'}
def get_video_file_path(dirpath: str) -> Union[None, str]:
filenames = [f for f in os.listdir(dirpath) if is_video_file(f)]
if len(filenames) == 0:
return None
assert len(filenames) == 1
filepath = os.path.join(dirpath, filenames[0])
return filepath
def save_to_npz(target_path, data: dict):
assert os.path.exists(target_path)
path = os.path.join(target_path, "events.npz")
np.savez(path, **data)
return path
def save_to_video(target_path, shape, data: dict):
# just convert to 30 fps, non-overlapping windows
fps = 30
tmin, tmax = data['t'][[0,-1]]
t0 = np.arange(tmin, tmax, 1e6/fps)
t1, t0 = t0[1:], t0[:-1]
idx0 = np.searchsorted(data['t'], t0)
idx1 = np.searchsorted(data['t'], t1)
path = os.path.join(target_path, "outputvideo.mp4")
writer = skvideo.io.FFmpegWriter(path)
red = np.array([255, 0, 0], dtype="uint8")
blue = np.array([0,0,255], dtype="uint8")
pbar = tqdm.tqdm(total=len(idx0))
for i0, i1 in zip(idx0, idx1):
sub_data = {k: v[i0:i1].cpu().numpy().astype("int32") for k, v in data.items()}
frame = np.full(shape=shape + (3,), fill_value=255, dtype="uint8")
event_processor(sub_data['x'], sub_data['y'], sub_data['p'], red, blue, frame)
writer.writeFrame(frame)
pbar.update(1)
writer.close()
return path
def save_to_h5(target_path, data: dict):
assert os.path.exists(target_path)
path = str(target_path+"events.h5")
with h5py.File(path, 'w') as h5f:
for k, v in data.items():
h5f.create_dataset(k, data=v)
return path
@numba.jit(nopython=True)
def event_processor(x, y, p, red, blue, output_frame):
for x_,y_,p_ in zip(x, y, p):
if p_ == 1:
output_frame[y_,x_] = blue
else:
output_frame[y_,x_] = red
return output_frame
def print_inventory(dct):
print("Items held:")
for item, amount in dct.items(): # dct.iteritems() in Python 2
print("{} ({})".format(item, amount))
def process_dir(outdir, indir, args):
print(f"Processing folder {indir}... Generating events in {outdir}")
os.makedirs(outdir, exist_ok=True)
# constructor
esim = EventSimulator_torch(args["contrast_threshold_negative"],
args["contrast_threshold_positive"],
args["refractory_period_ns"])
timestamps = np.genfromtxt(os.path.join(indir, "video_upload/timestamps.txt"), dtype="float64")
timestamps_ns = (timestamps * 1e9).astype("int64")
timestamps_ns = torch.from_numpy(timestamps_ns).cuda()
image_files = sorted(glob.glob(os.path.join(indir, "video_upload/imgs", "*.png")))
# pbar = tqdm.tqdm(total=len(image_files)-1)
# num_events = 0
# counter = 0
# for image_file, timestamp_ns in zip(image_files, timestamps_ns):
# image = cv2.imread(image_file, cv2.IMREAD_GRAYSCALE)
# log_image = np.log(image.astype("float32") / 255 + 1e-5)
# log_image = torch.from_numpy(log_image).cuda()
# sub_events = esim.forward(log_image, timestamp_ns)
# # for the first image, no events are generated, so this needs to be skipped
# if sub_events is None:
# continue
# sub_events = {k: v.cpu() for k, v in sub_events.items()}
# num_events += len(sub_events['t'])
# # do something with the events
# np.savez(os.path.join(outdir, "%010d.npz" % counter), **sub_events)
# pbar.set_description(f"Num events generated: {num_events}")
# pbar.update(1)
# counter += 1
images = np.stack([cv2.imread(f, cv2.IMREAD_GRAYSCALE) for f in image_files])
shape = images.shape[1:]
log_images = np.log(images.astype("float32") / 255 + 1e-4)
log_images = torch.from_numpy(log_images).cuda()
# generate events with GPU support
print("Generating events")
generated_events = esim.forward(log_images, timestamps_ns)
#events = esim.forward(log_image, timestamp_ns)
#print_inventory(generated_events)
generated_events = {k: v.cpu() for k, v in generated_events.items()}
generated_events['t'] = (generated_events['t'] / 1e3).long()
print_inventory(generated_events)
if args['format'] == "HDF5":
return save_to_h5(args["output_dir"], generated_events)
elif args['format'] == "NPZ":
return save_to_npz(args["output_dir"], generated_events)
elif args['format'] == "Rendered Video":
return save_to_video(args["output_dir"], shape, generated_events)
else:
raise ValueError
args = {
"contrast_threshold_negative": float(ct_n),
"contrast_threshold_positive": float(ct_p),
"refractory_period_ns": 0,
"input_dir": "data/upsampled/",
"output_dir": "data/events/",
"format": format
}
generate_button = st.button("Generate Events")
if generate_button:
#Step 1:
if video_file is not None:
#read the video file
#video_object = VideoSequence(video_file.getvalue())
#saving the uploaded file on the server
save_path = "data/original/video_upload/"
completeName = os.path.join(save_path, video_file.name)
with open(completeName, "wb") as file:
file.write(video_file.getvalue())
st.success("Uploaded video file saved on the server")
#Step 2:
if upsampling:
print("inside upsampling")
os.system("python ../upsampling/upsample.py --input_dir=data/original/ --output_dir=data/upsampled --device=cuda:0")
#Step 3: Event Generation
file_path = process_dir(args["output_dir"], args["input_dir"], args)
#Step 4: Download Option
with open(file_path, "rb") as file:
btn = st.download_button(label="Download Events", data=file, file_name=os.path.basename(file_path))
if btn:
st.markdown(':beer::beer::beer::beer::beer::beer::beer::beer::beer::beer::beer:')
elif add_selectbox == "Live Webcam":
st.title("ESIM Web App Playground")
# st.markdown(':camera::movie_camera::camera::movie_camera::camera::movie_camera::camera::movie_camera::camera::movie_camera:')
st.sidebar.subheader('ESIM Settings')
ct_options = ['0.1', '0.2', '0.3', '0.4', '0.5', '0.6', '0.7', '0.8', '0.9', '1.0', '1.1', '1.2', '1.3', '1.4', '1.5', '1.6', '1.7', '1.8', '1.9', '2.0']
st.sidebar.subheader('Positive Contrast Threshold')
ct_p = st.sidebar.select_slider("Choose the +ve contrast threshold", options=ct_options)
st.sidebar.subheader('Negative Contrast Threshold')
ct_n = st.sidebar.select_slider("Choose the -ve contrast threshold", options=ct_options)
window_size_options = [24, 30, 60, 80, 100, 120]
st.sidebar.subheader('Window Size (Number of Frames)')
window_size = st.sidebar.select_slider("Choose the window size or number of frames for event generation", options=window_size_options)
esim = EventSimulator_torch(float(ct_n), float(ct_p), 0)
FRAME_WINDOW = st.image([])
cam = cv2.VideoCapture(0)
frame_count = 0
#frames = np.empty(shape=(7,))
#timestamps = np.empty(shape=(7,))
frame_log = []
timestamps_ns = []
window_size = 10
while True:
# esim = EventSimulator_torch(0.4, 0.4, 0)
if frame_count <= window_size:
ret, frame = cam.read()
if not ret:
continue
if ret:
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
#FRAME_WINDOW.image(frame)
#st.write(cam.get(cv2.CAP_PROP_POS_MSEC))
#st.markdown(f"FPS: {cam.get(cv2.CAP_PROP_POS_MSEC)}")
frame_log_new = np.log(frame.astype("float32") / 255 + 1e-5)
#frame_log = torch.from_numpy(frame_log).cuda()
#frame_log.append(frame_log_new)
frame_log.append(frame_log_new)
timestamp_curr = cam.get(cv2.CAP_PROP_POS_MSEC)
timestamps_ns_new = (timestamp_curr * 1e9)
#timestamps_ns_new = np.array(timestamps_ns, dtype="int64")
#print(type(timestamp_curr))
#timestamps_ns = torch.from_numpy(timestamps_ns).cuda()
# timestamps_ns.append(timestamps_ns_new)
timestamps_ns.append(timestamps_ns_new)
# print(cam.get(cv2.CAP_PROP_FRAME_WIDTH))
# print(cam.get(cv2.CAP_PROP_FRAME_HEIGHT))
frame_count += 1
else:
#Event Generation
# esim = EventSimulator_torch(0.2, 0.2, 0)
#render events as frames
frame_log_torch_tensor = torch.from_numpy(np.array(frame_log)).cuda()
timestamps_ns_torch_tensor = torch.from_numpy(np.array(timestamps_ns, dtype="int64")).cuda()
generated_events = esim.forward(frame_log_torch_tensor, timestamps_ns_torch_tensor)
generated_events = {k: v.cpu() for k, v in generated_events.items()}
generated_events = np.stack([generated_events['x'], generated_events['y'], generated_events['t'], generated_events['p']], -1)
generated_events = Events((int(cam.get(cv2.CAP_PROP_FRAME_HEIGHT)), int(cam.get(cv2.CAP_PROP_FRAME_WIDTH))), generated_events)
event_rendering = generated_events.render()
# print(event_rendering.shape)
FRAME_WINDOW.image(event_rendering)
frame_count = 0
frame_log.clear()
timestamps_ns.clear()
\ No newline at end of file
scripts/dist_test.sh 0 → 100755
View file @ a75d2bda
#!/usr/bin/env bash
# export HIP_VISIBLE_DEVICES=3
GPUS=$1
CONFIG=$2
PORT=${PORT:-4321}
# usage
if [ $# -ne 2 ] ;then
echo "usage:"
echo "./scripts/dist_test.sh [number of gpu] [path to option file]"
exit
fi
# PYTHONPATH="$(dirname $0)/..:${PYTHONPATH}" \
# python -m torch.distributed.launch --nproc_per_node=$GPUS --master_port=$PORT \
# basicsr/test.py -opt $CONFIG --launcher pytorch
torchrun --nnodes=1 --nproc_per_node=$GPUS --rdzv_id=100 --rdzv_backend=c10d --rdzv_endpoint=localhost:29400 basicsr/test.py -opt $CONFIG
\ No newline at end of file
scripts/dist_train.sh 0 → 100755
View file @ a75d2bda
#!/usr/bin/env bash
GPUS=$1
CONFIG=$2
PORT=${PORT:-4321}
# usage
if [ $# -lt 2 ] ;then
echo "usage:"
echo "./scripts/dist_train.sh [number of gpu] [path to option file]"
exit
fi
PYTHONPATH="$(dirname $0)/..:${PYTHONPATH}" \
python -m torch.distributed.launch --nproc_per_node=$GPUS --master_port=$PORT \
basicsr/train.py -opt $CONFIG --launcher pytorch ${@:3}
setup.cfg 0 → 100644
View file @ a75d2bda
[flake8]
ignore =
# line break before binary operator (W503)
W503,
# line break after binary operator (W504)
W504,
max-line-length=120
[yapf]
based_on_style = pep8
column_limit = 120
blank_line_before_nested_class_or_def = true
split_before_expression_after_opening_paren = true
[isort]
line_length = 120
multi_line_output = 0
known_standard_library = pkg_resources,setuptools
known_first_party = basicsr
known_third_party = PIL,cv2,lmdb,numpy,pytest,requests,scipy,skimage,torch,torchvision,tqdm,yaml
no_lines_before = STDLIB,LOCALFOLDER
default_section = THIRDPARTY
[codespell]
skip = .git,./docs/build,*.cfg
count =
quiet-level = 3
ignore-words-list = gool
[aliases]
test=pytest
[tool:pytest]
addopts=tests/
setup.py 0 → 100644
View file @ a75d2bda
#!/usr/bin/env python
from setuptools import find_packages, setup
import os
import subprocess
import time
version_file = 'basicsr/version.py'
def readme():
with open('README.md', encoding='utf-8') as f:
content = f.read()
return content
def get_git_hash():
def _minimal_ext_cmd(cmd):
# construct minimal environment
env = {}
for k in ['SYSTEMROOT', 'PATH', 'HOME']:
v = os.environ.get(k)
if v is not None:
env[k] = v
# LANGUAGE is used on win32
env['LANGUAGE'] = 'C'
env['LANG'] = 'C'
env['LC_ALL'] = 'C'
out = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=env).communicate()[0]
return out
try:
out = _minimal_ext_cmd(['git', 'rev-parse', 'HEAD'])
sha = out.strip().decode('ascii')
except OSError:
sha = 'unknown'
return sha
def get_hash():
if os.path.exists('.git'):
sha = get_git_hash()[:7]
# currently ignore this
# elif os.path.exists(version_file):
# try:
# from basicsr.version import __version__
# sha = __version__.split('+')[-1]
# except ImportError:
# raise ImportError('Unable to get git version')
else:
sha = 'unknown'
return sha
def write_version_py():
content = """# GENERATED VERSION FILE
# TIME: {}
__version__ = '{}'
__gitsha__ = '{}'
version_info = ({})
"""
sha = get_hash()
with open('VERSION', 'r') as f:
SHORT_VERSION = f.read().strip()
VERSION_INFO = ', '.join([x if x.isdigit() else f'"{x}"' for x in SHORT_VERSION.split('.')])
version_file_str = content.format(time.asctime(), SHORT_VERSION, sha, VERSION_INFO)
with open(version_file, 'w') as f:
f.write(version_file_str)
def get_version():
with open(version_file, 'r') as f:
exec(compile(f.read(), version_file, 'exec'))
return locals()['__version__']
def make_cuda_ext(name, module, sources, sources_cuda=None):
if sources_cuda is None:
sources_cuda = []
define_macros = []
extra_compile_args = {'cxx': []}
if torch.cuda.is_available() or os.getenv('FORCE_CUDA', '0') == '1':
define_macros += [('WITH_CUDA', None)]
extension = CUDAExtension
extra_compile_args['nvcc'] = [
'-D__CUDA_NO_HALF_OPERATORS__',
'-D__CUDA_NO_HALF_CONVERSIONS__',
'-D__CUDA_NO_HALF2_OPERATORS__',
]
sources += sources_cuda
else:
print(f'Compiling {name} without CUDA')
extension = CppExtension
return extension(
name=f'{module}.{name}',
sources=[os.path.join(*module.split('.'), p) for p in sources],
define_macros=define_macros,
extra_compile_args=extra_compile_args)
def get_requirements(filename='requirements.txt'):
here = os.path.dirname(os.path.realpath(__file__))
with open(os.path.join(here, filename), 'r') as f:
requires = [line.replace('\n', '') for line in f.readlines()]
return requires
if __name__ == '__main__':
cuda_ext = os.getenv('BASICSR_EXT') # whether compile cuda ext
if cuda_ext == 'True':
try:
import torch
from torch.utils.cpp_extension import BuildExtension, CppExtension, CUDAExtension
except ImportError:
raise ImportError('Unable to import torch - torch is needed to build cuda extensions')
ext_modules = [
make_cuda_ext(
name='deform_conv_ext',
module='basicsr.ops.dcn',
sources=['src/deform_conv_ext.cpp'],
sources_cuda=['src/deform_conv_cuda.cpp', 'src/deform_conv_cuda_kernel.cu']),
make_cuda_ext(
name='fused_act_ext',
module='basicsr.ops.fused_act',
sources=['src/fused_bias_act.cpp'],
sources_cuda=['src/fused_bias_act_kernel.cu']),
make_cuda_ext(
name='upfirdn2d_ext',
module='basicsr.ops.upfirdn2d',
sources=['src/upfirdn2d.cpp'],
sources_cuda=['src/upfirdn2d_kernel.cu']),
]
setup_kwargs = dict(cmdclass={'build_ext': BuildExtension})
else:
ext_modules = []
setup_kwargs = dict()
write_version_py()
setup(
name='basicsr',
version=get_version(),
description='Open Source Image and Video Super-Resolution Toolbox',
long_description=readme(),
long_description_content_type='text/markdown',
keywords='computer vision, restoration, super resolution',
url='https://github.com/xinntao/BasicSR',
include_package_data=True,
packages=find_packages(exclude=('options', 'datasets', 'experiments', 'results', 'tb_logger', 'wandb')),
classifiers=[
'Development Status :: 4 - Beta',
'License :: OSI Approved :: Apache Software License',
'Operating System :: OS Independent',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
],
license='Apache License 2.0',
setup_requires=['cython', 'numpy', 'torch'],
install_requires=get_requirements(),
ext_modules=ext_modules,
zip_safe=False,
**setup_kwargs)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment