Skip to content

Free Motion Capture for Everyone πŸ’€βœ¨

TypeScript

Use Black to format all of `src` and add Contribution Guidelines (#188)

Endurance Idehen04529f746d41
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 00000000..be32fb7a
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,58 @@
+# Contributing to Freemocap
+We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+- Becoming a maintainer
+
+## We Develop with Github
+We use github to host code, to track issues and feature requests, as well as accept pull requests.
+
+## We Use [Github Flow](https://guides.github.com/introduction/flow/index.html), So All Code Changes Happen Through Pull Requests
+Pull requests are the best way to propose changes to the codebase (we use [Github Flow](https://guides.github.com/introduction/flow/index.html)). We actively welcome your pull requests:
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. Issue that pull request!
+
+## Any contributions you make will be under the AGPL Software License
+In short, when you submit code changes, your submissions are understood to be under the same [AGPL](LICENSE) that covers the project. Feel free to contact the maintainers to understand what that means.
+
+## Report bugs using Github's [issues](https://github.com/freemocap/freemocap/issues)
+We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/freemocap/freemocap/issues/new); it's that easy!
+
+## Write bug reports with detail, background, and sample code
+[This is an example](http://stackoverflow.com/q/12488905/180626) of a "great" bug report.
+
+**Great Bug Reports** tend to have:
+
+- A quick summary and/or background
+- Steps to reproduce
+ - Be specific!
+ - Include an uploaded ZIP of the freemocap data session you ran that produced the issue.
+ - Give sample code if you can. [This stackoverflow question](http://stackoverflow.com/q/12488905/180626) demonstrates the user giving as much information as possible.
+- What you expected would happen
+- What actually happens
+- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
+
+People *love* thorough bug reports. The likelihood that the community and/or maintainers will help you with your issues stems from how well you can help them understand you. :)
+
+## Use a Consistent Coding Style
+We use the [Black](https://black.readthedocs.io/en/stable/) autoformatter as the de-facto syntax style guide of choice.
+
+### PyQT Style Guide
+COMING SOON
+
+### API Style Guide
+COMING SOON
+
+## License
+By contributing, you agree that your contributions will be licensed under its [AGPL](LICENSE) License.
+
+## References
+This document was adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/main/CONTRIBUTING.md)
\ No newline at end of file
diff --git a/README.md b/README.md
index e2796fe3..96f97bb3 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
<p align="center">
- <img src="https://raw.githubusercontent.com/freemocap/freemocap/main/logo/freemocap-logo-black-border.svg" height="64" alt="Project Logo">
+ <img src="https://raw.githubusercontent.com/freemocap/freemocap/main/assets/logo/freemocap-logo-black-border.svg" height="64" alt="Project Logo">
</p>
<h3 align="center">freemocap</h3>
<p align="center">πŸ“ The FreeMoCap Project: A free-and-open-source, hardware-and-software-agnostic, minimal-cost, research-grade, motion capture system and platform for decentralized scientific research, education, and training</p>
@@ -16,6 +16,9 @@
<a href="https://github.com/freemocap/freemocap/issues">
<img src="https://img.shields.io/badge/contributions-welcome-ff69b4.svg" alt="Contributions Welcome">
</a>
+ <a href="https://github.com/psf/black">
+ <img alt="https://img.shields.io/badge/code%20style-black-000000.svg" src="https://img.shields.io/badge/code%20style-black-000000.svg">
+ </a>
</p>
## Pardon our dust as we transtion to the `alpha` phase of this project!
@@ -33,7 +36,7 @@ If you would like to access the `pre-alpha` code, you can find it here under the
- At least 2 simple webcameras (more cameras, better quality)
- [$10 camera here](https://www.amazon.com/1080P-Full-Webcam-Microphone-Conferencing/dp/B09JFPFVY4/ref=asc_df_B09JFPFVY4/?tag=hyprod-20&linkCode=df0&hvadid=563548498710&hvpos=&hvnetw=g&hvrand=17033983634274965573&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9028321&hvtargid=pla-1629009316502&psc=1)
- Each Camera must have a clean, unobstructed view of a Charuco board during initialization.
-
+
### Easy MacOSX Install Instructions
1. Install the Homebrew Package manager (Skip this step if you already have it)
@@ -93,9 +96,22 @@ That's it! You're ready to run the freemocap application, and create your own di
## For Developers
+### Dev Setup
+
+After you've done the easy install instructions, you'll be able to run our repo commands
+
+1. Run the "setup" command to set up your environment
+```bash
+task setup
+```
+2. Run the tests to ensure that everything works appropriate
+```bash
+task test
+```
+
### Contribution Guidelines
-COMING SOON
+Please read our contribution doc: [CONTRIBUTING.md](CONTRIBUTING.md)
### Creating a new binary
diff --git a/pyproject.toml b/pyproject.toml
index 3898388b..278c391d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,4 +1,7 @@
[tool.taskipy.tasks]
+setup = "pre-commit install"
test = "python -m unittest src/tests/**/test_*"
installer = "./bin/installer.sh"
+# This command ensures that your code is formatted appropriately
+format = "black src/"
\ No newline at end of file
diff --git a/requirements.txt b/requirements.txt
index 3c343e55..ab15832a 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -26,4 +26,5 @@ pytest==7.1.2
taskipy==1.10.2
pytest-cov==3.0.0
pyinstaller==5.2
-pyinstaller-hooks-contrib==2022.8
\ No newline at end of file
+pyinstaller-hooks-contrib==2022.8
+black[d]==22.6.0
diff --git a/src/api/routes/__init__.py b/src/api/routes/__init__.py
index 8837c62c..299ceb02 100644
--- a/src/api/routes/__init__.py
+++ b/src/api/routes/__init__.py
@@ -6,4 +6,11 @@ from src.api.routes.session.session_router import session_router
from src.api.routes.startup.startup import startup_router
# REGISTER NEW ROUTES HERE
-enabled_routers = [healthcheck_router, camera_router, cam_ws_router, startup_router, home_router, session_router]
+enabled_routers = [
+ healthcheck_router,
+ camera_router,
+ cam_ws_router,
+ startup_router,
+ home_router,
+ session_router,
+]
diff --git a/src/api/routes/camera/cam_data_ws.py b/src/api/routes/camera/cam_data_ws.py
index b30b603f..78da6d75 100644
--- a/src/api/routes/camera/cam_data_ws.py
+++ b/src/api/routes/camera/cam_data_ws.py
@@ -30,8 +30,6 @@ async def preview_webcam(web_socket: WebSocket):
byte_data = await web_socket.receive_bytes()
time_since = time.perf_counter()
print(f"Per Frame Time: {time_since - last_read:.4f}")
- buffer = np.frombuffer(byte_data, dtype='uint8').reshape((600, 800, 4))
+ buffer = np.frombuffer(byte_data, dtype="uint8").reshape((600, 800, 4))
buffer = cv2.cvtColor(buffer, cv2.COLOR_BGR2RGB)
print(f"image shape: {buffer.shape}")
-
-
diff --git a/src/api/routes/camera/camera_route.py b/src/api/routes/camera/camera_route.py
index f239a52a..7541168a 100644
--- a/src/api/routes/camera/camera_route.py
+++ b/src/api/routes/camera/camera_route.py
@@ -24,12 +24,16 @@ async def config_cam(webcam_config_model: WebcamConfigModel, session_id):
@camera_router.post("/camera/show_camera/{webcam_id}")
-async def show_camera(webcam_id: str, camera_preview_model: CameraPreviewModel = CameraPreviewModel()):
+async def show_camera(
+ webcam_id: str, camera_preview_model: CameraPreviewModel = CameraPreviewModel()
+):
cv_cam_manager = OpenCVCameraManager(session_id=camera_preview_model.session_id)
- with cv_cam_manager.start_capture_session_single_cam(webcam_id) as connected_camera_and_writer:
+ with cv_cam_manager.start_capture_session_single_cam(
+ webcam_id
+ ) as connected_camera_and_writer:
should_continue = True
this_camera = connected_camera_and_writer.cv_camera
- cv2_window_name = '(ESC to close) PREVIEWING camera_' + webcam_id
+ cv2_window_name = "(ESC to close) PREVIEWING camera_" + webcam_id
while should_continue:
if not this_camera.new_frame_ready:
continue
@@ -43,10 +47,12 @@ async def show_camera(webcam_id: str, camera_preview_model: CameraPreviewModel =
@camera_router.post("/camera/show_cameras")
-async def cv2_imshow_all_camera(camera_preview_model: CameraPreviewModel = CameraPreviewModel()):
+async def cv2_imshow_all_camera(
+ camera_preview_model: CameraPreviewModel = CameraPreviewModel(),
+):
cv_cam_manager = OpenCVCameraManager(session_id=camera_preview_model.session_id)
with cv_cam_manager.start_capture_session_all_cams() as connected_cameras_dict:
- logger.info(f'Available cameras: {connected_cameras_dict}')
+ logger.info(f"Available cameras: {connected_cameras_dict}")
should_continue = True
while should_continue:
for this_webcam_id, this_open_cv_camera in connected_cameras_dict.items():
@@ -57,7 +63,7 @@ async def cv2_imshow_all_camera(camera_preview_model: CameraPreviewModel = Camer
cv2.destroyWindow(cv2_window_name)
if not this_open_cv_camera.new_frame_ready:
continue
- cv2_window_name = '(ESC to close) PREVIEWING camera_' + this_webcam_id
+ cv2_window_name = "(ESC to close) PREVIEWING camera_" + this_webcam_id
cv2.imshow(cv2_window_name, this_open_cv_camera.latest_frame.image)
exit_key = cv2.waitKey(1)
if exit_key == 27:
@@ -74,4 +80,3 @@ async def get_cameras():
@camera_router.get("/camera/redetect")
async def redetect_cameras():
return get_or_create_cams(always_create=True)
-
diff --git a/src/api/routes/home/home.py b/src/api/routes/home/home.py
index d1a0cec2..090c895f 100644
--- a/src/api/routes/home/home.py
+++ b/src/api/routes/home/home.py
@@ -5,6 +5,6 @@ home_router = APIRouter()
@home_router.get("/")
async def hello():
- response_string = 'hello :D'
- print(f'response string: {response_string}')
+ response_string = "hello :D"
+ print(f"response string: {response_string}")
return response_string
diff --git a/src/api/routes/session/session_router.py b/src/api/routes/session/session_router.py
index acaf76af..d8a399cc 100644
--- a/src/api/routes/session/session_router.py
+++ b/src/api/routes/session/session_router.py
@@ -6,15 +6,28 @@ from typing import Optional, Union
from fastapi import APIRouter
from pydantic import BaseModel
-from jon_scratch.pupil_calibration_pipeline.qt_gl_laser_skeleton_visualizer import QtGlLaserSkeletonVisualizer
+from jon_scratch.pupil_calibration_pipeline.qt_gl_laser_skeleton_visualizer import (
+ QtGlLaserSkeletonVisualizer,
+)
from src.cameras.launch_camera_frame_loop import launch_camera_frame_loop
from src.cameras.multicam_manager.cv_camera_manager import OpenCVCameraManager
-from src.config.home_dir import create_session_id, get_session_folder_path, get_most_recent_session_id, \
- create_session_folder
-from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import MediaPipeSkeletonDetector
-from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import CalibrationPipelineOrchestrator
-from src.pipelines.session_pipeline.session_pipeline_orchestrator import SessionPipelineOrchestrator, \
- load_mediapipe3d_skeleton_data, load_mediapipe2d_data
+from src.config.home_dir import (
+ create_session_id,
+ get_session_folder_path,
+ get_most_recent_session_id,
+ create_session_folder,
+)
+from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import (
+ MediaPipeSkeletonDetector,
+)
+from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import (
+ CalibrationPipelineOrchestrator,
+)
+from src.pipelines.session_pipeline.session_pipeline_orchestrator import (
+ SessionPipelineOrchestrator,
+ load_mediapipe3d_skeleton_data,
+ load_mediapipe2d_data,
+)
logger = logging.getLogger(__name__)
@@ -40,36 +53,43 @@ class SessionCalibrateModel(TweakedPydanticBaseModel):
session_id: str, ID for the session we're calibrating
charuco_square_size:Union[int, float], the size of one of the black squares on the charuco board (preferably) in milimeters
"""
+
session_id: str = None
webcam_configs_dict: dict = None
opencv_camera_manager: OpenCVCameraManager = None
charuco_square_size: Union[int, float] = 39
+
class SessionRecordModel(TweakedPydanticBaseModel):
session_id: str = None
webcam_configs_dict: dict = None
class SessionIdModel(BaseModel):
- """
- """
+ """ """
+
session_id: str = None
@session_router.post("/session/create")
-async def create_session(session_create_model: SessionCreateModel = SessionCreateModel()) -> SessionResponse:
+async def create_session(
+ session_create_model: SessionCreateModel = SessionCreateModel(),
+) -> SessionResponse:
session_id = create_session_id()
if session_create_model.user_session_tag_str is not None:
session_id = session_id + session_create_model.user_session_tag_str
create_session_folder(session_id)
- return SessionResponse(session_id=session_id,
- session_path=get_session_folder_path(session_id))
+ return SessionResponse(
+ session_id=session_id, session_path=get_session_folder_path(session_id)
+ )
@session_router.post("/session/calibrate")
-def calibrate_session(session_calibrate_model: SessionCalibrateModel = SessionCalibrateModel()):
+def calibrate_session(
+ session_calibrate_model: SessionCalibrateModel = SessionCalibrateModel(),
+):
"""calibate capture volume - record synchronized videos (from all available camras wtih default parameters for now) and process with Anipose to produce a camera calibration (saved as a `.toml` file in the session folder"""
session_id = session_calibrate_model.session_id
@@ -83,17 +103,19 @@ def calibrate_session(session_calibrate_model: SessionCalibrateModel = SessionCa
# show_camera_views_in_windows=True,
# )
- launch_camera_frame_loop(session_id=session_id,
- webcam_configs_dict=session_calibrate_model.webcam_configs_dict,
- show_camera_views_in_windows=True,
- calibration_videos_bool=True,
- detect_charuco_in_image=True,
- )
+ launch_camera_frame_loop(
+ session_id=session_id,
+ webcam_configs_dict=session_calibrate_model.webcam_configs_dict,
+ show_camera_views_in_windows=True,
+ calibration_videos_bool=True,
+ detect_charuco_in_image=True,
+ )
try:
calibration_orchestrator.run_anipose_camera_calibration(
charuco_square_size=session_calibrate_model.charuco_square_size,
- pin_camera_0_to_origin=True)
+ pin_camera_0_to_origin=True,
+ )
except:
logger.error("Printing Traceback")
traceback.print_exc()
@@ -101,25 +123,30 @@ def calibrate_session(session_calibrate_model: SessionCalibrateModel = SessionCa
return calibration_orchestrator
-
@session_router.post("/session/record")
def record_session(session_record_model: SessionRecordModel = SessionRecordModel()):
- launch_camera_frame_loop(session_id=session_record_model.session_id,
- webcam_configs_dict=session_record_model.webcam_configs_dict,
- show_camera_views_in_windows=True,
- calibration_videos_bool=False,
- detect_charuco_in_image=True,
- )
+ launch_camera_frame_loop(
+ session_id=session_record_model.session_id,
+ webcam_configs_dict=session_record_model.webcam_configs_dict,
+ show_camera_views_in_windows=True,
+ calibration_videos_bool=False,
+ detect_charuco_in_image=True,
+ )
+
@session_router.post("/session/mediapipe_track_skeletons_offline")
-def mediapipe_track_2D_skeletons_offline(session_id_model: SessionIdModel = SessionIdModel()):
+def mediapipe_track_2D_skeletons_offline(
+ session_id_model: SessionIdModel = SessionIdModel(),
+):
if session_id_model.session_id is None or session_id_model.session_id == "string":
this_session_id = get_most_recent_session_id()
logger.info(f"loading most recent session:{this_session_id}")
else:
this_session_id = session_id_model.session_id
- logger.info(f"tracking 2D mediapipe skeletons in videos from session: {this_session_id}")
+ logger.info(
+ f"tracking 2D mediapipe skeletons in videos from session: {this_session_id}"
+ )
mediapipe_skeleton_detector = MediaPipeSkeletonDetector(this_session_id)
mediapipe_skeleton_detector.process_session_folder()
@@ -132,9 +159,12 @@ def mediapipe_reconstruct_3D_skeletons_offline(session_id_model: SessionIdModel
session_id = session_id_model.session_id
this_session_orchestrator = SessionPipelineOrchestrator(session_id=session_id)
- this_session_orchestrator.anipose_camera_calibration_object = CalibrationPipelineOrchestrator().load_calibration_from_session_id(
- session_id)
- this_session_orchestrator.mediapipe2d_numCams_numFrames_numTrackedPoints_XY = load_mediapipe2d_data(session_id)
+ this_session_orchestrator.anipose_camera_calibration_object = (
+ CalibrationPipelineOrchestrator().load_calibration_from_session_id(session_id)
+ )
+ this_session_orchestrator.mediapipe2d_numCams_numFrames_numTrackedPoints_XY = (
+ load_mediapipe2d_data(session_id)
+ )
this_session_orchestrator.reconstruct3d_from_2d_data_offline()
@@ -147,11 +177,16 @@ def visualize_session_offline(session_id_model: SessionIdModel = None):
mediapipe3d_data_payload = load_mediapipe3d_skeleton_data(session_id)
- mediapipe3d_skeleton_nFrames_nTrajectories_xyz = mediapipe3d_data_payload.data3d_numFrames_numTrackedPoints_XYZ
- mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError = mediapipe3d_data_payload.data3d_numFrames_numTrackedPoint_reprojectionError
+ mediapipe3d_skeleton_nFrames_nTrajectories_xyz = (
+ mediapipe3d_data_payload.data3d_numFrames_numTrackedPoints_XYZ
+ )
+ mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError = (
+ mediapipe3d_data_payload.data3d_numFrames_numTrackedPoint_reprojectionError
+ )
qt_gl_laser_skeleton = QtGlLaserSkeletonVisualizer(
- mediapipe_skel_fr_mar_xyz=mediapipe3d_skeleton_nFrames_nTrajectories_xyz)
+ mediapipe_skel_fr_mar_xyz=mediapipe3d_skeleton_nFrames_nTrajectories_xyz
+ )
qt_gl_laser_skeleton.start_animation()
diff --git a/src/api/services/board_detect_service.py b/src/api/services/board_detect_service.py
index 5bfdcf43..4077396d 100644
--- a/src/api/services/board_detect_service.py
+++ b/src/api/services/board_detect_service.py
@@ -7,4 +7,4 @@ class ImageResponse(NamedTuple):
class BoardDetectService:
- pass
\ No newline at end of file
+ pass
diff --git a/src/api/services/mediapipe_detect_service.py b/src/api/services/mediapipe_detect_service.py
index 085e3f2b..d000c853 100644
--- a/src/api/services/mediapipe_detect_service.py
+++ b/src/api/services/mediapipe_detect_service.py
@@ -2,4 +2,4 @@ from src.cameras.multicam_manager.cv_camera_manager import OpenCVCameraManager
class MediapipeSkeletonDetectionService:
- pass
\ No newline at end of file
+ pass
diff --git a/src/api/services/user_config.py b/src/api/services/user_config.py
index f0547da4..f189911a 100644
--- a/src/api/services/user_config.py
+++ b/src/api/services/user_config.py
@@ -8,6 +8,7 @@ from src.config.home_dir import get_session_folder_path
logger = logging.getLogger(__name__)
+
class WebcamConfigModel(BaseModel):
webcam_id: str
exposure: int = -6
@@ -22,13 +23,15 @@ def get_camera_name(camera_id: str):
class UserConfigService:
"""take things like WebcamConfigModel and save it to a text file or whatever"""
- def save_webcam_config_to_disk(self, webcam_config_model: WebcamConfigModel, session_id: str):
+ def save_webcam_config_to_disk(
+ self, webcam_config_model: WebcamConfigModel, session_id: str
+ ):
webcam_config_json = webcam_config_model.json()
camera_name = get_camera_name(webcam_config_model.webcam_id)
- json_file_name = camera_name + '_config.json'
+ json_file_name = camera_name + "_config.json"
session_path = get_session_folder_path(session_id)
webcam_config_json_path = Path(session_path) / json_file_name
- with open(webcam_config_json_path, 'w') as outfile:
+ with open(webcam_config_json_path, "w") as outfile:
outfile.write(str(webcam_config_json))
return webcam_config_json
@@ -39,14 +42,15 @@ class UserConfigService:
camera_name = get_camera_name(webcam_id)
print(f"webcam id (again): {webcam_id}")
print(f"camera name: {camera_name}")
- json_file_name = camera_name + '_config.json'
+ json_file_name = camera_name + "_config.json"
webcam_config_json_path = Path(session_path) / json_file_name
try:
webcam_config_model = WebcamConfigModel.parse_file(webcam_config_json_path)
- logger.info(f'loading webcam_config from: {webcam_config_json_path}')
+ logger.info(f"loading webcam_config from: {webcam_config_json_path}")
return webcam_config_model
except FileNotFoundError:
- logger.info(f'No webcam config file found for this camera, using default camera parameters')
+ logger.info(
+ f"No webcam config file found for this camera, using default camera parameters"
+ )
return WebcamConfigModel(webcam_id=webcam_id)
-
diff --git a/src/cameras/capture/dataclasses/frame_payload.py b/src/cameras/capture/dataclasses/frame_payload.py
index ccef9d6a..30b38030 100644
--- a/src/cameras/capture/dataclasses/frame_payload.py
+++ b/src/cameras/capture/dataclasses/frame_payload.py
@@ -6,9 +6,7 @@ import numpy as np
class FramePayload(NamedTuple):
success: bool = False
image: np.ndarray = None
- timestamp_in_seconds_from_record_start: float = None
+ timestamp_in_seconds_from_record_start: float = None
timestamp_unix_time_seconds: float = None
frame_number: int = None
webcam_id: str = None
-
-
diff --git a/src/cameras/capture/opencv_camera/camera_stream_thread_handler.py b/src/cameras/capture/opencv_camera/camera_stream_thread_handler.py
index 217f12ce..6af5ecdd 100644
--- a/src/cameras/capture/opencv_camera/camera_stream_thread_handler.py
+++ b/src/cameras/capture/opencv_camera/camera_stream_thread_handler.py
@@ -13,10 +13,9 @@ logger = logging.getLogger(__name__)
class VideoCaptureThread(threading.Thread):
def __init__(
- self,
- get_next_frame,
- webcam_id:str = 'unknown',
-
+ self,
+ get_next_frame,
+ webcam_id: str = "unknown",
):
super().__init__()
self._webcam_id = webcam_id
@@ -34,9 +33,13 @@ class VideoCaptureThread(threading.Thread):
@property
def median_framerate(self):
if self._num_frames_processed == 0:
- logger.warning(f"No Frames processed yet, cannot calculate median_framerate")
+ logger.warning(
+ f"No Frames processed yet, cannot calculate median_framerate"
+ )
else:
- self._median_framerate = np.nanmedian((np.diff(self._timestamps_npy) ** -1) / 1e9)
+ self._median_framerate = np.nanmedian(
+ (np.diff(self._timestamps_npy) ** -1) / 1e9
+ )
return self._median_framerate
@@ -68,7 +71,9 @@ class VideoCaptureThread(threading.Thread):
start = time.perf_counter_ns()
self._frame = self._get_next_frame()
self._elapsed_during_frame_grab.append(time.perf_counter_ns() - start)
- self._timestamps_npy.append(self._frame.timestamp_in_seconds_from_record_start)
+ self._timestamps_npy.append(
+ self._frame.timestamp_in_seconds_from_record_start
+ )
self._num_frames_processed += 1
# logger.debug(f"Camera {self._webcam_id}: captured frame# {self._num_frames_processed}")
except:
diff --git a/src/cameras/capture/opencv_camera/opencv_camera.py b/src/cameras/capture/opencv_camera/opencv_camera.py
index a60ea372..d56d7e81 100644
--- a/src/cameras/capture/opencv_camera/opencv_camera.py
+++ b/src/cameras/capture/opencv_camera/opencv_camera.py
@@ -11,7 +11,9 @@ from src.cameras.capture.dataclasses.frame_payload import FramePayload
from src.cameras.persistence.video_writer.video_recorder import VideoRecorder
from src.cameras.viewer.cv_cam_viewer import CvCamViewer
from src.config.webcam_config import WebcamConfig
-from src.cameras.capture.opencv_camera.camera_stream_thread_handler import VideoCaptureThread
+from src.cameras.capture.opencv_camera.camera_stream_thread_handler import (
+ VideoCaptureThread,
+)
from src.core_processor.timestamp_manager.timestamp_manager import TimestampLogger
logger = logging.getLogger(__name__)
@@ -22,11 +24,13 @@ class OpenCVCamera:
Performant implementation of video capture against webcams
"""
- def __init__(self,
- config: WebcamConfig,
- session_id: str=None,
- session_start_time_perf_counter_ns:int=0,
- calibration_video_bool: bool = False):
+ def __init__(
+ self,
+ config: WebcamConfig,
+ session_id: str = None,
+ session_start_time_perf_counter_ns: int = 0,
+ calibration_video_bool: bool = False,
+ ):
self._config = config
self._name = f"Camera_{self._config.webcam_id}"
self._opencv_video_capture_object: cv2.VideoCapture = None
@@ -117,22 +121,24 @@ class OpenCVCamera:
)
raise Exception
-
logger.debug(f"Camera found at port number {self._config.webcam_id}")
if self.session_id is False:
logger.info(
- f"No `session_id` specified for {self._name}, video_recorder will not be created (because we won't know where to save the videos)")
+ f"No `session_id` specified for {self._name}, video_recorder will not be created (because we won't know where to save the videos)"
+ )
self._video_recorder = None
else:
image_height = image.shape[0]
image_width = image.shape[1]
- self._video_recorder = VideoRecorder(self._name,
- image_width=image_width,
- image_height=image_height,
- session_id=self.session_id,
- calibration_video_bool=self._calibration_video_bool)
+ self._video_recorder = VideoRecorder(
+ self._name,
+ image_width=image_width,
+ image_height=image_height,
+ session_id=self.session_id,
+ calibration_video_bool=self._calibration_video_bool,
+ )
return success
@@ -182,7 +188,9 @@ class OpenCVCamera:
### A - see -> https://stackoverflow.com/questions/57716962/difference-between-video-capture-read-and-grab
self._opencv_video_capture_object.grab()
success, image = self._opencv_video_capture_object.retrieve()
- this_frame_timestamp_perf_counter_ns = time.perf_counter_ns() - self._session_start_time_perf_counter_ns
+ this_frame_timestamp_perf_counter_ns = (
+ time.perf_counter_ns() - self._session_start_time_perf_counter_ns
+ )
# timestamp_ns_post = time.perf_counter_ns()
# it_took_this_many_seconds_to_grab_the_frame = (timestamp_ns_post-timestamp_ns_pre)/1e9
@@ -190,7 +198,6 @@ class OpenCVCamera:
logger.error(f"Failed to read frame from Camera: {self.webcam_id_as_str}")
raise Exception
-
self._new_frame_ready = success
if success:
@@ -198,12 +205,15 @@ class OpenCVCamera:
# if self._camera_view_update_function is not None:
# self._camera_view_update_function(self.webcam_id_as_str, cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
- return FramePayload(success=success,
- image=image,
- timestamp_in_seconds_from_record_start=this_frame_timestamp_perf_counter_ns/1e9,
- timestamp_unix_time_seconds = time.time(),
- frame_number=self.latest_frame_number,
- webcam_id=self.webcam_id_as_str)
+ return FramePayload(
+ success=success,
+ image=image,
+ timestamp_in_seconds_from_record_start=this_frame_timestamp_perf_counter_ns
+ / 1e9,
+ timestamp_unix_time_seconds=time.time(),
+ frame_number=self.latest_frame_number,
+ webcam_id=self.webcam_id_as_str,
+ )
def stop_frame_capture(self):
self.close()
diff --git a/src/cameras/detection/cam_detection.py b/src/cameras/detection/cam_detection.py
index 2d7d81d0..5c62403f 100644
--- a/src/cameras/detection/cam_detection.py
+++ b/src/cameras/detection/cam_detection.py
@@ -21,11 +21,7 @@ class DetectPossibleCameras:
if success and image is not None:
try:
- cams_to_use_list.append(
- RawCamera(
- webcam_id=str(cam_id)
- )
- )
+ cams_to_use_list.append(RawCamera(webcam_id=str(cam_id)))
finally:
cap.release()
diff --git a/src/cameras/launch_camera_frame_loop.py b/src/cameras/launch_camera_frame_loop.py
index 9d4cd4c0..95ab1fb6 100644
--- a/src/cameras/launch_camera_frame_loop.py
+++ b/src/cameras/launch_camera_frame_loop.py
@@ -12,88 +12,113 @@ import asyncio
from src.core_processor.show_cam_window import show_cam_window
from src.core_processor.utils.image_fps_writer import write_fps_to_image
-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import CharucoBoardDetector
+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (
+ CharucoBoardDetector,
+)
from src.qt_visualizer_and_gui.qt_visualizer_and_gui import QTVisualizerAndGui
logger = logging.getLogger(__name__)
def launch_camera_frame_loop(
- session_id: str,
- webcam_configs_dict: Dict[str, WebcamConfig] = None,
- show_camera_views_in_windows: bool = True,
- calibration_videos_bool: bool = False,
- detect_charuco_in_image: bool = True,
- record_frames_bool: bool = True,
- show_visualizer_gui:bool = False,
+ session_id: str,
+ webcam_configs_dict: Dict[str, WebcamConfig] = None,
+ show_camera_views_in_windows: bool = True,
+ calibration_videos_bool: bool = False,
+ detect_charuco_in_image: bool = True,
+ record_frames_bool: bool = True,
+ show_visualizer_gui: bool = False,
):
any_frames_recorded = False
if detect_charuco_in_image:
charuco_board_detector = CharucoBoardDetector()
-
-
opencv_camera_manager = OpenCVCameraManager(session_id=session_id)
with opencv_camera_manager.start_capture_session_all_cams(
- webcam_configs_dict=webcam_configs_dict,
- calibration_videos=calibration_videos_bool,
+ webcam_configs_dict=webcam_configs_dict,
+ calibration_videos=calibration_videos_bool,
) as connected_cameras_dict:
-
- timestamp_manager = opencv_camera_manager.timestamp_manager# <-this is defines the start-time of the timestamp loggers for each camera
+ timestamp_manager = (
+ opencv_camera_manager.timestamp_manager
+ ) # <-this is defines the start-time of the timestamp loggers for each camera
try:
if show_visualizer_gui:
visualizer_gui = QTVisualizerAndGui()
- visualizer_gui.setup_and_launch(opencv_camera_manager.available_webcam_ids)
+ visualizer_gui.setup_and_launch(
+ opencv_camera_manager.available_webcam_ids
+ )
should_continue = True
while should_continue:
-
if not opencv_camera_manager.new_multi_frame_ready():
continue
this_multi_frame_payload = opencv_camera_manager.latest_multi_frame
- timestamp_manager.multi_frame_timestamp_logger.log_new_timestamp_seconds_from_unspecified_zero(this_multi_frame_payload.multi_frame_timestamp_seconds)
+ timestamp_manager.multi_frame_timestamp_logger.log_new_timestamp_seconds_from_unspecified_zero(
+ this_multi_frame_payload.multi_frame_timestamp_seconds
+ )
- for this_webcam_id, this_open_cv_camera in connected_cameras_dict.items():
+ for (
+ this_webcam_id,
+ this_open_cv_camera,
+ ) in connected_cameras_dict.items():
- this_cam_latest_frame = this_multi_frame_payload.frames_dict[this_webcam_id]
+ this_cam_latest_frame = this_multi_frame_payload.frames_dict[
+ this_webcam_id
+ ]
if this_cam_latest_frame is None:
continue
- this_cam_timestamp_logger = timestamp_manager.timestamp_logger_for_webcam_id(this_webcam_id)
- this_cam_timestamp_logger.log_new_timestamp_seconds_from_unspecified_zero(this_cam_latest_frame.timestamp_in_seconds_from_record_start)
+ this_cam_timestamp_logger = (
+ timestamp_manager.timestamp_logger_for_webcam_id(this_webcam_id)
+ )
+ this_cam_timestamp_logger.log_new_timestamp_seconds_from_unspecified_zero(
+ this_cam_latest_frame.timestamp_in_seconds_from_record_start
+ )
# save frame to video file
if record_frames_bool:
any_frames_recorded = True
- this_open_cv_camera.video_recorder.append_frame_payload_to_list(this_cam_latest_frame)
+ this_open_cv_camera.video_recorder.append_frame_payload_to_list(
+ this_cam_latest_frame
+ )
image_to_display = this_cam_latest_frame.image.copy()
if detect_charuco_in_image:
# detect charuco board
- this_charuco_frame_payload = charuco_board_detector.detect_charuco_board(
- this_cam_latest_frame)
- image_to_display = this_charuco_frame_payload.annotated_image.copy()
+ this_charuco_frame_payload = (
+ charuco_board_detector.detect_charuco_board(
+ this_cam_latest_frame
+ )
+ )
+ image_to_display = (
+ this_charuco_frame_payload.annotated_image.copy()
+ )
image_to_display = write_fps_to_image(
image_to_display,
- timestamp_manager.timestamp_logger_for_webcam_id(this_webcam_id).median_frames_per_second,
+ timestamp_manager.timestamp_logger_for_webcam_id(
+ this_webcam_id
+ ).median_frames_per_second,
)
if show_camera_views_in_windows:
- should_continue = show_cam_window(this_webcam_id, image_to_display )
+ should_continue = show_cam_window(
+ this_webcam_id, image_to_display
+ )
if show_visualizer_gui:
- visualizer_gui.update_camera_view_image(this_webcam_id,
- image_to_display)
+ visualizer_gui.update_camera_view_image(
+ this_webcam_id, image_to_display
+ )
# exit loop when user presses ESC key
exit_key = cv2.waitKey(1)
@@ -113,14 +138,15 @@ def launch_camera_frame_loop(
# cv2.destroyWindow(this_open_cv_camera.webcam_id_as_str)
cv2.destroyAllWindows()
-
for this_open_cv_camera in connected_cameras_dict.values():
if any_frames_recorded:
- logger.info(f"Saving {this_open_cv_camera.webcam_id_as_str} video to disk")
+ logger.info(
+ f"Saving {this_open_cv_camera.webcam_id_as_str} video to disk"
+ )
this_open_cv_camera.video_recorder.save_list_of_frames_to_video_file(
- this_open_cv_camera.video_recorder.frame_list)
+ this_open_cv_camera.video_recorder.frame_list
+ )
this_open_cv_camera.video_recorder.close()
logger.info("Creating camera timestamp diagnostic plot")
timestamp_manager.create_diagnostic_plots()
-
diff --git a/src/cameras/multicam_manager/cv_camera_manager.py b/src/cameras/multicam_manager/cv_camera_manager.py
index 9c43ac6f..5afbcc14 100644
--- a/src/cameras/multicam_manager/cv_camera_manager.py
+++ b/src/cameras/multicam_manager/cv_camera_manager.py
@@ -12,8 +12,13 @@ from src.cameras.capture.opencv_camera.opencv_camera import OpenCVCamera
from src.cameras.detection.cam_singleton import get_or_create_cams
from src.cameras.persistence.video_writer.video_recorder import VideoRecorder
from src.config.webcam_config import WebcamConfig
-from src.core_processor.timestamp_manager.timestamp_manager import TimestampManager, TimestampLogger
-from src.pipelines.session_pipeline.data_classes.multi_frame_payload import MultiFramePayload
+from src.core_processor.timestamp_manager.timestamp_manager import (
+ TimestampManager,
+ TimestampLogger,
+)
+from src.pipelines.session_pipeline.data_classes.multi_frame_payload import (
+ MultiFramePayload,
+)
logger = logging.getLogger(__name__)
@@ -27,10 +32,11 @@ class CamAndWriterResponse(BaseModel):
class OpenCVCameraManager:
- def __init__(self,
- session_id: str = None,
- shut_down_event_bool:bool = None,
- ):
+ def __init__(
+ self,
+ session_id: str = None,
+ shut_down_event_bool: bool = None,
+ ):
self._session_id = session_id
self._config_service = UserConfigService()
self._detected_cams_data = get_or_create_cams()
@@ -41,11 +47,10 @@ class OpenCVCameraManager:
self._shut_down_event_bool = shut_down_event_bool
self._number_of_multi_frames = 0
-
@property
def timestamp_manager(self):
if self._timestamp_manager is None:
- logger.error('timestamp manager has not been created yet')
+ logger.error("timestamp manager has not been created yet")
raise Exception
else:
return self._timestamp_manager
@@ -60,25 +65,30 @@ class OpenCVCameraManager:
logging.error("a multi_frame was requested before it was ready!")
raise Exception
- this_multi_frame_timestamp_sec = (time.perf_counter_ns() - self._session_start_time_perf_counter_ns)/1e9
+ this_multi_frame_timestamp_sec = (
+ time.perf_counter_ns() - self._session_start_time_perf_counter_ns
+ ) / 1e9
this_multi_frame_dict = {}
each_cam_timestamp = []
for this_cam in self._connected_cameras_dict.values():
if not this_cam.new_frame_ready:
- logger.error('It shouldnt be able to get here if the `new_multi_frame`')
+ logger.error("It shouldnt be able to get here if the `new_multi_frame`")
this_multi_frame_dict[this_cam.webcam_id_as_str] = this_cam.latest_frame
- each_cam_timestamp.append(this_cam.latest_frame.timestamp_in_seconds_from_record_start)
+ each_cam_timestamp.append(
+ this_cam.latest_frame.timestamp_in_seconds_from_record_start
+ )
- self._number_of_multi_frames +=1
+ self._number_of_multi_frames += 1
- return MultiFramePayload(frames_dict=this_multi_frame_dict,
- multi_frame_number=self._number_of_multi_frames,
- each_frame_timestamp=each_cam_timestamp,
- multi_frame_timestamp_seconds=this_multi_frame_timestamp_sec
- )
+ return MultiFramePayload(
+ frames_dict=this_multi_frame_dict,
+ multi_frame_number=self._number_of_multi_frames,
+ each_frame_timestamp=each_cam_timestamp,
+ multi_frame_timestamp_seconds=this_multi_frame_timestamp_sec,
+ )
def new_multi_frame_ready(self):
"""cycle through connected cameras and return false if one isn't read yet
@@ -92,13 +102,15 @@ class OpenCVCameraManager:
for this_raw_camera in self._detected_cams_data.cameras_found_list:
this_webcam_config = WebcamConfig()
this_webcam_config.webcam_id = this_raw_camera.webcam_id
- self._available_cameras_dict[this_webcam_config.webcam_id] = this_webcam_config
+ self._available_cameras_dict[
+ this_webcam_config.webcam_id
+ ] = this_webcam_config
return self._available_cameras_dict
@contextmanager
def start_capture_session_single_cam(
- self, webcam_id: str
+ self, webcam_id: str
) -> ContextManager[CamAndWriterResponse]:
"""
Context manager for easy start up, usage, and cleanup of camera resources.
@@ -106,7 +118,7 @@ class OpenCVCameraManager:
"""
cv_camera = self._create_single_opencv_cam(webcam_id)
- self._connected_cameras_dict['0'] = cv_camera
+ self._connected_cameras_dict["0"] = cv_camera
try:
self._start_frame_capture_on_cam_id(cv_camera)
self._initialize_timestamp_logger() # start timestamp logger and wha
@@ -118,17 +130,18 @@ class OpenCVCameraManager:
@contextmanager
def start_capture_session_all_cams(
- self,
- webcam_configs_dict: Dict[str, WebcamConfig] = None,
- camera_view_update_function=None,
- calibration_videos: bool = False,
+ self,
+ webcam_configs_dict: Dict[str, WebcamConfig] = None,
+ camera_view_update_function=None,
+ calibration_videos: bool = False,
) -> ContextManager[Dict[str, OpenCVCamera]]:
self._initialize_timestamp_logger()
open_cv_camera_objects = self._create_opencv_cameras(
session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns,
webcam_configs_dict=webcam_configs_dict,
- calibration_videos=calibration_videos, )
+ calibration_videos=calibration_videos,
+ )
try:
for cv_cam in open_cv_camera_objects:
@@ -142,17 +155,16 @@ class OpenCVCameraManager:
logger.error("Printing traceback from starting capture session by cam")
traceback.print_exc()
- def _create_opencv_cameras(self,
- session_start_time_perf_counter_ns: int,
- webcam_configs_dict: Dict[str, WebcamConfig] = None,
- calibration_videos: bool = False,
- ):
+ def _create_opencv_cameras(
+ self,
+ session_start_time_perf_counter_ns: int,
+ webcam_configs_dict: Dict[str, WebcamConfig] = None,
+ calibration_videos: bool = False,
+ ):
raw_camera_objects = self._detected_cams_data.cameras_found_list
open_cv_cameras: List[OpenCVCamera] = []
-
-
for this_raw_cam in raw_camera_objects:
if webcam_configs_dict is None:
@@ -160,19 +172,22 @@ class OpenCVCameraManager:
else:
this_webcam_config = webcam_configs_dict[this_raw_cam.webcam_id]
- opencv_cam_obj = self._create_single_opencv_cam(webcam_id=this_raw_cam.webcam_id,
- webcam_config=this_webcam_config,
- session_start_time_perf_counter_ns = session_start_time_perf_counter_ns,
- calibration_video_bool=calibration_videos)
+ opencv_cam_obj = self._create_single_opencv_cam(
+ webcam_id=this_raw_cam.webcam_id,
+ webcam_config=this_webcam_config,
+ session_start_time_perf_counter_ns=session_start_time_perf_counter_ns,
+ calibration_video_bool=calibration_videos,
+ )
open_cv_cameras.append(opencv_cam_obj)
return open_cv_cameras
- def _create_single_opencv_cam(self,
- webcam_id: str,
- webcam_config: WebcamConfig(),
- session_start_time_perf_counter_ns:int,
- calibration_video_bool: bool = False,
- ):
+ def _create_single_opencv_cam(
+ self,
+ webcam_id: str,
+ webcam_config: WebcamConfig(),
+ session_start_time_perf_counter_ns: int,
+ calibration_video_bool: bool = False,
+ ):
#
# JSM NOTE - This method seems to want to load webcam_config from disk? It might be useful, but I'll bypass it for now
#
@@ -182,23 +197,28 @@ class OpenCVCameraManager:
# resolution_width=webcam_config_model.resolution_width,
# resolution_height=webcam_config_model.resolution_height, )
webcam_config.webcam_id = webcam_id
- return OpenCVCamera(config=webcam_config,
- session_id=self._session_id,
- session_start_time_perf_counter_ns = session_start_time_perf_counter_ns,
- calibration_video_bool=calibration_video_bool)
+ return OpenCVCamera(
+ config=webcam_config,
+ session_id=self._session_id,
+ session_start_time_perf_counter_ns=session_start_time_perf_counter_ns,
+ calibration_video_bool=calibration_video_bool,
+ )
def _initialize_timestamp_logger(self):
self._session_start_time_unix_ns = time.time_ns()
self._session_start_time_perf_counter_ns = time.perf_counter_ns()
- logger.info(f"Initializing timestamp loggers - _session_start_time_unix_ns: {self._session_start_time_unix_ns}, _session_start_time_perf_counter_ns: {self._session_start_time_perf_counter_ns} ")
-
- self._timestamp_manager = TimestampManager(self._session_id,
- self.available_webcam_ids,
- self._session_start_time_unix_ns,
- self._session_start_time_perf_counter_ns)
+ logger.info(
+ f"Initializing timestamp loggers - _session_start_time_unix_ns: {self._session_start_time_unix_ns}, _session_start_time_perf_counter_ns: {self._session_start_time_perf_counter_ns} "
+ )
+ self._timestamp_manager = TimestampManager(
+ self._session_id,
+ self.available_webcam_ids,
+ self._session_start_time_unix_ns,
+ self._session_start_time_perf_counter_ns,
+ )
def _start_frame_capture_on_cam_id(self, opencv_cam: OpenCVCamera):
opencv_cam.connect()
diff --git a/src/cameras/persistence/video_writer/save_options_dataclass.py b/src/cameras/persistence/video_writer/save_options_dataclass.py
index 80d18a65..d4d1c7b4 100644
--- a/src/cameras/persistence/video_writer/save_options_dataclass.py
+++ b/src/cameras/persistence/video_writer/save_options_dataclass.py
@@ -3,7 +3,7 @@ from pathlib import Path
@dataclass
-class SaveOptions():
+class SaveOptions:
path_to_save_video: Path
camera_name: str
frame_width: int
diff --git a/src/cameras/persistence/video_writer/video_recorder.py b/src/cameras/persistence/video_writer/video_recorder.py
index 40323e81..17c6e84a 100644
--- a/src/cameras/persistence/video_writer/video_recorder.py
+++ b/src/cameras/persistence/video_writer/video_recorder.py
@@ -11,23 +11,28 @@ import pandas as pd
from src.cameras.capture.dataclasses.frame_payload import FramePayload
from src.cameras.persistence.video_writer.save_options_dataclass import SaveOptions
from src.config.data_paths import freemocap_data_path
-from src.config.home_dir import get_session_folder_path, get_synchronized_videos_folder_path, get_calibration_videos_folder_path, \
- get_mediapipe_annotated_videos_folder_path
+from src.config.home_dir import (
+ get_session_folder_path,
+ get_synchronized_videos_folder_path,
+ get_calibration_videos_folder_path,
+ get_mediapipe_annotated_videos_folder_path,
+)
from rich.progress import Progress
logger = logging.getLogger(__name__)
class VideoRecorder:
- def __init__(self,
- video_name: str,
- image_width: int,
- image_height: int,
- session_id: str,
- fourcc: str = "MP4V",
- calibration_video_bool: bool = False,
- mediapipe_annotated_video_bool: bool = False,
- ):
+ def __init__(
+ self,
+ video_name: str,
+ image_width: int,
+ image_height: int,
+ session_id: str,
+ fourcc: str = "MP4V",
+ calibration_video_bool: bool = False,
+ mediapipe_annotated_video_bool: bool = False,
+ ):
self._video_name = video_name
self._image_width = image_width
self._image_height = image_height
@@ -73,10 +78,12 @@ class VideoRecorder:
def append_frame_payload_to_list(self, frame_payload: FramePayload):
self._frame_payload_list.append(frame_payload)
- def save_list_of_frames_to_video_file(self,
- list_of_frames:List[FramePayload]=None,
- frames_per_second:float=None,
- calibration_videos:bool=False):
+ def save_list_of_frames_to_video_file(
+ self,
+ list_of_frames: List[FramePayload] = None,
+ frames_per_second: float = None,
+ calibration_videos: bool = False,
+ ):
if list_of_frames is None:
list_of_frames = self._frame_payload_list
@@ -88,13 +95,15 @@ class VideoRecorder:
self._timestamps_npy = self._gather_timestamps(list_of_frames)
frames_per_second = np.nanmedian((np.diff(self._timestamps_npy) ** -1))
- self._initialize_video_writer(frames_per_second=frames_per_second, calibration_videos=calibration_videos)
+ self._initialize_video_writer(
+ frames_per_second=frames_per_second, calibration_videos=calibration_videos
+ )
self._write_frame_list_to_video_file(list_of_frames)
self._save_timestamps(self._timestamps_npy)
- def save_image_list_to_disk(self,
- image_list: List[np.ndarray],
- frames_per_second:float):
+ def save_image_list_to_disk(
+ self, image_list: List[np.ndarray], frames_per_second: float
+ ):
if len(image_list) == 0:
logging.error(f"No frames to save for : {self._video_name}")
return
@@ -102,27 +111,37 @@ class VideoRecorder:
self._initialize_video_writer(frames_per_second=frames_per_second)
self._write_image_list_to_video_file(image_list)
- def _initialize_video_writer(self, frames_per_second: Union[int, float] = None, calibration_videos:bool=False):
+ def _initialize_video_writer(
+ self,
+ frames_per_second: Union[int, float] = None,
+ calibration_videos: bool = False,
+ ):
- video_file_name = self._video_name + '.mp4'
+ video_file_name = self._video_name + ".mp4"
if calibration_videos:
- self._video_folder_path = Path(get_calibration_videos_folder_path(self._session_id))
+ self._video_folder_path = Path(
+ get_calibration_videos_folder_path(self._session_id)
+ )
elif self._mediapipe_annotated_video_bool:
- self._video_folder_path = Path(get_mediapipe_annotated_videos_folder_path(self._session_id))
+ self._video_folder_path = Path(
+ get_mediapipe_annotated_videos_folder_path(self._session_id)
+ )
else:
- self._video_folder_path = Path(get_synchronized_videos_folder_path(self._session_id))
+ self._video_folder_path = Path(
+ get_synchronized_videos_folder_path(self._session_id)
+ )
self._path_to_save_video_file = self._video_folder_path / video_file_name
-
self._cv2_video_writer = cv2.VideoWriter(
str(self.path_to_save_video_file),
cv2.VideoWriter_fourcc(*self._fourcc),
frames_per_second,
- (int(self._image_width), int(self._image_height)))
+ (int(self._image_width), int(self._image_height)),
+ )
- def _write_frame_list_to_video_file(self, list_of_frames:List[FramePayload]):
+ def _write_frame_list_to_video_file(self, list_of_frames: List[FramePayload]):
try:
for frame in list_of_frames:
self._cv2_video_writer.write(frame.image)
@@ -135,30 +154,34 @@ class VideoRecorder:
logger.info(f"Saved video to path: {self.path_to_save_video_file}")
self._cv2_video_writer.release()
- def _write_image_list_to_video_file(self, image_list:List[np.ndarray]):
+ def _write_image_list_to_video_file(self, image_list: List[np.ndarray]):
try:
for image in image_list:
self._cv2_video_writer.write(image)
except Exception as e:
- logger.error(f"Failed during save in video writer: {self.path_to_save_video_file}")
+ logger.error(
+ f"Failed during save in video writer: {self.path_to_save_video_file}"
+ )
traceback.print_exc()
raise e
finally:
logger.info(f"Saved video to path: {self.path_to_save_video_file}")
self._cv2_video_writer.release()
- def _gather_timestamps(self, list_of_frames:List[FramePayload])->np.ndarray:
+ def _gather_timestamps(self, list_of_frames: List[FramePayload]) -> np.ndarray:
timestamps_npy = np.empty(0)
try:
for frame in list_of_frames:
- timestamps_npy = np.append(timestamps_npy, frame.timestamp_in_seconds_from_record_start)
+ timestamps_npy = np.append(
+ timestamps_npy, frame.timestamp_in_seconds_from_record_start
+ )
except:
logger.error("Error gathering timestamps")
return timestamps_npy
- def _save_timestamps(self, timestamps_npy:np.ndarray):
+ def _save_timestamps(self, timestamps_npy: np.ndarray):
timestamp_file_name_npy = self._video_name + "_timestamps_binary.npy"
timestamp_npy_full_save_path = self._video_folder_path / timestamp_file_name_npy
np.save(str(timestamp_npy_full_save_path), timestamps_npy)
diff --git a/src/cameras/save_synchronized_videos.py b/src/cameras/save_synchronized_videos.py
index 8231699d..98af4a3d 100644
--- a/src/cameras/save_synchronized_videos.py
+++ b/src/cameras/save_synchronized_videos.py
@@ -8,7 +8,11 @@ from src.cameras.persistence.video_writer.video_recorder import VideoRecorder
from src.gui.main.state.app_state import APP_STATE
-def save_synchronized_videos(list_of_video_recorders:List[VideoRecorder], calibration_videos:bool=False, mocap_videos:bool=False):
+def save_synchronized_videos(
+ list_of_video_recorders: List[VideoRecorder],
+ calibration_videos: bool = False,
+ mocap_videos: bool = False,
+):
each_cam_frame_list = []
first_frame_timestamps = []
final_frame_timestamps = []
@@ -46,13 +50,19 @@ def save_synchronized_videos(list_of_video_recorders:List[VideoRecorder], calibr
continue
each_cam_clipped_frame_list[-1].append(frame)
- each_cam_clipped_timestamp_list[-1].append(frame.timestamp_unix_time_seconds)
+ each_cam_clipped_timestamp_list[-1].append(
+ frame.timestamp_unix_time_seconds
+ )
number_of_frames_per_camera_clipped = [len(f) for f in each_cam_clipped_frame_list]
min_number_of_frames = np.min(number_of_frames_per_camera_clipped)
- index_of_the_camera_with_fewest_frames = np.argmin(number_of_frames_per_camera_clipped)
+ index_of_the_camera_with_fewest_frames = np.argmin(
+ number_of_frames_per_camera_clipped
+ )
- reference_frame_list = each_cam_clipped_frame_list[index_of_the_camera_with_fewest_frames]
+ reference_frame_list = each_cam_clipped_frame_list[
+ index_of_the_camera_with_fewest_frames
+ ]
number_of_frames_per_camera = len(reference_frame_list)
each_cam_synchronized_frame_list = []
@@ -63,31 +73,41 @@ def save_synchronized_videos(list_of_video_recorders:List[VideoRecorder], calibr
this_cam_synchronized_frame_list.append(closest_frame)
each_cam_synchronized_frame_list.append(this_cam_synchronized_frame_list)
- print(f" (clipped) number_of_frames_per_camera: {number_of_frames_per_camera_clipped}, min:{min_number_of_frames}")
+ print(
+ f" (clipped) number_of_frames_per_camera: {number_of_frames_per_camera_clipped}, min:{min_number_of_frames}"
+ )
if mocap_videos:
APP_STATE.number_of_frames_in_the_mocap_videos = number_of_frames_per_camera
- final_frame_timestamps = [frame_list[-1].timestamp_unix_time_seconds for frame_list in each_cam_synchronized_frame_list]
+ final_frame_timestamps = [
+ frame_list[-1].timestamp_unix_time_seconds
+ for frame_list in each_cam_synchronized_frame_list
+ ]
print(f"np.diff(final_frame_timestamps): {np.diff(final_frame_timestamps)}")
- for video_recoder, frame_list in zip(list_of_video_recorders, each_cam_synchronized_frame_list):
- video_recoder.save_list_of_frames_to_video_file(list_of_frames=frame_list, calibration_videos=calibration_videos)
+ for video_recoder, frame_list in zip(
+ list_of_video_recorders, each_cam_synchronized_frame_list
+ ):
+ video_recoder.save_list_of_frames_to_video_file(
+ list_of_frames=frame_list, calibration_videos=calibration_videos
+ )
-
-def get_nearest_frame(frame_list, reference_frame)->FramePayload:
+def get_nearest_frame(frame_list, reference_frame) -> FramePayload:
timestamps = gather_timestamps(frame_list)
- close_frame_index = np.argmin(np.abs(timestamps-reference_frame.timestamp_unix_time_seconds))
+ close_frame_index = np.argmin(
+ np.abs(timestamps - reference_frame.timestamp_unix_time_seconds)
+ )
return frame_list[close_frame_index]
-def gather_timestamps( frame_list: List[FramePayload]) -> np.ndarray:
+def gather_timestamps(frame_list: List[FramePayload]) -> np.ndarray:
timestamps_npy = np.empty(0)
for frame in frame_list:
timestamps_npy = np.append(timestamps_npy, frame.timestamp_unix_time_seconds)
- return timestamps_npy
\ No newline at end of file
+ return timestamps_npy
diff --git a/src/cameras/viewer/cv_cam_viewer.py b/src/cameras/viewer/cv_cam_viewer.py
index de48b016..c6c12647 100644
--- a/src/cameras/viewer/cv_cam_viewer.py
+++ b/src/cameras/viewer/cv_cam_viewer.py
@@ -22,10 +22,7 @@ class CvCamViewer:
cv2.waitKey(1)
def begin_viewer(self, webcam_id: Union[str, int]):
- self._process = Process(
- target=CvCamViewer._begin,
- args=(self._recv, webcam_id)
- )
+ self._process = Process(target=CvCamViewer._begin, args=(self._recv, webcam_id))
self._process.start()
diff --git a/src/config/home_dir.py b/src/config/home_dir.py
index a0197ed3..c76c54c4 100644
--- a/src/config/home_dir.py
+++ b/src/config/home_dir.py
@@ -11,16 +11,17 @@ MOST_RECENT_SESSION_ID_FILENAME = "most_recent_session_id.toml"
def create_session_id(string_tag: str = None):
- session_id = 'session_' + time.strftime("%m-%d-%Y-%H_%M_%S")
+ session_id = "session_" + time.strftime("%m-%d-%Y-%H_%M_%S")
if string_tag is not None:
- session_id = session_id + '_' + string_tag
+ session_id = session_id + "_" + string_tag
+
+ return session_id
- return session_id
def save_most_recent_session_id_to_a_toml_in_the_freemocap_data_folder(session_id: str):
session_id_dict = {}
- session_id_dict['session_id'] = session_id
+ session_id_dict["session_id"] = session_id
output_file_name = MOST_RECENT_SESSION_ID_FILENAME
output_file_path = Path(get_freemocap_data_folder_path()) / output_file_name
@@ -28,10 +29,12 @@ def save_most_recent_session_id_to_a_toml_in_the_freemocap_data_folder(session_i
toml.dump(session_id_dict, toml_file)
-def get_most_recent_session_id()->str:
- session_id_toml_path = Path(get_freemocap_data_folder_path()) / MOST_RECENT_SESSION_ID_FILENAME
+def get_most_recent_session_id() -> str:
+ session_id_toml_path = (
+ Path(get_freemocap_data_folder_path()) / MOST_RECENT_SESSION_ID_FILENAME
+ )
session_id_dict = toml.load(str(session_id_toml_path))
- return session_id_dict['session_id']
+ return session_id_dict["session_id"]
def os_independent_home_dir():
@@ -52,42 +55,55 @@ def get_session_folder_path(session_id: str, create_folder: bool = False):
return str(session_path)
-def create_session_folder(session_id:str):
+def create_session_folder(session_id: str):
session_path = Path(get_session_folder_path(session_id, create_folder=True))
- logger.info(f'Creating session folder at: {str(session_path)}')
+ logger.info(f"Creating session folder at: {str(session_path)}")
+
def get_synchronized_videos_folder_path(session_id: str, create_folder: bool = True):
- synchronized_videos_path = Path(get_session_folder_path(session_id)) / 'synchronized_videos'
+ synchronized_videos_path = (
+ Path(get_session_folder_path(session_id)) / "synchronized_videos"
+ )
if create_folder:
synchronized_videos_path.mkdir(exist_ok=create_folder, parents=True)
- #this now counts as a "proper" session, so save it to the toml
+ # this now counts as a "proper" session, so save it to the toml
save_most_recent_session_id_to_a_toml_in_the_freemocap_data_folder(session_id)
return str(synchronized_videos_path)
def get_calibration_videos_folder_path(session_id: str, create_folder: bool = True):
- calibration_videos_path = Path(get_session_folder_path(session_id)) / 'calibration_videos'
+ calibration_videos_path = (
+ Path(get_session_folder_path(session_id)) / "calibration_videos"
+ )
if create_folder:
calibration_videos_path.mkdir(exist_ok=create_folder, parents=True)
return str(calibration_videos_path)
-def get_mediapipe_annotated_videos_folder_path(session_id: str, create_folder: bool = True):
- mediapipe_annotated_videos_path = Path(get_session_folder_path(session_id)) / 'mediapipe_annotated_videos'
+def get_mediapipe_annotated_videos_folder_path(
+ session_id: str, create_folder: bool = True
+):
+ mediapipe_annotated_videos_path = (
+ Path(get_session_folder_path(session_id)) / "mediapipe_annotated_videos"
+ )
if create_folder:
mediapipe_annotated_videos_path.mkdir(exist_ok=create_folder, parents=True)
return str(mediapipe_annotated_videos_path)
def get_session_output_data_folder_path(session_id: str, create_folder: bool = True):
- output_data_folder_path = Path(get_session_folder_path(session_id)) / 'output_data_files'
+ output_data_folder_path = (
+ Path(get_session_folder_path(session_id)) / "output_data_files"
+ )
if create_folder:
output_data_folder_path.mkdir(exist_ok=create_folder, parents=True)
return str(output_data_folder_path)
-def get_session_calibration_file_path(session_id:str)->str:
+
+def get_session_calibration_file_path(session_id: str) -> str:
calibration_file_name = f"{session_id}_camera_calibration.toml"
- calibration_file_path = Path(get_session_folder_path(session_id))/calibration_file_name
+ calibration_file_path = (
+ Path(get_session_folder_path(session_id)) / calibration_file_name
+ )
return str(calibration_file_path)
-
diff --git a/src/config/webcam_config.py b/src/config/webcam_config.py
index 2afb9cf6..c2558167 100644
--- a/src/config/webcam_config.py
+++ b/src/config/webcam_config.py
@@ -16,7 +16,9 @@ class WebcamConfig(BaseModel):
base_save_video_dir = os_independent_home_dir()
-def webcam_config_to_qt_parameter_list(webcam_config:WebcamConfig = WebcamConfig())->List:
+def webcam_config_to_qt_parameter_list(
+ webcam_config: WebcamConfig = WebcamConfig(),
+) -> List:
"""
take in a webcam_config and return a parameter_tree_group that you can feed into a pyqtgraph Parameter tree like this:
@@ -29,10 +31,18 @@ def webcam_config_to_qt_parameter_list(webcam_config:WebcamConfig = WebcamConfig
"""
this_webcam_id = webcam_config.webcam_id
parameter_list = [
- {'name': 'use this camera❓', 'type': 'bool', 'value': True},
- {'name': 'exposure', 'type': 'int', 'value': webcam_config.exposure},
- {'name': 'resolution_width', 'type': 'int', 'value': webcam_config.resolution_width},
- {'name': 'resolution_height', 'type': 'int', 'value': webcam_config.resolution_height},
+ {"name": "use this camera❓", "type": "bool", "value": True},
+ {"name": "exposure", "type": "int", "value": webcam_config.exposure},
+ {
+ "name": "resolution_width",
+ "type": "int",
+ "value": webcam_config.resolution_width,
+ },
+ {
+ "name": "resolution_height",
+ "type": "int",
+ "value": webcam_config.resolution_height,
+ },
]
- return parameter_list
\ No newline at end of file
+ return parameter_list
diff --git a/src/core_processor/camera_calibration/calibration_dataclasses.py b/src/core_processor/camera_calibration/calibration_dataclasses.py
index 85a19ea8..c74da9af 100644
--- a/src/core_processor/camera_calibration/calibration_dataclasses.py
+++ b/src/core_processor/camera_calibration/calibration_dataclasses.py
@@ -18,11 +18,15 @@ class CameraCalibrationData:
def __post_init__(self):
if self.camera_matrix is None:
- self.camera_matrix = np.array([[float(self.image_width), 0., self.image_width / 2.],
- [0., float(self.image_width), self.image_height / 2.],
- [0., 0., 1.]])
+ self.camera_matrix = np.array(
+ [
+ [float(self.image_width), 0.0, self.image_width / 2.0],
+ [0.0, float(self.image_width), self.image_height / 2.0],
+ [0.0, 0.0, 1.0],
+ ]
+ )
if self.lens_distortion_coefficients is None:
- self.lens_distortion_coefficients = np.zeros((self.number_of_lens_distortion_coefficients, 1))
-
-
+ self.lens_distortion_coefficients = np.zeros(
+ (self.number_of_lens_distortion_coefficients, 1)
+ )
diff --git a/src/core_processor/camera_calibration/calibration_diagnostics_visualizer.py b/src/core_processor/camera_calibration/calibration_diagnostics_visualizer.py
index 6a965497..f6b590d3 100644
--- a/src/core_processor/camera_calibration/calibration_diagnostics_visualizer.py
+++ b/src/core_processor/camera_calibration/calibration_diagnostics_visualizer.py
@@ -7,8 +7,9 @@ from pyqtgraph.Qt import QtWidgets
import cv2
-from src.core_processor.camera_calibration.calibration_dataclasses import CameraCalibrationData
-
+from src.core_processor.camera_calibration.calibration_dataclasses import (
+ CameraCalibrationData,
+)
class CalibrationDiagnosticsVisualizer:
@@ -25,7 +26,9 @@ class CalibrationDiagnosticsVisualizer:
self.dock_area = DockArea()
self.main_window_widget.setCentralWidget(self.dock_area)
self.main_window_widget.resize(1000, 500)
- self.main_window_widget.setWindowTitle('Lens Distortion Calibration Diagnostics (pyqtgraph)')
+ self.main_window_widget.setWindowTitle(
+ "Lens Distortion Calibration Diagnostics (pyqtgraph)"
+ )
# create widget/dock for image subplot (based on `GraphicsItems - ImageItem - video` example `python -m pyqtgraph.examples`)
self.image_view_widget = pg.ImageView()
@@ -36,43 +39,61 @@ class CalibrationDiagnosticsVisualizer:
# create widget/dock for reprojection error subplot
self.reprojection_error_plot_widget = pg.PlotWidget(title="Reprojection Error")
- self.reprojection_error_plot_widget.setLabel('left', "Reprojection Error", units='pixels')
- self.reprojection_error_plot_widget.setLabel('bottom', "Time", units='Frame#')
+ self.reprojection_error_plot_widget.setLabel(
+ "left", "Reprojection Error", units="pixels"
+ )
+ self.reprojection_error_plot_widget.setLabel("bottom", "Time", units="Frame#")
self.reprojection_error_plot_widget.setYRange(0, 2)
self.reprojection_error_plot_widget.addLegend()
- self.reprojection_error_all_line = self.reprojection_error_plot_widget.plot(self.reprojection_error_all,
- pen=(255, 0, 128),
- name="reprojection error- all")
- self.reprojection_error_current_best_line = self.reprojection_error_plot_widget.plot(
- self.reprojection_error_current_best,
- name="current best")
+ self.reprojection_error_all_line = self.reprojection_error_plot_widget.plot(
+ self.reprojection_error_all,
+ pen=(255, 0, 128),
+ name="reprojection error- all",
+ )
+ self.reprojection_error_current_best_line = (
+ self.reprojection_error_plot_widget.plot(
+ self.reprojection_error_current_best, name="current best"
+ )
+ )
self.reprojection_error_dock = Dock("Reprojection Error")
self.reprojection_error_dock.addWidget(self.reprojection_error_plot_widget)
- self.dock_area.addDock(self.reprojection_error_dock, position='right')
+ self.dock_area.addDock(self.reprojection_error_dock, position="right")
# create widget/dock lens distortion remapping subplot
- self.image_point_remapping_plot_widget = pg.PlotWidget(title="Image Point Remapping")
- self.image_point_remapping_plot_widget.setLabel('left', "Y-position", units='pixels')
- self.image_point_remapping_plot_widget.setLabel('bottom', "X-position", units='pixels')
+ self.image_point_remapping_plot_widget = pg.PlotWidget(
+ title="Image Point Remapping"
+ )
+ self.image_point_remapping_plot_widget.setLabel(
+ "left", "Y-position", units="pixels"
+ )
+ self.image_point_remapping_plot_widget.setLabel(
+ "bottom", "X-position", units="pixels"
+ )
self.image_point_remapping_plot_widget.addLegend()
- self.image_point_original_dottos = self.image_point_remapping_plot_widget.plot(self.image_point_original,
- self.image_point_original,
- pen=None,
- symbol='o',
- symbolBrush=None,
- symbolPen=(150, 150, 150),
- name="image points - original")
- self.image_point_remapped_dottos = self.image_point_remapping_plot_widget.plot(self.image_point_remapped,
- self.image_point_remapped,
- symbol='+',
- pen=None,
- symbolBrush=None,
- symbolPen=(50, 150, 210),
- name="image points - remapped")
+ self.image_point_original_dottos = self.image_point_remapping_plot_widget.plot(
+ self.image_point_original,
+ self.image_point_original,
+ pen=None,
+ symbol="o",
+ symbolBrush=None,
+ symbolPen=(150, 150, 150),
+ name="image points - original",
+ )
+ self.image_point_remapped_dottos = self.image_point_remapping_plot_widget.plot(
+ self.image_point_remapped,
+ self.image_point_remapped,
+ symbol="+",
+ pen=None,
+ symbolBrush=None,
+ symbolPen=(50, 150, 210),
+ name="image points - remapped",
+ )
self.image_point_remapping_plot_dock = Dock("Image Point Remapping")
- self.image_point_remapping_plot_dock.addWidget(self.image_point_remapping_plot_widget)
+ self.image_point_remapping_plot_dock.addWidget(
+ self.image_point_remapping_plot_widget
+ )
self.dock_area.addDock(self.image_point_remapping_plot_dock)
###########
@@ -86,27 +107,28 @@ class CalibrationDiagnosticsVisualizer:
y_axis_line_array = np.array([[0, 0, 0], [0, 1, 0]])
z_axis_line_array = np.array([[0, 0, 0], [0, 0, 1]])
- self.origin_x_axis_gl_lineplot_item = gl.GLLinePlotItem(pos=x_axis_line_array,
- color=(1, 0, 0, 1),
- width=1.,
- antialias=True)
- self.origin_y_axis_gl_lineplot_item = gl.GLLinePlotItem(pos=y_axis_line_array,
- color=(0, 1, 0, 1),
- width=1.,
- antialias=True)
- self.origin_z_axis_gl_lineplot_item = gl.GLLinePlotItem(pos=z_axis_line_array,
- color=(0, 0, 1, 1),
- width=1.,
- antialias=True)
+ self.origin_x_axis_gl_lineplot_item = gl.GLLinePlotItem(
+ pos=x_axis_line_array, color=(1, 0, 0, 1), width=1.0, antialias=True
+ )
+ self.origin_y_axis_gl_lineplot_item = gl.GLLinePlotItem(
+ pos=y_axis_line_array, color=(0, 1, 0, 1), width=1.0, antialias=True
+ )
+ self.origin_z_axis_gl_lineplot_item = gl.GLLinePlotItem(
+ pos=z_axis_line_array, color=(0, 0, 1, 1), width=1.0, antialias=True
+ )
self.opengl_3d_plot_widget.addItem(self.origin_x_axis_gl_lineplot_item)
self.opengl_3d_plot_widget.addItem(self.origin_y_axis_gl_lineplot_item)
self.opengl_3d_plot_widget.addItem(self.origin_z_axis_gl_lineplot_item)
- self.opengl_charuco_scatter_item = gl.GLScatterPlotItem(pos=(0, 0, 0), color=(1, 0, 1), size=1, pxMode=False)
+ self.opengl_charuco_scatter_item = gl.GLScatterPlotItem(
+ pos=(0, 0, 0), color=(1, 0, 1), size=1, pxMode=False
+ )
self.opengl_3d_plot_dock = Dock("Image Point Remapping")
self.opengl_3d_plot_dock.addWidget(self.opengl_3d_plot_widget)
- self.dock_area.addDock(self.opengl_3d_plot_dock, 'right', self.image_point_remapping_plot_dock)
+ self.dock_area.addDock(
+ self.opengl_3d_plot_dock, "right", self.image_point_remapping_plot_dock
+ )
###
# create widget/dock for calibration text window
@@ -114,7 +136,9 @@ class CalibrationDiagnosticsVisualizer:
self.calibration_text_dock = Dock("Calibration Data")
self.calibration_text_dock.addWidget(self.calibration_text_widget)
- self.dock_area.addDock(self.calibration_text_dock, 'above', self.image_point_remapping_plot_dock)
+ self.dock_area.addDock(
+ self.calibration_text_dock, "above", self.image_point_remapping_plot_dock
+ )
self._display_main_window()
@@ -124,28 +148,47 @@ class CalibrationDiagnosticsVisualizer:
def update_image_subplot(self, new_image):
self.image_view_widget.setImage(
- cv2.cvtColor(cv2.rotate(new_image, cv2.ROTATE_90_COUNTERCLOCKWISE), cv2.COLOR_BGR2RGB))
-
- def update_reprojection_error_subplot(self, reprojection_error_new, reprojection_error_current_best):
- self.reprojection_error_all = np.append(self.reprojection_error_all, reprojection_error_new)
- self.reprojection_error_current_best = np.append(self.reprojection_error_current_best,
- reprojection_error_current_best)
+ cv2.cvtColor(
+ cv2.rotate(new_image, cv2.ROTATE_90_COUNTERCLOCKWISE), cv2.COLOR_BGR2RGB
+ )
+ )
+
+ def update_reprojection_error_subplot(
+ self, reprojection_error_new, reprojection_error_current_best
+ ):
+ self.reprojection_error_all = np.append(
+ self.reprojection_error_all, reprojection_error_new
+ )
+ self.reprojection_error_current_best = np.append(
+ self.reprojection_error_current_best, reprojection_error_current_best
+ )
self.reprojection_error_all_line.setData(self.reprojection_error_all)
- self.reprojection_error_current_best_line.setData(self.reprojection_error_current_best)
+ self.reprojection_error_current_best_line.setData(
+ self.reprojection_error_current_best
+ )
def update_calibration_text_overlay(self, calibration_data: CameraCalibrationData):
- lens_distortion_as_str = [f"{d[0]:.3f}" for d in calibration_data.lens_distortion_coefficients]
- calibration_text = f'reprojection error:{calibration_data.reprojection_error:.3f} \n {lens_distortion_as_str} \n {str(calibration_data.camera_matrix)}'
+ lens_distortion_as_str = [
+ f"{d[0]:.3f}" for d in calibration_data.lens_distortion_coefficients
+ ]
+ calibration_text = f"reprojection error:{calibration_data.reprojection_error:.3f} \n {lens_distortion_as_str} \n {str(calibration_data.camera_matrix)}"
self.calibration_text_widget.setText(calibration_text)
- def update_image_point_remapping_subplot(self, image_point_original_x, image_point_original_y,
- image_point_remapped_x, image_point_remapped_y):
- self.image_point_original_dottos.setData(x=image_point_original_x,
- y=image_point_original_y)
-
- self.image_point_remapped_dottos.setData(x=image_point_remapped_x,
- y=image_point_remapped_y)
+ def update_image_point_remapping_subplot(
+ self,
+ image_point_original_x,
+ image_point_original_y,
+ image_point_remapped_x,
+ image_point_remapped_y,
+ ):
+ self.image_point_original_dottos.setData(
+ x=image_point_original_x, y=image_point_original_y
+ )
+
+ self.image_point_remapped_dottos.setData(
+ x=image_point_remapped_x, y=image_point_remapped_y
+ )
def update_3d_subplot(self, charuco_xyz: np.ndarray = None):
if charuco_xyz is None:
diff --git a/src/core_processor/camera_calibration/camera_calibrator.py b/src/core_processor/camera_calibration/camera_calibrator.py
index d3776dfb..ba39c5b6 100644
--- a/src/core_processor/camera_calibration/camera_calibrator.py
+++ b/src/core_processor/camera_calibration/camera_calibrator.py
@@ -4,19 +4,27 @@ import cv2
from src.cameras.capture.dataclasses.frame_payload import FramePayload
from src.cameras.capture.opencv_camera.opencv_camera import OpenCVCamera
-from src.core_processor.camera_calibration.calibration_dataclasses import CameraCalibrationData
-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import CharucoBoardDetector
-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_image_annotator import \
- annotate_image_with_charuco_data
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import CharucoViewData
+from src.core_processor.camera_calibration.calibration_dataclasses import (
+ CameraCalibrationData,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (
+ CharucoBoardDetector,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_image_annotator import (
+ annotate_image_with_charuco_data,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import (
+ CharucoViewData,
+)
logger = logging.getLogger(__name__)
class CameraCalibrator:
- def __init__(self,
- charuco_board_detector: CharucoBoardDetector = CharucoBoardDetector(),
- ):
+ def __init__(
+ self,
+ charuco_board_detector: CharucoBoardDetector = CharucoBoardDetector(),
+ ):
self._current_calibration = None
self._charuco_board_detector = charuco_board_detector
@@ -31,13 +39,16 @@ class CameraCalibrator:
def _process_incoming_frame(self, raw_frame_payload: FramePayload):
- charuco_frame_payload = self._charuco_board_detector.detect_charuco_board(raw_frame_payload)
+ charuco_frame_payload = self._charuco_board_detector.detect_charuco_board(
+ raw_frame_payload
+ )
charuco_view_data = charuco_frame_payload.charuco_view_data
if self._current_calibration is None:
- self._current_calibration = CameraCalibrationData(charuco_view_data.image_width,
- charuco_view_data.image_height)
+ self._current_calibration = CameraCalibrationData(
+ charuco_view_data.image_width, charuco_view_data.image_height
+ )
if not charuco_view_data.full_board_found and self._current_calibration is None:
return charuco_frame_payload.annotated_image, self._current_calibration
@@ -49,13 +60,14 @@ class CameraCalibrator:
annotate_image_with_charuco_data(
raw_frame_payload.image, # this image will have stuff drawn on top of it inside this function
charuco_view_data,
- self._charuco_board_detector.number_of_charuco_corners
+ self._charuco_board_detector.number_of_charuco_corners,
)
- undistorted_image = cv2.undistort(raw_frame_payload.image,
- self._current_calibration.camera_matrix,
- self._current_calibration.lens_distortion_coefficients,
- )
+ undistorted_image = cv2.undistort(
+ raw_frame_payload.image,
+ self._current_calibration.camera_matrix,
+ self._current_calibration.lens_distortion_coefficients,
+ )
return undistorted_image, self._current_calibration
@@ -78,26 +90,29 @@ class CameraCalibrator:
image_height = charuco_view.image_height
# flag definitions -> https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#ga3207604e4b1a1758aa66acb6ed5aa65d
- flags = (cv2.CALIB_USE_INTRINSIC_GUESS +
- cv2.CALIB_ZERO_TANGENT_DIST +
- cv2.cv2.CALIB_FIX_ASPECT_RATIO +
- cv2.CALIB_FIX_PRINCIPAL_POINT +
- # cv2.CALIB_FIX_FOCAL_LENGTH
- cv2.CALIB_RATIONAL_MODEL
- # cv2.CALIB_THIN_PRISM_MODEL
- # cv2.CALIB_TILTED_MODEL
- )
+ flags = (
+ cv2.CALIB_USE_INTRINSIC_GUESS
+ + cv2.CALIB_ZERO_TANGENT_DIST
+ + cv2.cv2.CALIB_FIX_ASPECT_RATIO
+ + cv2.CALIB_FIX_PRINCIPAL_POINT
+ +
+ # cv2.CALIB_FIX_FOCAL_LENGTH
+ cv2.CALIB_RATIONAL_MODEL
+ # cv2.CALIB_THIN_PRISM_MODEL
+ # cv2.CALIB_TILTED_MODEL
+ )
# https://docs.opencv.org/4.x/d9/d6a/group__aruco.html#gacf03e5afb0bc516b73028cf209984a06
- (this_reprojection_error,
- this_camera_matrix,
- these_lens_distortion_coefficients,
- these_rotation_vectors_of_the_board,
- these_translation_vectors_of_the_board,
- this_lens_distortion_std_dev,
- this_camera_location_std_dev,
- these_reprojection_error_per_view
- ) = cv2.aruco.calibrateCameraCharucoExtended(
+ (
+ this_reprojection_error,
+ this_camera_matrix,
+ these_lens_distortion_coefficients,
+ these_rotation_vectors_of_the_board,
+ these_translation_vectors_of_the_board,
+ this_lens_distortion_std_dev,
+ this_camera_location_std_dev,
+ these_reprojection_error_per_view,
+ ) = cv2.aruco.calibrateCameraCharucoExtended(
charucoCorners=[charuco_view.charuco_corners],
charucoIds=[charuco_view.charuco_ids],
board=self._charuco_board_detector.cv2_aruco_charuco_board,
@@ -105,7 +120,8 @@ class CameraCalibrator:
cameraMatrix=self._current_calibration.camera_matrix,
distCoeffs=self._current_calibration.lens_distortion_coefficients,
flags=flags,
- criteria=(cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT, 10000, 1e-9))
+ criteria=(cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT, 10000, 1e-9),
+ )
if this_reprojection_error < self._current_calibration.reprojection_error:
self._current_calibration = CameraCalibrationData(
@@ -117,7 +133,8 @@ class CameraCalibrator:
rotation_vectors_of_the_board=these_rotation_vectors_of_the_board,
translation_vectors_of_the_board=these_translation_vectors_of_the_board,
lens_distortion_std_dev=this_lens_distortion_std_dev,
- camera_location_std_dev=this_camera_location_std_dev)
+ camera_location_std_dev=this_camera_location_std_dev,
+ )
def reconstruct_3d_charuco(self, charuco_data_per_camera_dictionary):
pass
diff --git a/src/core_processor/camera_calibration/lens_distortion_calibrator.py b/src/core_processor/camera_calibration/lens_distortion_calibrator.py
index ae0a45b9..d00815c9 100644
--- a/src/core_processor/camera_calibration/lens_distortion_calibrator.py
+++ b/src/core_processor/camera_calibration/lens_distortion_calibrator.py
@@ -3,32 +3,44 @@ import numpy as np
from rich import print
from src.cameras.capture.dataclasses.frame_payload import FramePayload
-from src.core_processor.camera_calibration.calibration_dataclasses import LensDistortionCalibrationData
+from src.core_processor.camera_calibration.calibration_dataclasses import (
+ LensDistortionCalibrationData,
+)
from src.pipelines.calibration_pipeline.charuco_board_detection import BoardDetector
-from src.core_processor.camera_calibration.calibration_diagnostics_visualizer import CalibrationDiagnosticsVisualizer
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses import CharucoViewData
-from src.pipelines.calibration_pipeline.charuco_board_detection import \
- annotate_image_with_charuco_data
+from src.core_processor.camera_calibration.calibration_diagnostics_visualizer import (
+ CalibrationDiagnosticsVisualizer,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses import (
+ CharucoViewData,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection import (
+ annotate_image_with_charuco_data,
+)
class LensDistortionCalibrator:
- def __init__(self,
- board_detector: BoardDetector = BoardDetector(),
- show_calibration_diagnostics_visualizer=False, ):
+ def __init__(
+ self,
+ board_detector: BoardDetector = BoardDetector(),
+ show_calibration_diagnostics_visualizer=False,
+ ):
self._board_detector = board_detector
self._current_calibration = None
self._all_charuco_views = []
self._num_charuco_views = 0
self._best_combo_of_charuco_views = []
self._previous_best_calibrations = []
- self.show_calibration_diagnostics_visualizer = show_calibration_diagnostics_visualizer
+ self.show_calibration_diagnostics_visualizer = (
+ show_calibration_diagnostics_visualizer
+ )
if self.show_calibration_diagnostics_visualizer:
self.calibration_diagnostics_visualizer = CalibrationDiagnosticsVisualizer()
- def process_incoming_frame(self,
- raw_frame_payload: FramePayload):
+ def process_incoming_frame(self, raw_frame_payload: FramePayload):
- charuco_frame_payload = self._board_detector.detect_charuco_board(raw_frame_payload)
+ charuco_frame_payload = self._board_detector.detect_charuco_board(
+ raw_frame_payload
+ )
charuco_view_data = charuco_frame_payload.charuco_view_data
@@ -42,29 +54,36 @@ class LensDistortionCalibrator:
if self._num_charuco_views < 2:
return charuco_frame_payload.annotated_image
- self.estimate_lens_distortion(charuco_view_data) # this is where the magic happens
+ self.estimate_lens_distortion(
+ charuco_view_data
+ ) # this is where the magic happens
if self._current_calibration is None:
- self._current_calibration = LensDistortionCalibrationData(charuco_view_data.image_width,
- charuco_view_data.image_height)
+ self._current_calibration = LensDistortionCalibrationData(
+ charuco_view_data.image_width, charuco_view_data.image_height
+ )
return raw_frame_payload.image
- undistorted_image_for_debug = self.undistort_image_with_invalid_pixels_as_black(raw_frame_payload.image,
- self._current_calibration)
+ undistorted_image_for_debug = self.undistort_image_with_invalid_pixels_as_black(
+ raw_frame_payload.image, self._current_calibration
+ )
if charuco_view_data.some_charuco_corners_found:
annotate_image_with_charuco_data(
undistorted_image_for_debug, # this image will have stuff drawn on top of it inside this function
charuco_view_data,
- self._board_detector.number_of_charuco_corners
+ self._board_detector.number_of_charuco_corners,
)
if self.show_calibration_diagnostics_visualizer:
- self.calibration_diagnostics_visualizer.update_image_subplot(undistorted_image_for_debug)
+ self.calibration_diagnostics_visualizer.update_image_subplot(
+ undistorted_image_for_debug
+ )
- undistorted_image = cv2.undistort(raw_frame_payload.image,
- self._current_calibration.camera_matrix,
- self._current_calibration.lens_distortion_coefficients,
- )
+ undistorted_image = cv2.undistort(
+ raw_frame_payload.image,
+ self._current_calibration.camera_matrix,
+ self._current_calibration.lens_distortion_coefficients,
+ )
return undistorted_image
@@ -84,9 +103,13 @@ class LensDistortionCalibrator:
print("CAMERA CALIBRATION")
if self._current_calibration is None:
- current_best_reprojection_error = 1e9 # put a big fake number on the first frame
+ current_best_reprojection_error = (
+ 1e9 # put a big fake number on the first frame
+ )
else:
- current_best_reprojection_error = self._current_calibration.reprojection_error
+ current_best_reprojection_error = (
+ self._current_calibration.reprojection_error
+ )
image_width = new_charuco_view.image_width
image_height = new_charuco_view.image_height
@@ -97,34 +120,38 @@ class LensDistortionCalibrator:
if self._num_charuco_views < num_charuco_views_per_combo:
list_of_combos_of_charuco_views = [self._all_charuco_views]
else:
- list_of_combos_of_charuco_views = self.generate_combos_of_charuco_views(new_charuco_view,
- num_combos_to_generate=10,
- num_views_per_combo=num_charuco_views_per_combo,
- )
+ list_of_combos_of_charuco_views = self.generate_combos_of_charuco_views(
+ new_charuco_view,
+ num_combos_to_generate=10,
+ num_views_per_combo=num_charuco_views_per_combo,
+ )
# flag definitions -> https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#ga3207604e4b1a1758aa66acb6ed5aa65d
- flags = (cv2.CALIB_USE_INTRINSIC_GUESS +
- cv2.CALIB_ZERO_TANGENT_DIST +
- cv2.cv2.CALIB_FIX_ASPECT_RATIO +
- cv2.CALIB_FIX_PRINCIPAL_POINT +
- # cv2.CALIB_FIX_FOCAL_LENGTH
- cv2.CALIB_RATIONAL_MODEL
- # cv2.CALIB_THIN_PRISM_MODEL
- # cv2.CALIB_TILTED_MODEL
- )
+ flags = (
+ cv2.CALIB_USE_INTRINSIC_GUESS
+ + cv2.CALIB_ZERO_TANGENT_DIST
+ + cv2.cv2.CALIB_FIX_ASPECT_RATIO
+ + cv2.CALIB_FIX_PRINCIPAL_POINT
+ +
+ # cv2.CALIB_FIX_FOCAL_LENGTH
+ cv2.CALIB_RATIONAL_MODEL
+ # cv2.CALIB_THIN_PRISM_MODEL
+ # cv2.CALIB_TILTED_MODEL
+ )
for this_combo_of_charuco_views in list_of_combos_of_charuco_views:
for this_charuco_view in this_combo_of_charuco_views:
# https://docs.opencv.org/4.x/d9/d6a/group__aruco.html#gacf03e5afb0bc516b73028cf209984a06
- (this_reprojection_error,
- this_camera_matrix,
- these_lens_distortion_coefficients,
- these_rotation_vectors_of_the_board,
- these_translation_vectors_of_the_board,
- this_lens_distortion_std_dev,
- this_camera_location_std_dev,
- these_reprojection_error_per_view
- ) = cv2.aruco.calibrateCameraCharucoExtended(
+ (
+ this_reprojection_error,
+ this_camera_matrix,
+ these_lens_distortion_coefficients,
+ these_rotation_vectors_of_the_board,
+ these_translation_vectors_of_the_board,
+ this_lens_distortion_std_dev,
+ this_camera_location_std_dev,
+ these_reprojection_error_per_view,
+ ) = cv2.aruco.calibrateCameraCharucoExtended(
charucoCorners=[this_charuco_view.charuco_corners],
charucoIds=[this_charuco_view.charuco_ids],
board=self._board_detector.cv2_aruco_charuco_board,
@@ -132,14 +159,22 @@ class LensDistortionCalibrator:
cameraMatrix=default_calibration.camera_matrix,
distCoeffs=default_calibration.lens_distortion_coefficients,
flags=flags,
- criteria=(cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT, 10000, 1e-9))
+ criteria=(
+ cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT,
+ 10000,
+ 1e-9,
+ ),
+ )
if self.show_calibration_diagnostics_visualizer:
- self.calibration_diagnostics_visualizer.update_reprojection_error_subplot(this_reprojection_error,
- current_best_reprojection_error,
- )
+ self.calibration_diagnostics_visualizer.update_reprojection_error_subplot(
+ this_reprojection_error,
+ current_best_reprojection_error,
+ )
if self._current_calibration is not None:
- self.calibration_diagnostics_visualizer.update_calibration_text_overlay(self._current_calibration)
+ self.calibration_diagnostics_visualizer.update_calibration_text_overlay(
+ self._current_calibration
+ )
if this_reprojection_error < current_best_reprojection_error:
@@ -152,16 +187,18 @@ class LensDistortionCalibrator:
rotation_vectors_of_the_board=these_rotation_vectors_of_the_board,
translation_vectors_of_the_board=these_translation_vectors_of_the_board,
lens_distortion_std_dev=this_lens_distortion_std_dev,
- camera_location_std_dev=this_camera_location_std_dev)
+ camera_location_std_dev=this_camera_location_std_dev,
+ )
if self.is_this_calibration_valid(this_calibration):
self._current_calibration = this_calibration
- self._previous_best_calibrations.append(self._current_calibration)
+ self._previous_best_calibrations.append(
+ self._current_calibration
+ )
- def generate_combos_of_charuco_views(self,
- new_charuco_view,
- num_combos_to_generate=10,
- num_views_per_combo=5):
+ def generate_combos_of_charuco_views(
+ self, new_charuco_view, num_combos_to_generate=10, num_views_per_combo=5
+ ):
"""
return -
1x current best combo plus new view
@@ -176,7 +213,9 @@ class LensDistortionCalibrator:
# add previous best view
if self._best_combo_of_charuco_views:
- list_of_combos_of_charuco_views.append(self._best_combo_of_charuco_views.copy())
+ list_of_combos_of_charuco_views.append(
+ self._best_combo_of_charuco_views.copy()
+ )
# add previous best view + new view
best_combo_plus_new = self._best_combo_of_charuco_views.copy()
@@ -188,12 +227,16 @@ class LensDistortionCalibrator:
# and a some random combos - half with the new view and half without it
for this_iter in range(num_combos_to_generate):
- this_random_sample_of_charuco_views = np.random.choice(num_views_per_combo,
- size=min(len(self._all_charuco_views),
- num_views_per_combo),
- replace=False)
+ this_random_sample_of_charuco_views = np.random.choice(
+ num_views_per_combo,
+ size=min(len(self._all_charuco_views), num_views_per_combo),
+ replace=False,
+ )
- this_combo = [self._all_charuco_views[cc] for cc in this_random_sample_of_charuco_views]
+ this_combo = [
+ self._all_charuco_views[cc]
+ for cc in this_random_sample_of_charuco_views
+ ]
if this_iter < np.ceil(num_views_per_combo / 2):
this_combo.append(new_charuco_view)
@@ -205,7 +248,9 @@ class LensDistortionCalibrator:
def is_this_calibration_valid(self, this_calibration):
is_valid = False
- if not self.lens_distortion_is_monotonic(this_calibration): # not currently working - returns True every time
+ if not self.lens_distortion_is_monotonic(
+ this_calibration
+ ): # not currently working - returns True every time
return False
if self.check_if_too_many_invalid_pixels(this_calibration):
@@ -213,7 +258,9 @@ class LensDistortionCalibrator:
return True
- def lens_distortion_is_monotonic(self, this_calibration, assume_wide_angle_lens=True) -> bool:
+ def lens_distortion_is_monotonic(
+ self, this_calibration, assume_wide_angle_lens=True
+ ) -> bool:
"""
check if the lens distortion coefficients are valid by checking it the resulting undistorted image is monotonically increasing along each diagonal (i.e. the image doesn't wrap around itself)
@@ -240,9 +287,12 @@ class LensDistortionCalibrator:
# following naming conventions of open cv docs
k1 = this_calibration.lens_distortion_coefficients[0]
k2 = this_calibration.lens_distortion_coefficients[1]
- p1 = this_calibration.lens_distortion_coefficients[2] # I think p1 and p2 are for tangential distortion...
+ p1 = this_calibration.lens_distortion_coefficients[
+ 2
+ ] # I think p1 and p2 are for tangential distortion...
p2 = this_calibration.lens_distortion_coefficients[
- 3] # ...which I might want to check on later? Currently fixed at Zero
+ 3
+ ] # ...which I might want to check on later? Currently fixed at Zero
k3 = this_calibration.lens_distortion_coefficients[4]
number_of_points = 40
@@ -251,34 +301,57 @@ class LensDistortionCalibrator:
original_points_xx, original_points_yy = np.meshgrid(original_x, original_y)
- original_points_x = original_points_xx.flatten() # transform 2D grid with 1D array
- original_points_y = original_points_yy.flatten() # transform 2D grid with 1D array
+ original_points_x = (
+ original_points_xx.flatten()
+ ) # transform 2D grid with 1D array
+ original_points_y = (
+ original_points_yy.flatten()
+ ) # transform 2D grid with 1D array
original_points_xy = np.vstack((original_points_x, original_points_y))
- distorted_points_xy = cv2.undistortPoints(original_points_xy,
- this_calibration.camera_matrix,
- this_calibration.lens_distortion_coefficients)
+ distorted_points_xy = cv2.undistortPoints(
+ original_points_xy,
+ this_calibration.camera_matrix,
+ this_calibration.lens_distortion_coefficients,
+ )
- distorted_points_x = np.squeeze(distorted_points_xy[:, :, 0])*this_calibration.image_width+this_calibration.image_width/2
- distorted_points_y = np.squeeze(distorted_points_xy[:, :, 1])*this_calibration.image_height+this_calibration.image_height/2
+ distorted_points_x = (
+ np.squeeze(distorted_points_xy[:, :, 0]) * this_calibration.image_width
+ + this_calibration.image_width / 2
+ )
+ distorted_points_y = (
+ np.squeeze(distorted_points_xy[:, :, 1]) * this_calibration.image_height
+ + this_calibration.image_height / 2
+ )
if self.show_calibration_diagnostics_visualizer:
- self.calibration_diagnostics_visualizer.update_image_point_remapping_subplot(original_points_x,
- original_points_y,
- distorted_points_x,
- distorted_points_y)
+ self.calibration_diagnostics_visualizer.update_image_point_remapping_subplot(
+ original_points_x,
+ original_points_y,
+ distorted_points_x,
+ distorted_points_y,
+ )
return True
- def check_if_too_many_invalid_pixels(self, this_calibration, how_many_is_too_many=.1) -> bool:
+ def check_if_too_many_invalid_pixels(
+ self, this_calibration, how_many_is_too_many=0.1
+ ) -> bool:
"""
create new camera matrix that will show full undistored image with black pixels in spots with no data.
return FALSE if black pixels are greater than `how_many_is_too_many` proportion of the full image
"""
- dummy_image = np.ones((this_calibration.image_width, this_calibration.image_height), np.uint8) * 255
+ dummy_image = (
+ np.ones(
+ (this_calibration.image_width, this_calibration.image_height), np.uint8
+ )
+ * 255
+ )
- undistorted_image = self.undistort_image_with_invalid_pixels_as_black(dummy_image, this_calibration)
+ undistorted_image = self.undistort_image_with_invalid_pixels_as_black(
+ dummy_image, this_calibration
+ )
num_invalid_pixels = np.sum(undistorted_image == 0)
proportion_invalid_pixels = num_invalid_pixels / dummy_image.size
@@ -294,15 +367,17 @@ class LensDistortionCalibrator:
calibration.lens_distortion_coefficients,
(calibration.image_width, calibration.image_height),
1,
- centerPrincipalPoint=True
+ centerPrincipalPoint=True,
)
# https://docs.opencv.org/4.5.5/d9/d0c/group__calib3d.html#ga69f2545a8b62a6b0fc2ee060dc30559d
- undistorted_image = cv2.undistort(image,
- calibration.camera_matrix,
- calibration.lens_distortion_coefficients,
- None,
- new_camera_matrix)
+ undistorted_image = cv2.undistort(
+ image,
+ calibration.camera_matrix,
+ calibration.lens_distortion_coefficients,
+ None,
+ new_camera_matrix,
+ )
return undistorted_image
diff --git a/src/core_processor/mediapipe_skeleton_detector/medaipipe_tracked_points_names_dict.py b/src/core_processor/mediapipe_skeleton_detector/medaipipe_tracked_points_names_dict.py
index fef4281b..7e92fae0 100644
--- a/src/core_processor/mediapipe_skeleton_detector/medaipipe_tracked_points_names_dict.py
+++ b/src/core_processor/mediapipe_skeleton_detector/medaipipe_tracked_points_names_dict.py
@@ -86,6 +86,6 @@ mediapipe_tracked_point_names_dict = {
"nose_tip",
"mouth_center",
"right_ear_tragion",
- "left_ear_tragion"
- ]
+ "left_ear_tragion",
+ ],
}
diff --git a/src/core_processor/mediapipe_skeleton_detector/mediapipe_skeleton_detector.py b/src/core_processor/mediapipe_skeleton_detector/mediapipe_skeleton_detector.py
index 5c3080dc..895634d1 100644
--- a/src/core_processor/mediapipe_skeleton_detector/mediapipe_skeleton_detector.py
+++ b/src/core_processor/mediapipe_skeleton_detector/mediapipe_skeleton_detector.py
@@ -9,10 +9,14 @@ import mediapipe as mp
from src.cameras.capture.dataclasses.frame_payload import FramePayload
from src.cameras.persistence.video_writer.video_recorder import VideoRecorder
-from src.config.home_dir import get_session_folder_path, get_synchronized_videos_folder_path, \
- get_session_output_data_folder_path
-from src.core_processor.mediapipe_skeleton_detector.medaipipe_tracked_points_names_dict import \
- mediapipe_tracked_point_names_dict
+from src.config.home_dir import (
+ get_session_folder_path,
+ get_synchronized_videos_folder_path,
+ get_session_output_data_folder_path,
+)
+from src.core_processor.mediapipe_skeleton_detector.medaipipe_tracked_points_names_dict import (
+ mediapipe_tracked_point_names_dict,
+)
from src.gui.main.state.app_state import APP_STATE
logger = logging.getLogger(__name__)
@@ -39,21 +43,32 @@ class Mediapipe2dNumpyArrays:
# if there's no body data, there's no hand or face data either
return
- if len(self.body2d_frameNumber_trackedPointNumber_XY.shape) == 3: #multiple frames
- return np.hstack([self.body2d_frameNumber_trackedPointNumber_XY,
- self.rightHand2d_frameNumber_trackedPointNumber_XY,
- self.leftHand2d_frameNumber_trackedPointNumber_XY,
- self.face2d_frameNumber_trackedPointNumber_XY])
- elif len(self.body2d_frameNumber_trackedPointNumber_XY.shape) == 2: #single frame
- return np.vstack([self.body2d_frameNumber_trackedPointNumber_XY,
- self.rightHand2d_frameNumber_trackedPointNumber_XY,
- self.leftHand2d_frameNumber_trackedPointNumber_XY,
- self.face2d_frameNumber_trackedPointNumber_XY])
+ if (
+ len(self.body2d_frameNumber_trackedPointNumber_XY.shape) == 3
+ ): # multiple frames
+ return np.hstack(
+ [
+ self.body2d_frameNumber_trackedPointNumber_XY,
+ self.rightHand2d_frameNumber_trackedPointNumber_XY,
+ self.leftHand2d_frameNumber_trackedPointNumber_XY,
+ self.face2d_frameNumber_trackedPointNumber_XY,
+ ]
+ )
+ elif (
+ len(self.body2d_frameNumber_trackedPointNumber_XY.shape) == 2
+ ): # single frame
+ return np.vstack(
+ [
+ self.body2d_frameNumber_trackedPointNumber_XY,
+ self.rightHand2d_frameNumber_trackedPointNumber_XY,
+ self.leftHand2d_frameNumber_trackedPointNumber_XY,
+ self.face2d_frameNumber_trackedPointNumber_XY,
+ ]
+ )
else:
logger.error("data should have either 2 or 3 dimensions")
-
@dataclass
class Mediapipe2dDataPayload:
raw_frame_payload: FramePayload = None
@@ -66,8 +81,8 @@ class MediaPipeSkeletonDetector:
def __init__(self, session_id: str = None):
self._session_id = session_id
self.model_complexity = 2 # can be 0,1, or 2 - higher numbers are more accurate but heavier computationally
- self.min_detection_confidence = .5
- self.min_tracking_confidence = .5
+ self.min_detection_confidence = 0.5
+ self.min_tracking_confidence = 0.5
self._mediapipe_payload_list = []
@@ -75,18 +90,30 @@ class MediaPipeSkeletonDetector:
self._mp_drawing_styles = mp.solutions.drawing_styles
self._mp_holistic = mp.solutions.holistic
- self._body_drawing_spec = self._mp_drawing.DrawingSpec(thickness=1, circle_radius=1)
- self._hand_drawing_spec = self._mp_drawing.DrawingSpec(thickness=1, circle_radius=1)
- self._face_drawing_spec = self._mp_drawing.DrawingSpec(thickness=1, circle_radius=1)
-
- self._holistic_tracker = self._mp_holistic.Holistic(model_complexity=self.model_complexity,
- min_detection_confidence=self.min_detection_confidence,
- min_tracking_confidence=self.min_tracking_confidence)
+ self._body_drawing_spec = self._mp_drawing.DrawingSpec(
+ thickness=1, circle_radius=1
+ )
+ self._hand_drawing_spec = self._mp_drawing.DrawingSpec(
+ thickness=1, circle_radius=1
+ )
+ self._face_drawing_spec = self._mp_drawing.DrawingSpec(
+ thickness=1, circle_radius=1
+ )
+
+ self._holistic_tracker = self._mp_holistic.Holistic(
+ model_complexity=self.model_complexity,
+ min_detection_confidence=self.min_detection_confidence,
+ min_tracking_confidence=self.min_tracking_confidence,
+ )
self._mediapipe_tracked_point_names_dict = mediapipe_tracked_point_names_dict
self.body_names_list = self._mediapipe_tracked_point_names_dict["body"]
- self.right_hand_names_list = self._mediapipe_tracked_point_names_dict["right_hand"]
- self.left_hand_names_list = self._mediapipe_tracked_point_names_dict["left_hand"]
+ self.right_hand_names_list = self._mediapipe_tracked_point_names_dict[
+ "right_hand"
+ ]
+ self.left_hand_names_list = self._mediapipe_tracked_point_names_dict[
+ "left_hand"
+ ]
self.face_names_list = self._mediapipe_tracked_point_names_dict["face"]
# TODO - build a better iterator and list of `face_marker_names` that will only pull out the face_counters & iris edges (mp.python.solutions.face_mesh_connections.FACEMESH_CONTOURS, FACE_MESH_IRISES)
@@ -94,55 +121,72 @@ class MediaPipeSkeletonDetector:
self.number_of_body_tracked_points = len(self.body_names_list)
self.number_of_right_hand_tracked_points = len(self.right_hand_names_list)
self.number_of_left_hand_tracked_points = len(self.left_hand_names_list)
- self.number_of_face_tracked_points = mp.python.solutions.face_mesh.FACEMESH_NUM_LANDMARKS_WITH_IRISES
-
- self.number_of_tracked_points_total = self.number_of_body_tracked_points + \
- self.number_of_left_hand_tracked_points + \
- self.number_of_right_hand_tracked_points + \
- self.number_of_face_tracked_points
-
- def detect_skeleton_in_image(self,
- raw_image: np.ndarray = None,
- raw_frame_payload: FramePayload = None,
- annotated_image: np.ndarray = None) -> Mediapipe2dDataPayload:
+ self.number_of_face_tracked_points = (
+ mp.python.solutions.face_mesh.FACEMESH_NUM_LANDMARKS_WITH_IRISES
+ )
+
+ self.number_of_tracked_points_total = (
+ self.number_of_body_tracked_points
+ + self.number_of_left_hand_tracked_points
+ + self.number_of_right_hand_tracked_points
+ + self.number_of_face_tracked_points
+ )
+
+ def detect_skeleton_in_image(
+ self,
+ raw_image: np.ndarray = None,
+ raw_frame_payload: FramePayload = None,
+ annotated_image: np.ndarray = None,
+ ) -> Mediapipe2dDataPayload:
if raw_frame_payload is not None:
raw_image = raw_frame_payload.image.copy()
mediapipe_results = self._holistic_tracker.process(
- raw_image) # <-this is where the magic happens, i.e. where the raw image is processed by a convolutional neural network to provide an estimate of joint position in pixel coordinates. Please don't forget that this is insane and should not be possible lol
+ raw_image
+ ) # <-this is where the magic happens, i.e. where the raw image is processed by a convolutional neural network to provide an estimate of joint position in pixel coordinates. Please don't forget that this is insane and should not be possible lol
if annotated_image is None:
annotated_image = raw_image.copy()
annotated_image = self._annotate_image(annotated_image, mediapipe_results)
- mediapipe_single_frame_npy_data = self._list_of_mediapipe_results_to_npy_arrays([mediapipe_results],
- image_width=
- annotated_image.shape[0],
- image_height=
- annotated_image.shape[1])
+ mediapipe_single_frame_npy_data = self._list_of_mediapipe_results_to_npy_arrays(
+ [mediapipe_results],
+ image_width=annotated_image.shape[0],
+ image_height=annotated_image.shape[1],
+ )
mediapipe_single_frame_npy_data.body2d_frameNumber_trackedPointNumber_XY = self._threshold_by_confidence(
mediapipe_single_frame_npy_data.body2d_frameNumber_trackedPointNumber_XY,
mediapipe_single_frame_npy_data.body2d_frameNumber_trackedPointNumber_confidence,
- confidence_threshold=.5)
-
- return Mediapipe2dDataPayload(raw_frame_payload=raw_frame_payload,
- mediapipe_results=mediapipe_results,
- annotated_image=annotated_image,
- pixel_data_numpy_arrays=mediapipe_single_frame_npy_data)
-
- def process_session_folder(self,
- save_annotated_videos: bool = True):
- synchronized_videos_path = Path(get_synchronized_videos_folder_path(self._session_id))
+ confidence_threshold=0.5,
+ )
+
+ return Mediapipe2dDataPayload(
+ raw_frame_payload=raw_frame_payload,
+ mediapipe_results=mediapipe_results,
+ annotated_image=annotated_image,
+ pixel_data_numpy_arrays=mediapipe_single_frame_npy_data,
+ )
+
+ def process_session_folder(self, save_annotated_videos: bool = True):
+ synchronized_videos_path = Path(
+ get_synchronized_videos_folder_path(self._session_id)
+ )
logger.info(f"loading synchronized videos from: {synchronized_videos_path}")
each_video_frame_width_list = []
each_video_frame_height_list = []
mediapipe2d_single_camera_npy_arrays_list = []
- for video_number, this_synchronized_video_file_path in enumerate(synchronized_videos_path.glob('*.mp4')):
- logger.info(f"Running `mediapipe` skeleton detection on video: {str(this_synchronized_video_file_path)}")
- this_video_capture_object = cv2.VideoCapture(str(this_synchronized_video_file_path))
+ for video_number, this_synchronized_video_file_path in enumerate(
+ synchronized_videos_path.glob("*.mp4")
+ ):
+ logger.info(
+ f"Running `mediapipe` skeleton detection on video: {str(this_synchronized_video_file_path)}"
+ )
+ this_video_capture_object = cv2.VideoCapture(
+ str(this_synchronized_video_file_path)
+ )
this_video_width = this_video_capture_object.get(cv2.CAP_PROP_FRAME_WIDTH)
this_video_height = this_video_capture_object.get(cv2.CAP_PROP_FRAME_HEIGHT)
@@ -152,125 +196,169 @@ class MediaPipeSkeletonDetector:
success, image = this_video_capture_object.read()
if not success or image is None:
- logger.error(f"Failed to load an image from: {str(this_synchronized_video_file_path)}")
+ logger.error(
+ f"Failed to load an image from: {str(this_synchronized_video_file_path)}"
+ )
raise Exception
frame_number = 0
while success and image is not None:
- frame_number+=1
- if frame_number%5==0:
- print(f"frame {frame_number} out of {APP_STATE.number_of_frames_in_the_mocap_videos}")
-
- mediapipe2d_data_payload = self.detect_skeleton_in_image(raw_image=image)
- this_video_mediapipe_results_list.append(mediapipe2d_data_payload.mediapipe_results)
- annotated_image = self._annotate_image(image, mediapipe2d_data_payload.mediapipe_results)
+ frame_number += 1
+ if frame_number % 5 == 0:
+ print(
+ f"frame {frame_number} out of {APP_STATE.number_of_frames_in_the_mocap_videos}"
+ )
+
+ mediapipe2d_data_payload = self.detect_skeleton_in_image(
+ raw_image=image
+ )
+ this_video_mediapipe_results_list.append(
+ mediapipe2d_data_payload.mediapipe_results
+ )
+ annotated_image = self._annotate_image(
+ image, mediapipe2d_data_payload.mediapipe_results
+ )
this_video_annotated_images_list.append(annotated_image)
success, image = this_video_capture_object.read()
if save_annotated_videos:
- self.save_annotated_videos(this_video_annotated_images_list,
- this_synchronized_video_file_path.stem,
- this_video_width,
- this_video_height,
- )
-
- this_camera_mediapipe_2d_single_camera_npy_arrays = self._list_of_mediapipe_results_to_npy_arrays(
- this_video_mediapipe_results_list,
- image_width=this_video_width,
- image_height=this_video_height)
-
- mediapipe2d_single_camera_npy_arrays_list.append(this_camera_mediapipe_2d_single_camera_npy_arrays)
-
- all_cameras_data2d_list = [m2d.all_data2d_nFrames_nTrackedPts_XY for m2d in
- mediapipe2d_single_camera_npy_arrays_list]
+ self.save_annotated_videos(
+ this_video_annotated_images_list,
+ this_synchronized_video_file_path.stem,
+ this_video_width,
+ this_video_height,
+ )
+
+ this_camera_mediapipe_2d_single_camera_npy_arrays = (
+ self._list_of_mediapipe_results_to_npy_arrays(
+ this_video_mediapipe_results_list,
+ image_width=this_video_width,
+ image_height=this_video_height,
+ )
+ )
+
+ mediapipe2d_single_camera_npy_arrays_list.append(
+ this_camera_mediapipe_2d_single_camera_npy_arrays
+ )
+
+ all_cameras_data2d_list = [
+ m2d.all_data2d_nFrames_nTrackedPts_XY
+ for m2d in mediapipe2d_single_camera_npy_arrays_list
+ ]
number_of_cameras = len(all_cameras_data2d_list)
number_of_frames = all_cameras_data2d_list[0].shape[0]
number_of_tracked_points = all_cameras_data2d_list[0].shape[1]
- number_of_spatial_dimensions = all_cameras_data2d_list[0].shape[2] # XY, 2d data
+ number_of_spatial_dimensions = all_cameras_data2d_list[0].shape[
+ 2
+ ] # XY, 2d data
if not number_of_spatial_dimensions == 2:
- logger.error(f"this should be 2D data (XY pixel coordinates), but we founds {number_of_spatial_dimensions}")
+ logger.error(
+ f"this should be 2D data (XY pixel coordinates), but we founds {number_of_spatial_dimensions}"
+ )
raise Exception
- data2d_numCams_numFrames_numTrackedPts_XY = np.empty((number_of_cameras,
- number_of_frames,
- number_of_tracked_points,
- number_of_spatial_dimensions))
+ data2d_numCams_numFrames_numTrackedPts_XY = np.empty(
+ (
+ number_of_cameras,
+ number_of_frames,
+ number_of_tracked_points,
+ number_of_spatial_dimensions,
+ )
+ )
for this_cam_num in range(number_of_cameras):
- data2d_numCams_numFrames_numTrackedPts_XY[this_cam_num, :, :, :] = all_cameras_data2d_list[this_cam_num]
+ data2d_numCams_numFrames_numTrackedPts_XY[
+ this_cam_num, :, :, :
+ ] = all_cameras_data2d_list[this_cam_num]
self._save_mediapipe2d_data_to_npy(data2d_numCams_numFrames_numTrackedPts_XY)
return data2d_numCams_numFrames_numTrackedPts_XY
def _save_mediapipe2d_data_to_npy(self, data2d_numCams_numFrames_numTrackedPts_XY):
output_data_folder = Path(get_session_output_data_folder_path(self._session_id))
- mediapipe_2dData_save_path = output_data_folder / "mediapipe_2dData_numCams_numFrames_numTrackedPoints_pixelXY.npy"
+ mediapipe_2dData_save_path = (
+ output_data_folder
+ / "mediapipe_2dData_numCams_numFrames_numTrackedPoints_pixelXY.npy"
+ )
logger.info(f"saving: {mediapipe_2dData_save_path}")
- np.save(str(mediapipe_2dData_save_path), data2d_numCams_numFrames_numTrackedPts_XY)
-
- def save_annotated_videos(self,
- annotated_images_list: List[np.ndarray],
- video_file_name: str,
- image_width: Union[int, float],
- image_height: Union[int, float],
- ):
+ np.save(
+ str(mediapipe_2dData_save_path), data2d_numCams_numFrames_numTrackedPts_XY
+ )
+
+ def save_annotated_videos(
+ self,
+ annotated_images_list: List[np.ndarray],
+ video_file_name: str,
+ image_width: Union[int, float],
+ image_height: Union[int, float],
+ ):
this_video_name = video_file_name + "_mediapipe.mp4"
- video_recorder = VideoRecorder(this_video_name,
- image_width,
- image_height,
- self._session_id,
- mediapipe_annotated_video_bool=True)
-
- logger.info(f'Saving mediapipe annotated video: {this_video_name}')
- video_recorder.save_image_list_to_disk(annotated_images_list, frames_per_second=30)
- logger.info(f'mediapipe annotated video saved to: {video_recorder.path_to_save_video_file}')
+ video_recorder = VideoRecorder(
+ this_video_name,
+ image_width,
+ image_height,
+ self._session_id,
+ mediapipe_annotated_video_bool=True,
+ )
+
+ logger.info(f"Saving mediapipe annotated video: {this_video_name}")
+ video_recorder.save_image_list_to_disk(
+ annotated_images_list, frames_per_second=30
+ )
+ logger.info(
+ f"mediapipe annotated video saved to: {video_recorder.path_to_save_video_file}"
+ )
def _annotate_image(self, image, mediapipe_results):
- self._mp_drawing.draw_landmarks(image=image,
- landmark_list=mediapipe_results.face_landmarks,
- connections=self._mp_holistic.FACEMESH_CONTOURS,
- landmark_drawing_spec=None,
- connection_drawing_spec=self._mp_drawing_styles
- .get_default_face_mesh_contours_style())
- self._mp_drawing.draw_landmarks(image=image,
- landmark_list=mediapipe_results.face_landmarks,
- connections=self._mp_holistic.FACEMESH_TESSELATION,
- landmark_drawing_spec=None,
- connection_drawing_spec=self._mp_drawing_styles
- .get_default_face_mesh_tesselation_style())
- self._mp_drawing.draw_landmarks(image=image,
- landmark_list=mediapipe_results.pose_landmarks,
- connections=self._mp_holistic.POSE_CONNECTIONS,
- landmark_drawing_spec=self._mp_drawing_styles
- .get_default_pose_landmarks_style())
+ self._mp_drawing.draw_landmarks(
+ image=image,
+ landmark_list=mediapipe_results.face_landmarks,
+ connections=self._mp_holistic.FACEMESH_CONTOURS,
+ landmark_drawing_spec=None,
+ connection_drawing_spec=self._mp_drawing_styles.get_default_face_mesh_contours_style(),
+ )
+ self._mp_drawing.draw_landmarks(
+ image=image,
+ landmark_list=mediapipe_results.face_landmarks,
+ connections=self._mp_holistic.FACEMESH_TESSELATION,
+ landmark_drawing_spec=None,
+ connection_drawing_spec=self._mp_drawing_styles.get_default_face_mesh_tesselation_style(),
+ )
+ self._mp_drawing.draw_landmarks(
+ image=image,
+ landmark_list=mediapipe_results.pose_landmarks,
+ connections=self._mp_holistic.POSE_CONNECTIONS,
+ landmark_drawing_spec=self._mp_drawing_styles.get_default_pose_landmarks_style(),
+ )
self._mp_drawing.draw_landmarks(
image=image,
landmark_list=mediapipe_results.left_hand_landmarks,
connections=self._mp_holistic.HAND_CONNECTIONS,
landmark_drawing_spec=None,
- connection_drawing_spec=self._mp_drawing_styles
- .get_default_hand_connections_style())
+ connection_drawing_spec=self._mp_drawing_styles.get_default_hand_connections_style(),
+ )
self._mp_drawing.draw_landmarks(
image=image,
landmark_list=mediapipe_results.right_hand_landmarks,
connections=self._mp_holistic.HAND_CONNECTIONS,
landmark_drawing_spec=None,
- connection_drawing_spec=self._mp_drawing_styles
- .get_default_hand_connections_style())
+ connection_drawing_spec=self._mp_drawing_styles.get_default_hand_connections_style(),
+ )
return image
- def _list_of_mediapipe_results_to_npy_arrays(self,
- mediapipe_results_list: List,
- image_width: Union[int, float],
- image_height: Union[int, float],
- ) -> Mediapipe2dNumpyArrays:
+ def _list_of_mediapipe_results_to_npy_arrays(
+ self,
+ mediapipe_results_list: List,
+ image_width: Union[int, float],
+ image_height: Union[int, float],
+ ) -> Mediapipe2dNumpyArrays:
# apparently `mediapipe_results.pose_landmarks.landmark` returns something iterable Β―\_(ツ)_/Β―
mediapipe_pose_landmark_iterator = mp.python.solutions.pose.PoseLandmark
@@ -280,26 +368,46 @@ class MediaPipeSkeletonDetector:
number_of_spatial_dimensions = 2 # this will be 2d XY pixel data
body2d_frameNumber_trackedPointNumber_XY = np.zeros(
- (number_of_frames, self.number_of_body_tracked_points, number_of_spatial_dimensions))
+ (
+ number_of_frames,
+ self.number_of_body_tracked_points,
+ number_of_spatial_dimensions,
+ )
+ )
body2d_frameNumber_trackedPointNumber_XY[:] = np.nan
body2d_frameNumber_trackedPointNumber_confidence = np.zeros(
- (number_of_frames, self.number_of_body_tracked_points))
- body2d_frameNumber_trackedPointNumber_confidence[:] = np.nan # only body markers get a 'confidence' value
-
- rightHand2d_frameNumber_trackedPointNumber_XY = np.zeros((number_of_frames,
- self.number_of_right_hand_tracked_points,
- number_of_spatial_dimensions))
+ (number_of_frames, self.number_of_body_tracked_points)
+ )
+ body2d_frameNumber_trackedPointNumber_confidence[
+ :
+ ] = np.nan # only body markers get a 'confidence' value
+
+ rightHand2d_frameNumber_trackedPointNumber_XY = np.zeros(
+ (
+ number_of_frames,
+ self.number_of_right_hand_tracked_points,
+ number_of_spatial_dimensions,
+ )
+ )
rightHand2d_frameNumber_trackedPointNumber_XY[:] = np.nan
- leftHand2d_frameNumber_trackedPointNumber_XY = np.zeros((number_of_frames,
- self.number_of_left_hand_tracked_points,
- number_of_spatial_dimensions))
+ leftHand2d_frameNumber_trackedPointNumber_XY = np.zeros(
+ (
+ number_of_frames,
+ self.number_of_left_hand_tracked_points,
+ number_of_spatial_dimensions,
+ )
+ )
leftHand2d_frameNumber_trackedPointNumber_XY[:] = np.nan
- face2d_frameNumber_trackedPointNumber_XY = np.zeros((number_of_frames,
- self.number_of_face_tracked_points,
- number_of_spatial_dimensions))
+ face2d_frameNumber_trackedPointNumber_XY = np.zeros(
+ (
+ number_of_frames,
+ self.number_of_face_tracked_points,
+ number_of_spatial_dimensions,
+ )
+ )
face2d_frameNumber_trackedPointNumber_XY[:] = np.nan
all_body_tracked_points_visible_on_this_frame_bool_list = []
@@ -313,83 +421,147 @@ class MediaPipeSkeletonDetector:
# get the Body data (aka 'pose')
if this_frame_results.pose_landmarks is not None:
- for this_landmark_number, this_landmark_data in enumerate(this_frame_results.pose_landmarks.landmark):
- body2d_frameNumber_trackedPointNumber_XY[this_frame_number,
- this_landmark_number,
- 0] = this_landmark_data.x * image_width
- body2d_frameNumber_trackedPointNumber_XY[this_frame_number,
- this_landmark_number,
- 1] = this_landmark_data.y * image_height
- body2d_frameNumber_trackedPointNumber_confidence[this_frame_number,
- this_landmark_number] = this_landmark_data.visibility # mediapipe calls their 'confidence' score 'visibility'
+ for this_landmark_number, this_landmark_data in enumerate(
+ this_frame_results.pose_landmarks.landmark
+ ):
+ body2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, this_landmark_number, 0
+ ] = (this_landmark_data.x * image_width)
+ body2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, this_landmark_number, 1
+ ] = (this_landmark_data.y * image_height)
+ body2d_frameNumber_trackedPointNumber_confidence[
+ this_frame_number, this_landmark_number
+ ] = (
+ this_landmark_data.visibility
+ ) # mediapipe calls their 'confidence' score 'visibility'
# get Right Hand data
if this_frame_results.right_hand_landmarks is not None:
for this_landmark_number, this_landmark_data in enumerate(
- this_frame_results.right_hand_landmarks.landmark):
- rightHand2d_frameNumber_trackedPointNumber_XY[this_frame_number,
- this_landmark_number,
- 0] = this_landmark_data.x * image_width
- rightHand2d_frameNumber_trackedPointNumber_XY[this_frame_number,
- this_landmark_number,
- 1] = this_landmark_data.y * image_height
+ this_frame_results.right_hand_landmarks.landmark
+ ):
+ rightHand2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, this_landmark_number, 0
+ ] = (this_landmark_data.x * image_width)
+ rightHand2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, this_landmark_number, 1
+ ] = (this_landmark_data.y * image_height)
# get Left Hand data
if this_frame_results.left_hand_landmarks is not None:
for this_landmark_number, this_landmark_data in enumerate(
- this_frame_results.left_hand_landmarks.landmark):
- leftHand2d_frameNumber_trackedPointNumber_XY[this_frame_number,
- this_landmark_number,
- 0] = this_landmark_data.x * image_width
- leftHand2d_frameNumber_trackedPointNumber_XY[this_frame_number,
- this_landmark_number,
- 1] = this_landmark_data.y * image_height
+ this_frame_results.left_hand_landmarks.landmark
+ ):
+ leftHand2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, this_landmark_number, 0
+ ] = (this_landmark_data.x * image_width)
+ leftHand2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, this_landmark_number, 1
+ ] = (this_landmark_data.y * image_height)
# get Face data
if this_frame_results.face_landmarks is not None:
- for this_landmark_number, this_landmark_data in enumerate(this_frame_results.face_landmarks.landmark):
- face2d_frameNumber_trackedPointNumber_XY[this_frame_number, this_landmark_number,
- 0] = this_landmark_data.x * image_width
- face2d_frameNumber_trackedPointNumber_XY[this_frame_number, this_landmark_number,
- 1] = this_landmark_data.y * image_height
+ for this_landmark_number, this_landmark_data in enumerate(
+ this_frame_results.face_landmarks.landmark
+ ):
+ face2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, this_landmark_number, 0
+ ] = (this_landmark_data.x * image_width)
+ face2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, this_landmark_number, 1
+ ] = (this_landmark_data.y * image_height)
# check if all tracked points are visible on this frame
- all_body_visible = all(sum(
- np.isnan(body2d_frameNumber_trackedPointNumber_XY[this_frame_number, :, :])) == 0)
- all_body_tracked_points_visible_on_this_frame_bool_list.append(all_body_visible)
-
- all_right_hand_visible = all(sum(
- np.isnan(rightHand2d_frameNumber_trackedPointNumber_XY[this_frame_number, :, :])) == 0)
- all_right_hand_points_visible_on_this_frame_bool_list.append(all_right_hand_visible)
-
- all_left_hand_visible = all(sum(
- np.isnan(leftHand2d_frameNumber_trackedPointNumber_XY[this_frame_number, :, :])) == 0)
- all_left_hand_points_visible_on_this_frame_bool_list.append(all_left_hand_visible)
-
- all_face_visible = all(sum(
- np.isnan(face2d_frameNumber_trackedPointNumber_XY[this_frame_number, :, :])) == 0)
+ all_body_visible = all(
+ sum(
+ np.isnan(
+ body2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, :, :
+ ]
+ )
+ )
+ == 0
+ )
+ all_body_tracked_points_visible_on_this_frame_bool_list.append(
+ all_body_visible
+ )
+
+ all_right_hand_visible = all(
+ sum(
+ np.isnan(
+ rightHand2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, :, :
+ ]
+ )
+ )
+ == 0
+ )
+ all_right_hand_points_visible_on_this_frame_bool_list.append(
+ all_right_hand_visible
+ )
+
+ all_left_hand_visible = all(
+ sum(
+ np.isnan(
+ leftHand2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, :, :
+ ]
+ )
+ )
+ == 0
+ )
+ all_left_hand_points_visible_on_this_frame_bool_list.append(
+ all_left_hand_visible
+ )
+
+ all_face_visible = all(
+ sum(
+ np.isnan(
+ face2d_frameNumber_trackedPointNumber_XY[
+ this_frame_number, :, :
+ ]
+ )
+ )
+ == 0
+ )
all_face_points_visible_on_this_frame_bool_list.append(all_face_visible)
- all_points_visible = all([all_body_visible,
- all_right_hand_visible,
- all_left_hand_visible,
- all_face_visible],
- )
+ all_points_visible = all(
+ [
+ all_body_visible,
+ all_right_hand_visible,
+ all_left_hand_visible,
+ all_face_visible,
+ ],
+ )
all_tracked_points_visible_on_this_frame_list.append(all_points_visible)
return Mediapipe2dNumpyArrays(
- body2d_frameNumber_trackedPointNumber_XY=np.squeeze(body2d_frameNumber_trackedPointNumber_XY),
- rightHand2d_frameNumber_trackedPointNumber_XY=np.squeeze(rightHand2d_frameNumber_trackedPointNumber_XY),
- leftHand2d_frameNumber_trackedPointNumber_XY=np.squeeze(leftHand2d_frameNumber_trackedPointNumber_XY),
- face2d_frameNumber_trackedPointNumber_XY=np.squeeze(face2d_frameNumber_trackedPointNumber_XY),
+ body2d_frameNumber_trackedPointNumber_XY=np.squeeze(
+ body2d_frameNumber_trackedPointNumber_XY
+ ),
+ rightHand2d_frameNumber_trackedPointNumber_XY=np.squeeze(
+ rightHand2d_frameNumber_trackedPointNumber_XY
+ ),
+ leftHand2d_frameNumber_trackedPointNumber_XY=np.squeeze(
+ leftHand2d_frameNumber_trackedPointNumber_XY
+ ),
+ face2d_frameNumber_trackedPointNumber_XY=np.squeeze(
+ face2d_frameNumber_trackedPointNumber_XY
+ ),
body2d_frameNumber_trackedPointNumber_confidence=np.squeeze(
- body2d_frameNumber_trackedPointNumber_confidence))
-
- def _threshold_by_confidence(self,
- data2d_trackedPoint_dim: np.ndarray,
- data2d_trackedPoint_confidence: np.ndarray,
- confidence_threshold: float):
+ body2d_frameNumber_trackedPointNumber_confidence
+ ),
+ )
+
+ def _threshold_by_confidence(
+ self,
+ data2d_trackedPoint_dim: np.ndarray,
+ data2d_trackedPoint_confidence: np.ndarray,
+ confidence_threshold: float,
+ ):
threshold_mask = data2d_trackedPoint_confidence < confidence_threshold
data2d_trackedPoint_dim[threshold_mask, :] = np.nan
diff --git a/src/core_processor/show_cam_window.py b/src/core_processor/show_cam_window.py
index 8a98c2ff..b89a97ec 100644
--- a/src/core_processor/show_cam_window.py
+++ b/src/core_processor/show_cam_window.py
@@ -9,6 +9,9 @@ from src.core_processor.utils.image_fps_writer import write_fps_to_image
logger = logging.getLogger(__name__)
-def show_cam_window(webcam_id: str, image: np.array,):
+def show_cam_window(
+ webcam_id: str,
+ image: np.array,
+):
cv2.imshow(webcam_id, image)
return True
diff --git a/src/core_processor/timestamp_manager/timestamp_manager.py b/src/core_processor/timestamp_manager/timestamp_manager.py
index 76760242..e05b6ae9 100644
--- a/src/core_processor/timestamp_manager/timestamp_manager.py
+++ b/src/core_processor/timestamp_manager/timestamp_manager.py
@@ -14,17 +14,17 @@ logger = logging.getLogger(__name__)
class TimestampLogger:
- def __init__(self,
- logger_name:str,
- session_start_time_perf_counter_ns:int,
- ):
+ def __init__(
+ self,
+ logger_name: str,
+ session_start_time_perf_counter_ns: int,
+ ):
self._logger_name = logger_name
self._session_start_time_perf_counter_ns = session_start_time_perf_counter_ns
self._num_frames_processed = 0
self._timestamps_perf_counter_ns = np.empty(0)
self._timestamps_from_zero = np.empty(0)
-
@property
def median_frames_per_second(self):
if self._num_frames_processed <= 10:
@@ -38,7 +38,9 @@ class TimestampLogger:
@property
def timestamps_in_seconds_from_session_start(self):
"""timestamps in seconds counting up from zero (where zero is the start time of the recording session)"""
- return (self._timestamps_perf_counter_ns - self._session_start_time_perf_counter_ns) / 1e9
+ return (
+ self._timestamps_perf_counter_ns - self._session_start_time_perf_counter_ns
+ ) / 1e9
@property
def latest_timestamp_in_seconds_from_record_start(self):
@@ -49,12 +51,13 @@ class TimestampLogger:
except:
logger.error(f"Failed to return latest timestamp")
-
- def log_new_timestamp_perf_counter_ns(self, timestamp:int):
- self._timestamps_perf_counter_ns = np.append(self._timestamps_perf_counter_ns, timestamp)
+ def log_new_timestamp_perf_counter_ns(self, timestamp: int):
+ self._timestamps_perf_counter_ns = np.append(
+ self._timestamps_perf_counter_ns, timestamp
+ )
self._num_frames_processed += 1
- def log_new_timestamp_seconds_from_unspecified_zero(self, timestamp:float):
+ def log_new_timestamp_seconds_from_unspecified_zero(self, timestamp: float):
self._timestamps_from_zero = np.append(self._timestamps_from_zero, timestamp)
self._num_frames_processed += 1
@@ -62,74 +65,89 @@ class TimestampLogger:
def timestamps_in_seconds_from_unspecified_zero(self):
return self._timestamps_from_zero
+
class TimestampManager:
- def __init__(self,
- session_id:str,
- webcam_ids: List[str],
- session_start_time_unix_ns: int,
- _session_start_time_perf_counter_ns:int):
+ def __init__(
+ self,
+ session_id: str,
+ webcam_ids: List[str],
+ session_start_time_unix_ns: int,
+ _session_start_time_perf_counter_ns: int,
+ ):
self._session_id = session_id
self._session_start_time_unix_ns = session_start_time_unix_ns
self._session_start_time_perf_counter_ns = _session_start_time_perf_counter_ns
self._webcam_ids = webcam_ids
- self._webcam_timestamp_loggers: Dict[str, TimestampLogger] = self._initialize_webcam_timestamp_loggers(webcam_ids)
- self._main_loop_timestamp_logger = TimestampLogger(logger_name = 'main_loop',
- session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns)
-
- self._multi_frame_timestamp_logger = TimestampLogger(logger_name = 'multi_frame',
- session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns)
-
+ self._webcam_timestamp_loggers: Dict[
+ str, TimestampLogger
+ ] = self._initialize_webcam_timestamp_loggers(webcam_ids)
+ self._main_loop_timestamp_logger = TimestampLogger(
+ logger_name="main_loop",
+ session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns,
+ )
+ self._multi_frame_timestamp_logger = TimestampLogger(
+ logger_name="multi_frame",
+ session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns,
+ )
- def timestamp_logger_for_webcam_id(self, webcam_id:str)->TimestampLogger:
+ def timestamp_logger_for_webcam_id(self, webcam_id: str) -> TimestampLogger:
return self._webcam_timestamp_loggers[webcam_id]
-
@property
def multi_frame_timestamp_logger(self):
return self._multi_frame_timestamp_logger
-
-
-
def _initialize_webcam_timestamp_loggers(self, webcam_ids: List[str]):
dictionary_of_timestamp_logger_objects = {}
for webcam_id in webcam_ids:
- dictionary_of_timestamp_logger_objects[webcam_id] = TimestampLogger(logger_name = 'camera_'+webcam_id,
- session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns)
+ dictionary_of_timestamp_logger_objects[webcam_id] = TimestampLogger(
+ logger_name="camera_" + webcam_id,
+ session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns,
+ )
return dictionary_of_timestamp_logger_objects
-
-
def _get_each_frame_timestamp(self, this_multi_frame_dict) -> List:
this_multi_frame_timestamps = []
for this_cam_id, this_cam_frame in this_multi_frame_dict.items():
- this_multi_frame_timestamps.append(this_cam_frame.timestamp_in_seconds_from_record_start)
+ this_multi_frame_timestamps.append(
+ this_cam_frame.timestamp_in_seconds_from_record_start
+ )
return this_multi_frame_timestamps
- def _get_intra_frame_interval_in_seconds(self, this_multi_frame_timestamps_in_seconds) -> float:
+ def _get_intra_frame_interval_in_seconds(
+ self, this_multi_frame_timestamps_in_seconds
+ ) -> float:
timestamp_min = np.min(this_multi_frame_timestamps_in_seconds)
timestamp_max = np.max(this_multi_frame_timestamps_in_seconds)
- return (timestamp_max - timestamp_min)
-
- def verify_multi_frame_is_synchronized(self,
- this_multi_frame_dict,) -> bool:
-
- this_multi_frame_timestamps = self._get_each_frame_timestamp(this_multi_frame_dict)
- this_multiframe_timestamp_interval_in_seconds = self._get_intra_frame_interval_in_seconds(this_multi_frame_timestamps)
+ return timestamp_max - timestamp_min
+ def verify_multi_frame_is_synchronized(
+ self,
+ this_multi_frame_dict,
+ ) -> bool:
+ this_multi_frame_timestamps = self._get_each_frame_timestamp(
+ this_multi_frame_dict
+ )
+ this_multiframe_timestamp_interval_in_seconds = (
+ self._get_intra_frame_interval_in_seconds(this_multi_frame_timestamps)
+ )
- print(f"this_multiframe_timestamp_interval: {this_multiframe_timestamp_interval_in_seconds*1e3:.6f} milliseconds")
+ print(
+ f"this_multiframe_timestamp_interval: {this_multiframe_timestamp_interval_in_seconds*1e3:.6f} milliseconds"
+ )
return True
def create_diagnostic_plots(self):
"""plot some diagnostics to assess quality of camera sync"""
- multi_frame_timestamp_list = self.multi_frame_timestamp_logger.timestamps_in_seconds_from_unspecified_zero
+ multi_frame_timestamp_list = (
+ self.multi_frame_timestamp_logger.timestamps_in_seconds_from_unspecified_zero
+ )
# multi_frame_interval_list = self.multi_frame_interval_list
fig = plt.figure(figsize=(18, 10))
@@ -175,7 +193,10 @@ class TimestampManager:
ylabel="Probability",
)
- for this_camera_id, this_camera_timestamp_logger in self._webcam_timestamp_loggers.items():
+ for (
+ this_camera_id,
+ this_camera_timestamp_logger,
+ ) in self._webcam_timestamp_loggers.items():
this_camera_timestamps = this_camera_timestamp_logger._timestamps_from_zero
ax1.plot(this_camera_timestamps, label=f"Camera# {this_camera_id}")
ax1.legend()
@@ -186,11 +207,7 @@ class TimestampManager:
alpha=0.5,
)
- ax4.plot(
- multi_frame_timestamp_list,
- color="darkslategrey",
- label=f"MultiFrame"
- )
+ ax4.plot(multi_frame_timestamp_list, color="darkslategrey", label=f"MultiFrame")
ax5.plot(
np.diff(np.asarray(multi_frame_timestamp_list)),
".",
@@ -219,7 +236,10 @@ class TimestampManager:
# )
ax5.legend()
- fig_save_path = Path(get_session_folder_path(self._session_id)) / "camera_timestamp_diagnostics.png"
+ fig_save_path = (
+ Path(get_session_folder_path(self._session_id))
+ / "camera_timestamp_diagnostics.png"
+ )
plt.savefig(str(fig_save_path))
logger.info(f"Saving diagnostic figure to - {str(fig_save_path)}")
# plt.show()
diff --git a/src/core_processor/utils/image_fps_writer.py b/src/core_processor/utils/image_fps_writer.py
index 8e926a44..e60c83fe 100644
--- a/src/core_processor/utils/image_fps_writer.py
+++ b/src/core_processor/utils/image_fps_writer.py
@@ -4,7 +4,7 @@ import cv2
import numpy as np
-def write_fps_to_image(image: np.array, fps_number: Union[int,float])->np.ndarray:
+def write_fps_to_image(image: np.array, fps_number: Union[int, float]) -> np.ndarray:
annotated_image = image.copy()
cv2.putText(
annotated_image,
diff --git a/src/debug_plot_makers/simple_plot_3d_points.py b/src/debug_plot_makers/simple_plot_3d_points.py
index 7ead59fd..006e9b3b 100644
--- a/src/debug_plot_makers/simple_plot_3d_points.py
+++ b/src/debug_plot_makers/simple_plot_3d_points.py
@@ -8,8 +8,8 @@ def simple_plot_3d(data_pt_xyz):
z = data_pt_xyz[:, 2]
fig = plt.figure()
- matplotlib.use('qtagg')
- ax3d = fig.add_subplot(projection='3d')
+ matplotlib.use("qtagg")
+ ax3d = fig.add_subplot(projection="3d")
ax3d.scatter(x, y, z)
- plt.pause(.1)
+ plt.pause(0.1)
plt.show()
diff --git a/src/experiments/cv_cam_multi_process.py b/src/experiments/cv_cam_multi_process.py
index e7b1ce89..69e8a89b 100644
--- a/src/experiments/cv_cam_multi_process.py
+++ b/src/experiments/cv_cam_multi_process.py
@@ -18,7 +18,6 @@ logger = logging.getLogger(__name__)
class CvCamByProcess:
-
def __init__(self, config: WebcamConfig = WebcamConfig()):
self._process: Process = None
self._recv, self._send = multiprocessing.Pipe(duplex=False)
@@ -39,10 +38,7 @@ class CvCamByProcess:
def begin_frame_capture(self):
if not self._is_capturing_frames:
self._is_capturing_frames = True
- self._process = Process(
- target=self._start_frame_loop,
- args=(self._send,)
- )
+ self._process = Process(target=self._start_frame_loop, args=(self._send,))
self._process.start()
def end_frame_capture(self):
@@ -56,9 +52,7 @@ class CvCamByProcess:
else:
cap_backend = cv2.CAP_ANY
- opencv_video_capture_object = cv2.VideoCapture(
- config.webcam_id, cap_backend
- )
+ opencv_video_capture_object = cv2.VideoCapture(config.webcam_id, cap_backend)
# self._apply_configuration()
if not self._test_cap_connection(opencv_video_capture_object):
raise Exception("Could not find camera")
@@ -108,11 +102,4 @@ class CvCamByProcess:
self._new_frame_ready = success
- return FramePayload(success=success,
- image=image,
- timestamp=timestamp_ns)
-
-
-
-
-
+ return FramePayload(success=success, image=image, timestamp=timestamp_ns)
diff --git a/src/gui/examples/main.py b/src/gui/examples/main.py
index 05de0631..1db19fe7 100644
--- a/src/gui/examples/main.py
+++ b/src/gui/examples/main.py
@@ -1,17 +1,13 @@
-
import sys
-from PyQt6.QtWidgets import (QWidget, QPushButton,
- QHBoxLayout, QVBoxLayout, QApplication)
+from PyQt6.QtWidgets import QWidget, QPushButton, QHBoxLayout, QVBoxLayout, QApplication
class Example(QWidget):
-
def __init__(self):
super().__init__()
self.initUI()
-
def initUI(self):
okButton = QPushButton("OK")
@@ -29,7 +25,7 @@ class Example(QWidget):
self.setLayout(vbox)
self.setGeometry(300, 300, 350, 250)
- self.setWindowTitle('Buttons')
+ self.setWindowTitle("Buttons")
self.show()
@@ -40,5 +36,5 @@ def main():
sys.exit(app.exec())
-if __name__ == '__main__':
- main()
\ No newline at end of file
+if __name__ == "__main__":
+ main()
diff --git a/src/gui/main/layout/main_window.py b/src/gui/main/layout/main_window.py
index 33780a7e..332939ef 100644
--- a/src/gui/main/layout/main_window.py
+++ b/src/gui/main/layout/main_window.py
@@ -1,16 +1,13 @@
from PyQt6.QtCore import Qt
from PyQt6.QtGui import QAction
-from PyQt6.QtWidgets import QMainWindow, QMenuBar, QVBoxLayout, \
- QWidget, QCheckBox
+from PyQt6.QtWidgets import QMainWindow, QMenuBar, QVBoxLayout, QWidget, QCheckBox
from src.gui.main.app import get_qt_app
from src.gui.main.qt_utils.clear_layout import clearLayout
from src.gui.main.state.app_state import APP_STATE
-from src.gui.main.workflows.calibration_instructions import \
- CalibrationInstructions
+from src.gui.main.workflows.calibration_instructions import CalibrationInstructions
from src.gui.main.workflows.camera_configuration import CameraConfiguration
-from src.gui.main.workflows.new_recording_session import \
- NewRecordingSession
+from src.gui.main.workflows.new_recording_session import NewRecordingSession
from src.gui.main.workflows.record_videos import RecordVideos
from src.gui.main.workflows.show_cams_charuco import ShowCamsCharuco
from src.gui.main.workflows.welcome import Welcome
@@ -19,7 +16,7 @@ from src.gui.main.workflows.welcome import Welcome
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
- self.setWindowTitle('freemocap')
+ self.setWindowTitle("freemocap")
self.setGeometry(0, 0, 800, 600)
self._create_menu_bar()
self.statusBar()
@@ -27,7 +24,6 @@ class MainWindow(QMainWindow):
self._show_welcome_screen()
-
def _create_basic_layout(self):
main_layout = QVBoxLayout()
widget = QWidget()
@@ -43,15 +39,15 @@ class MainWindow(QMainWindow):
def _add_file_menu(self, menubar: QMenuBar):
exit_action = self._create_exit_action()
record_action = self._create_new_recording_session_action()
- file_menu = menubar.addMenu('&File')
+ file_menu = menubar.addMenu("&File")
file_menu.addAction(record_action)
file_menu.addAction(exit_action)
def _create_exit_action(self):
# src/freemocap_qt_gui/conference/assets/new_recording_session.png
- action = QAction('&Quit', self)
- action.setShortcut('Ctrl+Q')
- action.setStatusTip('Exit application')
+ action = QAction("&Quit", self)
+ action.setShortcut("Ctrl+Q")
+ action.setStatusTip("Exit application")
# On triggered, quit the application
action.triggered.connect(get_qt_app().quit)
@@ -59,9 +55,9 @@ class MainWindow(QMainWindow):
return action
def _create_new_recording_session_action(self):
- action = QAction('&New Recording Session', self)
- action.setShortcut('Ctrl+N')
- action.setStatusTip('Begin a new recording session')
+ action = QAction("&New Recording Session", self)
+ action.setShortcut("Ctrl+N")
+ action.setStatusTip("Begin a new recording session")
# On triggered, quit the application
action.triggered.connect(self._show_recording_session_screen)
@@ -94,7 +90,6 @@ class MainWindow(QMainWindow):
else:
screen.config_accepted.clicked.connect(self._show_calibration_screen)
-
def _show_calibration_instructions_screen(self):
clearLayout(self._main_layout)
screen = CalibrationInstructions()
@@ -114,4 +109,3 @@ class MainWindow(QMainWindow):
screen = RecordVideos()
self._main_layout.addWidget(screen)
self._main_layout.setAlignment(Qt.AlignmentFlag.AlignTop)
-
diff --git a/src/gui/main/shared_widgets/primary_button.py b/src/gui/main/shared_widgets/primary_button.py
index 29ab6864..ad60b23f 100644
--- a/src/gui/main/shared_widgets/primary_button.py
+++ b/src/gui/main/shared_widgets/primary_button.py
@@ -1,7 +1,6 @@
from PyQt6.QtWidgets import QPushButton
-
active_primary_button_style_sheet = """
QPushButton {
background-color: #336e6d;
@@ -32,8 +31,8 @@ not_active_primary_button_style_sheet = """
}
"""
+
class PrimaryButton(QPushButton):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.setStyleSheet(active_primary_button_style_sheet)
-
diff --git a/src/gui/main/state/app_state.py b/src/gui/main/state/app_state.py
index afdcf39f..2c057e94 100644
--- a/src/gui/main/state/app_state.py
+++ b/src/gui/main/state/app_state.py
@@ -6,7 +6,7 @@ from pydantic import BaseModel
class AppState(BaseModel):
session_id: str = None
selected_cameras: List[str] = []
- use_previous_calibration:bool = False
+ use_previous_calibration: bool = False
number_of_frames_in_the_mocap_videos: int = None
diff --git a/src/gui/main/workers/cam_charuco_frame_worker.py b/src/gui/main/workers/cam_charuco_frame_worker.py
index 70cf908e..da0dc0b3 100644
--- a/src/gui/main/workers/cam_charuco_frame_worker.py
+++ b/src/gui/main/workers/cam_charuco_frame_worker.py
@@ -5,14 +5,17 @@ from PyQt6.QtGui import QImage
from src.cameras.capture.opencv_camera.opencv_camera import OpenCVCamera
from src.config.webcam_config import WebcamConfig
from src.gui.main.state.app_state import APP_STATE
-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import \
- CharucoBoardDetector
+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (
+ CharucoBoardDetector,
+)
class CamCharucoFrameWorker(QThread):
ImageUpdate = pyqtSignal(QImage)
- def __init__(self, cam_id=None, should_save_frames:bool=False, calibration_videos=False):
+ def __init__(
+ self, cam_id=None, should_save_frames: bool = False, calibration_videos=False
+ ):
super().__init__()
self._board_thingy = CharucoBoardDetector()
self._cam_id = cam_id
@@ -33,9 +36,7 @@ class CamCharucoFrameWorker(QThread):
def run(self):
cam = OpenCVCamera(
- WebcamConfig(
- webcam_id=self._cam_id
- ),
+ WebcamConfig(webcam_id=self._cam_id),
session_id=APP_STATE.session_id,
)
cam.connect()
@@ -62,12 +63,16 @@ class CamCharucoFrameWorker(QThread):
image_to_display.data,
image_to_display.shape[1],
image_to_display.shape[0],
- QImage.Format.Format_RGB888
+ QImage.Format.Format_RGB888,
+ )
+ converted_frame = converted_frame.scaled(
+ 640, 480, Qt.AspectRatioMode.KeepAspectRatio
)
- converted_frame = converted_frame.scaled(640, 480, Qt.AspectRatioMode.KeepAspectRatio)
self.ImageUpdate.emit(converted_frame)
finally:
- print(f"Closing the camera {self._cam.webcam_id_as_str}, and saving video to disk")
+ print(
+ f"Closing the camera {self._cam.webcam_id_as_str}, and saving video to disk"
+ )
self._cam.close()
# if any_frames_recorded:
# print(f"saving video for camera {self._cam.webcam_id_as_str}")
diff --git a/src/gui/main/workers/cam_frame_worker.py b/src/gui/main/workers/cam_frame_worker.py
index f1b8e5bb..60ec9857 100644
--- a/src/gui/main/workers/cam_frame_worker.py
+++ b/src/gui/main/workers/cam_frame_worker.py
@@ -14,11 +14,7 @@ class CamFrameWorker(QThread):
self._cam_id = cam_id
def run(self):
- cam = OpenCVCamera(
- WebcamConfig(
- webcam_id=self._cam_id
- )
- )
+ cam = OpenCVCamera(WebcamConfig(webcam_id=self._cam_id))
cam.connect()
cam.start_frame_capture_thread()
self._cam = cam
@@ -34,9 +30,11 @@ class CamFrameWorker(QThread):
image.data,
image.shape[1],
image.shape[0],
- QImage.Format.Format_RGB888
+ QImage.Format.Format_RGB888,
+ )
+ converted_frame = converted_frame.scaled(
+ 640, 480, Qt.AspectRatioMode.KeepAspectRatio
)
- converted_frame = converted_frame.scaled(640, 480, Qt.AspectRatioMode.KeepAspectRatio)
self.ImageUpdate.emit(converted_frame)
finally:
print("Closing the camera")
diff --git a/src/gui/main/workflows/available_camera.py b/src/gui/main/workflows/available_camera.py
index e275a4df..ce8ba19c 100644
--- a/src/gui/main/workflows/available_camera.py
+++ b/src/gui/main/workflows/available_camera.py
@@ -2,7 +2,6 @@ from PyQt6.QtWidgets import QCheckBox, QHBoxLayout, QLabel, QPushButton, QWidget
class AvailableCamera(QWidget):
-
def __init__(self, webcam_id: str = "0"):
super().__init__()
self._webcam_id = webcam_id
diff --git a/src/gui/main/workflows/available_cameras_list.py b/src/gui/main/workflows/available_cameras_list.py
index 0bf2f012..e4350ea6 100644
--- a/src/gui/main/workflows/available_cameras_list.py
+++ b/src/gui/main/workflows/available_cameras_list.py
@@ -37,7 +37,7 @@ class AvailableCamerasList(QWidget):
self._enable_accept_button_callback = None
- #automatically detect cameras on page load
+ # automatically detect cameras on page load
self._detect()
@property
@@ -70,8 +70,6 @@ class AvailableCamerasList(QWidget):
return
self._worker.start()
-
-
def _create_available_camera_widgets(self, detected: FoundCamerasResponse):
camera_widgets = []
for cam in detected.cameras_found_list:
@@ -95,4 +93,3 @@ class AvailableCamerasList(QWidget):
def _handle_camera_preview_click(self, cam_id):
self.PreviewClicked.emit(cam_id)
-
diff --git a/src/gui/main/workflows/calibration_instructions.py b/src/gui/main/workflows/calibration_instructions.py
index 3597e1e3..17000c5c 100644
--- a/src/gui/main/workflows/calibration_instructions.py
+++ b/src/gui/main/workflows/calibration_instructions.py
@@ -5,7 +5,6 @@ from src.gui.main.shared_widgets.primary_button import PrimaryButton
class CalibrationInstructions(QWidget):
-
def __init__(self):
super().__init__()
self._continue_button = PrimaryButton("Continue")
@@ -29,8 +28,12 @@ class CalibrationInstructions(QWidget):
def _create_description_layout(self):
description_layout = QVBoxLayout()
- description_1 = QLabel("Hold up the Charuko boards so all cameras can see every square. Don't have one? Get one here ")
- description_2 = QLabel("Note: You may need to adjust the cameras if you are having an issue")
+ description_1 = QLabel(
+ "Hold up the Charuko boards so all cameras can see every square. Don't have one? Get one here "
+ )
+ description_2 = QLabel(
+ "Note: You may need to adjust the cameras if you are having an issue"
+ )
description_layout.addWidget(description_1)
description_layout.addWidget(description_2)
return description_layout
diff --git a/src/gui/main/workflows/camera_configuration.py b/src/gui/main/workflows/camera_configuration.py
index deb0da83..0fa4bb44 100644
--- a/src/gui/main/workflows/camera_configuration.py
+++ b/src/gui/main/workflows/camera_configuration.py
@@ -4,21 +4,22 @@ from PyQt6.QtGui import QPixmap
from PyQt6.QtWidgets import QHBoxLayout, QLabel, QPushButton, QVBoxLayout, QWidget
from src.gui.main.shared_widgets.page_title import PageTitle
-from src.gui.main.shared_widgets.primary_button import PrimaryButton, \
- not_active_primary_button_style_sheet, active_primary_button_style_sheet
+from src.gui.main.shared_widgets.primary_button import (
+ PrimaryButton,
+ not_active_primary_button_style_sheet,
+ active_primary_button_style_sheet,
+)
from src.gui.main.state.app_state import APP_STATE
from src.gui.main.workers.cam_frame_worker import CamFrameWorker
from src.gui.main.workflows.available_cameras_list import AvailableCamerasList
class CameraConfiguration(QWidget):
-
def __init__(self):
super().__init__()
self._worker = self._init_frame_worker()
self._accept_button = self._create_accept_button()
-
container = QVBoxLayout()
# Holds the Camera Configuration Title
@@ -79,7 +80,7 @@ class CameraConfiguration(QWidget):
if self._worker.isRunning():
self._worker.quit()
while not self._worker.isFinished():
- time.sleep(.1)
+ time.sleep(0.1)
self._worker._cam_id = cam_id
self._worker.start()
diff --git a/src/gui/main/workflows/new_recording_session.py b/src/gui/main/workflows/new_recording_session.py
index 1f681848..a6f09527 100644
--- a/src/gui/main/workflows/new_recording_session.py
+++ b/src/gui/main/workflows/new_recording_session.py
@@ -1,5 +1,12 @@
-from PyQt6.QtWidgets import QHBoxLayout, QLabel, QLineEdit, QPushButton, QVBoxLayout, \
- QWidget, QCheckBox
+from PyQt6.QtWidgets import (
+ QHBoxLayout,
+ QLabel,
+ QLineEdit,
+ QPushButton,
+ QVBoxLayout,
+ QWidget,
+ QCheckBox,
+)
from src.config.home_dir import create_session_id
from src.gui.main.shared_widgets.page_title import PageTitle
@@ -8,12 +15,13 @@ from src.gui.main.state.app_state import APP_STATE
class NewRecordingSession(QWidget):
-
def __init__(self):
super().__init__()
self._submit_button = PrimaryButton("&Start Session")
- self._use_previous_calibration_checkbox = self._create_use_previous_calibration_checkbox()
+ self._use_previous_calibration_checkbox = (
+ self._create_use_previous_calibration_checkbox()
+ )
container = QVBoxLayout()
@@ -40,7 +48,9 @@ class NewRecordingSession(QWidget):
def _create_use_previous_calibration_checkbox(self):
previous_calibration_checkbox = QCheckBox("Use Previous Calibration")
previous_calibration_checkbox.setChecked(False)
- previous_calibration_checkbox.stateChanged.connect(self._use_previous_calibration_changed)
+ previous_calibration_checkbox.stateChanged.connect(
+ self._use_previous_calibration_changed
+ )
return previous_calibration_checkbox
def _create_record_sesion_title(self):
@@ -62,5 +72,6 @@ class NewRecordingSession(QWidget):
APP_STATE.session_id = self._session_input.text()
def _use_previous_calibration_changed(self):
- APP_STATE.use_previous_calibration = self._use_previous_calibration_checkbox.isChecked()
-
+ APP_STATE.use_previous_calibration = (
+ self._use_previous_calibration_checkbox.isChecked()
+ )
diff --git a/src/gui/main/workflows/record_instructions.py b/src/gui/main/workflows/record_instructions.py
index 27ee492b..80d3b77f 100644
--- a/src/gui/main/workflows/record_instructions.py
+++ b/src/gui/main/workflows/record_instructions.py
@@ -4,7 +4,6 @@ from src.gui.main.shared_widgets.primary_button import PrimaryButton
class RecordInstructions(QWidget):
-
def __init__(self):
super().__init__()
self._record_button = PrimaryButton("Record")
@@ -28,7 +27,9 @@ class RecordInstructions(QWidget):
def _create_description_layout(self):
description_layout = QVBoxLayout()
- description_1 = QLabel("Cameras are configured & calibrated. Hit the record button when you're ready! ")
+ description_1 = QLabel(
+ "Cameras are configured & calibrated. Hit the record button when you're ready! "
+ )
description_layout.addWidget(description_1)
return description_layout
diff --git a/src/gui/main/workflows/record_videos.py b/src/gui/main/workflows/record_videos.py
index 5cd98f84..ec9e2464 100644
--- a/src/gui/main/workflows/record_videos.py
+++ b/src/gui/main/workflows/record_videos.py
@@ -1,20 +1,29 @@
-from PyQt6.QtWidgets import QCheckBox, QHBoxLayout, QLabel, QPushButton, QVBoxLayout, QWidget
+from PyQt6.QtWidgets import (
+ QCheckBox,
+ QHBoxLayout,
+ QLabel,
+ QPushButton,
+ QVBoxLayout,
+ QWidget,
+)
from src.cameras.save_synchronized_videos import save_synchronized_videos
-from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import \
- MediaPipeSkeletonDetector
+from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import (
+ MediaPipeSkeletonDetector,
+)
from src.gui.main.state.app_state import APP_STATE
from src.gui.main.workflows.single_camera import SingleCamera
from src.open_in_blender.open_session_in_blender import open_session_in_blender
-from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import \
- CalibrationPipelineOrchestrator
-from src.pipelines.session_pipeline.session_pipeline_orchestrator import \
- SessionPipelineOrchestrator, \
- load_mediapipe2d_data
+from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import (
+ CalibrationPipelineOrchestrator,
+)
+from src.pipelines.session_pipeline.session_pipeline_orchestrator import (
+ SessionPipelineOrchestrator,
+ load_mediapipe2d_data,
+)
class RecordVideos(QWidget):
-
def __init__(self):
super().__init__()
# TODO: Take it in from init
@@ -39,8 +48,8 @@ class RecordVideos(QWidget):
# start/stop recording button layout
record_button_layout = QHBoxLayout()
- self._start_recording_button = QPushButton('Begin Recording')
- self._stop_recording_button = QPushButton('Stop Recording')
+ self._start_recording_button = QPushButton("Begin Recording")
+ self._stop_recording_button = QPushButton("Stop Recording")
self._stop_recording_button.setEnabled(False)
self._start_recording_button.clicked.connect(self._start_recording_frames)
@@ -55,7 +64,9 @@ class RecordVideos(QWidget):
self._detect_2d_skeletons_button = QPushButton("Detect 2d Skeletons in Videos")
self._reconstruct_3d_skeletons_button = QPushButton("Reconstruct 3d Skeletons")
- self._visualize_freemocap_session_button = QPushButton("Visualize Freemocap Session")
+ self._visualize_freemocap_session_button = QPushButton(
+ "Visualize Freemocap Session"
+ )
self._open_in_blender_button = QPushButton("Open Session in Blender")
self._detect_2d_skeletons_button.setEnabled(False)
@@ -63,9 +74,15 @@ class RecordVideos(QWidget):
self._visualize_freemocap_session_button.setEnabled(False)
self._detect_2d_skeletons_button.clicked.connect(self._detect_2d_skeletons)
- self._reconstruct_3d_skeletons_button.clicked.connect(self._reconstruct_3d_skeletons)
- self._visualize_freemocap_session_button.clicked.connect(self._visualize_freemocap_session)
- self._open_in_blender_button.clicked.connect(self._create_blender_scene_from_session_data)
+ self._reconstruct_3d_skeletons_button.clicked.connect(
+ self._reconstruct_3d_skeletons
+ )
+ self._visualize_freemocap_session_button.clicked.connect(
+ self._visualize_freemocap_session
+ )
+ self._open_in_blender_button.clicked.connect(
+ self._create_blender_scene_from_session_data
+ )
self._process_automatically_checkbox = QCheckBox("Process Videos Automatically")
self._process_automatically_checkbox.setChecked(True)
@@ -111,7 +128,9 @@ class RecordVideos(QWidget):
self._detect_2d_skeletons()
def _detect_2d_skeletons(self):
- print(f"tracking 2D mediapipe skeletons in videos from session: {APP_STATE.session_id}")
+ print(
+ f"tracking 2D mediapipe skeletons in videos from session: {APP_STATE.session_id}"
+ )
mediapipe_skeleton_detector = MediaPipeSkeletonDetector(APP_STATE.session_id)
mediapipe_skeleton_detector.process_session_folder()
@@ -123,19 +142,24 @@ class RecordVideos(QWidget):
def _reconstruct_3d_skeletons(self):
print(f"Reconstruct 3D Skeletons : {APP_STATE.session_id}")
- session_orchestrator = SessionPipelineOrchestrator(session_id=APP_STATE.session_id)
+ session_orchestrator = SessionPipelineOrchestrator(
+ session_id=APP_STATE.session_id
+ )
if APP_STATE.use_previous_calibration:
- session_orchestrator.anipose_camera_calibration_object = \
+ session_orchestrator.anipose_camera_calibration_object = (
CalibrationPipelineOrchestrator().load_most_recent_calibration()
+ )
else:
- session_orchestrator.anipose_camera_calibration_object = \
+ session_orchestrator.anipose_camera_calibration_object = (
CalibrationPipelineOrchestrator().load_calibration_from_session_id(
- APP_STATE.session_id)
+ APP_STATE.session_id
+ )
+ )
- session_orchestrator.mediapipe2d_numCams_numFrames_numTrackedPoints_XY = \
- load_mediapipe2d_data(
- APP_STATE.session_id)
+ session_orchestrator.mediapipe2d_numCams_numFrames_numTrackedPoints_XY = (
+ load_mediapipe2d_data(APP_STATE.session_id)
+ )
session_orchestrator.reconstruct3d_from_2d_data_offline()
self._reconstruct_3d_skeletons_button.setEnabled(False)
diff --git a/src/gui/main/workflows/show_cams_charuco.py b/src/gui/main/workflows/show_cams_charuco.py
index 867ff2df..ff5100ae 100644
--- a/src/gui/main/workflows/show_cams_charuco.py
+++ b/src/gui/main/workflows/show_cams_charuco.py
@@ -1,14 +1,22 @@
-from PyQt6.QtWidgets import QHBoxLayout, QLabel, QVBoxLayout, QWidget, QPushButton, QCheckBox
+from PyQt6.QtWidgets import (
+ QHBoxLayout,
+ QLabel,
+ QVBoxLayout,
+ QWidget,
+ QPushButton,
+ QCheckBox,
+)
from src.cameras.save_synchronized_videos import save_synchronized_videos
from src.gui.main.state.app_state import APP_STATE
from src.gui.main.workflows.single_camera import SingleCamera
-from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import CalibrationPipelineOrchestrator
+from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import (
+ CalibrationPipelineOrchestrator,
+)
class ShowCamsCharuco(QWidget):
-
def __init__(self):
super().__init__()
# TODO: Take it in from init
@@ -17,7 +25,9 @@ class ShowCamsCharuco(QWidget):
self._continue_button = QPushButton("Continue")
self._continue_button.setEnabled(False)
- self._use_previous_calibration_checkbox = self._create_use_previous_calibration_checkbox()
+ self._use_previous_calibration_checkbox = (
+ self._create_use_previous_calibration_checkbox()
+ )
container = QVBoxLayout()
@@ -40,8 +50,8 @@ class ShowCamsCharuco(QWidget):
# start/stop recording button layout
record_button_layout = QHBoxLayout()
- self._start_recording_button = QPushButton('Begin Recording')
- self._stop_recording_button = QPushButton('Stop Recording')
+ self._start_recording_button = QPushButton("Begin Recording")
+ self._stop_recording_button = QPushButton("Stop Recording")
self._stop_recording_button.setEnabled(False)
self._start_recording_button.clicked.connect(self._start_recording_frames)
@@ -63,7 +73,9 @@ class ShowCamsCharuco(QWidget):
def _create_use_previous_calibration_checkbox(self):
previous_calibration_checkbox = QCheckBox("Use Previous Calibration")
previous_calibration_checkbox.setChecked(False)
- previous_calibration_checkbox.stateChanged.connect(self._use_previous_calibration_changed)
+ previous_calibration_checkbox.stateChanged.connect(
+ self._use_previous_calibration_changed
+ )
return previous_calibration_checkbox
def _create_continue_button_container(self):
@@ -72,8 +84,12 @@ class ShowCamsCharuco(QWidget):
return continue_button_layout
def _use_previous_calibration_changed(self):
- self._continue_button.setEnabled(self._use_previous_calibration_checkbox.isChecked())
- APP_STATE.use_previous_calibration = self._use_previous_calibration_checkbox.isChecked()
+ self._continue_button.setEnabled(
+ self._use_previous_calibration_checkbox.isChecked()
+ )
+ APP_STATE.use_previous_calibration = (
+ self._use_previous_calibration_checkbox.isChecked()
+ )
def _start_recording_frames(self):
self._start_recording_button.setEnabled(False)
@@ -94,12 +110,12 @@ class ShowCamsCharuco(QWidget):
self._continue_button.setEnabled(True)
def _run_anipose_calibration(self):
- print('Beginning Anipose calibration')
+ print("Beginning Anipose calibration")
calibration_orchestrator = CalibrationPipelineOrchestrator(APP_STATE.session_id)
try:
calibration_orchestrator.run_anipose_camera_calibration(
- charuco_square_size=39,
- pin_camera_0_to_origin=True)
+ charuco_square_size=39, pin_camera_0_to_origin=True
+ )
except:
- print('something failed in the anipose calibration')
+ print("something failed in the anipose calibration")
raise Exception
diff --git a/src/gui/main/workflows/single_camera.py b/src/gui/main/workflows/single_camera.py
index 22977f0a..20958bdc 100644
--- a/src/gui/main/workflows/single_camera.py
+++ b/src/gui/main/workflows/single_camera.py
@@ -56,7 +56,6 @@ class SingleCamera(QWidget):
def quit(self):
self._worker.quit()
-
def _init_frame_worker(self):
worker = construct_worker(WorkerType.CHARUCO)(self._cam_id)
worker.ImageUpdate.connect(self._handle_image_update)
diff --git a/src/gui/main/workflows/visualize_session.py b/src/gui/main/workflows/visualize_session.py
index 58677769..bd162811 100644
--- a/src/gui/main/workflows/visualize_session.py
+++ b/src/gui/main/workflows/visualize_session.py
@@ -17,18 +17,17 @@ logger = logging.getLogger(__name__)
class VisualizeSkeleton(QDialog):
-
- def __init__(self,
- mediapipe_skel_fr_mar_xyz: np.ndarray,
- start_frame: int = None,
- end_frame: int = None,
- move_data_to_origin: bool = True):
+ def __init__(
+ self,
+ mediapipe_skel_fr_mar_xyz: np.ndarray,
+ start_frame: int = None,
+ end_frame: int = None,
+ move_data_to_origin: bool = True,
+ ):
super().__init__()
self.mediapipe_fr_mar_xyz = mediapipe_skel_fr_mar_xyz
-
-
if move_data_to_origin:
self.move_data_to_origin()
@@ -62,7 +61,7 @@ class VisualizeSkeleton(QDialog):
def create_app_window(self):
self.gl_view_widget = gl.GLViewWidget()
- self.gl_view_widget.opts['distance'] = 2000
+ self.gl_view_widget.opts["distance"] = 2000
# self.gl_view_widget.setWindowTitle('FreeMoCap Skeleton')
self.gl_view_widget.setGeometry(0, 110, 1920, 1080)
# self.gl_view_widget.show()
@@ -91,14 +90,22 @@ class VisualizeSkeleton(QDialog):
self.gl_view_widget.addItem(grid_plane_z)
def get_mediapipe_connections(self):
- self.mediapipe_body_connections = [this_connection for this_connection in mp_holistic.POSE_CONNECTIONS]
- self.mediapipe_hand_connections = [this_connection for this_connection in mp_holistic.HAND_CONNECTIONS]
- self.mediapipe_face_connections = [this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION]
+ self.mediapipe_body_connections = [
+ this_connection for this_connection in mp_holistic.POSE_CONNECTIONS
+ ]
+ self.mediapipe_hand_connections = [
+ this_connection for this_connection in mp_holistic.HAND_CONNECTIONS
+ ]
+ self.mediapipe_face_connections = [
+ this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION
+ ]
def initialize_skel_dottos(self):
self.skeleton_scatter_item = gl.GLScatterPlotItem(
- pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, :, :], color=(0, 1, 1, 1), size=10,
- pxMode=False
+ pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, :, :],
+ color=(0, 1, 1, 1),
+ size=10,
+ pxMode=False,
)
self.gl_view_widget.addItem(self.skeleton_scatter_item)
@@ -106,16 +113,17 @@ class VisualizeSkeleton(QDialog):
self.skeleton_connections_list = []
for this_connection in self.mediapipe_body_connections:
this_skel_line = gl.GLLinePlotItem(
- pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, this_connection, :])
+ pos=self.mediapipe_fr_mar_xyz[
+ self.current_frame_number, this_connection, :
+ ]
+ )
self.skeleton_connections_list.append(this_skel_line)
self.gl_view_widget.addItem(this_skel_line)
-
-
def update(self):
self.update_frame_number()
if self.current_frame_number % 10 == 0:
- print(f'frame number: {self.current_frame_number}')
+ print(f"frame number: {self.current_frame_number}")
self.update_skeleton()
@@ -126,9 +134,13 @@ class VisualizeSkeleton(QDialog):
)
# skel lines
- for this_skeleton_line_number, this_connection in enumerate(self.mediapipe_body_connections):
+ for this_skeleton_line_number, this_connection in enumerate(
+ self.mediapipe_body_connections
+ ):
self.skeleton_connections_list[this_skeleton_line_number].setData(
- pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, this_connection, :]
+ pos=self.mediapipe_fr_mar_xyz[
+ self.current_frame_number, this_connection, :
+ ]
)
def update_frame_number(self):
@@ -137,11 +149,11 @@ class VisualizeSkeleton(QDialog):
self.current_frame_number = self.start_frame_number
def start(self):
- print('starting animation')
+ print("starting animation")
pg.exec()
def start_animation(self):
- print('initializing display window')
+ print("initializing display window")
self.initialize_display_window()
timer = QtCore.QTimer()
timer.timeout.connect(self.update)
@@ -149,9 +161,10 @@ class VisualizeSkeleton(QDialog):
# self.start()
def move_data_to_origin(self):
- mean_position_xyz = np.nanmedian(np.nanmedian(self.mediapipe_fr_mar_xyz, axis=0), axis=0)
+ mean_position_xyz = np.nanmedian(
+ np.nanmedian(self.mediapipe_fr_mar_xyz, axis=0), axis=0
+ )
self.mediapipe_fr_mar_xyz[:, :, 0] -= mean_position_xyz[0]
self.mediapipe_fr_mar_xyz[:, :, 1] -= mean_position_xyz[1]
self.mediapipe_fr_mar_xyz[:, :, 2] -= mean_position_xyz[2]
-
diff --git a/src/gui/main/workflows/welcome.py b/src/gui/main/workflows/welcome.py
index 1f882310..1c103250 100644
--- a/src/gui/main/workflows/welcome.py
+++ b/src/gui/main/workflows/welcome.py
@@ -1,12 +1,19 @@
from PyQt6.QtCore import Qt
-from PyQt6.QtWidgets import QHBoxLayout, QLabel, QVBoxLayout, QWidget, QGridLayout, QSpacerItem, QCheckBox
+from PyQt6.QtWidgets import (
+ QHBoxLayout,
+ QLabel,
+ QVBoxLayout,
+ QWidget,
+ QGridLayout,
+ QSpacerItem,
+ QCheckBox,
+)
from src.gui.main.qt_utils.set_font_size import set_font_size
from src.gui.main.shared_widgets.primary_button import PrimaryButton
class Welcome(QWidget):
-
def __init__(self):
super().__init__()
diff --git a/src/open_in_blender/alpha_freemocap_blender_megascript.py b/src/open_in_blender/alpha_freemocap_blender_megascript.py
index 2ba7d649..b518b83e 100644
--- a/src/open_in_blender/alpha_freemocap_blender_megascript.py
+++ b/src/open_in_blender/alpha_freemocap_blender_megascript.py
@@ -4,7 +4,7 @@ import numpy as np
from pathlib import Path
import sys
-print(' - Starting (alpha) blender megascript - ')
+print(" - Starting (alpha) blender megascript - ")
#######################################################################
##% Activate necessary addons
@@ -17,7 +17,7 @@ try:
##% Session path
# #get session path as command line argument
argv = sys.argv
- argv = argv[argv.index("--") + 1:]
+ argv = argv[argv.index("--") + 1 :]
session_path = argv[0]
if len(argv) > 1:
@@ -25,20 +25,31 @@ try:
else:
good_clean_frame_number = 0
except:
- print('we appear to be running from the Blender Scripting tab! Manually enter your `session_path` at line 23')
+ print(
+ "we appear to be running from the Blender Scripting tab! Manually enter your `session_path` at line 23"
+ )
session_path = Path(
- r'C:\Users\jonma\Dropbox\FreeMoCapProject\FreeMocap_Data\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0')
+ r"C:\Users\jonma\Dropbox\FreeMoCapProject\FreeMocap_Data\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0"
+ )
good_clean_frame_number = 3341
print(str(session_path))
session_path = Path(session_path)
-path_to_mediapipe_npy = session_path / 'output_data_files' / 'mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy'
-path_to_mediapipe_3d_reproj = session_path / 'output_data_files' / 'mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy'
+path_to_mediapipe_npy = (
+ session_path
+ / "output_data_files"
+ / "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"
+)
+path_to_mediapipe_3d_reproj = (
+ session_path
+ / "output_data_files"
+ / "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"
+)
#######################################################################
# %% load mediapipe data
# %%
-print('loading data')
+print("loading data")
# %%
mediapipe_skel_fr_mar_dim = np.load(str(path_to_mediapipe_npy))
mediapipe_skel_fr_mar_dim = mediapipe_skel_fr_mar_dim / 1000 # convert to meters
@@ -49,12 +60,30 @@ mediapipe_reproj_fr_mar = mediapipe_reproj_fr_mar / 1000 # convert to meters
body_marker_range = np.arange(0, 33)
right_hand_marker_range = np.arange(33, 33 + 21)
left_hand_marker_range = np.arange(33 + 21, 33 + 21 + 21)
-face_marker_range = np.arange(left_hand_marker_range[-1], mediapipe_skel_fr_mar_dim.shape[1])
-
-body_skel_fr_mar_dim = mediapipe_skel_fr_mar_dim[:, body_marker_range, :, ]
-right_hand_fr_mar_dim = mediapipe_skel_fr_mar_dim[:, right_hand_marker_range, :, ]
-left_hand_fr_mar_dim = mediapipe_skel_fr_mar_dim[:, left_hand_marker_range, :, ]
-face_fr_mar_dim = mediapipe_skel_fr_mar_dim[:, face_marker_range, :, ]
+face_marker_range = np.arange(
+ left_hand_marker_range[-1], mediapipe_skel_fr_mar_dim.shape[1]
+)
+
+body_skel_fr_mar_dim = mediapipe_skel_fr_mar_dim[
+ :,
+ body_marker_range,
+ :,
+]
+right_hand_fr_mar_dim = mediapipe_skel_fr_mar_dim[
+ :,
+ right_hand_marker_range,
+ :,
+]
+left_hand_fr_mar_dim = mediapipe_skel_fr_mar_dim[
+ :,
+ left_hand_marker_range,
+ :,
+]
+face_fr_mar_dim = mediapipe_skel_fr_mar_dim[
+ :,
+ face_marker_range,
+ :,
+]
# %% Session Specific stuff
number_of_frames = mediapipe_skel_fr_mar_dim.shape[0]
@@ -71,8 +100,12 @@ if good_clean_frame_number == 0:
frame_mean_reproj_error = []
for this_frame in range(mediapipe_skel_fr_mar_dim.shape[0]):
- frame_nan_counts.append(np.sum(np.isnan(mediapipe_skel_fr_mar_dim[this_frame, :, 0])))
- frame_mean_reproj_error.append(np.nanmean(mediapipe_reproj_fr_mar[this_frame, :]))
+ frame_nan_counts.append(
+ np.sum(np.isnan(mediapipe_skel_fr_mar_dim[this_frame, :, 0]))
+ )
+ frame_mean_reproj_error.append(
+ np.nanmean(mediapipe_reproj_fr_mar[this_frame, :])
+ )
nan_times_vis = np.array(frame_nan_counts) * np.array(frame_mean_reproj_error)
num_frames = len(frame_nan_counts)
@@ -80,7 +113,8 @@ if good_clean_frame_number == 0:
# nan_times_vis[-int(num_frames/5):-1] = np.nanmax(nan_times_vis)
good_clean_frame_number = np.nanargmin(
- nan_times_vis) # the frame with the fewest nans (i.e. hopefully a frame where all tracked points are visible)
+ nan_times_vis
+ ) # the frame with the fewest nans (i.e. hopefully a frame where all tracked points are visible)
print(f"----good_clean_frame_number: {good_clean_frame_number}----")
@@ -171,14 +205,14 @@ mediapipe_tracked_point_names = [
try:
#####################################################################
###%% clear scene
- print('clearing scene')
+ print("clearing scene")
try:
- bpy.ops.object.mode_set(mode='OBJECT')
+ bpy.ops.object.mode_set(mode="OBJECT")
except:
pass
try:
- bpy.ops.object.select_all(action='SELECT') # select all objects
+ bpy.ops.object.select_all(action="SELECT") # select all objects
bpy.ops.object.delete(use_global=True) # delete all objects from all scenes
except:
pass
@@ -186,30 +220,34 @@ try:
#######################################################################
# %% create world origin
- bpy.ops.object.empty_add(type='ARROWS')
+ bpy.ops.object.empty_add(type="ARROWS")
world_origin_axes = bpy.context.active_object
- world_origin_axes.name = 'world_origin' # will stay at origin
+ world_origin_axes.name = "world_origin" # will stay at origin
# inspect(world_origin_axes, methods=True)
- bpy.ops.object.empty_add(type='ARROWS')
+ bpy.ops.object.empty_add(type="ARROWS")
freemocap_origin_axes = bpy.context.active_object
- freemocap_origin_axes.name = 'freemocap_origin_axes' # will translate to put skelly on ground symmetric-ish about origin
+ freemocap_origin_axes.name = "freemocap_origin_axes" # will translate to put skelly on ground symmetric-ish about origin
#######################################################################
# %% load empties
- print('loading {} empties on {} frames'.format(len(mediapipe_tracked_point_names), number_of_frames))
+ print(
+ "loading {} empties on {} frames".format(
+ len(mediapipe_tracked_point_names), number_of_frames
+ )
+ )
- empty_size = .01
+ empty_size = 0.01
for this_point_index, this_point_name in enumerate(mediapipe_tracked_point_names):
- print(f'loading {this_point_name}...')
- bpy.ops.object.empty_add(type='SPHERE')
+ print(f"loading {this_point_name}...")
+ bpy.ops.object.empty_add(type="SPHERE")
this_empty = bpy.context.active_object
this_empty.name = this_point_name
- if 'face' in this_point_name:
+ if "face" in this_point_name:
this_empty.scale = [empty_size / 4] * 3
- if 'hand' in this_point_name:
+ if "hand" in this_point_name:
this_empty.scale = [empty_size / 2] * 3
else:
this_empty.scale = [empty_size] * 3
@@ -219,288 +257,306 @@ try:
this_point_fr_xyz = mediapipe_skel_fr_mar_dim[:, this_point_index, :]
for frame_num in range(end_frame):
- this_empty.location = [this_point_fr_xyz[frame_num, 0],
- this_point_fr_xyz[frame_num, 1],
- this_point_fr_xyz[frame_num, 2]]
+ this_empty.location = [
+ this_point_fr_xyz[frame_num, 0],
+ this_point_fr_xyz[frame_num, 1],
+ this_point_fr_xyz[frame_num, 2],
+ ]
bpy.context.view_layer.update()
- this_empty.keyframe_insert(data_path='location', frame=frame_num)
+ this_empty.keyframe_insert(data_path="location", frame=frame_num)
#######################################################################
# %% create virtual markers
left_ear_index = 7
right_ear_index = 8
#######################################################################
- print('creating virtual markers')
- print('head_center - midway between left and right ears')
- hips_xyz = (mediapipe_skel_fr_mar_dim[:, left_ear_index, :] + mediapipe_skel_fr_mar_dim[:, right_ear_index, :]) / 2
- bpy.ops.object.empty_add(type='SPHERE')
+ print("creating virtual markers")
+ print("head_center - midway between left and right ears")
+ hips_xyz = (
+ mediapipe_skel_fr_mar_dim[:, left_ear_index, :]
+ + mediapipe_skel_fr_mar_dim[:, right_ear_index, :]
+ ) / 2
+ bpy.ops.object.empty_add(type="SPHERE")
this_empty = bpy.context.active_object
- this_empty.name = 'head_center'
+ this_empty.name = "head_center"
this_empty.scale = [empty_size] * 3
this_empty.parent = freemocap_origin_axes
mediapipe_tracked_point_names.append(this_empty.name)
for frame_num in range(end_frame):
- this_empty.location = [hips_xyz[frame_num, 0],
- hips_xyz[frame_num, 1],
- hips_xyz[frame_num, 2]]
- this_empty.keyframe_insert(data_path='location', frame=frame_num)
-
- print('neck_center - midway between left and right shoulders')
+ this_empty.location = [
+ hips_xyz[frame_num, 0],
+ hips_xyz[frame_num, 1],
+ hips_xyz[frame_num, 2],
+ ]
+ this_empty.keyframe_insert(data_path="location", frame=frame_num)
+
+ print("neck_center - midway between left and right shoulders")
left_shoulder_index = 11
right_shoulder_index = 12
- neck_xyz = (mediapipe_skel_fr_mar_dim[:, left_shoulder_index, :] + mediapipe_skel_fr_mar_dim[:,
- right_shoulder_index, :]) / 2
- bpy.ops.object.empty_add(type='PLAIN_AXES')
+ neck_xyz = (
+ mediapipe_skel_fr_mar_dim[:, left_shoulder_index, :]
+ + mediapipe_skel_fr_mar_dim[:, right_shoulder_index, :]
+ ) / 2
+ bpy.ops.object.empty_add(type="PLAIN_AXES")
this_empty = bpy.context.active_object
- this_empty.name = 'neck_center'
+ this_empty.name = "neck_center"
this_empty.scale = [empty_size] * 3
this_empty.parent = freemocap_origin_axes
mediapipe_tracked_point_names.append(this_empty.name)
for frame_num in range(end_frame):
- this_empty.location = [neck_xyz[frame_num, 0],
- neck_xyz[frame_num, 1],
- neck_xyz[frame_num, 2]]
- this_empty.keyframe_insert(data_path='location', frame=frame_num)
-
- print('hip_center - midway between left and right hips')
+ this_empty.location = [
+ neck_xyz[frame_num, 0],
+ neck_xyz[frame_num, 1],
+ neck_xyz[frame_num, 2],
+ ]
+ this_empty.keyframe_insert(data_path="location", frame=frame_num)
+
+ print("hip_center - midway between left and right hips")
left_hip_index = 23
right_hip_index = 24
- hips_xyz = (mediapipe_skel_fr_mar_dim[:, left_hip_index, :] + mediapipe_skel_fr_mar_dim[:, right_hip_index, :]) / 2
- bpy.ops.object.empty_add(type='PLAIN_AXES')
+ hips_xyz = (
+ mediapipe_skel_fr_mar_dim[:, left_hip_index, :]
+ + mediapipe_skel_fr_mar_dim[:, right_hip_index, :]
+ ) / 2
+ bpy.ops.object.empty_add(type="PLAIN_AXES")
this_empty = bpy.context.active_object
- this_empty.name = 'hip_center'
+ this_empty.name = "hip_center"
this_empty.scale = [empty_size] * 3
this_empty.parent = freemocap_origin_axes
mediapipe_tracked_point_names.append(this_empty.name)
rig_constraint_dict_of_dicts = {
"spine": {
- 'COPY_LOCATION': 'hip_center',
- 'STRETCH_TO': 'chest_center',
+ "COPY_LOCATION": "hip_center",
+ "STRETCH_TO": "chest_center",
},
"spine.001": {
- 'STRETCH_TO': 'neck_center',
+ "STRETCH_TO": "neck_center",
},
"spine.002": {
- 'STRETCH_TO': 'head_center',
+ "STRETCH_TO": "head_center",
},
"spine.003": {
- 'STRETCH_TO': 'nose',
+ "STRETCH_TO": "nose",
},
"pelvis.L": {
- 'STRETCH_TO': 'left_hip',
+ "STRETCH_TO": "left_hip",
},
"thigh.L": {
- 'COPY_LOCATION': 'left_hip',
- 'STRETCH_TO': 'left_knee',
+ "COPY_LOCATION": "left_hip",
+ "STRETCH_TO": "left_knee",
},
"shin.L": {
- 'STRETCH_TO': 'left_ankle',
+ "STRETCH_TO": "left_ankle",
},
"foot.L": {
- 'STRETCH_TO': 'left_foot_index',
+ "STRETCH_TO": "left_foot_index",
},
"heel.02.L": {
- 'STRETCH_TO': 'left_ankle',
+ "STRETCH_TO": "left_ankle",
},
"shoulder.L": {
- 'COPY_LOCATION': 'neck_center',
- 'STRETCH_TO': 'left_shoulder',
+ "COPY_LOCATION": "neck_center",
+ "STRETCH_TO": "left_shoulder",
},
"upper_arm.L": {
- 'STRETCH_TO': 'left_elbow',
+ "STRETCH_TO": "left_elbow",
},
"forearm.L": {
- 'STRETCH_TO': 'left_wrist',
+ "STRETCH_TO": "left_wrist",
},
"hand.L": {
- 'STRETCH_TO': 'left_index',
+ "STRETCH_TO": "left_index",
},
"pelvis.R": {
- 'STRETCH_TO': 'right_hip',
+ "STRETCH_TO": "right_hip",
},
"thigh.R": {
- 'COPY_LOCATION': 'right_hip',
- 'STRETCH_TO': 'right_knee',
+ "COPY_LOCATION": "right_hip",
+ "STRETCH_TO": "right_knee",
},
"shin.R": {
- 'STRETCH_TO': 'right_ankle',
+ "STRETCH_TO": "right_ankle",
},
"foot.R": {
- 'STRETCH_TO': 'right_foot_index',
+ "STRETCH_TO": "right_foot_index",
},
"heel.02.R": {
- 'STRETCH_TO': 'right_ankle',
+ "STRETCH_TO": "right_ankle",
},
"shoulder.R": {
- 'COPY_LOCATION': 'neck_center',
- 'STRETCH_TO': 'right_shoulder',
+ "COPY_LOCATION": "neck_center",
+ "STRETCH_TO": "right_shoulder",
},
"upper_arm.R": {
- 'STRETCH_TO': 'right_elbow',
+ "STRETCH_TO": "right_elbow",
},
"forearm.R": {
- 'STRETCH_TO': 'right_wrist',
+ "STRETCH_TO": "right_wrist",
},
"hand.R": {
- 'STRETCH_TO': 'right_index',
+ "STRETCH_TO": "right_index",
},
"thumb.01.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'STRETCH_TO': 'right_hand_thumb_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "STRETCH_TO": "right_hand_thumb_mcp",
},
"thumb.02.R": {
- 'STRETCH_TO': 'right_hand_thumb_ip',
+ "STRETCH_TO": "right_hand_thumb_ip",
},
"thumb.02.R": {
- 'STRETCH_TO': 'right_hand_thumb_tip',
+ "STRETCH_TO": "right_hand_thumb_tip",
},
"palm.01.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'STRETCH_TO': 'right_hand_index_finger_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "STRETCH_TO": "right_hand_index_finger_mcp",
},
"f_index.01.R": {
- 'STRETCH_TO': 'right_hand_index_finger_pip',
+ "STRETCH_TO": "right_hand_index_finger_pip",
},
"f_index.02.R": {
- 'STRETCH_TO': 'right_hand_index_finger_dip',
+ "STRETCH_TO": "right_hand_index_finger_dip",
},
"f_index.03.R": {
- 'STRETCH_TO': 'right_hand_index_finger_tip',
+ "STRETCH_TO": "right_hand_index_finger_tip",
},
"palm.02.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'STRETCH_TO': 'right_hand_middle_finger_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "STRETCH_TO": "right_hand_middle_finger_mcp",
},
"f_middle.01.R": {
- 'STRETCH_TO': 'right_hand_middle_finger_pip',
+ "STRETCH_TO": "right_hand_middle_finger_pip",
},
"f_middle.02.R": {
- 'STRETCH_TO': 'right_hand_middle_finger_dip',
+ "STRETCH_TO": "right_hand_middle_finger_dip",
},
"f_middle.03.R": {
- 'STRETCH_TO': 'right_hand_middle_finger_tip',
+ "STRETCH_TO": "right_hand_middle_finger_tip",
},
"palm.03.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'STRETCH_TO': 'right_hand_ring_finger_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "STRETCH_TO": "right_hand_ring_finger_mcp",
},
"f_ring.01.R": {
- 'STRETCH_TO': 'right_hand_ring_finger_pip',
+ "STRETCH_TO": "right_hand_ring_finger_pip",
},
"f_ring.02.R": {
- 'STRETCH_TO': 'right_hand_ring_finger_dip',
+ "STRETCH_TO": "right_hand_ring_finger_dip",
},
"f_ring.03.R": {
- 'STRETCH_TO': 'right_hand_ring_finger_tip',
+ "STRETCH_TO": "right_hand_ring_finger_tip",
},
"palm.04.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'STRETCH_TO': 'right_hand_pinky_finger_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "STRETCH_TO": "right_hand_pinky_finger_mcp",
},
"f_pinky.01.R": {
- 'STRETCH_TO': 'right_hand_pinky_finger_pip',
+ "STRETCH_TO": "right_hand_pinky_finger_pip",
},
"f_pinky.02.R": {
- 'STRETCH_TO': 'right_hand_pinky_finger_dip',
+ "STRETCH_TO": "right_hand_pinky_finger_dip",
},
"f_pinky.03.R": {
- 'STRETCH_TO': 'right_hand_pinky_finger_tip',
+ "STRETCH_TO": "right_hand_pinky_finger_tip",
},
"thumb.01.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'STRETCH_TO': 'left_hand_thumb_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "STRETCH_TO": "left_hand_thumb_mcp",
},
"thumb.02.L": {
- 'STRETCH_TO': 'left_hand_thumb_ip',
+ "STRETCH_TO": "left_hand_thumb_ip",
},
"thumb.03.L": {
- 'STRETCH_TO': 'left_hand_thumb_tip',
+ "STRETCH_TO": "left_hand_thumb_tip",
},
"palm.01.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'STRETCH_TO': 'left_hand_index_finger_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "STRETCH_TO": "left_hand_index_finger_mcp",
},
"f_index.01.L": {
- 'STRETCH_TO': 'left_hand_index_finger_pip',
+ "STRETCH_TO": "left_hand_index_finger_pip",
},
"f_index.02.L": {
- 'STRETCH_TO': 'left_hand_index_finger_dip',
+ "STRETCH_TO": "left_hand_index_finger_dip",
},
"f_index.03.L": {
- 'STRETCH_TO': 'left_hand_index_finger_tip',
+ "STRETCH_TO": "left_hand_index_finger_tip",
},
"palm.02.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'STRETCH_TO': 'left_hand_middle_finger_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "STRETCH_TO": "left_hand_middle_finger_mcp",
},
"f_middle.01.L": {
- 'STRETCH_TO': 'left_hand_middle_finger_pip',
+ "STRETCH_TO": "left_hand_middle_finger_pip",
},
"f_middle.02.L": {
- 'STRETCH_TO': 'left_hand_middle_finger_dip',
+ "STRETCH_TO": "left_hand_middle_finger_dip",
},
"f_middle.03.L": {
- 'STRETCH_TO': 'left_hand_middle_finger_tip',
+ "STRETCH_TO": "left_hand_middle_finger_tip",
},
"palm.03.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'STRETCH_TO': 'left_hand_ring_finger_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "STRETCH_TO": "left_hand_ring_finger_mcp",
},
"f_ring.01.L": {
- 'STRETCH_TO': 'left_hand_ring_finger_pip',
+ "STRETCH_TO": "left_hand_ring_finger_pip",
},
"f_ring.02.L": {
- 'STRETCH_TO': 'left_hand_ring_finger_dip',
+ "STRETCH_TO": "left_hand_ring_finger_dip",
},
"f_ring.03.L": {
- 'STRETCH_TO': 'left_hand_ring_finger_tip',
+ "STRETCH_TO": "left_hand_ring_finger_tip",
},
"palm.04.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'STRETCH_TO': 'left_hand_pinky_finger_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "STRETCH_TO": "left_hand_pinky_finger_mcp",
},
"f_pinky.01.L": {
- 'STRETCH_TO': 'left_hand_pinky_finger_pip',
+ "STRETCH_TO": "left_hand_pinky_finger_pip",
},
"f_pinky.02.L": {
- 'STRETCH_TO': 'left_hand_pinky_finger_dip',
+ "STRETCH_TO": "left_hand_pinky_finger_dip",
},
"f_pinky.03.L": {
- 'STRETCH_TO': 'left_hand_pinky_finger_tip',
+ "STRETCH_TO": "left_hand_pinky_finger_tip",
},
}
# loop through dictionary applying constraints
try:
- bpy.ops.object.mode_set(mode='POSE')
+ bpy.ops.object.mode_set(mode="POSE")
except:
pass
for frame_num in range(end_frame):
- this_empty.location = [hips_xyz[frame_num, 0],
- hips_xyz[frame_num, 1],
- hips_xyz[frame_num, 2]]
- this_empty.keyframe_insert(data_path='location', frame=frame_num)
-
- print('chest_center - mean of R/L shoulders and R/L hips')
+ this_empty.location = [
+ hips_xyz[frame_num, 0],
+ hips_xyz[frame_num, 1],
+ hips_xyz[frame_num, 2],
+ ]
+ this_empty.keyframe_insert(data_path="location", frame=frame_num)
+
+ print("chest_center - mean of R/L shoulders and R/L hips")
chest_xyz = (neck_xyz + hips_xyz) / 2
- bpy.ops.object.empty_add(type='PLAIN_AXES')
+ bpy.ops.object.empty_add(type="PLAIN_AXES")
this_empty = bpy.context.active_object
- this_empty.name = 'chest_center'
+ this_empty.name = "chest_center"
this_empty.scale = [empty_size] * 3
this_empty.parent = freemocap_origin_axes
mediapipe_tracked_point_names.append(this_empty.name)
for frame_num in range(end_frame):
- this_empty.location = [chest_xyz[frame_num, 0],
- chest_xyz[frame_num, 1],
- chest_xyz[frame_num, 2]]
- this_empty.keyframe_insert(data_path='location', frame=frame_num)
+ this_empty.location = [
+ chest_xyz[frame_num, 0],
+ chest_xyz[frame_num, 1],
+ chest_xyz[frame_num, 2],
+ ]
+ this_empty.keyframe_insert(data_path="location", frame=frame_num)
- print('Done loading empties :D')
+ print("Done loading empties :D")
######################################################################
## Fit Rigify metarig armatures to empties
@@ -510,308 +566,301 @@ try:
# each bone name has to match one in the Blender Rigify Human Metarig, or it will be ignored
metarig_bone_empty_pairs_dict_of_dicts = {
"nose.001": {
- 'head': 'head_center',
- 'tail': 'nose',
+ "head": "head_center",
+ "tail": "nose",
},
"nose.L.001": {
- 'head': 'nose',
- 'tail': 'left_eye_inner',
+ "head": "nose",
+ "tail": "left_eye_inner",
},
"eye.L": {
- 'head': 'left_eye_inner',
- 'tail': 'left_eye',
+ "head": "left_eye_inner",
+ "tail": "left_eye",
},
"lid.B.L": {
- 'head': 'left_eye',
- 'tail': 'left_eye_outer',
+ "head": "left_eye",
+ "tail": "left_eye_outer",
},
"ear.L": {
- 'head': 'left_eye_outer',
- 'tail': 'left_ear',
+ "head": "left_eye_outer",
+ "tail": "left_ear",
},
"nose.R.001": {
- 'head': 'nose',
- 'tail': 'right_eye_inner',
+ "head": "nose",
+ "tail": "right_eye_inner",
},
"eye.R": {
- 'head': 'right_eye_inner',
- 'tail': 'right_eye',
+ "head": "right_eye_inner",
+ "tail": "right_eye",
},
"lid.B.R": {
- 'head': 'right_eye',
- 'tail': 'right_eye_outer',
+ "head": "right_eye",
+ "tail": "right_eye_outer",
},
"ear.R": {
- 'head': 'right_eye_outer',
- 'tail': 'right_ear',
+ "head": "right_eye_outer",
+ "tail": "right_ear",
},
"lip.T.R": {
- 'head': 'nose',
- 'tail': 'mouth_right',
+ "head": "nose",
+ "tail": "mouth_right",
},
"lip.B.R": {
- 'head': 'mouth_right',
- 'tail': 'mouth_left',
+ "head": "mouth_right",
+ "tail": "mouth_left",
},
"lip.T.L": {
- 'head': 'nose',
- 'tail': 'mouth_left',
+ "head": "nose",
+ "tail": "mouth_left",
},
"hand.L": {
- 'head': 'left_wrist',
- 'tail': 'left_index',
+ "head": "left_wrist",
+ "tail": "left_index",
},
"forearm.L": {
- 'head': 'left_elbow',
- 'tail': 'left_wrist',
+ "head": "left_elbow",
+ "tail": "left_wrist",
},
"upper_arm.L": {
- 'head': 'left_shoulder',
- 'tail': 'left_elbow',
+ "head": "left_shoulder",
+ "tail": "left_elbow",
},
"shoulder.L": {
- 'head': 'neck_center',
- 'tail': 'left_shoulder',
+ "head": "neck_center",
+ "tail": "left_shoulder",
},
"hand.R": {
- 'head': 'right_wrist',
- 'tail': 'right_index',
+ "head": "right_wrist",
+ "tail": "right_index",
},
"forearm.R": {
- 'head': 'right_elbow',
- 'tail': 'right_wrist',
+ "head": "right_elbow",
+ "tail": "right_wrist",
},
"upper_arm.R": {
- 'head': 'right_shoulder',
- 'tail': 'right_elbow',
+ "head": "right_shoulder",
+ "tail": "right_elbow",
},
"shoulder.R": {
- 'head': 'neck_center',
- 'tail': 'right_shoulder',
+ "head": "neck_center",
+ "tail": "right_shoulder",
},
"thigh.L": {
- 'head': 'left_hip',
- 'tail': 'left_knee',
+ "head": "left_hip",
+ "tail": "left_knee",
},
"shin.L": {
- 'head': 'left_knee',
- 'tail': 'left_ankle',
+ "head": "left_knee",
+ "tail": "left_ankle",
},
"heel.02.L": {
- 'head': 'left_heel',
- 'tail': 'left_ankle',
+ "head": "left_heel",
+ "tail": "left_ankle",
},
"foot.L": {
- 'head': 'left_ankle',
- 'tail': 'left_foot_index',
+ "head": "left_ankle",
+ "tail": "left_foot_index",
},
"toe.L": {
- 'head': 'left_foot_index',
- 'tail': 'left_heel',
+ "head": "left_foot_index",
+ "tail": "left_heel",
},
"thigh.R": {
- 'head': 'right_hip',
- 'tail': 'right_knee',
+ "head": "right_hip",
+ "tail": "right_knee",
},
"shin.R": {
- 'head': 'right_knee',
- 'tail': 'right_ankle',
+ "head": "right_knee",
+ "tail": "right_ankle",
},
"heel.02.R": {
- 'head': 'right_heel',
- 'tail': 'right_ankle',
+ "head": "right_heel",
+ "tail": "right_ankle",
},
"foot.R": {
- 'head': 'right_ankle',
- 'tail': 'right_foot_index',
+ "head": "right_ankle",
+ "tail": "right_foot_index",
},
"toe.R": {
- 'head': 'right_foot_index',
- 'tail': 'right_heel',
+ "head": "right_foot_index",
+ "tail": "right_heel",
},
"pelvis.L": {
- 'head': 'hip_center',
- 'tail': 'left_hip',
+ "head": "hip_center",
+ "tail": "left_hip",
},
"pelvis.R": {
- 'head': 'hip_center',
- 'tail': 'right_hip',
+ "head": "hip_center",
+ "tail": "right_hip",
},
"spine": {
- 'head': 'hip_center',
- 'tail': 'chest_center',
+ "head": "hip_center",
+ "tail": "chest_center",
},
"spine.001": {
- 'head': 'chest_center',
- 'tail': 'neck_center',
+ "head": "chest_center",
+ "tail": "neck_center",
},
"spine.002": {
- 'head': 'neck_center',
- 'tail': 'head_center',
+ "head": "neck_center",
+ "tail": "head_center",
},
"spine.003": {
- 'head': 'head_center',
- 'tail': 'nose',
+ "head": "head_center",
+ "tail": "nose",
},
"palm.01.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_index_finger_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_index_finger_mcp",
},
"palm.02.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_middle_finger_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_middle_finger_mcp",
},
"palm.03.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_ring_finger_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_ring_finger_mcp",
},
"palm.04.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_pinky_finger_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_pinky_finger_mcp",
},
-
"thumb.01.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_thumb_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_thumb_mcp",
},
"thumb.02.R": {
- 'head': 'right_hand_thumb_mcp',
- 'tail': 'right_hand_thumb_ip',
+ "head": "right_hand_thumb_mcp",
+ "tail": "right_hand_thumb_ip",
},
"thumb.03.R": {
- 'head': 'right_hand_thumb_ip',
- 'tail': 'right_hand_thumb_tip',
+ "head": "right_hand_thumb_ip",
+ "tail": "right_hand_thumb_tip",
},
-
"f_index.01.R": {
- 'head': 'right_hand_index_finger_mcp',
- 'tail': 'right_hand_index_finger_pip',
+ "head": "right_hand_index_finger_mcp",
+ "tail": "right_hand_index_finger_pip",
},
"f_index.02.R": {
- 'head': 'right_hand_index_finger_pip',
- 'tail': 'right_hand_index_finger_dip',
+ "head": "right_hand_index_finger_pip",
+ "tail": "right_hand_index_finger_dip",
},
"f_index.03.R": {
- 'head': 'right_hand_index_finger_dip',
- 'tail': 'right_hand_index_finger_tip',
+ "head": "right_hand_index_finger_dip",
+ "tail": "right_hand_index_finger_tip",
},
-
"f_middle.01.R": {
- 'head': 'right_hand_middle_finger_mcp',
- 'tail': 'right_hand_middle_finger_pip',
+ "head": "right_hand_middle_finger_mcp",
+ "tail": "right_hand_middle_finger_pip",
},
"f_middle.02.R": {
- 'head': 'right_hand_middle_finger_pip',
- 'tail': 'right_hand_middle_finger_dip',
+ "head": "right_hand_middle_finger_pip",
+ "tail": "right_hand_middle_finger_dip",
},
"f_middle.03.R": {
- 'head': 'right_hand_middle_finger_dip',
- 'tail': 'right_hand_middle_finger_tip',
+ "head": "right_hand_middle_finger_dip",
+ "tail": "right_hand_middle_finger_tip",
},
-
"f_ring.01.R": {
- 'head': 'right_hand_ring_finger_mcp',
- 'tail': 'right_hand_ring_finger_pip',
+ "head": "right_hand_ring_finger_mcp",
+ "tail": "right_hand_ring_finger_pip",
},
"f_ring.02.R": {
- 'head': 'right_hand_ring_finger_pip',
- 'tail': 'right_hand_ring_finger_dip',
+ "head": "right_hand_ring_finger_pip",
+ "tail": "right_hand_ring_finger_dip",
},
"f_ring.03.R": {
- 'head': 'right_hand_ring_finger_dip',
- 'tail': 'right_hand_ring_finger_tip',
+ "head": "right_hand_ring_finger_dip",
+ "tail": "right_hand_ring_finger_tip",
},
-
"f_pinky.01.R": {
- 'head': 'right_hand_pinky_finger_mcp',
- 'tail': 'right_hand_pinky_finger_pip',
+ "head": "right_hand_pinky_finger_mcp",
+ "tail": "right_hand_pinky_finger_pip",
},
"f_pinky.02.R": {
- 'head': 'right_hand_pinky_finger_pip',
- 'tail': 'right_hand_pinky_finger_dip',
+ "head": "right_hand_pinky_finger_pip",
+ "tail": "right_hand_pinky_finger_dip",
},
"f_pinky.03.R": {
- 'head': 'right_hand_pinky_finger_dip',
- 'tail': 'right_hand_pinky_finger_tip',
+ "head": "right_hand_pinky_finger_dip",
+ "tail": "right_hand_pinky_finger_tip",
},
"thumb.01.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_thumb_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_thumb_mcp",
},
"thumb.02.L": {
- 'head': 'left_hand_thumb_mcp',
- 'tail': 'left_hand_thumb_ip',
+ "head": "left_hand_thumb_mcp",
+ "tail": "left_hand_thumb_ip",
},
"thumb.03.L": {
- 'head': 'left_hand_thumb_ip',
- 'tail': 'left_hand_thumb_tip',
+ "head": "left_hand_thumb_ip",
+ "tail": "left_hand_thumb_tip",
},
"palm.01.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_index_finger_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_index_finger_mcp",
},
"palm.02.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_middle_finger_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_middle_finger_mcp",
},
"palm.03.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_ring_finger_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_ring_finger_mcp",
},
"palm.04.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_pinky_finger_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_pinky_finger_mcp",
},
"f_index.01.L": {
- 'head': 'left_hand_index_finger_mcp',
- 'tail': 'left_hand_index_finger_pip',
+ "head": "left_hand_index_finger_mcp",
+ "tail": "left_hand_index_finger_pip",
},
"f_index.02.L": {
- 'head': 'left_hand_index_finger_pip',
- 'tail': 'left_hand_index_finger_dip',
+ "head": "left_hand_index_finger_pip",
+ "tail": "left_hand_index_finger_dip",
},
"f_index.03.L": {
- 'head': 'left_hand_index_finger_dip',
- 'tail': 'left_hand_index_finger_tip',
+ "head": "left_hand_index_finger_dip",
+ "tail": "left_hand_index_finger_tip",
},
-
"f_middle.01.L": {
- 'head': 'left_hand_middle_finger_mcp',
- 'tail': 'left_hand_middle_finger_pip',
+ "head": "left_hand_middle_finger_mcp",
+ "tail": "left_hand_middle_finger_pip",
},
"f_middle.02.L": {
- 'head': 'left_hand_middle_finger_pip',
- 'tail': 'left_hand_middle_finger_dip',
+ "head": "left_hand_middle_finger_pip",
+ "tail": "left_hand_middle_finger_dip",
},
"f_middle.03.L": {
- 'head': 'left_hand_middle_finger_dip',
- 'tail': 'left_hand_middle_finger_tip',
+ "head": "left_hand_middle_finger_dip",
+ "tail": "left_hand_middle_finger_tip",
},
"f_ring.01.L": {
- 'head': 'left_hand_ring_finger_mcp',
- 'tail': 'left_hand_ring_finger_pip',
+ "head": "left_hand_ring_finger_mcp",
+ "tail": "left_hand_ring_finger_pip",
},
"f_ring.02.L": {
- 'head': 'left_hand_ring_finger_pip',
- 'tail': 'left_hand_ring_finger_dip',
+ "head": "left_hand_ring_finger_pip",
+ "tail": "left_hand_ring_finger_dip",
},
"f_ring.03.L": {
- 'head': 'left_hand_ring_finger_dip',
- 'tail': 'left_hand_ring_finger_tip',
+ "head": "left_hand_ring_finger_dip",
+ "tail": "left_hand_ring_finger_tip",
},
"f_pinky.01.L": {
- 'head': 'left_hand_pinky_finger_mcp',
- 'tail': 'left_hand_pinky_finger_pip',
+ "head": "left_hand_pinky_finger_mcp",
+ "tail": "left_hand_pinky_finger_pip",
},
"f_pinky.02.L": {
- 'head': 'left_hand_pinky_finger_pip',
- 'tail': 'left_hand_pinky_finger_dip',
+ "head": "left_hand_pinky_finger_pip",
+ "tail": "left_hand_pinky_finger_dip",
},
"f_pinky.03.L": {
- 'head': 'left_hand_pinky_finger_dip',
- 'tail': 'left_hand_pinky_finger_tip',
+ "head": "left_hand_pinky_finger_dip",
+ "tail": "left_hand_pinky_finger_tip",
},
-
}
# %% create metarig
@@ -821,313 +870,328 @@ try:
# %% loop through dictionary - set head and tail of each bone's to the relevant empty
# bpy.ops.object.posemode_toggle()
try:
- bpy.ops.object.mode_set(mode='EDIT')
+ bpy.ops.object.mode_set(mode="EDIT")
except:
pass
for this_bone in list(this_metarig.data.bones):
this_bone_name = this_bone.name
- print(f'--{this_bone_name}--')
+ print(f"--{this_bone_name}--")
this_edit_bone = this_metarig.data.edit_bones[
- this_bone_name] # note this is using EDIT_bones, not regular bones b/c blender is a wild wacky place
+ this_bone_name
+ ] # note this is using EDIT_bones, not regular bones b/c blender is a wild wacky place
if this_bone_name in metarig_bone_empty_pairs_dict_of_dicts:
this_bone_dict = metarig_bone_empty_pairs_dict_of_dicts[this_bone_name]
- this_head_empty_name = this_bone_dict['head']
- this_tail_empty_name = this_bone_dict['tail']
- print('setting (bone:head->tail){}:{}->{})'.format(this_bone_name, this_head_empty_name,
- this_tail_empty_name))
+ this_head_empty_name = this_bone_dict["head"]
+ this_tail_empty_name = this_bone_dict["tail"]
+ print(
+ "setting (bone:head->tail){}:{}->{})".format(
+ this_bone_name, this_head_empty_name, this_tail_empty_name
+ )
+ )
if this_tail_empty_name is not None:
print(
- f'setting bone: {this_bone_name}: head: {bpy.data.objects[this_head_empty_name].location}, tail: {bpy.data.objects[this_tail_empty_name].location}')
- this_edit_bone.head = bpy.data.objects[this_head_empty_name].location
- this_edit_bone.tail = bpy.data.objects[this_tail_empty_name].location
+ f"setting bone: {this_bone_name}: head: {bpy.data.objects[this_head_empty_name].location}, tail: {bpy.data.objects[this_tail_empty_name].location}"
+ )
+ this_edit_bone.head = bpy.data.objects[
+ this_head_empty_name
+ ].location
+ this_edit_bone.tail = bpy.data.objects[
+ this_tail_empty_name
+ ].location
else: # remove bones with no tail (and/or head) until we figure out a better plan
- print(f'removing bone: {this_bone_name}')
+ print(f"removing bone: {this_bone_name}")
this_metarig.data.edit_bones.remove(this_edit_bone)
else:
- print(f'removing bone: {this_bone_name}')
+ print(f"removing bone: {this_bone_name}")
this_metarig.data.edit_bones.remove(this_edit_bone)
try:
- bpy.ops.object.mode_set(mode='OBJECT')
+ bpy.ops.object.mode_set(mode="OBJECT")
except:
pass
#######################################################
#### Damped track bones to empties
- bpy.ops.object.mode_set(mode='OBJECT')
- armature_name = 'metarig'
+ bpy.ops.object.mode_set(mode="OBJECT")
+ armature_name = "metarig"
this_armature = bpy.data.objects[armature_name]
rig_constraint_dict_of_dicts = {
"nose.001": {
- 'COPY_LOCATION': 'head_center',
- 'DAMPED_TRACK': 'nose',
+ "COPY_LOCATION": "head_center",
+ "DAMPED_TRACK": "nose",
},
"nose.L.001": {
- 'DAMPED_TRACK': 'left_eye_inner',
+ "DAMPED_TRACK": "left_eye_inner",
},
"eye.L": {
- 'DAMPED_TRACK': 'left_eye',
+ "DAMPED_TRACK": "left_eye",
},
"lid.B.L": {
- 'DAMPED_TRACK': 'left_eye_outer',
+ "DAMPED_TRACK": "left_eye_outer",
},
"ear.L": {
- 'DAMPED_TRACK': 'left_ear',
+ "DAMPED_TRACK": "left_ear",
},
"nose.R.001": {
- 'DAMPED_TRACK': 'right_eye_inner',
+ "DAMPED_TRACK": "right_eye_inner",
},
"eye.R": {
- 'DAMPED_TRACK': 'right_eye',
+ "DAMPED_TRACK": "right_eye",
},
"lid.B.R": {
- 'DAMPED_TRACK': 'right_eye_outer',
+ "DAMPED_TRACK": "right_eye_outer",
},
"ear.R": {
- 'DAMPED_TRACK': 'right_ear',
+ "DAMPED_TRACK": "right_ear",
},
"lip.T.R": {
- 'DAMPED_TRACK': 'mouth_right',
+ "DAMPED_TRACK": "mouth_right",
},
"lip.B.R": {
- 'DAMPED_TRACK': 'mouth_left',
+ "DAMPED_TRACK": "mouth_left",
},
"lip.T.L": {
- 'DAMPED_TRACK': 'nose',
+ "DAMPED_TRACK": "nose",
},
"spine": {
- 'COPY_LOCATION': 'hip_center',
- 'DAMPED_TRACK': 'chest_center',
+ "COPY_LOCATION": "hip_center",
+ "DAMPED_TRACK": "chest_center",
},
"spine.001": {
- 'DAMPED_TRACK': 'neck_center',
+ "DAMPED_TRACK": "neck_center",
},
"spine.002": {
- 'DAMPED_TRACK': 'head_center',
+ "DAMPED_TRACK": "head_center",
},
"spine.003": {
- 'DAMPED_TRACK': 'nose',
+ "DAMPED_TRACK": "nose",
},
"pelvis.L": {
- 'DAMPED_TRACK': 'left_hip',
+ "DAMPED_TRACK": "left_hip",
},
"thigh.L": {
- 'COPY_LOCATION': 'left_hip',
- 'DAMPED_TRACK': 'left_knee',
+ "COPY_LOCATION": "left_hip",
+ "DAMPED_TRACK": "left_knee",
},
"shin.L": {
- 'DAMPED_TRACK': 'left_ankle',
+ "DAMPED_TRACK": "left_ankle",
},
"foot.L": {
- 'DAMPED_TRACK': 'left_foot_index',
+ "DAMPED_TRACK": "left_foot_index",
},
"heel.02.L": {
- 'DAMPED_TRACK': 'left_ankle',
+ "DAMPED_TRACK": "left_ankle",
},
"shoulder.L": {
- 'COPY_LOCATION': 'neck_center',
- 'DAMPED_TRACK': 'left_shoulder',
+ "COPY_LOCATION": "neck_center",
+ "DAMPED_TRACK": "left_shoulder",
},
"upper_arm.L": {
- 'DAMPED_TRACK': 'left_elbow',
+ "DAMPED_TRACK": "left_elbow",
},
"forearm.L": {
- 'DAMPED_TRACK': 'left_wrist',
+ "DAMPED_TRACK": "left_wrist",
},
"hand.L": {
- 'DAMPED_TRACK': 'left_index',
+ "DAMPED_TRACK": "left_index",
},
"pelvis.R": {
- 'DAMPED_TRACK': 'right_hip',
+ "DAMPED_TRACK": "right_hip",
},
"thigh.R": {
- 'COPY_LOCATION': 'right_hip',
- 'DAMPED_TRACK': 'right_knee',
+ "COPY_LOCATION": "right_hip",
+ "DAMPED_TRACK": "right_knee",
},
"shin.R": {
- 'DAMPED_TRACK': 'right_ankle',
+ "DAMPED_TRACK": "right_ankle",
},
"foot.R": {
- 'DAMPED_TRACK': 'right_foot_index',
+ "DAMPED_TRACK": "right_foot_index",
},
"heel.02.R": {
- 'DAMPED_TRACK': 'right_ankle',
+ "DAMPED_TRACK": "right_ankle",
},
"shoulder.R": {
- 'COPY_LOCATION': 'neck_center',
- 'DAMPED_TRACK': 'right_shoulder',
+ "COPY_LOCATION": "neck_center",
+ "DAMPED_TRACK": "right_shoulder",
},
"upper_arm.R": {
- 'DAMPED_TRACK': 'right_elbow',
+ "DAMPED_TRACK": "right_elbow",
},
"forearm.R": {
- 'DAMPED_TRACK': 'right_wrist',
+ "DAMPED_TRACK": "right_wrist",
},
"hand.R": {
- 'DAMPED_TRACK': 'right_index',
+ "DAMPED_TRACK": "right_index",
},
"thumb.01.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'DAMPED_TRACK': 'right_hand_thumb_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "DAMPED_TRACK": "right_hand_thumb_mcp",
},
"thumb.02.R": {
- 'DAMPED_TRACK': 'right_hand_thumb_ip',
+ "DAMPED_TRACK": "right_hand_thumb_ip",
},
"thumb.02.R": {
- 'DAMPED_TRACK': 'right_hand_thumb_tip',
+ "DAMPED_TRACK": "right_hand_thumb_tip",
},
"palm.01.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'DAMPED_TRACK': 'right_hand_index_finger_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "DAMPED_TRACK": "right_hand_index_finger_mcp",
},
"f_index.01.R": {
- 'DAMPED_TRACK': 'right_hand_index_finger_pip',
+ "DAMPED_TRACK": "right_hand_index_finger_pip",
},
"f_index.02.R": {
- 'DAMPED_TRACK': 'right_hand_index_finger_dip',
+ "DAMPED_TRACK": "right_hand_index_finger_dip",
},
"f_index.03.R": {
- 'DAMPED_TRACK': 'right_hand_index_finger_tip',
+ "DAMPED_TRACK": "right_hand_index_finger_tip",
},
"palm.02.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'DAMPED_TRACK': 'right_hand_middle_finger_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "DAMPED_TRACK": "right_hand_middle_finger_mcp",
},
"f_middle.01.R": {
- 'DAMPED_TRACK': 'right_hand_middle_finger_pip',
+ "DAMPED_TRACK": "right_hand_middle_finger_pip",
},
"f_middle.02.R": {
- 'DAMPED_TRACK': 'right_hand_middle_finger_dip',
+ "DAMPED_TRACK": "right_hand_middle_finger_dip",
},
"f_middle.03.R": {
- 'DAMPED_TRACK': 'right_hand_middle_finger_tip',
+ "DAMPED_TRACK": "right_hand_middle_finger_tip",
},
"palm.03.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'DAMPED_TRACK': 'right_hand_ring_finger_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "DAMPED_TRACK": "right_hand_ring_finger_mcp",
},
"f_ring.01.R": {
- 'DAMPED_TRACK': 'right_hand_ring_finger_pip',
+ "DAMPED_TRACK": "right_hand_ring_finger_pip",
},
"f_ring.02.R": {
- 'DAMPED_TRACK': 'right_hand_ring_finger_dip',
+ "DAMPED_TRACK": "right_hand_ring_finger_dip",
},
"f_ring.03.R": {
- 'DAMPED_TRACK': 'right_hand_ring_finger_tip',
+ "DAMPED_TRACK": "right_hand_ring_finger_tip",
},
"palm.04.R": {
- 'COPY_LOCATION': 'right_wrist',
- 'DAMPED_TRACK': 'right_hand_pinky_finger_mcp',
+ "COPY_LOCATION": "right_wrist",
+ "DAMPED_TRACK": "right_hand_pinky_finger_mcp",
},
"f_pinky.01.R": {
- 'DAMPED_TRACK': 'right_hand_pinky_finger_pip',
+ "DAMPED_TRACK": "right_hand_pinky_finger_pip",
},
"f_pinky.02.R": {
- 'DAMPED_TRACK': 'right_hand_pinky_finger_dip',
+ "DAMPED_TRACK": "right_hand_pinky_finger_dip",
},
"f_pinky.03.R": {
- 'DAMPED_TRACK': 'right_hand_pinky_finger_tip',
+ "DAMPED_TRACK": "right_hand_pinky_finger_tip",
},
"thumb.01.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'DAMPED_TRACK': 'left_hand_thumb_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "DAMPED_TRACK": "left_hand_thumb_mcp",
},
"thumb.02.L": {
- 'DAMPED_TRACK': 'left_hand_thumb_ip',
+ "DAMPED_TRACK": "left_hand_thumb_ip",
},
"thumb.03.L": {
- 'DAMPED_TRACK': 'left_hand_thumb_tip',
+ "DAMPED_TRACK": "left_hand_thumb_tip",
},
"palm.01.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'DAMPED_TRACK': 'left_hand_index_finger_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "DAMPED_TRACK": "left_hand_index_finger_mcp",
},
"f_index.01.L": {
- 'DAMPED_TRACK': 'left_hand_index_finger_pip',
+ "DAMPED_TRACK": "left_hand_index_finger_pip",
},
"f_index.02.L": {
- 'DAMPED_TRACK': 'left_hand_index_finger_dip',
+ "DAMPED_TRACK": "left_hand_index_finger_dip",
},
"f_index.03.L": {
- 'DAMPED_TRACK': 'left_hand_index_finger_tip',
+ "DAMPED_TRACK": "left_hand_index_finger_tip",
},
"palm.02.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'DAMPED_TRACK': 'left_hand_middle_finger_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "DAMPED_TRACK": "left_hand_middle_finger_mcp",
},
"f_middle.01.L": {
- 'DAMPED_TRACK': 'left_hand_middle_finger_pip',
+ "DAMPED_TRACK": "left_hand_middle_finger_pip",
},
"f_middle.02.L": {
- 'DAMPED_TRACK': 'left_hand_middle_finger_dip',
+ "DAMPED_TRACK": "left_hand_middle_finger_dip",
},
"f_middle.03.L": {
- 'DAMPED_TRACK': 'left_hand_middle_finger_tip',
+ "DAMPED_TRACK": "left_hand_middle_finger_tip",
},
"palm.03.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'DAMPED_TRACK': 'left_hand_ring_finger_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "DAMPED_TRACK": "left_hand_ring_finger_mcp",
},
"f_ring.01.L": {
- 'DAMPED_TRACK': 'left_hand_ring_finger_pip',
+ "DAMPED_TRACK": "left_hand_ring_finger_pip",
},
"f_ring.02.L": {
- 'DAMPED_TRACK': 'left_hand_ring_finger_dip',
+ "DAMPED_TRACK": "left_hand_ring_finger_dip",
},
"f_ring.03.L": {
- 'DAMPED_TRACK': 'left_hand_ring_finger_tip',
+ "DAMPED_TRACK": "left_hand_ring_finger_tip",
},
"palm.04.L": {
- 'COPY_LOCATION': 'left_wrist',
- 'DAMPED_TRACK': 'left_hand_pinky_finger_mcp',
+ "COPY_LOCATION": "left_wrist",
+ "DAMPED_TRACK": "left_hand_pinky_finger_mcp",
},
"f_pinky.01.L": {
- 'DAMPED_TRACK': 'left_hand_pinky_finger_pip',
+ "DAMPED_TRACK": "left_hand_pinky_finger_pip",
},
"f_pinky.02.L": {
- 'DAMPED_TRACK': 'left_hand_pinky_finger_dip',
+ "DAMPED_TRACK": "left_hand_pinky_finger_dip",
},
"f_pinky.03.L": {
- 'DAMPED_TRACK': 'left_hand_pinky_finger_tip',
+ "DAMPED_TRACK": "left_hand_pinky_finger_tip",
},
}
# loop through dictionary applying constraints
try:
- bpy.ops.object.mode_set(mode='POSE')
+ bpy.ops.object.mode_set(mode="POSE")
except:
pass
for this_bone_name, this_bone_dict in rig_constraint_dict_of_dicts.items():
- print(f'---Setting constraints for bone:{this_bone_name}---')
+ print(f"---Setting constraints for bone:{this_bone_name}---")
- for this_constraint_name, this_constraint_target_empty_name in this_bone_dict.items():
- print(f'constraint: {this_constraint_name} with target:{this_constraint_target_empty_name}')
- print('grab bone')
+ for (
+ this_constraint_name,
+ this_constraint_target_empty_name,
+ ) in this_bone_dict.items():
+ print(
+ f"constraint: {this_constraint_name} with target:{this_constraint_target_empty_name}"
+ )
+ print("grab bone")
this_bone = this_armature.pose.bones[this_bone_name]
- print('apply bone')
+ print("apply bone")
this_constraint = this_bone.constraints.new(type=this_constraint_name)
this_constraint.name = this_constraint_name
this_constraint.target = bpy.data.objects[
- this_constraint_target_empty_name] # point constraint at relevant empty object
+ this_constraint_target_empty_name
+ ] # point constraint at relevant empty object
##############
## create extra bones
extra_bone_empty_pairs_dict_of_dicts = {
"hand.L": {
- 'head': 'left_wrist',
- 'tail': 'left_index',
+ "head": "left_wrist",
+ "tail": "left_index",
},
"forearm.L": {
- 'head': 'left_elbow',
- 'tail': 'left_wrist',
+ "head": "left_elbow",
+ "tail": "left_wrist",
},
}
@@ -1136,327 +1200,316 @@ try:
stick_figure_mesh_edge_pairs_dict_of_dicts = {
"nose.001": {
- 'head': 'head_center',
- 'tail': 'nose',
+ "head": "head_center",
+ "tail": "nose",
},
"nose.L.001": {
- 'head': 'nose',
- 'tail': 'left_eye_inner',
+ "head": "nose",
+ "tail": "left_eye_inner",
},
"eye.L": {
- 'head': 'left_eye_inner',
- 'tail': 'left_eye',
+ "head": "left_eye_inner",
+ "tail": "left_eye",
},
"lid.B.L": {
- 'head': 'left_eye',
- 'tail': 'left_eye_outer',
+ "head": "left_eye",
+ "tail": "left_eye_outer",
},
"ear.L": {
- 'head': 'left_eye_outer',
- 'tail': 'left_ear',
+ "head": "left_eye_outer",
+ "tail": "left_ear",
},
"nose.R.001": {
- 'head': 'nose',
- 'tail': 'right_eye_inner',
+ "head": "nose",
+ "tail": "right_eye_inner",
},
"eye.R": {
- 'head': 'right_eye_inner',
- 'tail': 'right_eye',
+ "head": "right_eye_inner",
+ "tail": "right_eye",
},
"lid.B.R": {
- 'head': 'right_eye',
- 'tail': 'right_eye_outer',
+ "head": "right_eye",
+ "tail": "right_eye_outer",
},
"ear.R": {
- 'head': 'right_eye_outer',
- 'tail': 'right_ear',
+ "head": "right_eye_outer",
+ "tail": "right_ear",
},
"lip.T.R": {
- 'head': 'nose',
- 'tail': 'mouth_right',
+ "head": "nose",
+ "tail": "mouth_right",
},
"lip.B.R": {
- 'head': 'mouth_right',
- 'tail': 'mouth_left',
+ "head": "mouth_right",
+ "tail": "mouth_left",
},
"lip.T.L": {
- 'head': 'nose',
- 'tail': 'mouth_left',
+ "head": "nose",
+ "tail": "mouth_left",
},
"forearm.L": {
- 'head': 'left_elbow',
- 'tail': 'left_wrist',
+ "head": "left_elbow",
+ "tail": "left_wrist",
},
"upper_arm.L": {
- 'head': 'left_shoulder',
- 'tail': 'left_elbow',
+ "head": "left_shoulder",
+ "tail": "left_elbow",
},
"shoulder.L": {
- 'head': 'neck_center',
- 'tail': 'left_shoulder',
+ "head": "neck_center",
+ "tail": "left_shoulder",
},
"forearm.R": {
- 'head': 'right_elbow',
- 'tail': 'right_wrist',
+ "head": "right_elbow",
+ "tail": "right_wrist",
},
"upper_arm.R": {
- 'head': 'right_shoulder',
- 'tail': 'right_elbow',
+ "head": "right_shoulder",
+ "tail": "right_elbow",
},
"shoulder.R": {
- 'head': 'neck_center',
- 'tail': 'right_shoulder',
+ "head": "neck_center",
+ "tail": "right_shoulder",
},
"thigh.L": {
- 'head': 'left_hip',
- 'tail': 'left_knee',
+ "head": "left_hip",
+ "tail": "left_knee",
},
"shin.L": {
- 'head': 'left_knee',
- 'tail': 'left_ankle',
+ "head": "left_knee",
+ "tail": "left_ankle",
},
"heel.02.L": {
- 'head': 'left_heel',
- 'tail': 'left_ankle',
+ "head": "left_heel",
+ "tail": "left_ankle",
},
"foot.L": {
- 'head': 'left_ankle',
- 'tail': 'left_foot_index',
+ "head": "left_ankle",
+ "tail": "left_foot_index",
},
"toe.L": {
- 'head': 'left_foot_index',
- 'tail': 'left_heel',
+ "head": "left_foot_index",
+ "tail": "left_heel",
},
"thigh.R": {
- 'head': 'right_hip',
- 'tail': 'right_knee',
+ "head": "right_hip",
+ "tail": "right_knee",
},
"shin.R": {
- 'head': 'right_knee',
- 'tail': 'right_ankle',
+ "head": "right_knee",
+ "tail": "right_ankle",
},
"heel.02.R": {
- 'head': 'right_heel',
- 'tail': 'right_ankle',
+ "head": "right_heel",
+ "tail": "right_ankle",
},
"foot.R": {
- 'head': 'right_ankle',
- 'tail': 'right_foot_index',
+ "head": "right_ankle",
+ "tail": "right_foot_index",
},
"toe.R": {
- 'head': 'right_foot_index',
- 'tail': 'right_heel',
+ "head": "right_foot_index",
+ "tail": "right_heel",
},
"pelvis.L": {
- 'head': 'hip_center',
- 'tail': 'left_hip',
+ "head": "hip_center",
+ "tail": "left_hip",
},
"pelvis.R": {
- 'head': 'hip_center',
- 'tail': 'right_hip',
+ "head": "hip_center",
+ "tail": "right_hip",
},
"spine": {
- 'head': 'hip_center',
- 'tail': 'chest_center',
+ "head": "hip_center",
+ "tail": "chest_center",
},
"spine.001": {
- 'head': 'chest_center',
- 'tail': 'neck_center',
+ "head": "chest_center",
+ "tail": "neck_center",
},
"spine.002": {
- 'head': 'neck_center',
- 'tail': 'head_center',
+ "head": "neck_center",
+ "tail": "head_center",
},
"spine.003": {
- 'head': 'head_center',
- 'tail': 'nose',
+ "head": "head_center",
+ "tail": "nose",
},
"palm.01.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_index_finger_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_index_finger_mcp",
},
"palm.02.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_middle_finger_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_middle_finger_mcp",
},
"palm.03.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_ring_finger_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_ring_finger_mcp",
},
"palm.04.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_pinky_finger_mcp',
+ "head": "right_wrist",
+ "tail": "right_hand_pinky_finger_mcp",
},
-
"thumb.01.R": {
- 'head': 'right_wrist',
- 'tail': 'right_hand_thumb_cmc',
+ "head": "right_wrist",
+ "tail": "right_hand_thumb_cmc",
},
"thumb.02.R": {
- 'head': 'right_hand_thumb_cmc',
- 'tail': 'right_hand_thumb_mcp',
+ "head": "right_hand_thumb_cmc",
+ "tail": "right_hand_thumb_mcp",
},
"thumb.03.R": {
- 'head': 'right_hand_thumb_mcp',
- 'tail': 'right_hand_thumb_ip',
+ "head": "right_hand_thumb_mcp",
+ "tail": "right_hand_thumb_ip",
},
"thumb.04.R": {
- 'head': 'right_hand_thumb_ip',
- 'tail': 'right_hand_thumb_tip',
+ "head": "right_hand_thumb_ip",
+ "tail": "right_hand_thumb_tip",
},
"f_index.01.R": {
- 'head': 'right_hand_index_finger_mcp',
- 'tail': 'right_hand_index_finger_pip',
+ "head": "right_hand_index_finger_mcp",
+ "tail": "right_hand_index_finger_pip",
},
"f_index.02.R": {
- 'head': 'right_hand_index_finger_pip',
- 'tail': 'right_hand_index_finger_dip',
+ "head": "right_hand_index_finger_pip",
+ "tail": "right_hand_index_finger_dip",
},
"f_index.03.R": {
- 'head': 'right_hand_index_finger_dip',
- 'tail': 'right_hand_index_finger_tip',
+ "head": "right_hand_index_finger_dip",
+ "tail": "right_hand_index_finger_tip",
},
"f_middle.01.R": {
- 'head': 'right_hand_middle_finger_mcp',
- 'tail': 'right_hand_middle_finger_pip',
+ "head": "right_hand_middle_finger_mcp",
+ "tail": "right_hand_middle_finger_pip",
},
"f_middle.02.R": {
- 'head': 'right_hand_middle_finger_pip',
- 'tail': 'right_hand_middle_finger_dip',
+ "head": "right_hand_middle_finger_pip",
+ "tail": "right_hand_middle_finger_dip",
},
"f_middle.03.R": {
- 'head': 'right_hand_middle_finger_dip',
- 'tail': 'right_hand_middle_finger_tip',
+ "head": "right_hand_middle_finger_dip",
+ "tail": "right_hand_middle_finger_tip",
},
"f_ring.01.R": {
- 'head': 'right_hand_ring_finger_mcp',
- 'tail': 'right_hand_ring_finger_pip',
+ "head": "right_hand_ring_finger_mcp",
+ "tail": "right_hand_ring_finger_pip",
},
"f_ring.02.R": {
- 'head': 'right_hand_ring_finger_pip',
- 'tail': 'right_hand_ring_finger_dip',
+ "head": "right_hand_ring_finger_pip",
+ "tail": "right_hand_ring_finger_dip",
},
"f_ring.03.R": {
- 'head': 'right_hand_ring_finger_dip',
- 'tail': 'right_hand_ring_finger_tip',
+ "head": "right_hand_ring_finger_dip",
+ "tail": "right_hand_ring_finger_tip",
},
-
"f_pinky.01.R": {
- 'head': 'right_hand_pinky_finger_mcp',
- 'tail': 'right_hand_pinky_finger_pip',
+ "head": "right_hand_pinky_finger_mcp",
+ "tail": "right_hand_pinky_finger_pip",
},
"f_pinky.02.R": {
- 'head': 'right_hand_pinky_finger_pip',
- 'tail': 'right_hand_pinky_finger_dip',
+ "head": "right_hand_pinky_finger_pip",
+ "tail": "right_hand_pinky_finger_dip",
},
"f_pinky.03.R": {
- 'head': 'right_hand_pinky_finger_dip',
- 'tail': 'right_hand_pinky_finger_tip',
+ "head": "right_hand_pinky_finger_dip",
+ "tail": "right_hand_pinky_finger_tip",
},
"thumb.01.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_thumb_cmc',
+ "head": "left_wrist",
+ "tail": "left_hand_thumb_cmc",
},
"thumb.02.L": {
- 'head': 'left_hand_thumb_cmc',
- 'tail': 'left_hand_thumb_mcp',
+ "head": "left_hand_thumb_cmc",
+ "tail": "left_hand_thumb_mcp",
},
"thumb.03.L": {
- 'head': 'left_hand_thumb_mcp',
- 'tail': 'left_hand_thumb_ip',
+ "head": "left_hand_thumb_mcp",
+ "tail": "left_hand_thumb_ip",
},
"thumb.04.L": {
- 'head': 'left_hand_thumb_ip',
- 'tail': 'left_hand_thumb_tip',
+ "head": "left_hand_thumb_ip",
+ "tail": "left_hand_thumb_tip",
},
"palm.01.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_index_finger_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_index_finger_mcp",
},
"palm.02.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_middle_finger_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_middle_finger_mcp",
},
"palm.03.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_ring_finger_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_ring_finger_mcp",
},
"palm.04.L": {
- 'head': 'left_wrist',
- 'tail': 'left_hand_pinky_finger_mcp',
+ "head": "left_wrist",
+ "tail": "left_hand_pinky_finger_mcp",
},
"f_index.01.L": {
- 'head': 'left_hand_index_finger_mcp',
- 'tail': 'left_hand_index_finger_pip',
+ "head": "left_hand_index_finger_mcp",
+ "tail": "left_hand_index_finger_pip",
},
"f_index.02.L": {
- 'head': 'left_hand_index_finger_pip',
- 'tail': 'left_hand_index_finger_dip',
+ "head": "left_hand_index_finger_pip",
+ "tail": "left_hand_index_finger_dip",
},
"f_index.03.L": {
- 'head': 'left_hand_index_finger_dip',
- 'tail': 'left_hand_index_finger_tip',
+ "head": "left_hand_index_finger_dip",
+ "tail": "left_hand_index_finger_tip",
},
"f_middle.01.L": {
- 'head': 'left_hand_middle_finger_mcp',
- 'tail': 'left_hand_middle_finger_pip',
+ "head": "left_hand_middle_finger_mcp",
+ "tail": "left_hand_middle_finger_pip",
},
"f_middle.02.L": {
- 'head': 'left_hand_middle_finger_pip',
- 'tail': 'left_hand_middle_finger_dip',
+ "head": "left_hand_middle_finger_pip",
+ "tail": "left_hand_middle_finger_dip",
},
"f_middle.03.L": {
- 'head': 'left_hand_middle_finger_dip',
- 'tail': 'left_hand_middle_finger_tip',
+ "head": "left_hand_middle_finger_dip",
+ "tail": "left_hand_middle_finger_tip",
},
"f_ring.01.L": {
- 'head': 'left_hand_ring_finger_mcp',
- 'tail': 'left_hand_ring_finger_pip',
+ "head": "left_hand_ring_finger_mcp",
+ "tail": "left_hand_ring_finger_pip",
},
"f_ring.02.L": {
- 'head': 'left_hand_ring_finger_pip',
- 'tail': 'left_hand_ring_finger_dip',
+ "head": "left_hand_ring_finger_pip",
+ "tail": "left_hand_ring_finger_dip",
},
"f_ring.03.L": {
- 'head': 'left_hand_ring_finger_dip',
- 'tail': 'left_hand_ring_finger_tip',
+ "head": "left_hand_ring_finger_dip",
+ "tail": "left_hand_ring_finger_tip",
},
"f_pinky.01.L": {
- 'head': 'left_hand_pinky_finger_mcp',
- 'tail': 'left_hand_pinky_finger_pip',
+ "head": "left_hand_pinky_finger_mcp",
+ "tail": "left_hand_pinky_finger_pip",
},
"f_pinky.02.L": {
- 'head': 'left_hand_pinky_finger_pip',
- 'tail': 'left_hand_pinky_finger_dip',
+ "head": "left_hand_pinky_finger_pip",
+ "tail": "left_hand_pinky_finger_dip",
},
"f_pinky.03.L": {
- 'head': 'left_hand_pinky_finger_dip',
- 'tail': 'left_hand_pinky_finger_tip',
+ "head": "left_hand_pinky_finger_dip",
+ "tail": "left_hand_pinky_finger_tip",
},
}
mediapipe_virtual_markers_weights_dict = {
- 'head_center': {
- 'right_ear': .5,
- 'left_ear': .5
- },
- 'neck_center': {
- 'right_shoulder': .5,
- 'left_shoulder': .5
- },
- 'hip_center': {
- 'right_hip': .5,
- 'left_hip': .5
- },
- 'chest_center': {
- 'right_shoulder': .25,
- 'left_shoulder': .25,
- 'right_hip': .25,
- 'left_hip': .25,
+ "head_center": {"right_ear": 0.5, "left_ear": 0.5},
+ "neck_center": {"right_shoulder": 0.5, "left_shoulder": 0.5},
+ "hip_center": {"right_hip": 0.5, "left_hip": 0.5},
+ "chest_center": {
+ "right_shoulder": 0.25,
+ "left_shoulder": 0.25,
+ "right_hip": 0.25,
+ "left_hip": 0.25,
},
}
- print('loading data as stick figure mesh \o/')
+ print("loading data as stick figure mesh \o/")
#######################################################################
# %% create virtual marker points (redundant from above, I think ? )
@@ -1472,17 +1525,23 @@ try:
this_weight = this_weighted_point_dict[1]
these_weighted_points.append(
mediapipe_skel_fr_mar_dim[
- :,
- mediapipe_tracked_point_names.index(this_point_name),
- :
- ] * this_weight
+ :, mediapipe_tracked_point_names.index(this_point_name), :
+ ]
+ * this_weight
)
this_virtual_point_fr_xyz = np.sum(these_weighted_points, axis=0)
- this_virtual_point_fr_mar_xyz = np.expand_dims(this_virtual_point_fr_xyz, axis=1)
- mediapipe_skel_fr_mar_dim = np.append(mediapipe_skel_fr_mar_dim, this_virtual_point_fr_mar_xyz, axis=1)
- virtual_point_index_dict[this_virtual_marker_name] = mediapipe_skel_fr_mar_dim.shape[1] - 1
+ this_virtual_point_fr_mar_xyz = np.expand_dims(
+ this_virtual_point_fr_xyz, axis=1
+ )
+ mediapipe_skel_fr_mar_dim = np.append(
+ mediapipe_skel_fr_mar_dim, this_virtual_point_fr_mar_xyz, axis=1
+ )
+ virtual_point_index_dict[this_virtual_marker_name] = (
+ mediapipe_skel_fr_mar_dim.shape[1] - 1
+ )
print(
- f'Virtual marker: {this_virtual_marker_name} created at marker index {mediapipe_skel_fr_mar_dim.shape[1] - 1}')
+ f"Virtual marker: {this_virtual_marker_name} created at marker index {mediapipe_skel_fr_mar_dim.shape[1] - 1}"
+ )
# %% define edges
@@ -1490,29 +1549,34 @@ try:
# regular data
for this_bone_dict in stick_figure_mesh_edge_pairs_dict_of_dicts.items():
edge_name = this_bone_dict[0]
- head_name = this_bone_dict[1]['head']
- tail_name = this_bone_dict[1]['tail']
+ head_name = this_bone_dict[1]["head"]
+ tail_name = this_bone_dict[1]["tail"]
try:
head_index = mediapipe_tracked_point_names.index(head_name)
tail_index = mediapipe_tracked_point_names.index(tail_name)
- if 'center' in head_name: # virtual markers are acting weird so we're brute forcing it :-/ this will fail if there is a virtual marker without 'center' in the name....
+ if (
+ "center" in head_name
+ ): # virtual markers are acting weird so we're brute forcing it :-/ this will fail if there is a virtual marker without 'center' in the name....
head_index = virtual_point_index_dict[head_name]
- if 'center' in tail_name:
+ if "center" in tail_name:
tail_index = virtual_point_index_dict[tail_name]
this_edge = (head_index, tail_index)
edges.append(this_edge)
print(
- f'edge created for bone: ({this_bone_dict[0]} : {head_name}-{tail_name}), indicies({this_edge[0]},{this_edge[1]})')
+ f"edge created for bone: ({this_bone_dict[0]} : {head_name}-{tail_name}), indicies({this_edge[0]},{this_edge[1]})"
+ )
except:
- print(f'edge FAILED for for bone:{this_bone_dict[0]} : {head_name}-{tail_name}')
+ print(
+ f"edge FAILED for for bone:{this_bone_dict[0]} : {head_name}-{tail_name}"
+ )
pass
- print('edges created!')
+ print("edges created!")
# #
# # ##############################
# # # %% Create mesh
@@ -1575,31 +1639,41 @@ try:
#####################
## Load nSynched Videos
try:
- print('loading videos as planes...')
- annotated_videos_path = session_path / 'mediapipe_annotated_videos'
+ print("loading videos as planes...")
+ annotated_videos_path = session_path / "mediapipe_annotated_videos"
if annotated_videos_path.is_dir():
vidFolderPath = annotated_videos_path
else:
- vidFolderPath = session_path / 'synchronized_videos'
+ vidFolderPath = session_path / "synchronized_videos"
- world_origin = bpy.data.objects['world_origin']
+ world_origin = bpy.data.objects["world_origin"]
- number_of_videos = len(list(vidFolderPath.glob('*.mp4')))
+ number_of_videos = len(list(vidFolderPath.glob("*.mp4")))
vid_location_scale = 1
- for vid_number, thisVidPath, in enumerate(vidFolderPath.glob('*.mp4')):
+ for (
+ vid_number,
+ thisVidPath,
+ ) in enumerate(vidFolderPath.glob("*.mp4")):
print(thisVidPath)
# use 'images as planes' add on to load in the video files as planes
- bpy.ops.import_image.to_plane(files=[{"name": thisVidPath.name}], directory=str(thisVidPath.parent),
- shader='EMISSION', )
+ bpy.ops.import_image.to_plane(
+ files=[{"name": thisVidPath.name}],
+ directory=str(thisVidPath.parent),
+ shader="EMISSION",
+ )
this_vid_as_plane = bpy.context.active_object
- this_vid_as_plane.name = 'vid_' + str(vid_number)
+ this_vid_as_plane.name = "vid_" + str(vid_number)
vid_x = (vid_number - number_of_videos / 2) * vid_location_scale
- this_vid_as_plane.location = [vid_x, vid_location_scale, vid_location_scale]
+ this_vid_as_plane.location = [
+ vid_x,
+ vid_location_scale,
+ vid_location_scale,
+ ]
this_vid_as_plane.rotation_euler = [np.pi / 2, 0, 0]
this_vid_as_plane.scale = [vid_location_scale * 1.5] * 3
this_vid_as_plane.parent = world_origin
@@ -1607,16 +1681,18 @@ try:
# bpy.ops.object.light_add(type='POINT', radius=1, align='WORLD')
except:
print(
- 'Failed to load videos to Blender scene - You might need to install the "images as planes" addon to this version of Blender')
+ 'Failed to load videos to Blender scene - You might need to install the "images as planes" addon to this version of Blender'
+ )
except Exception as e:
print(e)
print(
- 'Something broke in the Blender Megascript - Maybe it failed to find a good clean frame and then that borked something downstream?')
+ "Something broke in the Blender Megascript - Maybe it failed to find a good clean frame and then that borked something downstream?"
+ )
# set shading type to 'Material Preview' so you can see the videos
# save .blend file
sessionID = session_path.stem
-blend_file_save_path = session_path / (sessionID + '.blend')
+blend_file_save_path = session_path / (sessionID + ".blend")
print(f"Saving Blender output file to: {str(blend_file_save_path)}")
-bpy.ops.wm.save_as_mainfile(filepath=str(blend_file_save_path))
\ No newline at end of file
+bpy.ops.wm.save_as_mainfile(filepath=str(blend_file_save_path))
diff --git a/src/open_in_blender/open_session_in_blender.py b/src/open_in_blender/open_session_in_blender.py
index d6eb21ba..eee14b77 100644
--- a/src/open_in_blender/open_session_in_blender.py
+++ b/src/open_in_blender/open_session_in_blender.py
@@ -8,28 +8,36 @@ from src.config.home_dir import get_session_folder_path, get_most_recent_session
blender_exe_path = r"C:\Users\jonma\Blender Foundation\Blender 3.1\blender.exe"
-def open_session_in_blender(session_id:str):
+def open_session_in_blender(session_id: str):
create_session(session_id)
- blender_file_name = session_id + '.blend'
- blender_file_path = Path(get_session_folder_path(session_id)) / blender_file_name
+ blender_file_name = session_id + ".blend"
+ blender_file_path = Path(get_session_folder_path(session_id)) / blender_file_name
os.startfile(str(blender_file_path))
-def create_session(session_id:str, good_clean_frame_number:int = 0):
+
+def create_session(session_id: str, good_clean_frame_number: int = 0):
path_to_this_py_file = Path(__file__).parent.resolve()
- freemocap_blender_megascript_path = path_to_this_py_file / 'alpha_freemocap_blender_megascript.py'
+ freemocap_blender_megascript_path = (
+ path_to_this_py_file / "alpha_freemocap_blender_megascript.py"
+ )
print(str(freemocap_blender_megascript_path))
print(f"sending {session_id} data to Blender")
- command_str = str(blender_exe_path) + " --background" + " --python " + str(freemocap_blender_megascript_path) + " -- " +\
- get_session_folder_path(session_id) + ' ' + str(good_clean_frame_number)
+ command_str = (
+ str(blender_exe_path)
+ + " --background"
+ + " --python "
+ + str(freemocap_blender_megascript_path)
+ + " -- "
+ + get_session_folder_path(session_id)
+ + " "
+ + str(good_clean_frame_number)
+ )
print(command_str)
blender_process = subprocess.Popen(
- command_str,
- shell=False,
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE
+ command_str, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
while True:
output = blender_process.stdout.readline()
@@ -43,5 +51,6 @@ def create_session(session_id:str, good_clean_frame_number:int = 0):
print(blender_process.stderr.read().decode())
print(f"done with blender stuff :D")
+
if __name__ == "__main__":
- open_session_in_blender(get_most_recent_session_id())
\ No newline at end of file
+ open_session_in_blender(get_most_recent_session_id())
diff --git a/src/pipelines/calibration_pipeline/anipose_camera_calibration/anipose_camera_calibration.py b/src/pipelines/calibration_pipeline/anipose_camera_calibration/anipose_camera_calibration.py
index 7ebb7cea..9556cc87 100644
--- a/src/pipelines/calibration_pipeline/anipose_camera_calibration/anipose_camera_calibration.py
+++ b/src/pipelines/calibration_pipeline/anipose_camera_calibration/anipose_camera_calibration.py
@@ -6,80 +6,118 @@ import numpy as np
from cv2 import aruco_CharucoBoard
from scipy.spatial.transform import Rotation
-from src.config.home_dir import get_session_folder_path, get_freemocap_data_folder_path, \
- get_calibration_videos_folder_path
-from src.pipelines.calibration_pipeline.anipose_camera_calibration import freemocap_anipose
+from src.config.home_dir import (
+ get_session_folder_path,
+ get_freemocap_data_folder_path,
+ get_calibration_videos_folder_path,
+)
+from src.pipelines.calibration_pipeline.anipose_camera_calibration import (
+ freemocap_anipose,
+)
from aniposelib.boards import CharucoBoard as AniposeCharucoBoard
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import \
- CharucoBoardDataClass
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import (
+ CharucoBoardDataClass,
+)
logger = logging.getLogger(__name__)
class AniposeCameraCalibrator:
- def __init__(self,
- session_id: str,
- charuco_board_object: CharucoBoardDataClass,
- charuco_square_size: Union[int, float] = 1,
- ):
+ def __init__(
+ self,
+ session_id: str,
+ charuco_board_object: CharucoBoardDataClass,
+ charuco_square_size: Union[int, float] = 1,
+ ):
self._session_id = session_id
self._charuco_board_object = charuco_board_object
if charuco_square_size == 1:
logger.warning(
- 'Charuco square size is not set, so units of 3d reconstructed data will be in units of `however_long_the_black_edge_of_the_charuco_square_was`. Please input `charuco_square_size` in millimeters (or your preferred unity of length)')
+ "Charuco square size is not set, so units of 3d reconstructed data will be in units of `however_long_the_black_edge_of_the_charuco_square_was`. Please input `charuco_square_size` in millimeters (or your preferred unity of length)"
+ )
self._charuco_square_size = charuco_square_size
self.get_paths_and_whatnot()
self.initialize_anipose_objects()
def get_paths_and_whatnot(self):
self.session_folder_path = Path(get_session_folder_path(self._session_id))
- calibration_videos_folder = Path(get_calibration_videos_folder_path(self._session_id))
- self._list_of_video_paths = [this_video_path for this_video_path in
- calibration_videos_folder.glob('*.mp4'.lower())]
+ calibration_videos_folder = Path(
+ get_calibration_videos_folder_path(self._session_id)
+ )
+ self._list_of_video_paths = [
+ this_video_path
+ for this_video_path in calibration_videos_folder.glob("*.mp4".lower())
+ ]
def initialize_anipose_objects(self):
- list_of_camera_names = [this_video_path.stem for this_video_path in self._list_of_video_paths]
- self._anipose_camera_group_object = freemocap_anipose.CameraGroup.from_names(list_of_camera_names)
- self._anipose_camera_group_object.metadata['session_id'] = self._session_id
- self._anipose_camera_group_object.metadata['charuco_square_size'] = self._charuco_square_size
- self._anipose_camera_group_object.metadata['charuco_board_object'] = str(self._charuco_board_object)
-
- self._anipose_charuco_board = AniposeCharucoBoard(self._charuco_board_object.number_of_squares_width,
- self._charuco_board_object.number_of_squares_height,
- square_length=self._charuco_square_size, # mm
- marker_length=self._charuco_square_size * .8,
- marker_bits=4,
- dict_size=250)
+ list_of_camera_names = [
+ this_video_path.stem for this_video_path in self._list_of_video_paths
+ ]
+ self._anipose_camera_group_object = freemocap_anipose.CameraGroup.from_names(
+ list_of_camera_names
+ )
+ self._anipose_camera_group_object.metadata["session_id"] = self._session_id
+ self._anipose_camera_group_object.metadata[
+ "charuco_square_size"
+ ] = self._charuco_square_size
+ self._anipose_camera_group_object.metadata["charuco_board_object"] = str(
+ self._charuco_board_object
+ )
+
+ self._anipose_charuco_board = AniposeCharucoBoard(
+ self._charuco_board_object.number_of_squares_width,
+ self._charuco_board_object.number_of_squares_height,
+ square_length=self._charuco_square_size, # mm
+ marker_length=self._charuco_square_size * 0.8,
+ marker_bits=4,
+ dict_size=250,
+ )
def calibrate_camera_capture_volume(self, pin_camera_0_to_origin: bool = False):
# anipose needs this to be a list of lists (which is annoying but whatevs)
- video_paths_list_of_list_of_strings = [[str(this_path)] for this_path in self._list_of_video_paths]
-
- error, charuco_frame_data, charuco_frame_numbers = self._anipose_camera_group_object.calibrate_videos(
- video_paths_list_of_list_of_strings,
- self._anipose_charuco_board)
+ video_paths_list_of_list_of_strings = [
+ [str(this_path)] for this_path in self._list_of_video_paths
+ ]
+
+ (
+ error,
+ charuco_frame_data,
+ charuco_frame_numbers,
+ ) = self._anipose_camera_group_object.calibrate_videos(
+ video_paths_list_of_list_of_strings, self._anipose_charuco_board
+ )
logger.info("Anipose Calibration Successful!")
if pin_camera_0_to_origin:
# translate cameras so camera0 is on `0,0,0`
- self._anipose_camera_group_object = self.pin_camera_zero_to_origin(self._anipose_camera_group_object)
+ self._anipose_camera_group_object = self.pin_camera_zero_to_origin(
+ self._anipose_camera_group_object
+ )
# self._anipose_camera_group_object = self.rotate_cameras_so_camera_zero_aligns_with_XYZ(self._anipose_camera_group_object)
# save calibration info to files
calibration_toml_filename = f"{self._session_id}_camera_calibration.toml"
- camera_calibration_toml_path = self.session_folder_path / calibration_toml_filename
+ camera_calibration_toml_path = (
+ self.session_folder_path / calibration_toml_filename
+ )
self._anipose_camera_group_object.dump(camera_calibration_toml_path)
- logger.info(f"anipose camera calibration data saved to {str(camera_calibration_toml_path)}")
+ logger.info(
+ f"anipose camera calibration data saved to {str(camera_calibration_toml_path)}"
+ )
- last_successful_calibration_path = Path(get_freemocap_data_folder_path(), "last_successful_calibration.toml")
+ last_successful_calibration_path = Path(
+ get_freemocap_data_folder_path(), "last_successful_calibration.toml"
+ )
self._anipose_camera_group_object.dump(last_successful_calibration_path)
- logger.info(f"anipose camera calibration data also saved to {str(last_successful_calibration_path)}")
+ logger.info(
+ f"anipose camera calibration data also saved to {str(last_successful_calibration_path)}"
+ )
return self._anipose_camera_group_object
# # convert charuco data into a format that can be 3d reconstructed (effectively providing dummy data for the rest of the 3d reconstruction pipeline)
@@ -101,16 +139,22 @@ class AniposeCameraCalibrator:
camera_0_translation = original_translation_vectors[0, :]
altered_translation_vectors = np.zeros(original_translation_vectors.shape)
for this_camera_number in range(original_translation_vectors.shape[0]):
- altered_translation_vectors[this_camera_number, :] = original_translation_vectors[this_camera_number,
- :] - camera_0_translation
+ altered_translation_vectors[this_camera_number, :] = (
+ original_translation_vectors[this_camera_number, :]
+ - camera_0_translation
+ )
_anipose_camera_group_object.set_translations(altered_translation_vectors)
logger.info(f"original translation vectors: {original_translation_vectors}")
- logger.info(f"altered translation vectors: {_anipose_camera_group_object.get_translations()}")
+ logger.info(
+ f"altered translation vectors: {_anipose_camera_group_object.get_translations()}"
+ )
return _anipose_camera_group_object
- def rotate_cameras_so_camera_zero_aligns_with_XYZ(self, _anipose_camera_group_object):
- logger.warning('this function does not work, dont use it lol ')
+ def rotate_cameras_so_camera_zero_aligns_with_XYZ(
+ self, _anipose_camera_group_object
+ ):
+ logger.warning("this function does not work, dont use it lol ")
pass
# original_rotations_euler = _anipose_camera_group_object.get_rotations()
# original_translation_vectors = _anipose_camera_group_object.get_translations()
diff --git a/src/pipelines/calibration_pipeline/anipose_camera_calibration/freemocap_anipose.py b/src/pipelines/calibration_pipeline/anipose_camera_calibration/freemocap_anipose.py
index fc252730..bf2f081a 100644
--- a/src/pipelines/calibration_pipeline/anipose_camera_calibration/freemocap_anipose.py
+++ b/src/pipelines/calibration_pipeline/anipose_camera_calibration/freemocap_anipose.py
@@ -15,14 +15,18 @@ from rich import print
import time
-from aniposelib.boards import merge_rows, extract_points, \
- extract_rtvecs, get_video_params
-from aniposelib.utils import get_initial_extrinsics, make_M, get_rtvec, \
- get_connections
-
-numba_logger = logging.getLogger('numba')
+from aniposelib.boards import (
+ merge_rows,
+ extract_points,
+ extract_rtvecs,
+ get_video_params,
+)
+from aniposelib.utils import get_initial_extrinsics, make_M, get_rtvec, get_connections
+
+numba_logger = logging.getLogger("numba")
numba_logger.setLevel(logging.WARNING)
+
@jit(nopython=True, parallel=True)
def triangulate_simple(points, camera_mats):
num_cams = len(camera_mats)
@@ -30,8 +34,8 @@ def triangulate_simple(points, camera_mats):
for i in range(num_cams):
x, y = points[i]
mat = camera_mats[i]
- A[(i * 2):(i * 2 + 1)] = x * mat[2] - mat[0]
- A[(i * 2 + 1):(i * 2 + 2)] = y * mat[2] - mat[1]
+ A[(i * 2) : (i * 2 + 1)] = x * mat[2] - mat[0]
+ A[(i * 2 + 1) : (i * 2 + 2)] = y * mat[2] - mat[1]
u, s, vh = np.linalg.svd(A, full_matrices=True)
p3d = vh[-1]
p3d = p3d[:3] / p3d[3]
@@ -69,36 +73,38 @@ def subset_extra(extra, ixs):
return None
new_extra = {
- 'objp': extra['objp'][ixs],
- 'ids': extra['ids'][ixs],
- 'rvecs': extra['rvecs'][:, ixs],
- 'tvecs': extra['tvecs'][:, ixs]
+ "objp": extra["objp"][ixs],
+ "ids": extra["ids"][ixs],
+ "rvecs": extra["rvecs"][:, ixs],
+ "tvecs": extra["tvecs"][:, ixs],
}
return new_extra
def resample_points_extra(imgp, extra, n_samp=25):
n_cams, n_points, _ = imgp.shape
- ids = remap_ids(extra['ids'])
+ ids = remap_ids(extra["ids"])
n_ids = np.max(ids) + 1
good = ~np.isnan(imgp[:, :, 0])
ixs = np.arange(n_points)
- cam_counts = np.zeros((n_ids, n_cams), dtype='int64')
+ cam_counts = np.zeros((n_ids, n_cams), dtype="int64")
for idnum in range(n_ids):
cam_counts[idnum] = np.sum(good[:, ids == idnum], axis=1)
cam_counts_random = cam_counts + np.random.random(size=cam_counts.shape)
best_boards = np.argsort(-cam_counts_random, axis=0)
- cam_totals = np.zeros(n_cams, dtype='int64')
+ cam_totals = np.zeros(n_cams, dtype="int64")
include = set()
for cam_num in range(n_cams):
for board_id in best_boards[:, cam_num]:
include.update(ixs[ids == board_id])
cam_totals += cam_counts[board_id]
- if cam_totals[cam_num] >= n_samp or \
- cam_counts_random[board_id, cam_num] < 1:
+ if (
+ cam_totals[cam_num] >= n_samp
+ or cam_counts_random[board_id, cam_num] < 1
+ ):
break
final_ixs = sorted(include)
@@ -125,7 +131,7 @@ def resample_points(imgp, extra=None, n_samp=25):
n_good = np.sum(subset)
if n_good > 0:
## pick points, prioritizing points seen by more cameras
- arr = np.copy(num_cams[subset]).astype('float64')
+ arr = np.copy(num_cams[subset]).astype("float64")
arr += np.random.random(size=arr.shape)
picked_ix = np.argsort(-arr)[:n_samp]
picked = ixs[subset][picked_ix]
@@ -139,7 +145,7 @@ def resample_points(imgp, extra=None, n_samp=25):
def medfilt_data(values, size=15):
padsize = size + 5
- vpad = np.pad(values, (padsize, padsize), mode='reflect')
+ vpad = np.pad(values, (padsize, padsize), mode="reflect")
vpadf = signal.medfilt(vpad, kernel_size=size)
return vpadf[padsize:-padsize]
@@ -171,29 +177,31 @@ def transform_points(points, rvecs, tvecs):
Rodrigues' rotation formula is used.
"""
theta = np.linalg.norm(rvecs, axis=1)[:, np.newaxis]
- with np.errstate(invalid='ignore'):
+ with np.errstate(invalid="ignore"):
v = rvecs / theta
v = np.nan_to_num(v)
dot = np.sum(points * v, axis=1)[:, np.newaxis]
cos_theta = np.cos(theta)
sin_theta = np.sin(theta)
- rotated = cos_theta * points + \
- sin_theta * np.cross(v, points) + \
- dot * (1 - cos_theta) * v
+ rotated = (
+ cos_theta * points + sin_theta * np.cross(v, points) + dot * (1 - cos_theta) * v
+ )
return rotated + tvecs
class Camera:
- def __init__(self,
- matrix=np.eye(3),
- dist=np.zeros(5),
- size=None,
- rvec=np.zeros(3),
- tvec=np.zeros(3),
- name=None,
- extra_dist=False):
+ def __init__(
+ self,
+ matrix=np.eye(3),
+ dist=np.zeros(5),
+ size=None,
+ rvec=np.zeros(3),
+ tvec=np.zeros(3),
+ name=None,
+ extra_dist=False,
+ ):
self.set_camera_matrix(matrix)
self.set_distortions(dist)
@@ -205,21 +213,21 @@ class Camera:
def get_dict(self):
return {
- 'name': self.get_name(),
- 'size': list(self.get_size()),
- 'matrix': self.get_camera_matrix().tolist(),
- 'distortions': self.get_distortions().tolist(),
- 'rotation': self.get_rotation().tolist(),
- 'translation': self.get_translation().tolist(),
+ "name": self.get_name(),
+ "size": list(self.get_size()),
+ "matrix": self.get_camera_matrix().tolist(),
+ "distortions": self.get_distortions().tolist(),
+ "rotation": self.get_rotation().tolist(),
+ "translation": self.get_translation().tolist(),
}
def load_dict(self, d):
- self.set_camera_matrix(d['matrix'])
- self.set_rotation(d['rotation'])
- self.set_translation(d['translation'])
- self.set_distortions(d['distortions'])
- self.set_name(d['name'])
- self.set_size(d['size'])
+ self.set_camera_matrix(d["matrix"])
+ self.set_rotation(d["rotation"])
+ self.set_translation(d["translation"])
+ self.set_distortions(d["distortions"])
+ self.set_name(d["name"])
+ self.set_size(d["size"])
def from_dict(d):
cam = Camera()
@@ -233,7 +241,7 @@ class Camera:
return self.dist
def set_camera_matrix(self, matrix):
- self.matrix = np.array(matrix, dtype='float64')
+ self.matrix = np.array(matrix, dtype="float64")
def set_focal_length(self, fx, fy=None):
if fy is None:
@@ -250,16 +258,16 @@ class Camera:
return (fx + fy) / 2.0
def set_distortions(self, dist):
- self.dist = np.array(dist, dtype='float64').ravel()
+ self.dist = np.array(dist, dtype="float64").ravel()
def set_rotation(self, rvec):
- self.rvec = np.array(rvec, dtype='float64').ravel()
+ self.rvec = np.array(rvec, dtype="float64").ravel()
def get_rotation(self):
return self.rvec
def set_translation(self, tvec):
- self.tvec = np.array(tvec, dtype='float64').ravel()
+ self.tvec = np.array(tvec, dtype="float64").ravel()
def get_translation(self):
return self.tvec
@@ -292,7 +300,7 @@ class Camera:
self.set_camera_matrix(new_matrix)
def get_params(self):
- params = np.zeros(8 + self.extra_dist, dtype='float64')
+ params = np.zeros(8 + self.extra_dist, dtype="float64")
params[0:3] = self.get_rotation()
params[3:6] = self.get_translation()
params[6] = self.get_focal_length()
@@ -307,7 +315,7 @@ class Camera:
self.set_translation(params[3:6])
self.set_focal_length(params[6])
- dist = np.zeros(5, dtype='float64')
+ dist = np.zeros(5, dtype="float64")
dist[0] = params[7]
if self.extra_dist:
dist[1] = params[8]
@@ -317,24 +325,32 @@ class Camera:
shape = points.shape
points = points.reshape(-1, 1, 2)
new_points = np.dstack([points, np.ones((points.shape[0], 1, 1))])
- out, _ = cv2.projectPoints(new_points, np.zeros(3), np.zeros(3),
- self.matrix.astype('float64'),
- self.dist.astype('float64'))
+ out, _ = cv2.projectPoints(
+ new_points,
+ np.zeros(3),
+ np.zeros(3),
+ self.matrix.astype("float64"),
+ self.dist.astype("float64"),
+ )
return out.reshape(shape)
def undistort_points(self, points):
shape = points.shape
points = points.reshape(-1, 1, 2)
- out = cv2.undistortPoints(points,
- self.matrix.astype('float64'),
- self.dist.astype('float64'))
+ out = cv2.undistortPoints(
+ points, self.matrix.astype("float64"), self.dist.astype("float64")
+ )
return out.reshape(shape)
def project(self, points):
points = points.reshape(-1, 1, 3)
- out, _ = cv2.projectPoints(points, self.rvec, self.tvec,
- self.matrix.astype('float64'),
- self.dist.astype('float64'))
+ out, _ = cv2.projectPoints(
+ points,
+ self.rvec,
+ self.tvec,
+ self.matrix.astype("float64"),
+ self.dist.astype("float64"),
+ )
return out
def reprojection_error(self, p3d, p2d):
@@ -342,25 +358,28 @@ class Camera:
return p2d - proj
def copy(self):
- return \
- Camera(matrix=self.get_camera_matrix().copy(),
- dist=self.get_distortions().copy(),
- size=self.get_size(),
- rvec=self.get_rotation().copy(),
- tvec=self.get_translation().copy(),
- name=self.get_name(),
- extra_dist=self.extra_dist)
+ return Camera(
+ matrix=self.get_camera_matrix().copy(),
+ dist=self.get_distortions().copy(),
+ size=self.get_size(),
+ rvec=self.get_rotation().copy(),
+ tvec=self.get_translation().copy(),
+ name=self.get_name(),
+ extra_dist=self.extra_dist,
+ )
class FisheyeCamera(Camera):
- def __init__(self,
- matrix=np.eye(3),
- dist=np.zeros(4),
- size=None,
- rvec=np.zeros(3),
- tvec=np.zeros(3),
- name=None,
- extra_dist=False):
+ def __init__(
+ self,
+ matrix=np.eye(3),
+ dist=np.zeros(4),
+ size=None,
+ rvec=np.zeros(3),
+ tvec=np.zeros(3),
+ name=None,
+ extra_dist=False,
+ ):
self.set_camera_matrix(matrix)
self.set_distortions(dist)
self.set_size(size)
@@ -376,33 +395,41 @@ class FisheyeCamera(Camera):
def get_dict(self):
d = super().get_dict()
- d['fisheye'] = True
+ d["fisheye"] = True
return d
def distort_points(self, points):
shape = points.shape
points = points.reshape(-1, 1, 2)
new_points = np.dstack([points, np.ones((points.shape[0], 1, 1))])
- out, _ = cv2.fisheye.projectPoints(new_points,
- np.zeros(3), np.zeros(3),
- self.matrix.astype('float64'),
- self.dist.astype('float64'))
+ out, _ = cv2.fisheye.projectPoints(
+ new_points,
+ np.zeros(3),
+ np.zeros(3),
+ self.matrix.astype("float64"),
+ self.dist.astype("float64"),
+ )
return out.reshape(shape)
def undistort_points(self, points):
shape = points.shape
points = points.reshape(-1, 1, 2)
- out = cv2.fisheye.undistortPoints(points.astype('float64'),
- self.matrix.astype('float64'),
- self.dist.astype('float64'))
+ out = cv2.fisheye.undistortPoints(
+ points.astype("float64"),
+ self.matrix.astype("float64"),
+ self.dist.astype("float64"),
+ )
return out.reshape(shape)
def project(self, points):
points = points.reshape(-1, 1, 3)
- out, _ = cv2.fisheye.projectPoints(points,
- self.rvec, self.tvec,
- self.matrix.astype('float64'),
- self.dist.astype('float64'))
+ out, _ = cv2.fisheye.projectPoints(
+ points,
+ self.rvec,
+ self.tvec,
+ self.matrix.astype("float64"),
+ self.dist.astype("float64"),
+ )
return out
def set_params(self, params):
@@ -410,7 +437,7 @@ class FisheyeCamera(Camera):
self.set_translation(params[3:6])
self.set_focal_length(params[6])
- dist = np.zeros(4, dtype='float64')
+ dist = np.zeros(4, dtype="float64")
dist[0] = params[7]
if self.extra_dist:
dist[1] = params[8]
@@ -419,7 +446,7 @@ class FisheyeCamera(Camera):
self.set_distortions(dist)
def get_params(self):
- params = np.zeros(8 + self.extra_dist, dtype='float64')
+ params = np.zeros(8 + self.extra_dist, dtype="float64")
params[0:3] = self.get_rotation()
params[3:6] = self.get_translation()
params[6] = self.get_focal_length()
@@ -439,7 +466,8 @@ class FisheyeCamera(Camera):
rvec=self.get_rotation().copy(),
tvec=self.get_translation().copy(),
name=self.get_name(),
- extra_dist=self.extra_dist)
+ extra_dist=self.extra_dist,
+ )
class CameraGroup:
@@ -458,9 +486,8 @@ class CameraGroup:
for name in names:
if name not in cur_names_dict:
raise IndexError(
- "name {} not part of camera names: {}".format(
- name, cur_names
- ))
+ "name {} not part of camera names: {}".format(name, cur_names)
+ )
indices.append(cur_names_dict[name])
return self.subset_cameras(indices)
@@ -471,7 +498,7 @@ class CameraGroup:
n_points = points.shape[0]
n_cams = len(self.cameras)
- out = np.empty((n_cams, n_points, 2), dtype='float64')
+ out = np.empty((n_cams, n_points, 2), dtype="float64")
for cnum, cam in enumerate(self.cameras):
out[cnum] = cam.project(points).reshape(n_points, 2)
@@ -481,11 +508,12 @@ class CameraGroup:
"""Given an CxNx2 array, this returns an Nx3 array of points,
where N is the number of points and C is the number of cameras"""
- assert points.shape[0] == len(self.cameras), \
- "Invalid points shape, first dim should be equal to" \
+ assert points.shape[0] == len(self.cameras), (
+ "Invalid points shape, first dim should be equal to"
" number of cameras ({}), but shape is {}".format(
len(self.cameras), points.shape
)
+ )
one_point = False
if len(points.shape) == 2:
@@ -523,8 +551,9 @@ class CameraGroup:
return out
- def triangulate_possible(self, points, undistort=True,
- min_cams=2, progress=False, threshold=0.5):
+ def triangulate_possible(
+ self, points, undistort=True, min_cams=2, progress=False, threshold=0.5
+ ):
"""Given an CxNxPx2 array, this returns an Nx3 array of points
by triangulating all possible points and picking the ones with
best reprojection error
@@ -534,21 +563,22 @@ class CameraGroup:
P: number of possible options per point
"""
- assert points.shape[0] == len(self.cameras), \
- "Invalid points shape, first dim should be equal to" \
+ assert points.shape[0] == len(self.cameras), (
+ "Invalid points shape, first dim should be equal to"
" number of cameras ({}), but shape is {}".format(
len(self.cameras), points.shape
)
+ )
n_cams, n_points, n_possible, _ = points.shape
- cam_nums, point_nums, possible_nums = np.where(
- ~np.isnan(points[:, :, :, 0]))
+ cam_nums, point_nums, possible_nums = np.where(~np.isnan(points[:, :, :, 0]))
all_iters = defaultdict(dict)
- for cam_num, point_num, possible_num in zip(cam_nums, point_nums,
- possible_nums):
+ for cam_num, point_num, possible_num in zip(
+ cam_nums, point_nums, possible_nums
+ ):
if cam_num not in all_iters[point_num]:
all_iters[point_num][cam_num] = []
all_iters[point_num][cam_num].append((cam_num, possible_num))
@@ -557,10 +587,10 @@ class CameraGroup:
for cam_num in all_iters[point_num].keys():
all_iters[point_num][cam_num].append(None)
- out = np.full((n_points, 3), np.nan, dtype='float64')
- picked_vals = np.zeros((n_cams, n_points, n_possible), dtype='bool')
- errors = np.zeros(n_points, dtype='float64')
- points_2d = np.full((n_cams, n_points, 2), np.nan, dtype='float64')
+ out = np.full((n_points, 3), np.nan, dtype="float64")
+ picked_vals = np.zeros((n_cams, n_points, n_possible), dtype="bool")
+ errors = np.zeros(n_points, dtype="float64")
+ points_2d = np.full((n_cams, n_points, 2), np.nan, dtype="float64")
if progress:
iterator = trange(n_points, ncols=70)
@@ -589,46 +619,46 @@ class CameraGroup:
if err < best_error:
best_point = {
- 'error': err,
- 'point': p3d[:3],
- 'points': pts,
- 'picked': picked,
- 'joint_ix': point_ix
+ "error": err,
+ "point": p3d[:3],
+ "points": pts,
+ "picked": picked,
+ "joint_ix": point_ix,
}
best_error = err
if best_error < threshold:
break
if best_point is not None:
- out[point_ix] = best_point['point']
- picked = best_point['picked']
+ out[point_ix] = best_point["point"]
+ picked = best_point["picked"]
cnums = [p[0] for p in picked]
xnums = [p[1] for p in picked]
picked_vals[cnums, point_ix, xnums] = True
- errors[point_ix] = best_point['error']
- points_2d[cnums, point_ix] = best_point['points']
+ errors[point_ix] = best_point["error"]
+ points_2d[cnums, point_ix] = best_point["points"]
# return out, picked_vals, points_2d, errors #original code from OG anipose
- return out #simplify output so that `triangulate_ransac` can be used exactly the same way as `triangulate`
+ return out # simplify output so that `triangulate_ransac` can be used exactly the same way as `triangulate`
def triangulate_ransac(self, points, undistort=True, min_cams=2, progress=False):
"""Given an CxNx2 array, this returns an Nx3 array of points,
where N is the number of points and C is the number of cameras"""
- assert points.shape[0] == len(self.cameras), \
- "Invalid points shape, first dim should be equal to" \
+ assert points.shape[0] == len(self.cameras), (
+ "Invalid points shape, first dim should be equal to"
" number of cameras ({}), but shape is {}".format(
len(self.cameras), points.shape
)
+ )
n_cams, n_points, _ = points.shape
points_ransac = points.reshape(n_cams, n_points, 1, 2)
- return self.triangulate_possible(points_ransac,
- undistort=undistort,
- min_cams=min_cams,
- progress=progress)
+ return self.triangulate_possible(
+ points_ransac, undistort=undistort, min_cams=min_cams, progress=progress
+ )
@jit(parallel=True, forceobj=True)
def reprojection_error(self, p3ds, p2ds, mean=False):
@@ -644,9 +674,12 @@ class CameraGroup:
one_point = True
n_cams, n_points, _ = p2ds.shape
- assert p3ds.shape == (n_points, 3), \
- "shapes of 2D and 3D points are not consistent: " \
- "2D={}, 3D={}".format(p2ds.shape, p3ds.shape)
+ assert p3ds.shape == (
+ n_points,
+ 3,
+ ), "shapes of 2D and 3D points are not consistent: " "2D={}, 3D={}".format(
+ p2ds.shape, p3ds.shape
+ )
errors = np.empty((n_cams, n_points, 2))
@@ -657,7 +690,7 @@ class CameraGroup:
errors_norm = np.linalg.norm(errors, axis=2)
good = ~np.isnan(errors_norm)
errors_norm[~good] = 0
- denom = np.sum(good, axis=0).astype('float64')
+ denom = np.sum(good, axis=0).astype("float64")
denom[denom < 1.5] = np.nan
errors = np.sum(errors_norm, axis=0) / denom
@@ -669,12 +702,20 @@ class CameraGroup:
return errors
- def bundle_adjust_iter(self, p2ds, extra=None,
- n_iters=10, start_mu=15, end_mu=1,
- max_nfev=200, ftol=1e-4,
- n_samp_iter=100, n_samp_full=1000,
- error_threshold=0.3,
- verbose=False):
+ def bundle_adjust_iter(
+ self,
+ p2ds,
+ extra=None,
+ n_iters=10,
+ start_mu=15,
+ end_mu=1,
+ max_nfev=200,
+ ftol=1e-4,
+ n_samp_iter=100,
+ n_samp_full=1000,
+ error_threshold=0.3,
+ verbose=False,
+ ):
"""Given an CxNx2 array of 2D points,
where N is the number of points and C is the number of cameras,
this performs iterative bundle adjustsment to fine-tune the parameters of the cameras.
@@ -684,30 +725,29 @@ class CameraGroup:
"""
error_list = []
- assert p2ds.shape[0] == len(self.cameras), \
- "Invalid points shape, first dim should be equal to" \
+ assert p2ds.shape[0] == len(self.cameras), (
+ "Invalid points shape, first dim should be equal to"
" number of cameras ({}), but shape is {}".format(
len(self.cameras), p2ds.shape
)
+ )
p2ds_full = p2ds
extra_full = extra
- p2ds, extra = resample_points(p2ds_full, extra_full,
- n_samp=n_samp_full)
+ p2ds, extra = resample_points(p2ds_full, extra_full, n_samp=n_samp_full)
error = self.average_error(p2ds, median=True)
if verbose:
- print('error: ', error)
+ print("error: ", error)
mus = np.exp(np.linspace(np.log(start_mu), np.log(end_mu), num=n_iters))
if verbose:
- print('n_samples: {}'.format(n_samp_iter))
+ print("n_samples: {}".format(n_samp_iter))
for i in range(n_iters):
- p2ds, extra = resample_points(p2ds_full, extra_full,
- n_samp=n_samp_full)
+ p2ds, extra = resample_points(p2ds_full, extra_full, n_samp=n_samp_full)
p3ds = self.triangulate(p2ds)
errors_full = self.reprojection_error(p3ds, p2ds, mean=False)
errors_norm = self.reprojection_error(p3ds, p2ds, mean=True)
@@ -724,7 +764,8 @@ class CameraGroup:
good = errors_norm < mu
extra_good = subset_extra(extra, good)
p2ds_samp, extra_samp = resample_points(
- p2ds[:, good], extra_good, n_samp=n_samp_iter)
+ p2ds[:, good], extra_good, n_samp=n_samp_iter
+ )
error = np.median(errors_norm)
error_list.append(error)
@@ -734,16 +775,23 @@ class CameraGroup:
if verbose:
print(error_dict)
- print('error: {:.2f}, mu: {:.1f}, ratio: {:.3f}'.format(error, mu, np.mean(good)))
- print(f'previous error scores: [magenta] {error_list}[/magenta]')
-
- self.bundle_adjust(p2ds_samp, extra_samp,
- loss='linear', ftol=ftol,
- max_nfev=max_nfev,
- verbose=verbose)
+ print(
+ "error: {:.2f}, mu: {:.1f}, ratio: {:.3f}".format(
+ error, mu, np.mean(good)
+ )
+ )
+ print(f"previous error scores: [magenta] {error_list}[/magenta]")
+
+ self.bundle_adjust(
+ p2ds_samp,
+ extra_samp,
+ loss="linear",
+ ftol=ftol,
+ max_nfev=max_nfev,
+ verbose=verbose,
+ )
- p2ds, extra = resample_points(p2ds_full, extra_full,
- n_samp=n_samp_full)
+ p2ds, extra = resample_points(p2ds_full, extra_full, n_samp=n_samp_full)
p3ds = self.triangulate(p2ds)
errors_full = self.reprojection_error(p3ds, p2ds, mean=False)
errors_norm = self.reprojection_error(p3ds, p2ds, mean=True)
@@ -761,10 +809,14 @@ class CameraGroup:
good = errors_norm < mu
extra_good = subset_extra(extra, good)
- self.bundle_adjust(p2ds[:, good], extra_good,
- loss='linear',
- ftol=ftol, max_nfev=max(200, max_nfev),
- verbose=verbose)
+ self.bundle_adjust(
+ p2ds[:, good],
+ extra_good,
+ loss="linear",
+ ftol=ftol,
+ max_nfev=max(200, max_nfev),
+ verbose=verbose,
+ )
error = self.average_error(p2ds, median=True)
@@ -775,30 +827,35 @@ class CameraGroup:
print(error_dict)
if verbose:
- print('error: ', error)
+ print("error: ", error)
return error
- def bundle_adjust(self, p2ds, extra=None,
- loss='linear',
- threshold=50,
- ftol=1e-4,
- max_nfev=1000,
- weights=None,
- start_params=None,
- verbose=True):
+ def bundle_adjust(
+ self,
+ p2ds,
+ extra=None,
+ loss="linear",
+ threshold=50,
+ ftol=1e-4,
+ max_nfev=1000,
+ weights=None,
+ start_params=None,
+ verbose=True,
+ ):
"""Given an CxNx2 array of 2D points,
where N is the number of points and C is the number of cameras,
this performs bundle adjustsment to fine-tune the parameters of the cameras"""
- assert p2ds.shape[0] == len(self.cameras), \
- "Invalid points shape, first dim should be equal to" \
+ assert p2ds.shape[0] == len(self.cameras), (
+ "Invalid points shape, first dim should be equal to"
" number of cameras ({}), but shape is {}".format(
len(self.cameras), p2ds.shape
)
+ )
if extra is not None:
- extra['ids_map'] = remap_ids(extra['ids'])
+ extra["ids_map"] = remap_ids(extra["ids"])
x0, n_cam_params = self._initialize_params_bundle(p2ds, extra)
@@ -811,18 +868,20 @@ class CameraGroup:
jac_sparse = self._jac_sparsity_bundle(p2ds, n_cam_params, extra)
f_scale = threshold
- opt = optimize.least_squares(error_fun,
- x0,
- jac_sparsity=jac_sparse,
- f_scale=f_scale,
- x_scale='jac',
- loss=loss,
- ftol=ftol,
- method='trf',
- tr_solver='lsmr',
- verbose=2 * verbose,
- max_nfev=max_nfev,
- args=(p2ds, n_cam_params, extra))
+ opt = optimize.least_squares(
+ error_fun,
+ x0,
+ jac_sparsity=jac_sparse,
+ f_scale=f_scale,
+ x_scale="jac",
+ loss=loss,
+ ftol=ftol,
+ method="trf",
+ tr_solver="lsmr",
+ verbose=2 * verbose,
+ max_nfev=max_nfev,
+ args=(p2ds, n_cam_params, extra),
+ )
best_params = opt.x
for i, cam in enumerate(self.cameras):
@@ -848,18 +907,18 @@ class CameraGroup:
n_cams = len(self.cameras)
sub = n_cam_params * n_cams
n3d = p2ds.shape[1] * 3
- p3ds_test = params[sub:sub + n3d].reshape(-1, 3)
+ p3ds_test = params[sub : sub + n3d].reshape(-1, 3)
errors = self.reprojection_error(p3ds_test, p2ds)
errors_reproj = errors[good]
if extra is not None:
- ids = extra['ids_map']
- objp = extra['objp']
+ ids = extra["ids_map"]
+ objp = extra["objp"]
min_scale = np.min(objp[objp > 0])
n_boards = int(np.max(ids)) + 1
a = sub + n3d
- rvecs = params[a:a + n_boards * 3].reshape(-1, 3)
- tvecs = params[a + n_boards * 3:a + n_boards * 6].reshape(-1, 3)
+ rvecs = params[a : a + n_boards * 3].reshape(-1, 3)
+ tvecs = params[a + n_boards * 3 : a + n_boards * 6].reshape(-1, 3)
expected = transform_points(objp, rvecs[ids], tvecs[ids])
errors_obj = 2 * (p3ds_test - expected).ravel() / min_scale
else:
@@ -872,8 +931,8 @@ class CameraGroup:
where N is the number of points and C is the number of cameras,
compute the sparsity structure of the jacobian for bundle adjustment"""
- point_indices = np.zeros(p2ds.shape, dtype='int32')
- cam_indices = np.zeros(p2ds.shape, dtype='int32')
+ point_indices = np.zeros(p2ds.shape, dtype="int32")
+ cam_indices = np.zeros(p2ds.shape, dtype="int32")
for i in range(p2ds.shape[1]):
point_indices[:, i] = i
@@ -884,7 +943,7 @@ class CameraGroup:
good = ~np.isnan(p2ds)
if extra is not None:
- ids = extra['ids_map']
+ ids = extra["ids_map"]
n_boards = int(np.max(ids)) + 1
total_board_params = n_boards * (3 + 3) # rvecs + tvecs
else:
@@ -902,7 +961,7 @@ class CameraGroup:
else:
n_errors = n_good_values
- A_sparse = dok_matrix((n_errors, n_params), dtype='int16')
+ A_sparse = dok_matrix((n_errors, n_params), dtype="int16")
cam_indices_good = cam_indices[good]
point_indices_good = point_indices[good]
@@ -929,15 +988,21 @@ class CameraGroup:
## update board rotation and translation based on error from expected
for i in range(3):
for j in range(3):
- A_sparse[n_good_values + point_ix * 3 + i,
- total_params_reproj + ids * 3 + j] = 1
- A_sparse[n_good_values + point_ix * 3 + i,
- total_params_reproj + n_boards * 3 + ids * 3 + j] = 1
+ A_sparse[
+ n_good_values + point_ix * 3 + i,
+ total_params_reproj + ids * 3 + j,
+ ] = 1
+ A_sparse[
+ n_good_values + point_ix * 3 + i,
+ total_params_reproj + n_boards * 3 + ids * 3 + j,
+ ] = 1
## update point position based on error from expected
for i in range(3):
- A_sparse[n_good_values + point_ix * 3 + i,
- n_cams * n_cam_params + point_ix * 3 + i] = 1
+ A_sparse[
+ n_good_values + point_ix * 3 + i,
+ n_cams * n_cam_params + point_ix * 3 + i,
+ ] = 1
return A_sparse
@@ -952,59 +1017,67 @@ class CameraGroup:
total_cam_params = len(cam_params)
n_cams, n_points, _ = p2ds.shape
- assert n_cams == len(self.cameras), \
- "number of cameras in CameraGroup does not " \
+ assert n_cams == len(self.cameras), (
+ "number of cameras in CameraGroup does not "
"match number of cameras in 2D points given"
+ )
p3ds = self.triangulate(p2ds)
if extra is not None:
- ids = extra['ids_map']
+ ids = extra["ids_map"]
n_boards = int(np.max(ids[~np.isnan(ids)])) + 1
total_board_params = n_boards * (3 + 3) # rvecs + tvecs
# initialize to 0
- rvecs = np.zeros((n_boards, 3), dtype='float64')
- tvecs = np.zeros((n_boards, 3), dtype='float64')
+ rvecs = np.zeros((n_boards, 3), dtype="float64")
+ tvecs = np.zeros((n_boards, 3), dtype="float64")
- if 'rvecs' in extra and 'tvecs' in extra:
- rvecs_all = extra['rvecs']
- tvecs_all = extra['tvecs']
+ if "rvecs" in extra and "tvecs" in extra:
+ rvecs_all = extra["rvecs"]
+ tvecs_all = extra["tvecs"]
for board_num in range(n_boards):
point_id = np.where(ids == board_num)[0][0]
cam_ids_possible = np.where(~np.isnan(p2ds[:, point_id, 0]))[0]
cam_id = np.random.choice(cam_ids_possible)
M_cam = self.cameras[cam_id].get_extrinsics_mat()
- M_board_cam = make_M(rvecs_all[cam_id, point_id],
- tvecs_all[cam_id, point_id])
+ M_board_cam = make_M(
+ rvecs_all[cam_id, point_id], tvecs_all[cam_id, point_id]
+ )
M_board = np.matmul(np.linalg.inv(M_cam), M_board_cam)
rvec, tvec = get_rtvec(M_board)
rvecs[board_num] = rvec
tvecs[board_num] = tvec
-
else:
total_board_params = 0
x0 = np.zeros(total_cam_params + p3ds.size + total_board_params)
x0[:total_cam_params] = cam_params
- x0[total_cam_params:total_cam_params + p3ds.size] = p3ds.ravel()
+ x0[total_cam_params : total_cam_params + p3ds.size] = p3ds.ravel()
if extra is not None:
start_board = total_cam_params + p3ds.size
- x0[start_board:start_board + n_boards * 3] = rvecs.ravel()
- x0[start_board + n_boards * 3:start_board + n_boards * 6] = \
- tvecs.ravel()
+ x0[start_board : start_board + n_boards * 3] = rvecs.ravel()
+ x0[start_board + n_boards * 3 : start_board + n_boards * 6] = tvecs.ravel()
return x0, n_cam_params
- def optim_points(self, points, p3ds,
- constraints=[],
- constraints_weak=[],
- scale_smooth=4,
- scale_length=2, scale_length_weak=0.5,
- reproj_error_threshold=15, reproj_loss='soft_l1',
- n_deriv_smooth=1, scores=None, verbose=False):
+ def optim_points(
+ self,
+ points,
+ p3ds,
+ constraints=[],
+ constraints_weak=[],
+ scale_smooth=4,
+ scale_length=2,
+ scale_length_weak=0.5,
+ reproj_error_threshold=15,
+ reproj_loss="soft_l1",
+ n_deriv_smooth=1,
+ scores=None,
+ verbose=False,
+ ):
"""
Take in an array of 2D points of shape CxNxJx2,
an array of 3D points of shape NxJx3,
@@ -1021,11 +1094,12 @@ class CameraGroup:
(meaning that lengths of segments 0->1, 1->2, 2->3 are all constant)
"""
- assert points.shape[0] == len(self.cameras), \
- "Invalid points shape, first dim should be equal to" \
+ assert points.shape[0] == len(self.cameras), (
+ "Invalid points shape, first dim should be equal to"
" number of cameras ({}), but shape is {}".format(
len(self.cameras), points.shape
)
+ )
n_cams, n_frames, n_joints, _ = points.shape
constraints = np.array(constraints)
@@ -1041,45 +1115,60 @@ class CameraGroup:
t1 = time.time()
x0 = self._initialize_params_triangulation(
- p3ds_intp, constraints, constraints_weak)
+ p3ds_intp, constraints, constraints_weak
+ )
x0[~np.isfinite(x0)] = 0
jac = self._jac_sparsity_triangulation(
- points, constraints, constraints_weak, n_deriv_smooth)
-
- opt2 = optimize.least_squares(self._error_fun_triangulation,
- x0=x0, jac_sparsity=jac,
- loss='linear',
- ftol=1e-3,
- verbose=2 * verbose,
- args=(points,
- constraints,
- constraints_weak,
- scores,
- scale_smooth_full,
- scale_length,
- scale_length_weak,
- reproj_error_threshold,
- reproj_loss,
- n_deriv_smooth))
-
- p3ds_new2 = opt2.x[:p3ds.size].reshape(p3ds.shape)
+ points, constraints, constraints_weak, n_deriv_smooth
+ )
+
+ opt2 = optimize.least_squares(
+ self._error_fun_triangulation,
+ x0=x0,
+ jac_sparsity=jac,
+ loss="linear",
+ ftol=1e-3,
+ verbose=2 * verbose,
+ args=(
+ points,
+ constraints,
+ constraints_weak,
+ scores,
+ scale_smooth_full,
+ scale_length,
+ scale_length_weak,
+ reproj_error_threshold,
+ reproj_loss,
+ n_deriv_smooth,
+ ),
+ )
+
+ p3ds_new2 = opt2.x[: p3ds.size].reshape(p3ds.shape)
t2 = time.time()
if verbose:
- print('optimization took {:.2f} seconds'.format(t2 - t1))
+ print("optimization took {:.2f} seconds".format(t2 - t1))
return p3ds_new2
- def optim_points_possible(self, points, p3ds,
- constraints=[],
- constraints_weak=[],
- scale_smooth=4,
- scale_length=2, scale_length_weak=0.5,
- reproj_error_threshold=15, reproj_loss='soft_l1',
- n_deriv_smooth=1, scores=None, verbose=False):
+ def optim_points_possible(
+ self,
+ points,
+ p3ds,
+ constraints=[],
+ constraints_weak=[],
+ scale_smooth=4,
+ scale_length=2,
+ scale_length_weak=0.5,
+ reproj_error_threshold=15,
+ reproj_loss="soft_l1",
+ n_deriv_smooth=1,
+ scores=None,
+ verbose=False,
+ ):
"""
Take in an array of 2D points of shape CxNxJxPx2,
an array of 3D points of shape NxJx3,
@@ -1097,11 +1186,12 @@ class CameraGroup:
(meaning that lengths of segments 0->1, 1->2, 2->3 are all constant)
"""
- assert points.shape[0] == len(self.cameras), \
- "Invalid points shape, first dim should be equal to" \
+ assert points.shape[0] == len(self.cameras), (
+ "Invalid points shape, first dim should be equal to"
" number of cameras ({}), but shape is {}".format(
len(self.cameras), points.shape
)
+ )
n_cams, n_frames, n_joints, n_possible, _ = points.shape
constraints = np.array(constraints)
@@ -1117,44 +1207,54 @@ class CameraGroup:
t1 = time.time()
x0 = self._initialize_params_triangulation_possible(
- p3ds_intp, points, constraints=constraints, constraints_weak=constraints_weak)
+ p3ds_intp,
+ points,
+ constraints=constraints,
+ constraints_weak=constraints_weak,
+ )
- print('getting jacobian...')
+ print("getting jacobian...")
jac = self._jac_sparsity_triangulation_possible(
points,
constraints=constraints,
constraints_weak=constraints_weak,
- n_deriv_smooth=n_deriv_smooth)
+ n_deriv_smooth=n_deriv_smooth,
+ )
beta = 5
- print('starting optimization...')
- opt2 = optimize.least_squares(self._error_fun_triangulation_possible,
- x0=x0, jac_sparsity=jac,
- loss='linear',
- ftol=1e-3,
- verbose=2 * verbose,
- args=(points,
- beta,
- constraints,
- constraints_weak,
- scores,
- scale_smooth_full,
- scale_length,
- scale_length_weak,
- reproj_error_threshold,
- reproj_loss,
- n_deriv_smooth))
+ print("starting optimization...")
+ opt2 = optimize.least_squares(
+ self._error_fun_triangulation_possible,
+ x0=x0,
+ jac_sparsity=jac,
+ loss="linear",
+ ftol=1e-3,
+ verbose=2 * verbose,
+ args=(
+ points,
+ beta,
+ constraints,
+ constraints_weak,
+ scores,
+ scale_smooth_full,
+ scale_length,
+ scale_length_weak,
+ reproj_error_threshold,
+ reproj_loss,
+ n_deriv_smooth,
+ ),
+ )
params = opt2.x
- p3ds_new2 = params[:p3ds.size].reshape(p3ds.shape)
+ p3ds_new2 = params[: p3ds.size].reshape(p3ds.shape)
bad = np.isnan(points[:, :, :, :, 0])
all_bad = np.all(bad, axis=3)
n_params_norm = p3ds.size + len(constraints) + len(constraints_weak)
- alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype='float64')
+ alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype="float64")
alphas[~bad] = params[n_params_norm:]
alphas_exp = np.exp(beta * alphas)
@@ -1167,12 +1267,13 @@ class CameraGroup:
t2 = time.time()
if verbose:
- print('optimization took {:.2f} seconds'.format(t2 - t1))
+ print("optimization took {:.2f} seconds".format(t2 - t1))
return p3ds_new2, alphas_norm
- def triangulate_optim(self, points, init_ransac=False, init_progress=False,
- **kwargs):
+ def triangulate_optim(
+ self, points, init_ransac=False, init_progress=False, **kwargs
+ ):
"""
Take in an array of 2D points of shape CxNxJx2, and an array of constraints of shape Kx2, where
C: number of camera
@@ -1188,11 +1289,12 @@ class CameraGroup:
"""
- assert points.shape[0] == len(self.cameras), \
- "Invalid points shape, first dim should be equal to" \
+ assert points.shape[0] == len(self.cameras), (
+ "Invalid points shape, first dim should be equal to"
" number of cameras ({}), but shape is {}".format(
len(self.cameras), points.shape
)
+ )
n_cams, n_frames, n_joints, _ = points.shape
# constraints = np.array(constraints)
@@ -1200,7 +1302,9 @@ class CameraGroup:
points_shaped = points.reshape(n_cams, n_frames * n_joints, 2)
if init_ransac:
- p3ds, picked, p2ds, errors = self.triangulate_ransac(points_shaped, progress=init_progress)
+ p3ds, picked, p2ds, errors = self.triangulate_ransac(
+ points_shaped, progress=init_progress
+ )
points = p2ds.reshape(points.shape)
else:
p3ds = self.triangulate(points_shaped, progress=init_progress)
@@ -1214,16 +1318,20 @@ class CameraGroup:
return self.optim_points(points, p3ds, **kwargs)
@jit(forceobj=True, parallel=True)
- def _error_fun_triangulation(self, params, p2ds,
- constraints=[],
- constraints_weak=[],
- scores=None,
- scale_smooth=10000,
- scale_length=1,
- scale_length_weak=0.2,
- reproj_error_threshold=100,
- reproj_loss='soft_l1',
- n_deriv_smooth=1):
+ def _error_fun_triangulation(
+ self,
+ params,
+ p2ds,
+ constraints=[],
+ constraints_weak=[],
+ scores=None,
+ scale_smooth=10000,
+ scale_length=1,
+ scale_length_weak=0.2,
+ reproj_error_threshold=100,
+ reproj_loss="soft_l1",
+ n_deriv_smooth=1,
+ ):
n_cams, n_frames, n_joints, _ = p2ds.shape
n_3d = n_frames * n_joints * 3
@@ -1232,8 +1340,8 @@ class CameraGroup:
# load params
p3ds = params[:n_3d].reshape((n_frames, n_joints, 3))
- joint_lengths = np.array(params[n_3d:n_3d + n_constraints])
- joint_lengths_weak = np.array(params[n_3d + n_constraints:])
+ joint_lengths = np.array(params[n_3d : n_3d + n_constraints])
+ joint_lengths_weak = np.array(params[n_3d + n_constraints :])
# reprojection errors
p3ds_flat = p3ds.reshape(-1, 3)
@@ -1246,40 +1354,39 @@ class CameraGroup:
rp = reproj_error_threshold
errors_reproj = np.abs(errors_reproj)
- if reproj_loss == 'huber':
+ if reproj_loss == "huber":
bad = errors_reproj > rp
errors_reproj[bad] = rp * (2 * np.sqrt(errors_reproj[bad] / rp) - 1)
- elif reproj_loss == 'linear':
+ elif reproj_loss == "linear":
pass
- elif reproj_loss == 'soft_l1':
+ elif reproj_loss == "soft_l1":
errors_reproj = rp * 2 * (np.sqrt(1 + errors_reproj / rp) - 1)
# temporal constraint
errors_smooth = np.diff(p3ds, n=n_deriv_smooth, axis=0).ravel() * scale_smooth
# joint length constraint
- errors_lengths = np.empty((n_constraints, n_frames), dtype='float64')
+ errors_lengths = np.empty((n_constraints, n_frames), dtype="float64")
for cix, (a, b) in enumerate(constraints):
lengths = np.linalg.norm(p3ds[:, a] - p3ds[:, b], axis=1)
expected = joint_lengths[cix]
errors_lengths[cix] = 100 * (lengths - expected) / expected
errors_lengths = errors_lengths.ravel() * scale_length
- errors_lengths_weak = np.empty((n_constraints_weak, n_frames), dtype='float64')
+ errors_lengths_weak = np.empty((n_constraints_weak, n_frames), dtype="float64")
for cix, (a, b) in enumerate(constraints_weak):
lengths = np.linalg.norm(p3ds[:, a] - p3ds[:, b], axis=1)
expected = joint_lengths_weak[cix]
errors_lengths_weak[cix] = 100 * (lengths - expected) / expected
errors_lengths_weak = errors_lengths_weak.ravel() * scale_length_weak
- return np.hstack([errors_reproj, errors_smooth,
- errors_lengths, errors_lengths_weak])
+ return np.hstack(
+ [errors_reproj, errors_smooth, errors_lengths, errors_lengths_weak]
+ )
- def _error_fun_triangulation_possible(self, params, p2ds,
- beta=2,
- constraints=[],
- constraints_weak=[],
- *args):
+ def _error_fun_triangulation_possible(
+ self, params, p2ds, beta=2, constraints=[], constraints_weak=[], *args
+ ):
# extract alphas from end of params
# soft argmax for picking the appropriate points from p2ds
# pass the points to error_fun_triangulate_possible for residuals
@@ -1297,7 +1404,7 @@ class CameraGroup:
bad = np.isnan(p2ds[:, :, :, :, 0])
all_bad = np.all(bad, axis=3)
- alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype='float64')
+ alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype="float64")
alphas[~bad] = params[n_params_norm:]
params_rest = np.array(params[:n_params_norm])
@@ -1314,19 +1421,20 @@ class CameraGroup:
p2ds_adj = np.sum(alphas_norm[:, :, :, :, None] * p2ds_test, axis=3)
p2ds_adj[all_bad] = np.nan
- errors = self._error_fun_triangulation(params_rest, p2ds_adj,
- constraints, constraints_weak, *args)
+ errors = self._error_fun_triangulation(
+ params_rest, p2ds_adj, constraints, constraints_weak, *args
+ )
alphas_test = alphas_norm[~all_bad]
errors_alphas = (1 - np.std(alphas_test, axis=1)) * 10
return np.hstack([errors, errors_alphas])
- def _initialize_params_triangulation(self, p3ds,
- constraints=[],
- constraints_weak=[]):
- joint_lengths = np.empty(len(constraints), dtype='float64')
- joint_lengths_weak = np.empty(len(constraints_weak), dtype='float64')
+ def _initialize_params_triangulation(
+ self, p3ds, constraints=[], constraints_weak=[]
+ ):
+ joint_lengths = np.empty(len(constraints), dtype="float64")
+ joint_lengths_weak = np.empty(len(constraints_weak), dtype="float64")
for cix, (a, b) in enumerate(constraints):
lengths = np.linalg.norm(p3ds[:, a] - p3ds[:, b], axis=1)
@@ -1357,7 +1465,7 @@ class CameraGroup:
n_cams, n_frames, n_joints, n_possible, _ = p2ds.shape
good = ~np.isnan(p2ds[:, :, :, :, 0])
- alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype='float64')
+ alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype="float64")
alphas[:, :, :, 0] = 0
params = self._initialize_params_triangulation(p3ds, **kwargs)
@@ -1365,22 +1473,20 @@ class CameraGroup:
return params_full
- def _jac_sparsity_triangulation(self, p2ds,
- constraints=[],
- constraints_weak=[],
- n_deriv_smooth=1):
+ def _jac_sparsity_triangulation(
+ self, p2ds, constraints=[], constraints_weak=[], n_deriv_smooth=1
+ ):
n_cams, n_frames, n_joints, _ = p2ds.shape
n_constraints = len(constraints)
n_constraints_weak = len(constraints_weak)
p2ds_flat = p2ds.reshape((n_cams, -1, 2))
- point_indices = np.zeros(p2ds_flat.shape, dtype='int32')
+ point_indices = np.zeros(p2ds_flat.shape, dtype="int32")
for i in range(p2ds_flat.shape[1]):
point_indices[:, i] = i
- point_indices_3d = np.arange(n_frames * n_joints) \
- .reshape((n_frames, n_joints))
+ point_indices_3d = np.arange(n_frames * n_joints).reshape((n_frames, n_joints))
good = ~np.isnan(p2ds_flat)
n_errors_reproj = np.sum(good)
@@ -1388,15 +1494,16 @@ class CameraGroup:
n_errors_lengths = n_constraints * n_frames
n_errors_lengths_weak = n_constraints_weak * n_frames
- n_errors = n_errors_reproj + n_errors_smooth + \
- n_errors_lengths + n_errors_lengths_weak
+ n_errors = (
+ n_errors_reproj + n_errors_smooth + n_errors_lengths + n_errors_lengths_weak
+ )
n_3d = n_frames * n_joints * 3
n_params = n_3d + n_constraints + n_constraints_weak
point_indices_good = point_indices[good]
- A_sparse = dok_matrix((n_errors, n_params), dtype='int16')
+ A_sparse = dok_matrix((n_errors, n_params), dtype="int16")
# constraints for reprojection errors
ix_reproj = np.arange(n_errors_reproj)
@@ -1463,17 +1570,22 @@ class CameraGroup:
n_errors, n_params = A_sparse.shape
- B_sparse = dok_matrix((n_errors + n_errors_alphas, n_params + n_alphas), dtype='int16')
+ B_sparse = dok_matrix(
+ (n_errors + n_errors_alphas, n_params + n_alphas), dtype="int16"
+ )
for r, c in zip(*A_sparse.nonzero()):
B_sparse[r, c] = A_sparse[r, c]
- point_indices_2d = np.arange(n_cams * n_frames * n_joints) \
- .reshape(n_cams, n_frames, n_joints)
+ point_indices_2d = np.arange(n_cams * n_frames * n_joints).reshape(
+ n_cams, n_frames, n_joints
+ )
point_indices_2d_rep = np.repeat(point_indices_2d[:, :, :, None], 2, axis=3)
point_indices_2d_good = point_indices_2d_rep[~np.isnan(p2ds)]
point_indices_good = point_indices_2d[any_good]
- alpha_indices = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype='int64')
+ alpha_indices = np.zeros(
+ (n_cams, n_frames, n_joints, n_possible), dtype="int64"
+ )
for pnum in range(n_possible):
alpha_indices[:, :, :, pnum] = point_indices_2d
@@ -1485,8 +1597,7 @@ class CameraGroup:
point_indices_2d_good_find[p].append(ix)
for ix, alpha_index in enumerate(alpha_indices_good):
- B_sparse[point_indices_2d_good_find[alpha_index],
- n_params + ix] = 1
+ B_sparse[point_indices_2d_good_find[alpha_index], n_params + ix] = 1
# alphas should change according to the alpha errors
point_indices_good_find = dict()
@@ -1542,23 +1653,34 @@ class CameraGroup:
else:
return np.mean(errors)
- def calibrate_rows(self, all_rows, board,
- init_intrinsics=True, init_extrinsics=True, verbose=True,
- **kwargs):
- assert len(all_rows) == len(self.cameras), \
- "Number of camera detections does not match number of cameras"
+ def calibrate_rows(
+ self,
+ all_rows,
+ board,
+ init_intrinsics=True,
+ init_extrinsics=True,
+ verbose=True,
+ **kwargs,
+ ):
+ assert len(all_rows) == len(
+ self.cameras
+ ), "Number of camera detections does not match number of cameras"
for rows, camera in zip(all_rows, self.cameras):
size = camera.get_size()
- assert size is not None, \
- "Camera with name {} has no specified frame size".format(camera.get_name())
-
+ assert (
+ size is not None
+ ), "Camera with name {} has no specified frame size".format(
+ camera.get_name()
+ )
if init_intrinsics:
objp, imgp = board.get_all_calibration_points(rows)
mixed = [(o, i) for (o, i) in zip(objp, imgp) if len(o) >= 7]
- assert len(objp) != 0 and len(imgp) != 0, "No Charuco board points detected"
+ assert (
+ len(objp) != 0 and len(imgp) != 0
+ ), "No Charuco board points detected"
objp, imgp = zip(*mixed)
matrix = cv2.initCameraMatrix2D(objp, imgp, tuple(size))
camera.set_camera_matrix(matrix)
@@ -1566,7 +1688,7 @@ class CameraGroup:
for i, (row, cam) in enumerate(zip(all_rows, self.cameras)):
all_rows[i] = board.estimate_pose_rows(cam, row)
- charuco_frames = [f['framenum'][1] for f in all_rows[0]]
+ charuco_frames = [f["framenum"][1] for f in all_rows[0]]
merged = merge_rows(all_rows)
imgp, extra = extract_points(merged, board, min_cameras=2)
@@ -1588,9 +1710,11 @@ class CameraGroup:
for cix, (cam, cam_videos) in enumerate(zip(self.cameras, videos)):
rows_cam = []
for vnum, vidname in enumerate(cam_videos):
- if verbose: print(vidname)
+ if verbose:
+ print(vidname)
rows = board.detect_video(vidname, prefix=vnum, progress=verbose)
- if verbose: print("{} boards detected".format(len(rows)))
+ if verbose:
+ print("{} boards detected".format(len(rows)))
rows_cam.extend(rows)
all_rows.append(rows_cam)
@@ -1601,12 +1725,18 @@ class CameraGroup:
rows_cam = []
for vnum, vidname in enumerate(cam_videos):
params = get_video_params(vidname)
- size = (params['width'], params['height'])
+ size = (params["width"], params["height"])
cam.set_size(size)
- def calibrate_videos(self, videos, board,
- init_intrinsics=True, init_extrinsics=True, verbose=True,
- **kwargs):
+ def calibrate_videos(
+ self,
+ videos,
+ board,
+ init_intrinsics=True,
+ init_extrinsics=True,
+ verbose=True,
+ **kwargs,
+ ):
"""Takes as input a list of list of video filenames, one list of each camera.
Also takes a board which specifies what should be detected in the videos"""
@@ -1614,10 +1744,14 @@ class CameraGroup:
if init_extrinsics:
self.set_camera_sizes_videos(videos)
- error, merged, charuco_frames = self.calibrate_rows(all_rows, board,
- init_intrinsics=init_intrinsics,
- init_extrinsics=init_extrinsics,
- verbose=verbose, **kwargs)
+ error, merged, charuco_frames = self.calibrate_rows(
+ all_rows,
+ board,
+ init_intrinsics=init_intrinsics,
+ init_extrinsics=init_extrinsics,
+ verbose=verbose,
+ **kwargs,
+ )
return error, merged, charuco_frames
def get_dicts(self):
@@ -1629,7 +1763,7 @@ class CameraGroup:
def from_dicts(arr):
cameras = []
for d in arr:
- if 'fisheye' in d and d['fisheye']:
+ if "fisheye" in d and d["fisheye"]:
cam = FisheyeCamera.from_dict(d)
else:
cam = Camera.from_dict(d)
@@ -1653,20 +1787,20 @@ class CameraGroup:
def dump(self, fname):
dicts = self.get_dicts()
- names = ['cam_{}'.format(i) for i in range(len(dicts))]
+ names = ["cam_{}".format(i) for i in range(len(dicts))]
master_dict = dict(zip(names, dicts))
- master_dict['metadata'] = self.metadata
- with open(fname, 'w') as f:
+ master_dict["metadata"] = self.metadata
+ with open(fname, "w") as f:
toml.dump(master_dict, f)
@staticmethod
def load(fname):
master_dict = toml.load(fname)
keys = sorted(master_dict.keys())
- items = [master_dict[k] for k in keys if k != 'metadata']
+ items = [master_dict[k] for k in keys if k != "metadata"]
cgroup = CameraGroup.from_dicts(items)
- if 'metadata' in master_dict:
- cgroup.metadata = master_dict['metadata']
+ if "metadata" in master_dict:
+ cgroup.metadata = master_dict["metadata"]
return cgroup
def resize_cameras(self, scale):
diff --git a/src/pipelines/calibration_pipeline/calibration_pipeline_orchestrator.py b/src/pipelines/calibration_pipeline/calibration_pipeline_orchestrator.py
index ea34efa3..2af920c0 100644
--- a/src/pipelines/calibration_pipeline/calibration_pipeline_orchestrator.py
+++ b/src/pipelines/calibration_pipeline/calibration_pipeline_orchestrator.py
@@ -7,31 +7,45 @@ from typing import Union
import cv2
from src.cameras.multicam_manager.cv_camera_manager import OpenCVCameraManager
-from src.config.home_dir import create_session_id, get_session_folder_path, get_freemocap_data_folder_path, \
- get_session_output_data_folder_path, get_session_calibration_file_path
+from src.config.home_dir import (
+ create_session_id,
+ get_session_folder_path,
+ get_freemocap_data_folder_path,
+ get_session_output_data_folder_path,
+ get_session_calibration_file_path,
+)
from src.core_processor.timestamp_manager.timestamp_manager import TimestampManager
from src.core_processor.show_cam_window import show_cam_window
from src.core_processor.utils.image_fps_writer import write_fps_to_image
-from src.pipelines.calibration_pipeline.anipose_camera_calibration import freemocap_anipose
-from src.pipelines.calibration_pipeline.anipose_camera_calibration.anipose_camera_calibration import \
- AniposeCameraCalibrator
-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import CharucoBoardDetector
+from src.pipelines.calibration_pipeline.anipose_camera_calibration import (
+ freemocap_anipose,
+)
+from src.pipelines.calibration_pipeline.anipose_camera_calibration.anipose_camera_calibration import (
+ AniposeCameraCalibrator,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (
+ CharucoBoardDetector,
+)
from src.qt_visualizer_and_gui.qt_visualizer_and_gui import QTVisualizerAndGui
logger = logging.getLogger(__name__)
class CalibrationPipelineOrchestrator:
-
- def __init__(self,
- session_id: str = None,
- expected_framerate: Union[int, float] = None,
- ):
+ def __init__(
+ self,
+ session_id: str = None,
+ expected_framerate: Union[int, float] = None,
+ ):
if session_id is None:
- self._session_id = create_session_id(string_tag='calibration') #create a 'calibration only' session
+ self._session_id = create_session_id(
+ string_tag="calibration"
+ ) # create a 'calibration only' session
else:
- self._session_id = session_id #add calibration folder to "full" session folder
+ self._session_id = (
+ session_id # add calibration folder to "full" session folder
+ )
self._calibration_start_time_unix_ns = time.time_ns()
self._charuco_board_detector = CharucoBoardDetector()
@@ -48,65 +62,87 @@ class CalibrationPipelineOrchestrator:
return get_session_folder_path(self._session_id)
def record_videos(
- self,
- show_visualizer_gui=False,
- show_camera_views_in_windows=True,
- save_video_in_frame_loop=False,
-
+ self,
+ show_visualizer_gui=False,
+ show_camera_views_in_windows=True,
+ save_video_in_frame_loop=False,
):
"""open all cameras and start recording, detect charuco boards until the user closes the windows, then send recorded videos to `aniposelib` for processing
produces `[session_id]_calibration.toml` file that can be used to estiamte 3d trajectorires from synchronized images"""
with self._open_cv_camera_manager.start_capture_session_all_cams(
- calibration_videos=True,
+ calibration_videos=True,
) as connected_cameras_dict:
timestamp_manager = self._open_cv_camera_manager.timestamp_manager
try:
if show_visualizer_gui:
self._visualizer_gui = QTVisualizerAndGui()
- self._visualizer_gui.setup_and_launch(self._open_cv_camera_manager.available_webcam_ids)
+ self._visualizer_gui.setup_and_launch(
+ self._open_cv_camera_manager.available_webcam_ids
+ )
should_continue = True
while should_continue:
- timestamp_manager.log_new_timestamp_for_main_loop_perf_coutner_ns(time.perf_counter_ns())
+ timestamp_manager.log_new_timestamp_for_main_loop_perf_coutner_ns(
+ time.perf_counter_ns()
+ )
if not self._open_cv_camera_manager.new_multi_frame_ready():
continue
- this_multi_frame_payload = self._open_cv_camera_manager.latest_multi_frame
+ this_multi_frame_payload = (
+ self._open_cv_camera_manager.latest_multi_frame
+ )
- for this_webcam_id, this_open_cv_camera in connected_cameras_dict.items():
+ for (
+ this_webcam_id,
+ this_open_cv_camera,
+ ) in connected_cameras_dict.items():
- this_cam_latest_frame = this_multi_frame_payload.frames_dict[this_webcam_id]
+ this_cam_latest_frame = this_multi_frame_payload.frames_dict[
+ this_webcam_id
+ ]
if this_cam_latest_frame is None:
continue
-
# save frame to video file
if save_video_in_frame_loop:
# save this frame straight to the video file (no risk of memory overflow, but can't handle higher numbers of cameras)
- this_open_cv_camera.video_recorder.save_frame_payload_to_video_file(this_cam_latest_frame)
+ this_open_cv_camera.video_recorder.save_frame_payload_to_video_file(
+ this_cam_latest_frame
+ )
else:
# can handle large numbers of cameras, but will eventually fill up RAM and cause a crash
- this_open_cv_camera.video_recorder.append_frame_payload_to_list(this_cam_latest_frame)
+ this_open_cv_camera.video_recorder.append_frame_payload_to_list(
+ this_cam_latest_frame
+ )
# detect charuco board
- this_charuco_frame_payload = self._charuco_board_detector.detect_charuco_board(
- this_cam_latest_frame)
+ this_charuco_frame_payload = (
+ self._charuco_board_detector.detect_charuco_board(
+ this_cam_latest_frame
+ )
+ )
if show_camera_views_in_windows:
should_continue = show_cam_window(
- this_webcam_id, this_charuco_frame_payload.annotated_image, timestamp_manager
+ this_webcam_id,
+ this_charuco_frame_payload.annotated_image,
+ timestamp_manager,
)
if show_visualizer_gui:
write_fps_to_image(
this_charuco_frame_payload.annotated_image,
- timestamp_manager.median_frames_per_second_for_webcam(this_webcam_id),
+ timestamp_manager.median_frames_per_second_for_webcam(
+ this_webcam_id
+ ),
+ )
+ self._visualizer_gui.update_camera_view_image(
+ this_webcam_id,
+ this_charuco_frame_payload.annotated_image,
)
- self._visualizer_gui.update_camera_view_image(this_webcam_id,
- this_charuco_frame_payload.annotated_image)
# exit loop when user presses ESC key
exit_key = cv2.waitKey(1)
@@ -124,50 +160,68 @@ class CalibrationPipelineOrchestrator:
cv2.destroyAllWindows()
for this_open_cv_camera in connected_cameras_dict.values():
if not save_video_in_frame_loop:
- this_open_cv_camera.video_recorder.save_list_of_frames_to_video_file(this_open_cv_camera.video_recorder.frame_list)
+ this_open_cv_camera.video_recorder.save_list_of_frames_to_video_file(
+ this_open_cv_camera.video_recorder.frame_list
+ )
this_open_cv_camera.video_recorder.close()
if show_visualizer_gui:
self._visualizer_gui.close()
- def run_anipose_camera_calibration(self,
- charuco_square_size: Union[int, float] = 1,
- pin_camera_0_to_origin: bool = False,
- ):
- anipose_camera_calibrator = AniposeCameraCalibrator(self.session_id,
- self._charuco_board_detector.charuco_board_data_class_object,
- charuco_square_size=charuco_square_size)
- return anipose_camera_calibrator.calibrate_camera_capture_volume(pin_camera_0_to_origin=pin_camera_0_to_origin)
+ def run_anipose_camera_calibration(
+ self,
+ charuco_square_size: Union[int, float] = 1,
+ pin_camera_0_to_origin: bool = False,
+ ):
+ anipose_camera_calibrator = AniposeCameraCalibrator(
+ self.session_id,
+ self._charuco_board_detector.charuco_board_data_class_object,
+ charuco_square_size=charuco_square_size,
+ )
+ return anipose_camera_calibrator.calibrate_camera_capture_volume(
+ pin_camera_0_to_origin=pin_camera_0_to_origin
+ )
def load_most_recent_calibration(self):
- last_successful_calibration_path = Path(get_freemocap_data_folder_path(), "last_successful_calibration.toml")
- logger.info(f"loading `most recent calibration from:{str(last_successful_calibration_path)}")
+ last_successful_calibration_path = Path(
+ get_freemocap_data_folder_path(), "last_successful_calibration.toml"
+ )
+ logger.info(
+ f"loading `most recent calibration from:{str(last_successful_calibration_path)}"
+ )
return freemocap_anipose.CameraGroup.load(str(last_successful_calibration_path))
- def load_calibration_from_session_id(self, session_id:str):
+ def load_calibration_from_session_id(self, session_id: str):
session_calibration_file_path = get_session_calibration_file_path(session_id)
- logger.info(f"loading camera calibration file from:{str(session_calibration_file_path)}")
+ logger.info(
+ f"loading camera calibration file from:{str(session_calibration_file_path)}"
+ )
if Path(session_calibration_file_path).is_file():
- return freemocap_anipose.CameraGroup.load(str(session_calibration_file_path))
+ return freemocap_anipose.CameraGroup.load(
+ str(session_calibration_file_path)
+ )
return self.load_most_recent_calibration()
if __name__ == "__main__":
- print('running `calibration_pipeline` as a `__main__` file')
+ print("running `calibration_pipeline` as a `__main__` file")
record_new = True
if record_new:
- session_id = create_session_id('calibration')
+ session_id = create_session_id("calibration")
session_path = Path(get_session_folder_path(session_id))
- logger.info(f'Creating `calibration only` session folder at: {str(session_path)}')
+ logger.info(
+ f"Creating `calibration only` session folder at: {str(session_path)}"
+ )
session_path.mkdir(parents=True, exist_ok=False)
calibration_orchestrator = CalibrationPipelineOrchestrator(session_id)
- calibration_orchestrator.record_videos(show_visualizer_gui=False,
- save_video_in_frame_loop=False,
- show_camera_views_in_windows=True,
- )
+ calibration_orchestrator.record_videos(
+ show_visualizer_gui=False,
+ save_video_in_frame_loop=False,
+ show_camera_views_in_windows=True,
+ )
else:
session_id = "session_06-05-2022-10_31_34_calibration"
diff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_board_detector.py b/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_board_detector.py
index 891e867b..d5ec696e 100644
--- a/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_board_detector.py
+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_board_detector.py
@@ -5,10 +5,15 @@ import cv2
import numpy as np
from src.cameras.capture.dataclasses.frame_payload import FramePayload
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import CharucoBoardDataClass
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_frame_payload import \
- CharucoFramePayload
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import CharucoViewData
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import (
+ CharucoBoardDataClass,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_frame_payload import (
+ CharucoFramePayload,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import (
+ CharucoViewData,
+)
from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_image_annotator import (
annotate_image_with_charuco_data,
@@ -20,30 +25,38 @@ logger = logging.getLogger(__name__)
class CharucoBoardDetector:
def __init__(self):
self.charuco_board_data_class_object = CharucoBoardDataClass()
- self.cv2_aruco_charuco_board = self.charuco_board_data_class_object.charuco_board
+ self.cv2_aruco_charuco_board = (
+ self.charuco_board_data_class_object.charuco_board
+ )
self.aruco_marker_dict = self.charuco_board_data_class_object.aruco_marker_dict
- self.number_of_charuco_corners = self.charuco_board_data_class_object.number_of_charuco_corners
+ self.number_of_charuco_corners = (
+ self.charuco_board_data_class_object.number_of_charuco_corners
+ )
- def detect_charuco_board(self, raw_frame_payload: FramePayload) -> CharucoFramePayload:
+ def detect_charuco_board(
+ self, raw_frame_payload: FramePayload
+ ) -> CharucoFramePayload:
if raw_frame_payload.image is None:
logger.error("Frame is empty")
return CharucoFramePayload()
- charuco_view_data = self.detect_charuco_board_in_an_image(raw_frame_payload.image)
+ charuco_view_data = self.detect_charuco_board_in_an_image(
+ raw_frame_payload.image
+ )
annotated_image = raw_frame_payload.image.copy()
if charuco_view_data.some_charuco_corners_found:
annotate_image_with_charuco_data(
annotated_image, # this image will have stuff drawn on top of it inside this function
charuco_view_data,
- self.number_of_charuco_corners
+ self.number_of_charuco_corners,
)
return CharucoFramePayload(
annotated_image=annotated_image,
raw_frame_payload=raw_frame_payload,
- charuco_view_data=charuco_view_data
+ charuco_view_data=charuco_view_data,
)
def detect_charuco_board_in_an_image(self, raw_image) -> CharucoViewData:
@@ -61,13 +74,19 @@ class CharucoBoardDetector:
charuco_ids = []
# SUB PIXEL CORNER DETECTION CRITERION
- termination_criteria_threshold = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.00001)
+ termination_criteria_threshold = (
+ cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER,
+ 100,
+ 0.00001,
+ )
grayscale_image = cv2.cvtColor(raw_image, cv2.COLOR_BGR2GRAY)
- aruco_square_corners, aruco_square_ids, rejected_image_points = cv2.aruco.detectMarkers(
- grayscale_image, self.aruco_marker_dict
- )
+ (
+ aruco_square_corners,
+ aruco_square_ids,
+ rejected_image_points,
+ ) = cv2.aruco.detectMarkers(grayscale_image, self.aruco_marker_dict)
full_board_found = False
any_markers_found = False
@@ -79,16 +98,24 @@ class CharucoBoardDetector:
# https://docs.opencv.org/4.x/dd/d1a/group__imgproc__feature.html#ga354e0d7c86d0d9da75de9b9701a9a87e
for this_corner in aruco_square_corners:
cv2.cornerSubPix(
- grayscale_image, this_corner, winSize=(3, 3), zeroZone=(-1, -1),
- criteria=termination_criteria_threshold
+ grayscale_image,
+ this_corner,
+ winSize=(3, 3),
+ zeroZone=(-1, -1),
+ criteria=termination_criteria_threshold,
)
- results = cv2.aruco.interpolateCornersCharuco(aruco_square_corners,
- aruco_square_ids,
- grayscale_image,
- self.cv2_aruco_charuco_board
- )
+ results = cv2.aruco.interpolateCornersCharuco(
+ aruco_square_corners,
+ aruco_square_ids,
+ grayscale_image,
+ self.cv2_aruco_charuco_board,
+ )
- if results[1] is not None and results[2] is not None and len(results[1]) > 3:
+ if (
+ results[1] is not None
+ and results[2] is not None
+ and len(results[1]) > 3
+ ):
some_charuco_corners_found = True
charuco_corners = results[1]
@@ -97,37 +124,51 @@ class CharucoBoardDetector:
if len(charuco_ids) == self.number_of_charuco_corners:
full_board_found = True
- return CharucoViewData(charuco_board_object=self.charuco_board_data_class_object,
- full_board_found=full_board_found,
- some_charuco_corners_found=some_charuco_corners_found,
- any_markers_found=any_markers_found,
- charuco_corners=charuco_corners,
- charuco_ids=charuco_ids,
- aruco_square_corners=aruco_square_corners,
- aruco_square_ids=aruco_square_ids,
- image_width=image_width,
- image_height=image_height)
+ return CharucoViewData(
+ charuco_board_object=self.charuco_board_data_class_object,
+ full_board_found=full_board_found,
+ some_charuco_corners_found=some_charuco_corners_found,
+ any_markers_found=any_markers_found,
+ charuco_corners=charuco_corners,
+ charuco_ids=charuco_ids,
+ aruco_square_corners=aruco_square_corners,
+ aruco_square_ids=aruco_square_ids,
+ image_width=image_width,
+ image_height=image_height,
+ )
def format_charuco2d_data(self, this_multi_frame_charuco_data) -> Dict:
number_of_tracked_points = this_multi_frame_charuco_data[
- 0].charuco_view_data.charuco_board_object.number_of_charuco_corners
+ 0
+ ].charuco_view_data.charuco_board_object.number_of_charuco_corners
charuco2d_data_per_cam_dict = {}
- base_charuco_data_npy_with_nans_for_missing_data_xy = np.zeros((number_of_tracked_points, 2))
+ base_charuco_data_npy_with_nans_for_missing_data_xy = np.zeros(
+ (number_of_tracked_points, 2)
+ )
base_charuco_data_npy_with_nans_for_missing_data_xy[:] = np.nan
for this_cam_data in this_multi_frame_charuco_data:
if this_cam_data.charuco_view_data.some_charuco_corners_found:
- this_frame_charuco_data_xy = base_charuco_data_npy_with_nans_for_missing_data_xy.copy()
- charuco_ids_in_this_frame_idx = this_cam_data.charuco_view_data.charuco_ids
- charuco_corners_in_this_frame_xy = this_cam_data.charuco_view_data.charuco_corners
- this_frame_charuco_data_xy[charuco_ids_in_this_frame_idx, :] = charuco_corners_in_this_frame_xy
+ this_frame_charuco_data_xy = (
+ base_charuco_data_npy_with_nans_for_missing_data_xy.copy()
+ )
+ charuco_ids_in_this_frame_idx = (
+ this_cam_data.charuco_view_data.charuco_ids
+ )
+ charuco_corners_in_this_frame_xy = (
+ this_cam_data.charuco_view_data.charuco_corners
+ )
+ this_frame_charuco_data_xy[
+ charuco_ids_in_this_frame_idx, :
+ ] = charuco_corners_in_this_frame_xy
this_webcam_id = this_cam_data.raw_frame_payload.webcam_id
charuco2d_data_per_cam_dict[this_webcam_id] = this_frame_charuco_data_xy
return charuco2d_data_per_cam_dict
+
if __name__ == "__main__":
pass
# ## Jon, Run me to easily start a charuco board detect run
diff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_image_annotator.py b/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_image_annotator.py
index 5dcef17b..ee958720 100644
--- a/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_image_annotator.py
+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_image_annotator.py
@@ -3,13 +3,15 @@ from typing import List
import cv2
import numpy as np
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import CharucoViewData
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import (
+ CharucoViewData,
+)
def _board_text(
- charuco_ids: List[int],
- num_charuco_corners: int,
- full_board_detected_bool: bool,
+ charuco_ids: List[int],
+ num_charuco_corners: int,
+ full_board_detected_bool: bool,
):
num_ids_as_str = str(len(charuco_ids))
num_corners_as_str = str(num_charuco_corners)
@@ -21,19 +23,17 @@ def _board_text(
def annotate_image_with_charuco_data(
- image,
- charuco_view_data: CharucoViewData,
- number_of_charuco_corners: int) -> bool:
+ image, charuco_view_data: CharucoViewData, number_of_charuco_corners: int
+) -> bool:
annotated_image = cv2.aruco.drawDetectedCornersCharuco(
image,
np.array(charuco_view_data.charuco_corners),
np.array(charuco_view_data.charuco_ids),
- (0, 255, 125, 255)
+ (0, 255, 125, 255),
) # yellow? I
# think cv2 uses BGR instead of RGB?
-
current_cam_corner_count_str = _board_text(
charuco_view_data.charuco_ids,
number_of_charuco_corners,
@@ -72,5 +72,9 @@ def annotate_image_with_charuco_data(
if charuco_view_data.full_board_found:
cv2.polylines(
- annotated_image, np.int32([charuco_view_data.charuco_corners]), False, (0, 100, 255), 2
+ annotated_image,
+ np.int32([charuco_view_data.charuco_corners]),
+ False,
+ (0, 100, 255),
+ 2,
)
diff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_board_definition.py b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_board_definition.py
index 1a88ec0e..0de571a0 100644
--- a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_board_definition.py
+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_board_definition.py
@@ -11,16 +11,17 @@ class CharucoBoardDataClass:
number_of_squares_width: int = 7
number_of_squares_height: int = 5
black_square_side_length: int = 1
- aruco_marker_length_proportional: float = .8
+ aruco_marker_length_proportional: float = 0.8
def __post_init__(self):
- self.charuco_board = cv2.aruco.CharucoBoard_create(self.number_of_squares_width,
- self.number_of_squares_height,
- self.black_square_side_length,
- self.aruco_marker_length_proportional,
- self.aruco_marker_dict)
-
- self.number_of_charuco_corners = (self.number_of_squares_width - 1) * (self.number_of_squares_height - 1)
-
-
-
+ self.charuco_board = cv2.aruco.CharucoBoard_create(
+ self.number_of_squares_width,
+ self.number_of_squares_height,
+ self.black_square_side_length,
+ self.aruco_marker_length_proportional,
+ self.aruco_marker_dict,
+ )
+
+ self.number_of_charuco_corners = (self.number_of_squares_width - 1) * (
+ self.number_of_squares_height - 1
+ )
diff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_frame_payload.py b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_frame_payload.py
index 8cc07b49..dee283e1 100644
--- a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_frame_payload.py
+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_frame_payload.py
@@ -3,8 +3,9 @@ from dataclasses import dataclass
import numpy as np
from src.cameras.capture.dataclasses.frame_payload import FramePayload
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import \
- CharucoViewData
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import (
+ CharucoViewData,
+)
@dataclass
diff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_view_data.py b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_view_data.py
index 46b7db8d..f7767413 100644
--- a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_view_data.py
+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_view_data.py
@@ -3,14 +3,18 @@ from dataclasses import dataclass
import numpy as np
-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import CharucoBoardDataClass
+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import (
+ CharucoBoardDataClass,
+)
logger = logging.getLogger(__name__)
@dataclass
class CharucoViewData:
- charuco_board_object: CharucoBoardDataClass = None # the charuco board definition this 'view' was trying to detect
+ charuco_board_object: CharucoBoardDataClass = (
+ None # the charuco board definition this 'view' was trying to detect
+ )
charuco_corners: np.ndarray = None
charuco_ids: np.ndarray = None
aruco_square_corners: np.ndarray = None
@@ -26,4 +30,5 @@ class CharucoViewData:
if self.some_charuco_corners_found:
if self.charuco_board_object is None:
logger.warning(
- 'If there is any data in this view point, you should also include a copy of the charcuo_board that was being detected')
+ "If there is any data in this view point, you should also include a copy of the charcuo_board that was being detected"
+ )
diff --git a/src/pipelines/session_pipeline/data_classes/data3d_full_session_data.py b/src/pipelines/session_pipeline/data_classes/data3d_full_session_data.py
index 00976d66..5435d3ea 100644
--- a/src/pipelines/session_pipeline/data_classes/data3d_full_session_data.py
+++ b/src/pipelines/session_pipeline/data_classes/data3d_full_session_data.py
@@ -10,6 +10,6 @@ class TweakedModel(BaseModel):
class Data3dFullSessionPayload(TweakedModel):
- has_data: bool = True,
+ has_data: bool = (True,)
data3d_numFrames_numTrackedPoints_XYZ: np.ndarray = None
data3d_numFrames_numTrackedPoint_reprojectionError: np.ndarray = None
diff --git a/src/pipelines/session_pipeline/data_classes/data_3d_single_frame_payload.py b/src/pipelines/session_pipeline/data_classes/data_3d_single_frame_payload.py
index ef58ac57..e8388045 100644
--- a/src/pipelines/session_pipeline/data_classes/data_3d_single_frame_payload.py
+++ b/src/pipelines/session_pipeline/data_classes/data_3d_single_frame_payload.py
@@ -10,6 +10,6 @@ class TweakedModel(BaseModel):
class Data3dMultiFramePayload(TweakedModel):
- has_data: bool = True,
+ has_data: bool = (True,)
data3d_trackedPointNum_xyz: np.ndarray = None
data3d_trackedPointNum_reprojectionError: np.ndarray = None
diff --git a/src/pipelines/session_pipeline/session_pipeline_orchestrator.py b/src/pipelines/session_pipeline/session_pipeline_orchestrator.py
index 2e8e719c..9fd2b37e 100644
--- a/src/pipelines/session_pipeline/session_pipeline_orchestrator.py
+++ b/src/pipelines/session_pipeline/session_pipeline_orchestrator.py
@@ -9,22 +9,31 @@ import numpy as np
from src.cameras.persistence.video_writer.save_options_dataclass import SaveOptions
from src.config.data_paths import freemocap_data_path
-from src.config.home_dir import create_session_folder, create_session_id, \
- get_most_recent_session_id, get_session_output_data_folder_path
-from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import \
- MediaPipeSkeletonDetector, \
- Mediapipe2dDataPayload
+from src.config.home_dir import (
+ create_session_folder,
+ create_session_id,
+ get_most_recent_session_id,
+ get_session_output_data_folder_path,
+)
+from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import (
+ MediaPipeSkeletonDetector,
+ Mediapipe2dDataPayload,
+)
from src.core_processor.show_cam_window import show_cam_window
from src.core_processor.timestamp_manager.timestamp_manager import TimestampManager
from src.core_processor.utils.image_fps_writer import write_fps_to_image
-from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import \
- CalibrationPipelineOrchestrator
-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import \
- CharucoBoardDetector
-from src.pipelines.session_pipeline.data_classes.data3d_full_session_data import \
- Data3dFullSessionPayload
-from src.pipelines.session_pipeline.data_classes.data_3d_single_frame_payload import \
- Data3dMultiFramePayload
+from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import (
+ CalibrationPipelineOrchestrator,
+)
+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (
+ CharucoBoardDetector,
+)
+from src.pipelines.session_pipeline.data_classes.data3d_full_session_data import (
+ Data3dFullSessionPayload,
+)
+from src.pipelines.session_pipeline.data_classes.data_3d_single_frame_payload import (
+ Data3dMultiFramePayload,
+)
from src.qt_visualizer_and_gui.qt_visualizer_and_gui import QTVisualizerAndGui
logger = logging.getLogger(__name__)
@@ -35,8 +44,9 @@ def get_canonical_time_str():
class SessionPipelineOrchestrator:
-
- def __init__(self, session_id: str = None, expected_framerate: Union[int, float] = None):
+ def __init__(
+ self, session_id: str = None, expected_framerate: Union[int, float] = None
+ ):
if session_id is not None:
self._session_id = session_id
@@ -71,40 +81,50 @@ class SessionPipelineOrchestrator:
return self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY
@mediapipe2d_numCams_numFrames_numTrackedPoints_XY.setter
- def mediapipe2d_numCams_numFrames_numTrackedPoints_XY(self,
- mediapipe2d_numCams_numFrames_numTrackedPoints_XY: np.ndarray):
- self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY = \
+ def mediapipe2d_numCams_numFrames_numTrackedPoints_XY(
+ self, mediapipe2d_numCams_numFrames_numTrackedPoints_XY: np.ndarray
+ ):
+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY = (
mediapipe2d_numCams_numFrames_numTrackedPoints_XY
+ )
@property
def session_folder_path(self):
return freemocap_data_path / self._session_id
- def calibrate_camera_capture_volume(self, use_most_recent_calibration: bool = False,
+ def calibrate_camera_capture_volume(
+ self,
+ use_most_recent_calibration: bool = False,
load_calibration_from_session_id: str = None,
charuco_square_size: int = 1,
- pin_camera_0_to_origin: bool = False):
+ pin_camera_0_to_origin: bool = False,
+ ):
if use_most_recent_calibration:
- self._anipose_camera_calibration_object = CalibrationPipelineOrchestrator(
-
- ).load_most_recent_calibration()
+ self._anipose_camera_calibration_object = (
+ CalibrationPipelineOrchestrator().load_most_recent_calibration()
+ )
return
elif load_calibration_from_session_id is not None:
calibration_orchestrator = CalibrationPipelineOrchestrator(
- load_calibration_from_session_id)
+ load_calibration_from_session_id
+ )
else: # create new calibration
- calibration_orchestrator = CalibrationPipelineOrchestrator(self.session_id,
- expected_framerate=self.expected_framerate)
- calibration_orchestrator.record_videos(show_visualizer_gui=False,
- save_video_in_frame_loop=False,
- show_camera_views_in_windows=True,
- )
-
- self._anipose_camera_calibration_object = \
+ calibration_orchestrator = CalibrationPipelineOrchestrator(
+ self.session_id, expected_framerate=self.expected_framerate
+ )
+ calibration_orchestrator.record_videos(
+ show_visualizer_gui=False,
+ save_video_in_frame_loop=False,
+ show_camera_views_in_windows=True,
+ )
+
+ self._anipose_camera_calibration_object = (
calibration_orchestrator.run_anipose_camera_calibration(
- charuco_square_size=charuco_square_size,
- pin_camera_0_to_origin=pin_camera_0_to_origin)
+ charuco_square_size=charuco_square_size,
+ pin_camera_0_to_origin=pin_camera_0_to_origin,
+ )
+ )
def record_new_session(
self,
@@ -117,23 +137,24 @@ class SessionPipelineOrchestrator:
):
if reconstruct_3d:
try:
- self.anipose_camera_calibration_object = CalibrationPipelineOrchestrator(
-
- ).load_calibration_from_session_id(
- self.session_id)
+ self.anipose_camera_calibration_object = (
+ CalibrationPipelineOrchestrator().load_calibration_from_session_id(
+ self.session_id
+ )
+ )
except:
logger.info(
f"No calibration found for session_id: {self.session_id}, loading most recent "
- f"calibration")
- self.anipose_camera_calibration_object = CalibrationPipelineOrchestrator(
-
- ).load_most_recent_calibration()
+ f"calibration"
+ )
+ self.anipose_camera_calibration_object = (
+ CalibrationPipelineOrchestrator().load_most_recent_calibration()
+ )
dictionary_of_charuco_frame_payloads_on_this_multi_frame = {}
dictionary_of_mediapipe_payloads_on_this_multi_frame = {}
- with self._open_cv_camera_manager.start_capture_session_all_cams() as \
- connected_cameras_dict:
+ with self._open_cv_camera_manager.start_capture_session_all_cams() as connected_cameras_dict:
self._number_of_cameras = len(connected_cameras_dict)
@@ -142,46 +163,77 @@ class SessionPipelineOrchestrator:
try:
if show_visualizer_gui:
self._visualizer_gui.setup_and_launch(
- self._open_cv_camera_manager.available_webcam_ids)
+ self._open_cv_camera_manager.available_webcam_ids
+ )
should_continue = True
while should_continue: # BIG FRAME LOOP STARTS HERE
if reconstruct_3d:
if detect_charuco:
- if len(dictionary_of_charuco_frame_payloads_on_this_multi_frame) > 0:
+ if (
+ len(
+ dictionary_of_charuco_frame_payloads_on_this_multi_frame
+ )
+ > 0
+ ):
charuco3d_multi_frame_payload = self._reconstruct_3d_charuco(
- dictionary_of_charuco_frame_payloads_on_this_multi_frame)
- if show_visualizer_gui and charuco3d_multi_frame_payload is not \
- None:
+ dictionary_of_charuco_frame_payloads_on_this_multi_frame
+ )
+ if (
+ show_visualizer_gui
+ and charuco3d_multi_frame_payload is not None
+ ):
self._visualizer_gui.update_charuco_3d_dottos(
- charuco3d_multi_frame_payload)
- dictionary_of_charuco_frame_payloads_on_this_multi_frame = {}
+ charuco3d_multi_frame_payload
+ )
+ dictionary_of_charuco_frame_payloads_on_this_multi_frame = (
+ {}
+ )
if detect_mediapipe:
- if len(dictionary_of_mediapipe_payloads_on_this_multi_frame) > 0:
+ if (
+ len(
+ dictionary_of_mediapipe_payloads_on_this_multi_frame
+ )
+ > 0
+ ):
mediapipe3d_multi_frame_payload = self._reconstruct_3d_mediapipe(
- dictionary_of_mediapipe_payloads_on_this_multi_frame)
- if show_visualizer_gui and mediapipe3d_multi_frame_payload is not\
- None:
+ dictionary_of_mediapipe_payloads_on_this_multi_frame
+ )
+ if (
+ show_visualizer_gui
+ and mediapipe3d_multi_frame_payload is not None
+ ):
self._visualizer_gui.update_mediapipe3d_skeleton(
- mediapipe3d_multi_frame_payload)
- dictionary_of_mediapipe_payloads_on_this_multi_frame = {}
+ mediapipe3d_multi_frame_payload
+ )
+ dictionary_of_mediapipe_payloads_on_this_multi_frame = (
+ {}
+ )
timestamp_manager.log_new_timestamp_for_main_loop_perf_coutner_ns(
- time.perf_counter_ns())
+ time.perf_counter_ns()
+ )
if not self._open_cv_camera_manager.new_multi_frame_ready():
continue
- this_multi_frame_payload = self._open_cv_camera_manager.latest_multi_frame
+ this_multi_frame_payload = (
+ self._open_cv_camera_manager.latest_multi_frame
+ )
if this_multi_frame_payload is None:
continue
- for this_webcam_id, this_open_cv_camera in connected_cameras_dict.items():
+ for (
+ this_webcam_id,
+ this_open_cv_camera,
+ ) in connected_cameras_dict.items():
- this_cam_latest_frame = this_multi_frame_payload.frames_dict[this_webcam_id]
+ this_cam_latest_frame = this_multi_frame_payload.frames_dict[
+ this_webcam_id
+ ]
if this_cam_latest_frame is None:
continue
@@ -192,7 +244,8 @@ class SessionPipelineOrchestrator:
create_session_folder(self.session_id)
print(
f"Recording: {this_open_cv_camera.webcam_id_as_str}, "
- f"Frame# {this_open_cv_camera.latest_frame_number}")
+ f"Frame# {this_open_cv_camera.latest_frame_number}"
+ )
# # save this frame straight to the video file (no risk of memory
# overflow, but can't handle higher numbers of cameras)
# this_open_cv_camera.video_recorder.save_frame_to_video_file(
@@ -201,30 +254,39 @@ class SessionPipelineOrchestrator:
# can handle large numbers of cameras, but will eventually fill up
# RAM and cause a crash
this_open_cv_camera.video_recorder.append_frame_payload_to_list(
- this_cam_latest_frame)
+ this_cam_latest_frame
+ )
if detect_charuco:
# detect charuco board
- this_charuco_frame_payload = \
+ this_charuco_frame_payload = (
self._charuco_board_detector.detect_charuco_board(
- this_cam_latest_frame)
- image_to_display = this_charuco_frame_payload.annotated_image
+ this_cam_latest_frame
+ )
+ )
+ image_to_display = (
+ this_charuco_frame_payload.annotated_image
+ )
if reconstruct_3d:
dictionary_of_charuco_frame_payloads_on_this_multi_frame[
- this_webcam_id] = this_charuco_frame_payload
+ this_webcam_id
+ ] = this_charuco_frame_payload
if detect_mediapipe:
# detect mediapipe skeleton
- this_mediapipe_frame_payload = \
- self._mediapipe_skeleton_detector.detect_skeleton_in_image(
+ this_mediapipe_frame_payload = self._mediapipe_skeleton_detector.detect_skeleton_in_image(
raw_frame_payload=this_cam_latest_frame,
- annotated_image=image_to_display)
- image_to_display = this_mediapipe_frame_payload.annotated_image
+ annotated_image=image_to_display,
+ )
+ image_to_display = (
+ this_mediapipe_frame_payload.annotated_image
+ )
if reconstruct_3d:
dictionary_of_mediapipe_payloads_on_this_multi_frame[
- this_webcam_id] = this_mediapipe_frame_payload
+ this_webcam_id
+ ] = this_mediapipe_frame_payload
if show_camera_views_in_windows:
should_continue = show_cam_window(
@@ -235,10 +297,12 @@ class SessionPipelineOrchestrator:
write_fps_to_image(
image_to_display,
timestamp_manager.median_frames_per_second_for_webcam(
- this_webcam_id),
+ this_webcam_id
+ ),
+ )
+ self._visualizer_gui.update_camera_view_image(
+ this_webcam_id, image_to_display
)
- self._visualizer_gui.update_camera_view_image(this_webcam_id,
- image_to_display)
if show_camera_views_in_windows:
should_continue = show_cam_window(
@@ -249,10 +313,12 @@ class SessionPipelineOrchestrator:
write_fps_to_image(
image_to_display,
timestamp_manager.median_frames_per_second_for_webcam(
- this_webcam_id),
+ this_webcam_id
+ ),
+ )
+ self._visualizer_gui.update_camera_view_image(
+ this_webcam_id, image_to_display
)
- self._visualizer_gui.update_camera_view_image(this_webcam_id,
- image_to_display)
# self._visualizer_gui.update_timestamp_plots(timestamp_manager)
if self._visualizer_gui.shut_it_down:
@@ -261,7 +327,8 @@ class SessionPipelineOrchestrator:
# save frames if 'record' button is pressed
this_open_cv_camera.record_frames(
- self._visualizer_gui.record_button_pressed)
+ self._visualizer_gui.record_button_pressed
+ )
# exit loop when user presses ESC key
exit_key = cv2.waitKey(1)
@@ -269,17 +336,19 @@ class SessionPipelineOrchestrator:
logger.info("ESC has been pressed.")
should_continue = False
-
except:
logger.error("Printing traceback")
traceback.print_exc()
finally:
for this_open_cv_camera in connected_cameras_dict.values():
this_open_cv_camera.video_recorder.save_list_of_frames_to_video_file(
- this_open_cv_camera.frame_list)
+ this_open_cv_camera.frame_list
+ )
if show_camera_views_in_windows:
- logger.info(f"Destroy window {this_open_cv_camera.webcam_id_as_str}")
+ logger.info(
+ f"Destroy window {this_open_cv_camera.webcam_id_as_str}"
+ )
cv2.destroyWindow(this_open_cv_camera.webcam_id_as_str)
if show_visualizer_gui:
@@ -287,81 +356,114 @@ class SessionPipelineOrchestrator:
timestamp_manager.create_diagnostic_plots()
- def _reconstruct_3d_charuco(self, dictionary_of_charuco_payloads_on_this_multiframe) -> \
- Union[None, Data3dMultiFramePayload]:
+ def _reconstruct_3d_charuco(
+ self, dictionary_of_charuco_payloads_on_this_multiframe
+ ) -> Union[None, Data3dMultiFramePayload]:
is_there_any_charuco_data = []
- for this_cam_charuco_data in dictionary_of_charuco_payloads_on_this_multiframe.values():
+ for (
+ this_cam_charuco_data
+ ) in dictionary_of_charuco_payloads_on_this_multiframe.values():
is_there_any_charuco_data.append(
- this_cam_charuco_data.charuco_view_data.some_charuco_corners_found)
+ this_cam_charuco_data.charuco_view_data.some_charuco_corners_found
+ )
min_cameras_to_reconstruct = 2
- if sum(
- is_there_any_charuco_data) >= min_cameras_to_reconstruct: # if at least 2 cameras
+ if (
+ sum(is_there_any_charuco_data) >= min_cameras_to_reconstruct
+ ): # if at least 2 cameras
# have any data, then we can try to make some 3d dottos
charuco2d_data_per_cam_dict = self._format_charuco2d_data(
- dictionary_of_charuco_payloads_on_this_multiframe)
- charuco_3d_data_payload = self._triangulate_2d_data(charuco2d_data_per_cam_dict,
- self._charuco_board_detector.number_of_charuco_corners)
+ dictionary_of_charuco_payloads_on_this_multiframe
+ )
+ charuco_3d_data_payload = self._triangulate_2d_data(
+ charuco2d_data_per_cam_dict,
+ self._charuco_board_detector.number_of_charuco_corners,
+ )
x = charuco_3d_data_payload.data3d_trackedPointNum_xyz[:, 0]
return charuco_3d_data_payload
- def _format_charuco2d_data(self, dictionary_of_charuco_payloads_on_this_multiframe) -> Dict:
+ def _format_charuco2d_data(
+ self, dictionary_of_charuco_payloads_on_this_multiframe
+ ) -> Dict:
- number_of_tracked_points = self._charuco_board_detector.number_of_charuco_corners
+ number_of_tracked_points = (
+ self._charuco_board_detector.number_of_charuco_corners
+ )
base_charuco_data_npy_with_nans_for_missing_data_xy = np.zeros(
- (number_of_tracked_points, 2))
+ (number_of_tracked_points, 2)
+ )
base_charuco_data_npy_with_nans_for_missing_data_xy[:] = np.nan
charuco2d_data_per_cam_dict = {}
for this_cam_data in dictionary_of_charuco_payloads_on_this_multiframe.values():
- this_frame_charuco_data_xy = base_charuco_data_npy_with_nans_for_missing_data_xy.copy()
+ this_frame_charuco_data_xy = (
+ base_charuco_data_npy_with_nans_for_missing_data_xy.copy()
+ )
charuco_ids_in_this_frame_idx = this_cam_data.charuco_view_data.charuco_ids
- charuco_corners_in_this_frame_xy = this_cam_data.charuco_view_data.charuco_corners
+ charuco_corners_in_this_frame_xy = (
+ this_cam_data.charuco_view_data.charuco_corners
+ )
if charuco_ids_in_this_frame_idx.__class__ != list:
- this_frame_charuco_data_xy[charuco_ids_in_this_frame_idx,
- :] = charuco_corners_in_this_frame_xy
+ this_frame_charuco_data_xy[
+ charuco_ids_in_this_frame_idx, :
+ ] = charuco_corners_in_this_frame_xy
this_webcam_id = this_cam_data.raw_frame_payload.webcam_id
charuco2d_data_per_cam_dict[this_webcam_id] = this_frame_charuco_data_xy
return charuco2d_data_per_cam_dict
- def _reconstruct_3d_mediapipe(self, dictionary_of_mediapipe_payloads_on_this_multi_frame) -> \
- Union[None, Data3dMultiFramePayload]:
+ def _reconstruct_3d_mediapipe(
+ self, dictionary_of_mediapipe_payloads_on_this_multi_frame
+ ) -> Union[None, Data3dMultiFramePayload]:
is_there_any_mediapipe_data = []
- for this_cam_medipipe_data in dictionary_of_mediapipe_payloads_on_this_multi_frame.values():
+ for (
+ this_cam_medipipe_data
+ ) in dictionary_of_mediapipe_payloads_on_this_multi_frame.values():
is_there_any_mediapipe_data.append(
- this_cam_medipipe_data.pixel_data_numpy_arrays.has_data)
+ this_cam_medipipe_data.pixel_data_numpy_arrays.has_data
+ )
min_cameras_to_reconstruct = 2
- if sum(
- is_there_any_mediapipe_data) >= min_cameras_to_reconstruct: # if at least 2 cameras
+ if (
+ sum(is_there_any_mediapipe_data) >= min_cameras_to_reconstruct
+ ): # if at least 2 cameras
# have any data, then we can try to make some 3d dottos
mediapipe2d_data_per_cam_dict = {}
- for this_webcam_id, this_mediapipe_payload in \
- dictionary_of_mediapipe_payloads_on_this_multi_frame.items():
+ for (
+ this_webcam_id,
+ this_mediapipe_payload,
+ ) in dictionary_of_mediapipe_payloads_on_this_multi_frame.items():
self._threshold_2d_data_by_confidence(
- this_mediapipe_payload.pixel_data_numpy_arrays, .5)
+ this_mediapipe_payload.pixel_data_numpy_arrays, 0.5
+ )
mediapipe2d_data_per_cam_dict[
- this_webcam_id] = \
+ this_webcam_id
+ ] = (
this_mediapipe_payload.pixel_data_numpy_arrays.all_data2d_nFrames_nTrackedPts_XY
+ )
- mediapipe_3d_data_payload = self._triangulate_2d_data(mediapipe2d_data_per_cam_dict,
- self._mediapipe_skeleton_detector.number_of_tracked_points_total)
+ mediapipe_3d_data_payload = self._triangulate_2d_data(
+ mediapipe2d_data_per_cam_dict,
+ self._mediapipe_skeleton_detector.number_of_tracked_points_total,
+ )
return mediapipe_3d_data_payload
- def _triangulate_2d_data(self, data2d_per_cam_dict: Dict,
- number_of_tracked_points: int) -> Data3dMultiFramePayload:
+ def _triangulate_2d_data(
+ self, data2d_per_cam_dict: Dict, number_of_tracked_points: int
+ ) -> Data3dMultiFramePayload:
- each_cam2d_data_list = [this_cam_charuco2d_frame_xy for this_cam_charuco2d_frame_xy in
- data2d_per_cam_dict.values()]
+ each_cam2d_data_list = [
+ this_cam_charuco2d_frame_xy
+ for this_cam_charuco2d_frame_xy in data2d_per_cam_dict.values()
+ ]
data2d_camNum_trackedPointNum_xy = np.asarray(each_cam2d_data_list)
@@ -369,10 +471,11 @@ class SessionPipelineOrchestrator:
# become a 3d estimate. Hurray! :`D
# simple triangulation
- data3d_trackedPointNum_xyz = self._anipose_camera_calibration_object.triangulate(
- data2d_camNum_trackedPointNum_xy,
- progress=False,
- undistort=True)
+ data3d_trackedPointNum_xyz = (
+ self._anipose_camera_calibration_object.triangulate(
+ data2d_camNum_trackedPointNum_xy, progress=False, undistort=True
+ )
+ )
# # triangulation with ransac (to find the best combo of cameras to minimize reprojection
# error, I think?
@@ -390,87 +493,122 @@ class SessionPipelineOrchestrator:
# they choose to call it) for thresholding, but it's problematic because of the
# neural_networks' propensity to apply high confidence to bonkers estimates, which leads
# to SPOOKY GHOST SKELETONS! :O
- data3d_trackedPointNum_reprojectionError = \
+ data3d_trackedPointNum_reprojectionError = (
self._anipose_camera_calibration_object.reprojection_error(
- data3d_trackedPointNum_xyz,
- data2d_camNum_trackedPointNum_xy,
- mean=True)
+ data3d_trackedPointNum_xyz, data2d_camNum_trackedPointNum_xy, mean=True
+ )
+ )
- return Data3dMultiFramePayload(has_data=True,
- data3d_trackedPointNum_xyz=data3d_trackedPointNum_xyz,
- data3d_trackedPointNum_reprojectionError=data3d_trackedPointNum_reprojectionError, )
+ return Data3dMultiFramePayload(
+ has_data=True,
+ data3d_trackedPointNum_xyz=data3d_trackedPointNum_xyz,
+ data3d_trackedPointNum_reprojectionError=data3d_trackedPointNum_reprojectionError,
+ )
def mediapipe_track_skeletons_offline(self):
- self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY = \
+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY = (
self._mediapipe_skeleton_detector.process_session_folder(
- save_annotated_videos=True)
+ save_annotated_videos=True
+ )
+ )
def reconstruct3d_from_2d_data_offline(self):
- number_of_cameras = self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[0]
- number_of_frames = self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[1]
- number_of_tracked_points = self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[2]
- number_of_spatial_dimensions = \
- self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[3]
+ number_of_cameras = (
+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[0]
+ )
+ number_of_frames = (
+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[1]
+ )
+ number_of_tracked_points = (
+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[2]
+ )
+ number_of_spatial_dimensions = (
+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[3]
+ )
if not number_of_spatial_dimensions == 2:
logger.error(
f"This is supposed to be 2D data but, number_of_spatial_dimensio"
- f"ns: {number_of_spatial_dimensions}")
+ f"ns: {number_of_spatial_dimensions}"
+ )
raise Exception
# reshape data to collapse across 'frames' so it becomes [number_of_cameras,
# number_of_2d_points(numFrames*numPoints), XY]
data2d_flat = self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.reshape(
- number_of_cameras,
- -1,
- 2)
-
- logger.info(f"Reconstructing 3d points from 2d points with shape: "
- f"number_of_cameras: {number_of_cameras},"
- f" number_of_frames: {number_of_frames}, "
- f" number_of_tracked_points: {number_of_tracked_points},"
- f" number_of_spatial_dimensions: {number_of_spatial_dimensions}")
-
- data3d_flat = self.anipose_camera_calibration_object.triangulate(data2d_flat, progress=True)
-
- data3d_reprojectionError_flat = self.anipose_camera_calibration_object.reprojection_error(
- data3d_flat,
- data2d_flat,
- mean=True)
-
- data3d_numFrames_numTrackedPoints_XYZ = data3d_flat.reshape(number_of_frames,
- number_of_tracked_points, 3)
- data3d_numFrames_numTrackedPoints_reprojectionError = data3d_reprojectionError_flat.reshape(
- number_of_frames,
- number_of_tracked_points)
-
- self._save_mediapipe3d_data_to_npy(data3d_numFrames_numTrackedPoints_XYZ,
- data3d_numFrames_numTrackedPoints_reprojectionError)
+ number_of_cameras, -1, 2
+ )
+
+ logger.info(
+ f"Reconstructing 3d points from 2d points with shape: "
+ f"number_of_cameras: {number_of_cameras},"
+ f" number_of_frames: {number_of_frames}, "
+ f" number_of_tracked_points: {number_of_tracked_points},"
+ f" number_of_spatial_dimensions: {number_of_spatial_dimensions}"
+ )
+
+ data3d_flat = self.anipose_camera_calibration_object.triangulate(
+ data2d_flat, progress=True
+ )
+
+ data3d_reprojectionError_flat = (
+ self.anipose_camera_calibration_object.reprojection_error(
+ data3d_flat, data2d_flat, mean=True
+ )
+ )
+
+ data3d_numFrames_numTrackedPoints_XYZ = data3d_flat.reshape(
+ number_of_frames, number_of_tracked_points, 3
+ )
+ data3d_numFrames_numTrackedPoints_reprojectionError = (
+ data3d_reprojectionError_flat.reshape(
+ number_of_frames, number_of_tracked_points
+ )
+ )
+
+ self._save_mediapipe3d_data_to_npy(
+ data3d_numFrames_numTrackedPoints_XYZ,
+ data3d_numFrames_numTrackedPoints_reprojectionError,
+ )
self._mediapipe3d_full_session_payload = Data3dFullSessionPayload(
data3d_numFrames_numTrackedPoints_XYZ=data3d_numFrames_numTrackedPoints_XYZ,
- data3d_numFrames_numTrackedPoint_reprojectionError
- =data3d_numFrames_numTrackedPoints_reprojectionError)
+ data3d_numFrames_numTrackedPoint_reprojectionError=data3d_numFrames_numTrackedPoints_reprojectionError,
+ )
- def _save_mediapipe3d_data_to_npy(self, data3d_numFrames_numTrackedPoints_XYZ: np.ndarray,
- data3d_numFrames_numTrackedPoints_reprojectionError: np.ndarray):
+ def _save_mediapipe3d_data_to_npy(
+ self,
+ data3d_numFrames_numTrackedPoints_XYZ: np.ndarray,
+ data3d_numFrames_numTrackedPoints_reprojectionError: np.ndarray,
+ ):
output_data_folder = Path(get_session_output_data_folder_path(self._session_id))
# save spatial XYZ data
- self._mediapipe_3dData_save_path = output_data_folder / \
- "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"
+ self._mediapipe_3dData_save_path = (
+ output_data_folder
+ / "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"
+ )
logger.info(f"saving: {self._mediapipe_3dData_save_path}")
- np.save(str(self._mediapipe_3dData_save_path), data3d_numFrames_numTrackedPoints_XYZ)
+ np.save(
+ str(self._mediapipe_3dData_save_path), data3d_numFrames_numTrackedPoints_XYZ
+ )
# save reprojection error
- self._mediapipe_reprojection_error_save_path = output_data_folder / \
- "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"
+ self._mediapipe_reprojection_error_save_path = (
+ output_data_folder
+ / "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"
+ )
logger.info(f"saving: {self._mediapipe_reprojection_error_save_path}")
- np.save(str(self._mediapipe_reprojection_error_save_path),
- data3d_numFrames_numTrackedPoints_reprojectionError)
+ np.save(
+ str(self._mediapipe_reprojection_error_save_path),
+ data3d_numFrames_numTrackedPoints_reprojectionError,
+ )
- def _threshold_2d_data_by_confidence(self, mediapipe2d_data_payload: Mediapipe2dDataPayload,
- confidence_threshold: float):
+ def _threshold_2d_data_by_confidence(
+ self,
+ mediapipe2d_data_payload: Mediapipe2dDataPayload,
+ confidence_threshold: float,
+ ):
pass
@@ -478,50 +616,64 @@ def load_mediapipe3d_skeleton_data(session_id: str = None):
if session_id is None:
session_id = get_most_recent_session_id()
output_data_folder = Path(get_session_output_data_folder_path(session_id))
- mediapipe3d_xyz_file_path = output_data_folder / \
- "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"
+ mediapipe3d_xyz_file_path = (
+ output_data_folder
+ / "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"
+ )
logger.info(f"loading: {mediapipe3d_xyz_file_path}")
- mediapipe3d_skeleton_nFrames_nTrajectories_xyz = np.load(str(mediapipe3d_xyz_file_path))
-
- mediapipe_reprojection_error_file_path = output_data_folder / \
- "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"
+ mediapipe3d_skeleton_nFrames_nTrajectories_xyz = np.load(
+ str(mediapipe3d_xyz_file_path)
+ )
+
+ mediapipe_reprojection_error_file_path = (
+ output_data_folder
+ / "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"
+ )
logger.info(f"loading: {mediapipe_reprojection_error_file_path}")
mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError = np.load(
- str(mediapipe_reprojection_error_file_path))
+ str(mediapipe_reprojection_error_file_path)
+ )
return Data3dFullSessionPayload(
data3d_numFrames_numTrackedPoints_XYZ=mediapipe3d_skeleton_nFrames_nTrajectories_xyz,
- data3d_numFrames_numTrackedPoint_reprojectionError
- =mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError)
+ data3d_numFrames_numTrackedPoint_reprojectionError=mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError,
+ )
def load_mediapipe2d_data(session_id: str = None):
if session_id is None:
session_id = get_most_recent_session_id()
output_data_folder = Path(get_session_output_data_folder_path(session_id))
- mediapipe2d_xy_file_path = output_data_folder / \
- "mediapipe_2dData_numCams_numFrames_numTrackedPoints_pixelXY.npy"
+ mediapipe2d_xy_file_path = (
+ output_data_folder
+ / "mediapipe_2dData_numCams_numFrames_numTrackedPoints_pixelXY.npy"
+ )
logger.info(f"loading: {mediapipe2d_xy_file_path}")
- mediapipe2d_numCams_numFrames_numTrackedPoints_pixelXY = np.load(mediapipe2d_xy_file_path)
+ mediapipe2d_numCams_numFrames_numTrackedPoints_pixelXY = np.load(
+ mediapipe2d_xy_file_path
+ )
# TODO - create a dataclass for 2d mediapipe data, including things split up by body, hands,
# and face (and maybe the raw `mediapipe results` data things if that's not too much data)
return mediapipe2d_numCams_numFrames_numTrackedPoints_pixelXY
if __name__ == "__main__":
- print('running `session_pipeline_orchestrator` as `__main__')
+ print("running `session_pipeline_orchestrator` as `__main__")
length_of_one_edge_of_a_black_square_on_the_charuco_board_in_mm = 39
expected_framerate = 25
calibrate_cameras = True
- this_session_orchestrator = SessionPipelineOrchestrator(expected_framerate=expected_framerate)
+ this_session_orchestrator = SessionPipelineOrchestrator(
+ expected_framerate=expected_framerate
+ )
if calibrate_cameras:
this_session_orchestrator.calibrate_camera_capture_volume(
charuco_square_size=length_of_one_edge_of_a_black_square_on_the_charuco_board_in_mm,
- pin_camera_0_to_origin=True)
+ pin_camera_0_to_origin=True,
+ )
this_session_orchestrator.record_new_session()
diff --git a/src/qt_visualizer_and_gui/qt_visualizer_and_gui.py b/src/qt_visualizer_and_gui/qt_visualizer_and_gui.py
index ddbed52e..47415c39 100644
--- a/src/qt_visualizer_and_gui/qt_visualizer_and_gui.py
+++ b/src/qt_visualizer_and_gui/qt_visualizer_and_gui.py
@@ -12,7 +12,9 @@ from pyqtgraph.Qt import QtWidgets
from mediapipe.python.solutions import holistic as mp_holistic
from src.core_processor.timestamp_manager.timestamp_manager import TimestampManager
-from src.pipelines.session_pipeline.data_classes.data_3d_single_frame_payload import Data3dMultiFramePayload
+from src.pipelines.session_pipeline.data_classes.data_3d_single_frame_payload import (
+ Data3dMultiFramePayload,
+)
logger = logging.getLogger(__name__)
@@ -22,8 +24,8 @@ class QTVisualizerAndGui:
# https://pyqtgraph.readthedocs.io/en/latest/config_options.html
self._skeleton_connections_list = None
self._mediapipe_skeleton_initialized = False
- self.pyqtgraph_app = pg.mkQApp('freemocap! :D')
- pg.setConfigOptions(imageAxisOrder='row-major')
+ self.pyqtgraph_app = pg.mkQApp("freemocap! :D")
+ pg.setConfigOptions(imageAxisOrder="row-major")
self._dict_of_camera_image_item_widgets = {}
self._number_of_cameras = None
@@ -54,14 +56,14 @@ class QTVisualizerAndGui:
self._main_window_widget.close()
def setup_and_launch(self, webcam_ids_list):
- logger.info('setting up QT Visualizer and GUI')
+ logger.info("setting up QT Visualizer and GUI")
self._number_of_cameras = len(webcam_ids_list)
self._webcam_ids_list = webcam_ids_list
self._setup_main_window()
self._setup_control_panel()
self._setup_camera_views_dock()
self._setup_3d_viewport()
- logger.info('launching QT Visualizer and GUI window')
+ logger.info("launching QT Visualizer and GUI window")
self._main_window_widget.show()
@@ -70,11 +72,15 @@ class QTVisualizerAndGui:
return
try:
- camera_image_item_widget = self._dict_of_camera_image_item_widgets[webcam_id]
+ camera_image_item_widget = self._dict_of_camera_image_item_widgets[
+ webcam_id
+ ]
except Exception as e:
- logger.warning(f'Could not find ViewBoxWidget for camera {webcam_id}')
+ logger.warning(f"Could not find ViewBoxWidget for camera {webcam_id}")
raise e
- camera_image_item_widget.setImage(cv2.cvtColor(image_to_display, cv2.COLOR_BGR2RGB))
+ camera_image_item_widget.setImage(
+ cv2.cvtColor(image_to_display, cv2.COLOR_BGR2RGB)
+ )
def _setup_main_window(self, window_width: int = 1000, window_height: int = 1000):
"""
@@ -86,22 +92,22 @@ class QTVisualizerAndGui:
self._main_dock_area = DockArea()
self._main_window_widget.setCentralWidget(self._main_dock_area)
- self._main_window_widget.setWindowTitle('Main Window ;D')
+ self._main_window_widget.setWindowTitle("Main Window ;D")
def _setup_control_panel(self):
- self._control_panel_dock = Dock('Control Panel', size=(1, 1))
+ self._control_panel_dock = Dock("Control Panel", size=(1, 1))
control_panel_layout_widget = pg.LayoutWidget()
self._control_panel_dock.addWidget(control_panel_layout_widget)
label = QtWidgets.QLabel("Blah Blah Blah put words here ")
control_panel_layout_widget.addWidget(label, row=0, col=0)
- self._pause_button = QtWidgets.QPushButton('Pause')
+ self._pause_button = QtWidgets.QPushButton("Pause")
self._pause_button.setEnabled(True)
self._pause_button.clicked.connect(self._pause)
control_panel_layout_widget.addWidget(self._pause_button, row=1, col=0)
- self._play_button = QtWidgets.QPushButton('Resume')
+ self._play_button = QtWidgets.QPushButton("Resume")
self._play_button.setEnabled(False)
self._play_button.clicked.connect(self._play)
control_panel_layout_widget.addWidget(self._play_button, row=2, col=0)
@@ -111,26 +117,30 @@ class QTVisualizerAndGui:
# self._reset_calibration_button.clicked.connect(self._reset_calibration)
# control_panel_layout_widget.addWidget(self._reset_calibration_button, row=3, col=0)
#
- self._start_record_button = QtWidgets.QPushButton('Start Recording Video')
+ self._start_record_button = QtWidgets.QPushButton("Start Recording Video")
self._start_record_button.setEnabled(True)
self._start_record_button.clicked.connect(self._record_button_pressed)
control_panel_layout_widget.addWidget(self._start_record_button, row=3, col=0)
- self._close_button = QtWidgets.QPushButton('Close All')
+ self._close_button = QtWidgets.QPushButton("Close All")
self._close_button.setEnabled(True)
self._close_button.clicked.connect(self._close_button_pressed)
control_panel_layout_widget.addWidget(self._close_button, row=4, col=0)
- self._main_dock_area.addDock(self._control_panel_dock, position='left')
+ self._main_dock_area.addDock(self._control_panel_dock, position="left")
def _setup_camera_views_dock(self):
self._camera_graphics_layout_window = pg.GraphicsLayoutWidget()
- self._camera_views_dock = Dock('Camera Views')
+ self._camera_views_dock = Dock("Camera Views")
self._camera_views_dock.addWidget(self._camera_graphics_layout_window)
- self._main_dock_area.addDock(self._camera_views_dock, 'right', self._control_panel_dock)
+ self._main_dock_area.addDock(
+ self._camera_views_dock, "right", self._control_panel_dock
+ )
for this_webcam_id in self._webcam_ids_list:
- self._dict_of_camera_image_item_widgets[this_webcam_id] = self._create_camera_view_widget()
+ self._dict_of_camera_image_item_widgets[
+ this_webcam_id
+ ] = self._create_camera_view_widget()
def _create_camera_view_widget(self):
camera_view_box_widget = pg.ViewBox(invertY=True, lockAspect=True)
@@ -159,13 +169,15 @@ class QTVisualizerAndGui:
self._start_record_button.setText("Recording! Click Again to stop")
else:
self._record_video_bool = False
- self._start_record_button.setText("Click to record more, or click Close All to stop")
+ self._start_record_button.setText(
+ "Click to record more, or click Close All to stop"
+ )
def _setup_3d_viewport(self):
self.opengl_3d_plot_widget = gl.GLViewWidget()
# self.opengl_3d_plot_widget.opts['center'] = (0,0,0)
- self.opengl_3d_plot_widget.opts['distance'] = 2e3
+ self.opengl_3d_plot_widget.opts["distance"] = 2e3
# self.opengl_3d_plot_widget.opts['azimuth'] = 0
# self.opengl_3d_plot_widget.opts['elevation'] = 0
@@ -176,32 +188,30 @@ class QTVisualizerAndGui:
y_axis_line_array = np.array([[0, 0, 0], [0, 100, 0]])
z_axis_line_array = np.array([[0, 0, 0], [0, 0, 100]])
- self.origin_x_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(pos=x_axis_line_array,
- color=(1, 0, 0, 1),
- width=1.,
- antialias=True)
- self.origin_y_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(pos=y_axis_line_array,
- color=(0, 1, 0, 1),
- width=1.,
- antialias=True)
- self.origin_z_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(pos=z_axis_line_array,
- color=(0, 0, 1, 1),
- width=1.,
- antialias=True)
+ self.origin_x_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(
+ pos=x_axis_line_array, color=(1, 0, 0, 1), width=1.0, antialias=True
+ )
+ self.origin_y_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(
+ pos=y_axis_line_array, color=(0, 1, 0, 1), width=1.0, antialias=True
+ )
+ self.origin_z_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(
+ pos=z_axis_line_array, color=(0, 0, 1, 1), width=1.0, antialias=True
+ )
self.opengl_3d_plot_widget.addItem(self.origin_x_axis_gl_lineplot_item)
self.opengl_3d_plot_widget.addItem(self.origin_y_axis_gl_lineplot_item)
self.opengl_3d_plot_widget.addItem(self.origin_z_axis_gl_lineplot_item)
- self.opengl_charuco_scatter_item = gl.GLScatterPlotItem(pos=(0, 0, 0),
- color=(1, 0, 1, 1),
- size=10,
- pxMode=False)
+ self.opengl_charuco_scatter_item = gl.GLScatterPlotItem(
+ pos=(0, 0, 0), color=(1, 0, 1, 1), size=10, pxMode=False
+ )
self.opengl_3d_plot_widget.addItem(self.opengl_charuco_scatter_item)
self.opengl_3d_plot_dock = Dock("3d View Port")
self.opengl_3d_plot_dock.addWidget(self.opengl_3d_plot_widget)
- self._main_dock_area.addDock(self.opengl_3d_plot_dock, 'bottom', self._camera_views_dock)
+ self._main_dock_area.addDock(
+ self.opengl_3d_plot_dock, "bottom", self._camera_views_dock
+ )
def create_grid_planes(self):
grid_scale = 2e3
@@ -229,9 +239,7 @@ class QTVisualizerAndGui:
def initialize_charuco_dottos(self, number_of_charuco_corners: int):
dummy_charuco_points = np.zeros((number_of_charuco_corners, 3))
self._charuco_scatter_item = gl.GLScatterPlotItem(
- pos=dummy_charuco_points,
- color=(1, 1, 0, 1),
- size=20
+ pos=dummy_charuco_points, color=(1, 1, 0, 1), size=20
)
self.opengl_3d_plot_widget.addItem(self._charuco_scatter_item)
@@ -246,14 +254,19 @@ class QTVisualizerAndGui:
)
def get_mediapipe_connections(self):
- self.mediapipe_body_connections = [this_connection for this_connection in mp_holistic.POSE_CONNECTIONS]
- self.mediapipe_hand_connections = [this_connection for this_connection in mp_holistic.HAND_CONNECTIONS]
- self.mediapipe_face_connections = [this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION]
+ self.mediapipe_body_connections = [
+ this_connection for this_connection in mp_holistic.POSE_CONNECTIONS
+ ]
+ self.mediapipe_hand_connections = [
+ this_connection for this_connection in mp_holistic.HAND_CONNECTIONS
+ ]
+ self.mediapipe_face_connections = [
+ this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION
+ ]
def initialize_skel_dottos(self, mediapipe_trackedPoint_xyz: np.ndarray):
self.skeleton_scatter_item = gl.GLScatterPlotItem(
- pos=mediapipe_trackedPoint_xyz, color=(0, 1, 1, 1), size=10,
- pxMode=False
+ pos=mediapipe_trackedPoint_xyz, color=(0, 1, 1, 1), size=10, pxMode=False
)
self.opengl_3d_plot_widget.addItem(self.skeleton_scatter_item)
@@ -261,12 +274,15 @@ class QTVisualizerAndGui:
self._skeleton_connections_list = []
for this_connection in self.mediapipe_body_connections:
this_skel_line = gl.GLLinePlotItem(
- pos=mediapipe_trackedPoint_xyz[this_connection, :])
+ pos=mediapipe_trackedPoint_xyz[this_connection, :]
+ )
self._skeleton_connections_list.append(this_skel_line)
self.opengl_3d_plot_widget.addItem(this_skel_line)
def update_mediapipe3d_skeleton(self, mediapipe3d_multi_frame_payload):
- mediapipe3d_trackedPoint_xyz = mediapipe3d_multi_frame_payload.data3d_trackedPointNum_xyz
+ mediapipe3d_trackedPoint_xyz = (
+ mediapipe3d_multi_frame_payload.data3d_trackedPointNum_xyz
+ )
if not self._mediapipe_skeleton_initialized:
self.get_mediapipe_connections()
self.initialize_skel_dottos(mediapipe3d_trackedPoint_xyz)
@@ -274,17 +290,16 @@ class QTVisualizerAndGui:
self._mediapipe_skeleton_initialized = True
# skel dottos
- self.skeleton_scatter_item.setData(
- pos=mediapipe3d_trackedPoint_xyz
- )
+ self.skeleton_scatter_item.setData(pos=mediapipe3d_trackedPoint_xyz)
- #skel lines
- for this_skeleton_line_number, this_connection in enumerate(self.mediapipe_body_connections):
+ # skel lines
+ for this_skeleton_line_number, this_connection in enumerate(
+ self.mediapipe_body_connections
+ ):
self._skeleton_connections_list[this_skeleton_line_number].setData(
pos=mediapipe3d_trackedPoint_xyz[this_connection, :]
)
-
if __name__ == "__main__":
pass
diff --git a/src/tests/video_writer/test_video_writer.py b/src/tests/video_writer/test_video_writer.py
index 4a1625ce..dab82fe9 100644
--- a/src/tests/video_writer/test_video_writer.py
+++ b/src/tests/video_writer/test_video_writer.py
@@ -21,7 +21,7 @@ class VideoWriterTestCase(TestCase):
def test_write_on_video_writer(self):
example_payload = FramePayload(
image=np.random.randint(0, 4, (720, 1280, 3)),
- timestamp_in_seconds_from_record_start=time.time_ns()
+ timestamp_in_seconds_from_record_start=time.time_ns(),
)
vw = VideoRecorder("nothing", 100, 100, "nothing")
vw.append_frame_payload_to_list(example_payload)
diff --git a/src/utils/get_canonical_time_str.py b/src/utils/get_canonical_time_str.py
index 2072c067..bd35e428 100644
--- a/src/utils/get_canonical_time_str.py
+++ b/src/utils/get_canonical_time_str.py
@@ -1,4 +1 @@
import time
-
-
-