Free Motion Capture for Everyone πβ¨
TypeScript
Use Black to format all of `src` and add Contribution Guidelines (#188)
Endurance Idehen04529f746d41
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdnew file mode 100644index 00000000..be32fb7a--- /dev/null+++ b/CONTRIBUTING.md@@ -0,0 +1,58 @@+# Contributing to Freemocap+We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:++- Reporting a bug+- Discussing the current state of the code+- Submitting a fix+- Proposing new features+- Becoming a maintainer++## We Develop with Github+We use github to host code, to track issues and feature requests, as well as accept pull requests.++## We Use [Github Flow](https://guides.github.com/introduction/flow/index.html), So All Code Changes Happen Through Pull Requests+Pull requests are the best way to propose changes to the codebase (we use [Github Flow](https://guides.github.com/introduction/flow/index.html)). We actively welcome your pull requests:++1. Fork the repo and create your branch from `main`.+2. If you've added code that should be tested, add tests.+3. If you've changed APIs, update the documentation.+4. Ensure the test suite passes.+5. Make sure your code lints.+6. Issue that pull request!++## Any contributions you make will be under the AGPL Software License+In short, when you submit code changes, your submissions are understood to be under the same [AGPL](LICENSE) that covers the project. Feel free to contact the maintainers to understand what that means.++## Report bugs using Github's [issues](https://github.com/freemocap/freemocap/issues)+We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/freemocap/freemocap/issues/new); it's that easy!++## Write bug reports with detail, background, and sample code+[This is an example](http://stackoverflow.com/q/12488905/180626) of a "great" bug report.++**Great Bug Reports** tend to have:++- A quick summary and/or background+- Steps to reproduce+ - Be specific!+ - Include an uploaded ZIP of the freemocap data session you ran that produced the issue.+ - Give sample code if you can. [This stackoverflow question](http://stackoverflow.com/q/12488905/180626) demonstrates the user giving as much information as possible.+- What you expected would happen+- What actually happens+- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)++People *love* thorough bug reports. The likelihood that the community and/or maintainers will help you with your issues stems from how well you can help them understand you. :)++## Use a Consistent Coding Style+We use the [Black](https://black.readthedocs.io/en/stable/) autoformatter as the de-facto syntax style guide of choice.++### PyQT Style Guide+COMING SOON++### API Style Guide+COMING SOON++## License+By contributing, you agree that your contributions will be licensed under its [AGPL](LICENSE) License.++## References+This document was adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/main/CONTRIBUTING.md)\ No newline at end of filediff --git a/README.md b/README.mdindex e2796fe3..96f97bb3 100644--- a/README.md+++ b/README.md@@ -1,5 +1,5 @@<p align="center">- <img src="https://raw.githubusercontent.com/freemocap/freemocap/main/logo/freemocap-logo-black-border.svg" height="64" alt="Project Logo">+ <img src="https://raw.githubusercontent.com/freemocap/freemocap/main/assets/logo/freemocap-logo-black-border.svg" height="64" alt="Project Logo"></p><h3 align="center">freemocap</h3><p align="center">π The FreeMoCap Project: A free-and-open-source, hardware-and-software-agnostic, minimal-cost, research-grade, motion capture system and platform for decentralized scientific research, education, and training</p>@@ -16,6 +16,9 @@<a href="https://github.com/freemocap/freemocap/issues"><img src="https://img.shields.io/badge/contributions-welcome-ff69b4.svg" alt="Contributions Welcome"></a>+ <a href="https://github.com/psf/black">+ <img alt="https://img.shields.io/badge/code%20style-black-000000.svg" src="https://img.shields.io/badge/code%20style-black-000000.svg">+ </a></p>## Pardon our dust as we transtion to the `alpha` phase of this project!@@ -33,7 +36,7 @@ If you would like to access the `pre-alpha` code, you can find it here under the- At least 2 simple webcameras (more cameras, better quality)- [$10 camera here](https://www.amazon.com/1080P-Full-Webcam-Microphone-Conferencing/dp/B09JFPFVY4/ref=asc_df_B09JFPFVY4/?tag=hyprod-20&linkCode=df0&hvadid=563548498710&hvpos=&hvnetw=g&hvrand=17033983634274965573&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9028321&hvtargid=pla-1629009316502&psc=1)- Each Camera must have a clean, unobstructed view of a Charuco board during initialization.-+### Easy MacOSX Install Instructions1. Install the Homebrew Package manager (Skip this step if you already have it)@@ -93,9 +96,22 @@ That's it! You're ready to run the freemocap application, and create your own di## For Developers+### Dev Setup++After you've done the easy install instructions, you'll be able to run our repo commands++1. Run the "setup" command to set up your environment+```bash+task setup+```+2. Run the tests to ensure that everything works appropriate+```bash+task test+```+### Contribution Guidelines-COMING SOON+Please read our contribution doc: [CONTRIBUTING.md](CONTRIBUTING.md)### Creating a new binarydiff --git a/pyproject.toml b/pyproject.tomlindex 3898388b..278c391d 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,4 +1,7 @@[tool.taskipy.tasks]+setup = "pre-commit install"test = "python -m unittest src/tests/**/test_*"installer = "./bin/installer.sh"+# This command ensures that your code is formatted appropriately+format = "black src/"\ No newline at end of filediff --git a/requirements.txt b/requirements.txtindex 3c343e55..ab15832a 100644--- a/requirements.txt+++ b/requirements.txt@@ -26,4 +26,5 @@ pytest==7.1.2taskipy==1.10.2pytest-cov==3.0.0pyinstaller==5.2-pyinstaller-hooks-contrib==2022.8\ No newline at end of file+pyinstaller-hooks-contrib==2022.8+black[d]==22.6.0diff --git a/src/api/routes/__init__.py b/src/api/routes/__init__.pyindex 8837c62c..299ceb02 100644--- a/src/api/routes/__init__.py+++ b/src/api/routes/__init__.py@@ -6,4 +6,11 @@ from src.api.routes.session.session_router import session_routerfrom src.api.routes.startup.startup import startup_router# REGISTER NEW ROUTES HERE-enabled_routers = [healthcheck_router, camera_router, cam_ws_router, startup_router, home_router, session_router]+enabled_routers = [+ healthcheck_router,+ camera_router,+ cam_ws_router,+ startup_router,+ home_router,+ session_router,+]diff --git a/src/api/routes/camera/cam_data_ws.py b/src/api/routes/camera/cam_data_ws.pyindex b30b603f..78da6d75 100644--- a/src/api/routes/camera/cam_data_ws.py+++ b/src/api/routes/camera/cam_data_ws.py@@ -30,8 +30,6 @@ async def preview_webcam(web_socket: WebSocket):byte_data = await web_socket.receive_bytes()time_since = time.perf_counter()print(f"Per Frame Time: {time_since - last_read:.4f}")- buffer = np.frombuffer(byte_data, dtype='uint8').reshape((600, 800, 4))+ buffer = np.frombuffer(byte_data, dtype="uint8").reshape((600, 800, 4))buffer = cv2.cvtColor(buffer, cv2.COLOR_BGR2RGB)print(f"image shape: {buffer.shape}")--diff --git a/src/api/routes/camera/camera_route.py b/src/api/routes/camera/camera_route.pyindex f239a52a..7541168a 100644--- a/src/api/routes/camera/camera_route.py+++ b/src/api/routes/camera/camera_route.py@@ -24,12 +24,16 @@ async def config_cam(webcam_config_model: WebcamConfigModel, session_id):@camera_router.post("/camera/show_camera/{webcam_id}")-async def show_camera(webcam_id: str, camera_preview_model: CameraPreviewModel = CameraPreviewModel()):+async def show_camera(+ webcam_id: str, camera_preview_model: CameraPreviewModel = CameraPreviewModel()+):cv_cam_manager = OpenCVCameraManager(session_id=camera_preview_model.session_id)- with cv_cam_manager.start_capture_session_single_cam(webcam_id) as connected_camera_and_writer:+ with cv_cam_manager.start_capture_session_single_cam(+ webcam_id+ ) as connected_camera_and_writer:should_continue = Truethis_camera = connected_camera_and_writer.cv_camera- cv2_window_name = '(ESC to close) PREVIEWING camera_' + webcam_id+ cv2_window_name = "(ESC to close) PREVIEWING camera_" + webcam_idwhile should_continue:if not this_camera.new_frame_ready:continue@@ -43,10 +47,12 @@ async def show_camera(webcam_id: str, camera_preview_model: CameraPreviewModel =@camera_router.post("/camera/show_cameras")-async def cv2_imshow_all_camera(camera_preview_model: CameraPreviewModel = CameraPreviewModel()):+async def cv2_imshow_all_camera(+ camera_preview_model: CameraPreviewModel = CameraPreviewModel(),+):cv_cam_manager = OpenCVCameraManager(session_id=camera_preview_model.session_id)with cv_cam_manager.start_capture_session_all_cams() as connected_cameras_dict:- logger.info(f'Available cameras: {connected_cameras_dict}')+ logger.info(f"Available cameras: {connected_cameras_dict}")should_continue = Truewhile should_continue:for this_webcam_id, this_open_cv_camera in connected_cameras_dict.items():@@ -57,7 +63,7 @@ async def cv2_imshow_all_camera(camera_preview_model: CameraPreviewModel = Camercv2.destroyWindow(cv2_window_name)if not this_open_cv_camera.new_frame_ready:continue- cv2_window_name = '(ESC to close) PREVIEWING camera_' + this_webcam_id+ cv2_window_name = "(ESC to close) PREVIEWING camera_" + this_webcam_idcv2.imshow(cv2_window_name, this_open_cv_camera.latest_frame.image)exit_key = cv2.waitKey(1)if exit_key == 27:@@ -74,4 +80,3 @@ async def get_cameras():@camera_router.get("/camera/redetect")async def redetect_cameras():return get_or_create_cams(always_create=True)-diff --git a/src/api/routes/home/home.py b/src/api/routes/home/home.pyindex d1a0cec2..090c895f 100644--- a/src/api/routes/home/home.py+++ b/src/api/routes/home/home.py@@ -5,6 +5,6 @@ home_router = APIRouter()@home_router.get("/")async def hello():- response_string = 'hello :D'- print(f'response string: {response_string}')+ response_string = "hello :D"+ print(f"response string: {response_string}")return response_stringdiff --git a/src/api/routes/session/session_router.py b/src/api/routes/session/session_router.pyindex acaf76af..d8a399cc 100644--- a/src/api/routes/session/session_router.py+++ b/src/api/routes/session/session_router.py@@ -6,15 +6,28 @@ from typing import Optional, Unionfrom fastapi import APIRouterfrom pydantic import BaseModel-from jon_scratch.pupil_calibration_pipeline.qt_gl_laser_skeleton_visualizer import QtGlLaserSkeletonVisualizer+from jon_scratch.pupil_calibration_pipeline.qt_gl_laser_skeleton_visualizer import (+ QtGlLaserSkeletonVisualizer,+)from src.cameras.launch_camera_frame_loop import launch_camera_frame_loopfrom src.cameras.multicam_manager.cv_camera_manager import OpenCVCameraManager-from src.config.home_dir import create_session_id, get_session_folder_path, get_most_recent_session_id, \- create_session_folder-from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import MediaPipeSkeletonDetector-from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import CalibrationPipelineOrchestrator-from src.pipelines.session_pipeline.session_pipeline_orchestrator import SessionPipelineOrchestrator, \- load_mediapipe3d_skeleton_data, load_mediapipe2d_data+from src.config.home_dir import (+ create_session_id,+ get_session_folder_path,+ get_most_recent_session_id,+ create_session_folder,+)+from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import (+ MediaPipeSkeletonDetector,+)+from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import (+ CalibrationPipelineOrchestrator,+)+from src.pipelines.session_pipeline.session_pipeline_orchestrator import (+ SessionPipelineOrchestrator,+ load_mediapipe3d_skeleton_data,+ load_mediapipe2d_data,+)logger = logging.getLogger(__name__)@@ -40,36 +53,43 @@ class SessionCalibrateModel(TweakedPydanticBaseModel):session_id: str, ID for the session we're calibratingcharuco_square_size:Union[int, float], the size of one of the black squares on the charuco board (preferably) in milimeters"""+session_id: str = Nonewebcam_configs_dict: dict = Noneopencv_camera_manager: OpenCVCameraManager = Nonecharuco_square_size: Union[int, float] = 39+class SessionRecordModel(TweakedPydanticBaseModel):session_id: str = Nonewebcam_configs_dict: dict = Noneclass SessionIdModel(BaseModel):- """- """+ """ """+session_id: str = None@session_router.post("/session/create")-async def create_session(session_create_model: SessionCreateModel = SessionCreateModel()) -> SessionResponse:+async def create_session(+ session_create_model: SessionCreateModel = SessionCreateModel(),+) -> SessionResponse:session_id = create_session_id()if session_create_model.user_session_tag_str is not None:session_id = session_id + session_create_model.user_session_tag_strcreate_session_folder(session_id)- return SessionResponse(session_id=session_id,- session_path=get_session_folder_path(session_id))+ return SessionResponse(+ session_id=session_id, session_path=get_session_folder_path(session_id)+ )@session_router.post("/session/calibrate")-def calibrate_session(session_calibrate_model: SessionCalibrateModel = SessionCalibrateModel()):+def calibrate_session(+ session_calibrate_model: SessionCalibrateModel = SessionCalibrateModel(),+):"""calibate capture volume - record synchronized videos (from all available camras wtih default parameters for now) and process with Anipose to produce a camera calibration (saved as a `.toml` file in the session folder"""session_id = session_calibrate_model.session_id@@ -83,17 +103,19 @@ def calibrate_session(session_calibrate_model: SessionCalibrateModel = SessionCa# show_camera_views_in_windows=True,# )- launch_camera_frame_loop(session_id=session_id,- webcam_configs_dict=session_calibrate_model.webcam_configs_dict,- show_camera_views_in_windows=True,- calibration_videos_bool=True,- detect_charuco_in_image=True,- )+ launch_camera_frame_loop(+ session_id=session_id,+ webcam_configs_dict=session_calibrate_model.webcam_configs_dict,+ show_camera_views_in_windows=True,+ calibration_videos_bool=True,+ detect_charuco_in_image=True,+ )try:calibration_orchestrator.run_anipose_camera_calibration(charuco_square_size=session_calibrate_model.charuco_square_size,- pin_camera_0_to_origin=True)+ pin_camera_0_to_origin=True,+ )except:logger.error("Printing Traceback")traceback.print_exc()@@ -101,25 +123,30 @@ def calibrate_session(session_calibrate_model: SessionCalibrateModel = SessionCareturn calibration_orchestrator-@session_router.post("/session/record")def record_session(session_record_model: SessionRecordModel = SessionRecordModel()):- launch_camera_frame_loop(session_id=session_record_model.session_id,- webcam_configs_dict=session_record_model.webcam_configs_dict,- show_camera_views_in_windows=True,- calibration_videos_bool=False,- detect_charuco_in_image=True,- )+ launch_camera_frame_loop(+ session_id=session_record_model.session_id,+ webcam_configs_dict=session_record_model.webcam_configs_dict,+ show_camera_views_in_windows=True,+ calibration_videos_bool=False,+ detect_charuco_in_image=True,+ )+@session_router.post("/session/mediapipe_track_skeletons_offline")-def mediapipe_track_2D_skeletons_offline(session_id_model: SessionIdModel = SessionIdModel()):+def mediapipe_track_2D_skeletons_offline(+ session_id_model: SessionIdModel = SessionIdModel(),+):if session_id_model.session_id is None or session_id_model.session_id == "string":this_session_id = get_most_recent_session_id()logger.info(f"loading most recent session:{this_session_id}")else:this_session_id = session_id_model.session_id- logger.info(f"tracking 2D mediapipe skeletons in videos from session: {this_session_id}")+ logger.info(+ f"tracking 2D mediapipe skeletons in videos from session: {this_session_id}"+ )mediapipe_skeleton_detector = MediaPipeSkeletonDetector(this_session_id)mediapipe_skeleton_detector.process_session_folder()@@ -132,9 +159,12 @@ def mediapipe_reconstruct_3D_skeletons_offline(session_id_model: SessionIdModelsession_id = session_id_model.session_idthis_session_orchestrator = SessionPipelineOrchestrator(session_id=session_id)- this_session_orchestrator.anipose_camera_calibration_object = CalibrationPipelineOrchestrator().load_calibration_from_session_id(- session_id)- this_session_orchestrator.mediapipe2d_numCams_numFrames_numTrackedPoints_XY = load_mediapipe2d_data(session_id)+ this_session_orchestrator.anipose_camera_calibration_object = (+ CalibrationPipelineOrchestrator().load_calibration_from_session_id(session_id)+ )+ this_session_orchestrator.mediapipe2d_numCams_numFrames_numTrackedPoints_XY = (+ load_mediapipe2d_data(session_id)+ )this_session_orchestrator.reconstruct3d_from_2d_data_offline()@@ -147,11 +177,16 @@ def visualize_session_offline(session_id_model: SessionIdModel = None):mediapipe3d_data_payload = load_mediapipe3d_skeleton_data(session_id)- mediapipe3d_skeleton_nFrames_nTrajectories_xyz = mediapipe3d_data_payload.data3d_numFrames_numTrackedPoints_XYZ- mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError = mediapipe3d_data_payload.data3d_numFrames_numTrackedPoint_reprojectionError+ mediapipe3d_skeleton_nFrames_nTrajectories_xyz = (+ mediapipe3d_data_payload.data3d_numFrames_numTrackedPoints_XYZ+ )+ mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError = (+ mediapipe3d_data_payload.data3d_numFrames_numTrackedPoint_reprojectionError+ )qt_gl_laser_skeleton = QtGlLaserSkeletonVisualizer(- mediapipe_skel_fr_mar_xyz=mediapipe3d_skeleton_nFrames_nTrajectories_xyz)+ mediapipe_skel_fr_mar_xyz=mediapipe3d_skeleton_nFrames_nTrajectories_xyz+ )qt_gl_laser_skeleton.start_animation()diff --git a/src/api/services/board_detect_service.py b/src/api/services/board_detect_service.pyindex 5bfdcf43..4077396d 100644--- a/src/api/services/board_detect_service.py+++ b/src/api/services/board_detect_service.py@@ -7,4 +7,4 @@ class ImageResponse(NamedTuple):class BoardDetectService:- pass\ No newline at end of file+ passdiff --git a/src/api/services/mediapipe_detect_service.py b/src/api/services/mediapipe_detect_service.pyindex 085e3f2b..d000c853 100644--- a/src/api/services/mediapipe_detect_service.py+++ b/src/api/services/mediapipe_detect_service.py@@ -2,4 +2,4 @@ from src.cameras.multicam_manager.cv_camera_manager import OpenCVCameraManagerclass MediapipeSkeletonDetectionService:- pass\ No newline at end of file+ passdiff --git a/src/api/services/user_config.py b/src/api/services/user_config.pyindex f0547da4..f189911a 100644--- a/src/api/services/user_config.py+++ b/src/api/services/user_config.py@@ -8,6 +8,7 @@ from src.config.home_dir import get_session_folder_pathlogger = logging.getLogger(__name__)+class WebcamConfigModel(BaseModel):webcam_id: strexposure: int = -6@@ -22,13 +23,15 @@ def get_camera_name(camera_id: str):class UserConfigService:"""take things like WebcamConfigModel and save it to a text file or whatever"""- def save_webcam_config_to_disk(self, webcam_config_model: WebcamConfigModel, session_id: str):+ def save_webcam_config_to_disk(+ self, webcam_config_model: WebcamConfigModel, session_id: str+ ):webcam_config_json = webcam_config_model.json()camera_name = get_camera_name(webcam_config_model.webcam_id)- json_file_name = camera_name + '_config.json'+ json_file_name = camera_name + "_config.json"session_path = get_session_folder_path(session_id)webcam_config_json_path = Path(session_path) / json_file_name- with open(webcam_config_json_path, 'w') as outfile:+ with open(webcam_config_json_path, "w") as outfile:outfile.write(str(webcam_config_json))return webcam_config_json@@ -39,14 +42,15 @@ class UserConfigService:camera_name = get_camera_name(webcam_id)print(f"webcam id (again): {webcam_id}")print(f"camera name: {camera_name}")- json_file_name = camera_name + '_config.json'+ json_file_name = camera_name + "_config.json"webcam_config_json_path = Path(session_path) / json_file_nametry:webcam_config_model = WebcamConfigModel.parse_file(webcam_config_json_path)- logger.info(f'loading webcam_config from: {webcam_config_json_path}')+ logger.info(f"loading webcam_config from: {webcam_config_json_path}")return webcam_config_modelexcept FileNotFoundError:- logger.info(f'No webcam config file found for this camera, using default camera parameters')+ logger.info(+ f"No webcam config file found for this camera, using default camera parameters"+ )return WebcamConfigModel(webcam_id=webcam_id)-diff --git a/src/cameras/capture/dataclasses/frame_payload.py b/src/cameras/capture/dataclasses/frame_payload.pyindex ccef9d6a..30b38030 100644--- a/src/cameras/capture/dataclasses/frame_payload.py+++ b/src/cameras/capture/dataclasses/frame_payload.py@@ -6,9 +6,7 @@ import numpy as npclass FramePayload(NamedTuple):success: bool = Falseimage: np.ndarray = None- timestamp_in_seconds_from_record_start: float = None+ timestamp_in_seconds_from_record_start: float = Nonetimestamp_unix_time_seconds: float = Noneframe_number: int = Nonewebcam_id: str = None--diff --git a/src/cameras/capture/opencv_camera/camera_stream_thread_handler.py b/src/cameras/capture/opencv_camera/camera_stream_thread_handler.pyindex 217f12ce..6af5ecdd 100644--- a/src/cameras/capture/opencv_camera/camera_stream_thread_handler.py+++ b/src/cameras/capture/opencv_camera/camera_stream_thread_handler.py@@ -13,10 +13,9 @@ logger = logging.getLogger(__name__)class VideoCaptureThread(threading.Thread):def __init__(- self,- get_next_frame,- webcam_id:str = 'unknown',-+ self,+ get_next_frame,+ webcam_id: str = "unknown",):super().__init__()self._webcam_id = webcam_id@@ -34,9 +33,13 @@ class VideoCaptureThread(threading.Thread):@propertydef median_framerate(self):if self._num_frames_processed == 0:- logger.warning(f"No Frames processed yet, cannot calculate median_framerate")+ logger.warning(+ f"No Frames processed yet, cannot calculate median_framerate"+ )else:- self._median_framerate = np.nanmedian((np.diff(self._timestamps_npy) ** -1) / 1e9)+ self._median_framerate = np.nanmedian(+ (np.diff(self._timestamps_npy) ** -1) / 1e9+ )return self._median_framerate@@ -68,7 +71,9 @@ class VideoCaptureThread(threading.Thread):start = time.perf_counter_ns()self._frame = self._get_next_frame()self._elapsed_during_frame_grab.append(time.perf_counter_ns() - start)- self._timestamps_npy.append(self._frame.timestamp_in_seconds_from_record_start)+ self._timestamps_npy.append(+ self._frame.timestamp_in_seconds_from_record_start+ )self._num_frames_processed += 1# logger.debug(f"Camera {self._webcam_id}: captured frame# {self._num_frames_processed}")except:diff --git a/src/cameras/capture/opencv_camera/opencv_camera.py b/src/cameras/capture/opencv_camera/opencv_camera.pyindex a60ea372..d56d7e81 100644--- a/src/cameras/capture/opencv_camera/opencv_camera.py+++ b/src/cameras/capture/opencv_camera/opencv_camera.py@@ -11,7 +11,9 @@ from src.cameras.capture.dataclasses.frame_payload import FramePayloadfrom src.cameras.persistence.video_writer.video_recorder import VideoRecorderfrom src.cameras.viewer.cv_cam_viewer import CvCamViewerfrom src.config.webcam_config import WebcamConfig-from src.cameras.capture.opencv_camera.camera_stream_thread_handler import VideoCaptureThread+from src.cameras.capture.opencv_camera.camera_stream_thread_handler import (+ VideoCaptureThread,+)from src.core_processor.timestamp_manager.timestamp_manager import TimestampLoggerlogger = logging.getLogger(__name__)@@ -22,11 +24,13 @@ class OpenCVCamera:Performant implementation of video capture against webcams"""- def __init__(self,- config: WebcamConfig,- session_id: str=None,- session_start_time_perf_counter_ns:int=0,- calibration_video_bool: bool = False):+ def __init__(+ self,+ config: WebcamConfig,+ session_id: str = None,+ session_start_time_perf_counter_ns: int = 0,+ calibration_video_bool: bool = False,+ ):self._config = configself._name = f"Camera_{self._config.webcam_id}"self._opencv_video_capture_object: cv2.VideoCapture = None@@ -117,22 +121,24 @@ class OpenCVCamera:)raise Exception-logger.debug(f"Camera found at port number {self._config.webcam_id}")if self.session_id is False:logger.info(- f"No `session_id` specified for {self._name}, video_recorder will not be created (because we won't know where to save the videos)")+ f"No `session_id` specified for {self._name}, video_recorder will not be created (because we won't know where to save the videos)"+ )self._video_recorder = Noneelse:image_height = image.shape[0]image_width = image.shape[1]- self._video_recorder = VideoRecorder(self._name,- image_width=image_width,- image_height=image_height,- session_id=self.session_id,- calibration_video_bool=self._calibration_video_bool)+ self._video_recorder = VideoRecorder(+ self._name,+ image_width=image_width,+ image_height=image_height,+ session_id=self.session_id,+ calibration_video_bool=self._calibration_video_bool,+ )return success@@ -182,7 +188,9 @@ class OpenCVCamera:### A - see -> https://stackoverflow.com/questions/57716962/difference-between-video-capture-read-and-grabself._opencv_video_capture_object.grab()success, image = self._opencv_video_capture_object.retrieve()- this_frame_timestamp_perf_counter_ns = time.perf_counter_ns() - self._session_start_time_perf_counter_ns+ this_frame_timestamp_perf_counter_ns = (+ time.perf_counter_ns() - self._session_start_time_perf_counter_ns+ )# timestamp_ns_post = time.perf_counter_ns()# it_took_this_many_seconds_to_grab_the_frame = (timestamp_ns_post-timestamp_ns_pre)/1e9@@ -190,7 +198,6 @@ class OpenCVCamera:logger.error(f"Failed to read frame from Camera: {self.webcam_id_as_str}")raise Exception-self._new_frame_ready = successif success:@@ -198,12 +205,15 @@ class OpenCVCamera:# if self._camera_view_update_function is not None:# self._camera_view_update_function(self.webcam_id_as_str, cv2.cvtColor(image, cv2.COLOR_BGR2RGB))- return FramePayload(success=success,- image=image,- timestamp_in_seconds_from_record_start=this_frame_timestamp_perf_counter_ns/1e9,- timestamp_unix_time_seconds = time.time(),- frame_number=self.latest_frame_number,- webcam_id=self.webcam_id_as_str)+ return FramePayload(+ success=success,+ image=image,+ timestamp_in_seconds_from_record_start=this_frame_timestamp_perf_counter_ns+ / 1e9,+ timestamp_unix_time_seconds=time.time(),+ frame_number=self.latest_frame_number,+ webcam_id=self.webcam_id_as_str,+ )def stop_frame_capture(self):self.close()diff --git a/src/cameras/detection/cam_detection.py b/src/cameras/detection/cam_detection.pyindex 2d7d81d0..5c62403f 100644--- a/src/cameras/detection/cam_detection.py+++ b/src/cameras/detection/cam_detection.py@@ -21,11 +21,7 @@ class DetectPossibleCameras:if success and image is not None:try:- cams_to_use_list.append(- RawCamera(- webcam_id=str(cam_id)- )- )+ cams_to_use_list.append(RawCamera(webcam_id=str(cam_id)))finally:cap.release()diff --git a/src/cameras/launch_camera_frame_loop.py b/src/cameras/launch_camera_frame_loop.pyindex 9d4cd4c0..95ab1fb6 100644--- a/src/cameras/launch_camera_frame_loop.py+++ b/src/cameras/launch_camera_frame_loop.py@@ -12,88 +12,113 @@ import asynciofrom src.core_processor.show_cam_window import show_cam_windowfrom src.core_processor.utils.image_fps_writer import write_fps_to_image-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import CharucoBoardDetector+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (+ CharucoBoardDetector,+)from src.qt_visualizer_and_gui.qt_visualizer_and_gui import QTVisualizerAndGuilogger = logging.getLogger(__name__)def launch_camera_frame_loop(- session_id: str,- webcam_configs_dict: Dict[str, WebcamConfig] = None,- show_camera_views_in_windows: bool = True,- calibration_videos_bool: bool = False,- detect_charuco_in_image: bool = True,- record_frames_bool: bool = True,- show_visualizer_gui:bool = False,+ session_id: str,+ webcam_configs_dict: Dict[str, WebcamConfig] = None,+ show_camera_views_in_windows: bool = True,+ calibration_videos_bool: bool = False,+ detect_charuco_in_image: bool = True,+ record_frames_bool: bool = True,+ show_visualizer_gui: bool = False,):any_frames_recorded = Falseif detect_charuco_in_image:charuco_board_detector = CharucoBoardDetector()--opencv_camera_manager = OpenCVCameraManager(session_id=session_id)with opencv_camera_manager.start_capture_session_all_cams(- webcam_configs_dict=webcam_configs_dict,- calibration_videos=calibration_videos_bool,+ webcam_configs_dict=webcam_configs_dict,+ calibration_videos=calibration_videos_bool,) as connected_cameras_dict:-- timestamp_manager = opencv_camera_manager.timestamp_manager# <-this is defines the start-time of the timestamp loggers for each camera+ timestamp_manager = (+ opencv_camera_manager.timestamp_manager+ ) # <-this is defines the start-time of the timestamp loggers for each cameratry:if show_visualizer_gui:visualizer_gui = QTVisualizerAndGui()- visualizer_gui.setup_and_launch(opencv_camera_manager.available_webcam_ids)+ visualizer_gui.setup_and_launch(+ opencv_camera_manager.available_webcam_ids+ )should_continue = Truewhile should_continue:-if not opencv_camera_manager.new_multi_frame_ready():continuethis_multi_frame_payload = opencv_camera_manager.latest_multi_frame- timestamp_manager.multi_frame_timestamp_logger.log_new_timestamp_seconds_from_unspecified_zero(this_multi_frame_payload.multi_frame_timestamp_seconds)+ timestamp_manager.multi_frame_timestamp_logger.log_new_timestamp_seconds_from_unspecified_zero(+ this_multi_frame_payload.multi_frame_timestamp_seconds+ )- for this_webcam_id, this_open_cv_camera in connected_cameras_dict.items():+ for (+ this_webcam_id,+ this_open_cv_camera,+ ) in connected_cameras_dict.items():- this_cam_latest_frame = this_multi_frame_payload.frames_dict[this_webcam_id]+ this_cam_latest_frame = this_multi_frame_payload.frames_dict[+ this_webcam_id+ ]if this_cam_latest_frame is None:continue- this_cam_timestamp_logger = timestamp_manager.timestamp_logger_for_webcam_id(this_webcam_id)- this_cam_timestamp_logger.log_new_timestamp_seconds_from_unspecified_zero(this_cam_latest_frame.timestamp_in_seconds_from_record_start)+ this_cam_timestamp_logger = (+ timestamp_manager.timestamp_logger_for_webcam_id(this_webcam_id)+ )+ this_cam_timestamp_logger.log_new_timestamp_seconds_from_unspecified_zero(+ this_cam_latest_frame.timestamp_in_seconds_from_record_start+ )# save frame to video fileif record_frames_bool:any_frames_recorded = True- this_open_cv_camera.video_recorder.append_frame_payload_to_list(this_cam_latest_frame)+ this_open_cv_camera.video_recorder.append_frame_payload_to_list(+ this_cam_latest_frame+ )image_to_display = this_cam_latest_frame.image.copy()if detect_charuco_in_image:# detect charuco board- this_charuco_frame_payload = charuco_board_detector.detect_charuco_board(- this_cam_latest_frame)- image_to_display = this_charuco_frame_payload.annotated_image.copy()+ this_charuco_frame_payload = (+ charuco_board_detector.detect_charuco_board(+ this_cam_latest_frame+ )+ )+ image_to_display = (+ this_charuco_frame_payload.annotated_image.copy()+ )image_to_display = write_fps_to_image(image_to_display,- timestamp_manager.timestamp_logger_for_webcam_id(this_webcam_id).median_frames_per_second,+ timestamp_manager.timestamp_logger_for_webcam_id(+ this_webcam_id+ ).median_frames_per_second,)if show_camera_views_in_windows:- should_continue = show_cam_window(this_webcam_id, image_to_display )+ should_continue = show_cam_window(+ this_webcam_id, image_to_display+ )if show_visualizer_gui:- visualizer_gui.update_camera_view_image(this_webcam_id,- image_to_display)+ visualizer_gui.update_camera_view_image(+ this_webcam_id, image_to_display+ )# exit loop when user presses ESC keyexit_key = cv2.waitKey(1)@@ -113,14 +138,15 @@ def launch_camera_frame_loop(# cv2.destroyWindow(this_open_cv_camera.webcam_id_as_str)cv2.destroyAllWindows()-for this_open_cv_camera in connected_cameras_dict.values():if any_frames_recorded:- logger.info(f"Saving {this_open_cv_camera.webcam_id_as_str} video to disk")+ logger.info(+ f"Saving {this_open_cv_camera.webcam_id_as_str} video to disk"+ )this_open_cv_camera.video_recorder.save_list_of_frames_to_video_file(- this_open_cv_camera.video_recorder.frame_list)+ this_open_cv_camera.video_recorder.frame_list+ )this_open_cv_camera.video_recorder.close()logger.info("Creating camera timestamp diagnostic plot")timestamp_manager.create_diagnostic_plots()-diff --git a/src/cameras/multicam_manager/cv_camera_manager.py b/src/cameras/multicam_manager/cv_camera_manager.pyindex 9c43ac6f..5afbcc14 100644--- a/src/cameras/multicam_manager/cv_camera_manager.py+++ b/src/cameras/multicam_manager/cv_camera_manager.py@@ -12,8 +12,13 @@ from src.cameras.capture.opencv_camera.opencv_camera import OpenCVCamerafrom src.cameras.detection.cam_singleton import get_or_create_camsfrom src.cameras.persistence.video_writer.video_recorder import VideoRecorderfrom src.config.webcam_config import WebcamConfig-from src.core_processor.timestamp_manager.timestamp_manager import TimestampManager, TimestampLogger-from src.pipelines.session_pipeline.data_classes.multi_frame_payload import MultiFramePayload+from src.core_processor.timestamp_manager.timestamp_manager import (+ TimestampManager,+ TimestampLogger,+)+from src.pipelines.session_pipeline.data_classes.multi_frame_payload import (+ MultiFramePayload,+)logger = logging.getLogger(__name__)@@ -27,10 +32,11 @@ class CamAndWriterResponse(BaseModel):class OpenCVCameraManager:- def __init__(self,- session_id: str = None,- shut_down_event_bool:bool = None,- ):+ def __init__(+ self,+ session_id: str = None,+ shut_down_event_bool: bool = None,+ ):self._session_id = session_idself._config_service = UserConfigService()self._detected_cams_data = get_or_create_cams()@@ -41,11 +47,10 @@ class OpenCVCameraManager:self._shut_down_event_bool = shut_down_event_boolself._number_of_multi_frames = 0-@propertydef timestamp_manager(self):if self._timestamp_manager is None:- logger.error('timestamp manager has not been created yet')+ logger.error("timestamp manager has not been created yet")raise Exceptionelse:return self._timestamp_manager@@ -60,25 +65,30 @@ class OpenCVCameraManager:logging.error("a multi_frame was requested before it was ready!")raise Exception- this_multi_frame_timestamp_sec = (time.perf_counter_ns() - self._session_start_time_perf_counter_ns)/1e9+ this_multi_frame_timestamp_sec = (+ time.perf_counter_ns() - self._session_start_time_perf_counter_ns+ ) / 1e9this_multi_frame_dict = {}each_cam_timestamp = []for this_cam in self._connected_cameras_dict.values():if not this_cam.new_frame_ready:- logger.error('It shouldnt be able to get here if the `new_multi_frame`')+ logger.error("It shouldnt be able to get here if the `new_multi_frame`")this_multi_frame_dict[this_cam.webcam_id_as_str] = this_cam.latest_frame- each_cam_timestamp.append(this_cam.latest_frame.timestamp_in_seconds_from_record_start)+ each_cam_timestamp.append(+ this_cam.latest_frame.timestamp_in_seconds_from_record_start+ )- self._number_of_multi_frames +=1+ self._number_of_multi_frames += 1- return MultiFramePayload(frames_dict=this_multi_frame_dict,- multi_frame_number=self._number_of_multi_frames,- each_frame_timestamp=each_cam_timestamp,- multi_frame_timestamp_seconds=this_multi_frame_timestamp_sec- )+ return MultiFramePayload(+ frames_dict=this_multi_frame_dict,+ multi_frame_number=self._number_of_multi_frames,+ each_frame_timestamp=each_cam_timestamp,+ multi_frame_timestamp_seconds=this_multi_frame_timestamp_sec,+ )def new_multi_frame_ready(self):"""cycle through connected cameras and return false if one isn't read yet@@ -92,13 +102,15 @@ class OpenCVCameraManager:for this_raw_camera in self._detected_cams_data.cameras_found_list:this_webcam_config = WebcamConfig()this_webcam_config.webcam_id = this_raw_camera.webcam_id- self._available_cameras_dict[this_webcam_config.webcam_id] = this_webcam_config+ self._available_cameras_dict[+ this_webcam_config.webcam_id+ ] = this_webcam_configreturn self._available_cameras_dict@contextmanagerdef start_capture_session_single_cam(- self, webcam_id: str+ self, webcam_id: str) -> ContextManager[CamAndWriterResponse]:"""Context manager for easy start up, usage, and cleanup of camera resources.@@ -106,7 +118,7 @@ class OpenCVCameraManager:"""cv_camera = self._create_single_opencv_cam(webcam_id)- self._connected_cameras_dict['0'] = cv_camera+ self._connected_cameras_dict["0"] = cv_cameratry:self._start_frame_capture_on_cam_id(cv_camera)self._initialize_timestamp_logger() # start timestamp logger and wha@@ -118,17 +130,18 @@ class OpenCVCameraManager:@contextmanagerdef start_capture_session_all_cams(- self,- webcam_configs_dict: Dict[str, WebcamConfig] = None,- camera_view_update_function=None,- calibration_videos: bool = False,+ self,+ webcam_configs_dict: Dict[str, WebcamConfig] = None,+ camera_view_update_function=None,+ calibration_videos: bool = False,) -> ContextManager[Dict[str, OpenCVCamera]]:self._initialize_timestamp_logger()open_cv_camera_objects = self._create_opencv_cameras(session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns,webcam_configs_dict=webcam_configs_dict,- calibration_videos=calibration_videos, )+ calibration_videos=calibration_videos,+ )try:for cv_cam in open_cv_camera_objects:@@ -142,17 +155,16 @@ class OpenCVCameraManager:logger.error("Printing traceback from starting capture session by cam")traceback.print_exc()- def _create_opencv_cameras(self,- session_start_time_perf_counter_ns: int,- webcam_configs_dict: Dict[str, WebcamConfig] = None,- calibration_videos: bool = False,- ):+ def _create_opencv_cameras(+ self,+ session_start_time_perf_counter_ns: int,+ webcam_configs_dict: Dict[str, WebcamConfig] = None,+ calibration_videos: bool = False,+ ):raw_camera_objects = self._detected_cams_data.cameras_found_listopen_cv_cameras: List[OpenCVCamera] = []--for this_raw_cam in raw_camera_objects:if webcam_configs_dict is None:@@ -160,19 +172,22 @@ class OpenCVCameraManager:else:this_webcam_config = webcam_configs_dict[this_raw_cam.webcam_id]- opencv_cam_obj = self._create_single_opencv_cam(webcam_id=this_raw_cam.webcam_id,- webcam_config=this_webcam_config,- session_start_time_perf_counter_ns = session_start_time_perf_counter_ns,- calibration_video_bool=calibration_videos)+ opencv_cam_obj = self._create_single_opencv_cam(+ webcam_id=this_raw_cam.webcam_id,+ webcam_config=this_webcam_config,+ session_start_time_perf_counter_ns=session_start_time_perf_counter_ns,+ calibration_video_bool=calibration_videos,+ )open_cv_cameras.append(opencv_cam_obj)return open_cv_cameras- def _create_single_opencv_cam(self,- webcam_id: str,- webcam_config: WebcamConfig(),- session_start_time_perf_counter_ns:int,- calibration_video_bool: bool = False,- ):+ def _create_single_opencv_cam(+ self,+ webcam_id: str,+ webcam_config: WebcamConfig(),+ session_start_time_perf_counter_ns: int,+ calibration_video_bool: bool = False,+ ):## JSM NOTE - This method seems to want to load webcam_config from disk? It might be useful, but I'll bypass it for now#@@ -182,23 +197,28 @@ class OpenCVCameraManager:# resolution_width=webcam_config_model.resolution_width,# resolution_height=webcam_config_model.resolution_height, )webcam_config.webcam_id = webcam_id- return OpenCVCamera(config=webcam_config,- session_id=self._session_id,- session_start_time_perf_counter_ns = session_start_time_perf_counter_ns,- calibration_video_bool=calibration_video_bool)+ return OpenCVCamera(+ config=webcam_config,+ session_id=self._session_id,+ session_start_time_perf_counter_ns=session_start_time_perf_counter_ns,+ calibration_video_bool=calibration_video_bool,+ )def _initialize_timestamp_logger(self):self._session_start_time_unix_ns = time.time_ns()self._session_start_time_perf_counter_ns = time.perf_counter_ns()- logger.info(f"Initializing timestamp loggers - _session_start_time_unix_ns: {self._session_start_time_unix_ns}, _session_start_time_perf_counter_ns: {self._session_start_time_perf_counter_ns} ")-- self._timestamp_manager = TimestampManager(self._session_id,- self.available_webcam_ids,- self._session_start_time_unix_ns,- self._session_start_time_perf_counter_ns)+ logger.info(+ f"Initializing timestamp loggers - _session_start_time_unix_ns: {self._session_start_time_unix_ns}, _session_start_time_perf_counter_ns: {self._session_start_time_perf_counter_ns} "+ )+ self._timestamp_manager = TimestampManager(+ self._session_id,+ self.available_webcam_ids,+ self._session_start_time_unix_ns,+ self._session_start_time_perf_counter_ns,+ )def _start_frame_capture_on_cam_id(self, opencv_cam: OpenCVCamera):opencv_cam.connect()diff --git a/src/cameras/persistence/video_writer/save_options_dataclass.py b/src/cameras/persistence/video_writer/save_options_dataclass.pyindex 80d18a65..d4d1c7b4 100644--- a/src/cameras/persistence/video_writer/save_options_dataclass.py+++ b/src/cameras/persistence/video_writer/save_options_dataclass.py@@ -3,7 +3,7 @@ from pathlib import Path@dataclass-class SaveOptions():+class SaveOptions:path_to_save_video: Pathcamera_name: strframe_width: intdiff --git a/src/cameras/persistence/video_writer/video_recorder.py b/src/cameras/persistence/video_writer/video_recorder.pyindex 40323e81..17c6e84a 100644--- a/src/cameras/persistence/video_writer/video_recorder.py+++ b/src/cameras/persistence/video_writer/video_recorder.py@@ -11,23 +11,28 @@ import pandas as pdfrom src.cameras.capture.dataclasses.frame_payload import FramePayloadfrom src.cameras.persistence.video_writer.save_options_dataclass import SaveOptionsfrom src.config.data_paths import freemocap_data_path-from src.config.home_dir import get_session_folder_path, get_synchronized_videos_folder_path, get_calibration_videos_folder_path, \- get_mediapipe_annotated_videos_folder_path+from src.config.home_dir import (+ get_session_folder_path,+ get_synchronized_videos_folder_path,+ get_calibration_videos_folder_path,+ get_mediapipe_annotated_videos_folder_path,+)from rich.progress import Progresslogger = logging.getLogger(__name__)class VideoRecorder:- def __init__(self,- video_name: str,- image_width: int,- image_height: int,- session_id: str,- fourcc: str = "MP4V",- calibration_video_bool: bool = False,- mediapipe_annotated_video_bool: bool = False,- ):+ def __init__(+ self,+ video_name: str,+ image_width: int,+ image_height: int,+ session_id: str,+ fourcc: str = "MP4V",+ calibration_video_bool: bool = False,+ mediapipe_annotated_video_bool: bool = False,+ ):self._video_name = video_nameself._image_width = image_widthself._image_height = image_height@@ -73,10 +78,12 @@ class VideoRecorder:def append_frame_payload_to_list(self, frame_payload: FramePayload):self._frame_payload_list.append(frame_payload)- def save_list_of_frames_to_video_file(self,- list_of_frames:List[FramePayload]=None,- frames_per_second:float=None,- calibration_videos:bool=False):+ def save_list_of_frames_to_video_file(+ self,+ list_of_frames: List[FramePayload] = None,+ frames_per_second: float = None,+ calibration_videos: bool = False,+ ):if list_of_frames is None:list_of_frames = self._frame_payload_list@@ -88,13 +95,15 @@ class VideoRecorder:self._timestamps_npy = self._gather_timestamps(list_of_frames)frames_per_second = np.nanmedian((np.diff(self._timestamps_npy) ** -1))- self._initialize_video_writer(frames_per_second=frames_per_second, calibration_videos=calibration_videos)+ self._initialize_video_writer(+ frames_per_second=frames_per_second, calibration_videos=calibration_videos+ )self._write_frame_list_to_video_file(list_of_frames)self._save_timestamps(self._timestamps_npy)- def save_image_list_to_disk(self,- image_list: List[np.ndarray],- frames_per_second:float):+ def save_image_list_to_disk(+ self, image_list: List[np.ndarray], frames_per_second: float+ ):if len(image_list) == 0:logging.error(f"No frames to save for : {self._video_name}")return@@ -102,27 +111,37 @@ class VideoRecorder:self._initialize_video_writer(frames_per_second=frames_per_second)self._write_image_list_to_video_file(image_list)- def _initialize_video_writer(self, frames_per_second: Union[int, float] = None, calibration_videos:bool=False):+ def _initialize_video_writer(+ self,+ frames_per_second: Union[int, float] = None,+ calibration_videos: bool = False,+ ):- video_file_name = self._video_name + '.mp4'+ video_file_name = self._video_name + ".mp4"if calibration_videos:- self._video_folder_path = Path(get_calibration_videos_folder_path(self._session_id))+ self._video_folder_path = Path(+ get_calibration_videos_folder_path(self._session_id)+ )elif self._mediapipe_annotated_video_bool:- self._video_folder_path = Path(get_mediapipe_annotated_videos_folder_path(self._session_id))+ self._video_folder_path = Path(+ get_mediapipe_annotated_videos_folder_path(self._session_id)+ )else:- self._video_folder_path = Path(get_synchronized_videos_folder_path(self._session_id))+ self._video_folder_path = Path(+ get_synchronized_videos_folder_path(self._session_id)+ )self._path_to_save_video_file = self._video_folder_path / video_file_name-self._cv2_video_writer = cv2.VideoWriter(str(self.path_to_save_video_file),cv2.VideoWriter_fourcc(*self._fourcc),frames_per_second,- (int(self._image_width), int(self._image_height)))+ (int(self._image_width), int(self._image_height)),+ )- def _write_frame_list_to_video_file(self, list_of_frames:List[FramePayload]):+ def _write_frame_list_to_video_file(self, list_of_frames: List[FramePayload]):try:for frame in list_of_frames:self._cv2_video_writer.write(frame.image)@@ -135,30 +154,34 @@ class VideoRecorder:logger.info(f"Saved video to path: {self.path_to_save_video_file}")self._cv2_video_writer.release()- def _write_image_list_to_video_file(self, image_list:List[np.ndarray]):+ def _write_image_list_to_video_file(self, image_list: List[np.ndarray]):try:for image in image_list:self._cv2_video_writer.write(image)except Exception as e:- logger.error(f"Failed during save in video writer: {self.path_to_save_video_file}")+ logger.error(+ f"Failed during save in video writer: {self.path_to_save_video_file}"+ )traceback.print_exc()raise efinally:logger.info(f"Saved video to path: {self.path_to_save_video_file}")self._cv2_video_writer.release()- def _gather_timestamps(self, list_of_frames:List[FramePayload])->np.ndarray:+ def _gather_timestamps(self, list_of_frames: List[FramePayload]) -> np.ndarray:timestamps_npy = np.empty(0)try:for frame in list_of_frames:- timestamps_npy = np.append(timestamps_npy, frame.timestamp_in_seconds_from_record_start)+ timestamps_npy = np.append(+ timestamps_npy, frame.timestamp_in_seconds_from_record_start+ )except:logger.error("Error gathering timestamps")return timestamps_npy- def _save_timestamps(self, timestamps_npy:np.ndarray):+ def _save_timestamps(self, timestamps_npy: np.ndarray):timestamp_file_name_npy = self._video_name + "_timestamps_binary.npy"timestamp_npy_full_save_path = self._video_folder_path / timestamp_file_name_npynp.save(str(timestamp_npy_full_save_path), timestamps_npy)diff --git a/src/cameras/save_synchronized_videos.py b/src/cameras/save_synchronized_videos.pyindex 8231699d..98af4a3d 100644--- a/src/cameras/save_synchronized_videos.py+++ b/src/cameras/save_synchronized_videos.py@@ -8,7 +8,11 @@ from src.cameras.persistence.video_writer.video_recorder import VideoRecorderfrom src.gui.main.state.app_state import APP_STATE-def save_synchronized_videos(list_of_video_recorders:List[VideoRecorder], calibration_videos:bool=False, mocap_videos:bool=False):+def save_synchronized_videos(+ list_of_video_recorders: List[VideoRecorder],+ calibration_videos: bool = False,+ mocap_videos: bool = False,+):each_cam_frame_list = []first_frame_timestamps = []final_frame_timestamps = []@@ -46,13 +50,19 @@ def save_synchronized_videos(list_of_video_recorders:List[VideoRecorder], calibrcontinueeach_cam_clipped_frame_list[-1].append(frame)- each_cam_clipped_timestamp_list[-1].append(frame.timestamp_unix_time_seconds)+ each_cam_clipped_timestamp_list[-1].append(+ frame.timestamp_unix_time_seconds+ )number_of_frames_per_camera_clipped = [len(f) for f in each_cam_clipped_frame_list]min_number_of_frames = np.min(number_of_frames_per_camera_clipped)- index_of_the_camera_with_fewest_frames = np.argmin(number_of_frames_per_camera_clipped)+ index_of_the_camera_with_fewest_frames = np.argmin(+ number_of_frames_per_camera_clipped+ )- reference_frame_list = each_cam_clipped_frame_list[index_of_the_camera_with_fewest_frames]+ reference_frame_list = each_cam_clipped_frame_list[+ index_of_the_camera_with_fewest_frames+ ]number_of_frames_per_camera = len(reference_frame_list)each_cam_synchronized_frame_list = []@@ -63,31 +73,41 @@ def save_synchronized_videos(list_of_video_recorders:List[VideoRecorder], calibrthis_cam_synchronized_frame_list.append(closest_frame)each_cam_synchronized_frame_list.append(this_cam_synchronized_frame_list)- print(f" (clipped) number_of_frames_per_camera: {number_of_frames_per_camera_clipped}, min:{min_number_of_frames}")+ print(+ f" (clipped) number_of_frames_per_camera: {number_of_frames_per_camera_clipped}, min:{min_number_of_frames}"+ )if mocap_videos:APP_STATE.number_of_frames_in_the_mocap_videos = number_of_frames_per_camera- final_frame_timestamps = [frame_list[-1].timestamp_unix_time_seconds for frame_list in each_cam_synchronized_frame_list]+ final_frame_timestamps = [+ frame_list[-1].timestamp_unix_time_seconds+ for frame_list in each_cam_synchronized_frame_list+ ]print(f"np.diff(final_frame_timestamps): {np.diff(final_frame_timestamps)}")- for video_recoder, frame_list in zip(list_of_video_recorders, each_cam_synchronized_frame_list):- video_recoder.save_list_of_frames_to_video_file(list_of_frames=frame_list, calibration_videos=calibration_videos)+ for video_recoder, frame_list in zip(+ list_of_video_recorders, each_cam_synchronized_frame_list+ ):+ video_recoder.save_list_of_frames_to_video_file(+ list_of_frames=frame_list, calibration_videos=calibration_videos+ )--def get_nearest_frame(frame_list, reference_frame)->FramePayload:+def get_nearest_frame(frame_list, reference_frame) -> FramePayload:timestamps = gather_timestamps(frame_list)- close_frame_index = np.argmin(np.abs(timestamps-reference_frame.timestamp_unix_time_seconds))+ close_frame_index = np.argmin(+ np.abs(timestamps - reference_frame.timestamp_unix_time_seconds)+ )return frame_list[close_frame_index]-def gather_timestamps( frame_list: List[FramePayload]) -> np.ndarray:+def gather_timestamps(frame_list: List[FramePayload]) -> np.ndarray:timestamps_npy = np.empty(0)for frame in frame_list:timestamps_npy = np.append(timestamps_npy, frame.timestamp_unix_time_seconds)- return timestamps_npy\ No newline at end of file+ return timestamps_npydiff --git a/src/cameras/viewer/cv_cam_viewer.py b/src/cameras/viewer/cv_cam_viewer.pyindex de48b016..c6c12647 100644--- a/src/cameras/viewer/cv_cam_viewer.py+++ b/src/cameras/viewer/cv_cam_viewer.py@@ -22,10 +22,7 @@ class CvCamViewer:cv2.waitKey(1)def begin_viewer(self, webcam_id: Union[str, int]):- self._process = Process(- target=CvCamViewer._begin,- args=(self._recv, webcam_id)- )+ self._process = Process(target=CvCamViewer._begin, args=(self._recv, webcam_id))self._process.start()diff --git a/src/config/home_dir.py b/src/config/home_dir.pyindex a0197ed3..c76c54c4 100644--- a/src/config/home_dir.py+++ b/src/config/home_dir.py@@ -11,16 +11,17 @@ MOST_RECENT_SESSION_ID_FILENAME = "most_recent_session_id.toml"def create_session_id(string_tag: str = None):- session_id = 'session_' + time.strftime("%m-%d-%Y-%H_%M_%S")+ session_id = "session_" + time.strftime("%m-%d-%Y-%H_%M_%S")if string_tag is not None:- session_id = session_id + '_' + string_tag+ session_id = session_id + "_" + string_tag++ return session_id- return session_iddef save_most_recent_session_id_to_a_toml_in_the_freemocap_data_folder(session_id: str):session_id_dict = {}- session_id_dict['session_id'] = session_id+ session_id_dict["session_id"] = session_idoutput_file_name = MOST_RECENT_SESSION_ID_FILENAMEoutput_file_path = Path(get_freemocap_data_folder_path()) / output_file_name@@ -28,10 +29,12 @@ def save_most_recent_session_id_to_a_toml_in_the_freemocap_data_folder(session_itoml.dump(session_id_dict, toml_file)-def get_most_recent_session_id()->str:- session_id_toml_path = Path(get_freemocap_data_folder_path()) / MOST_RECENT_SESSION_ID_FILENAME+def get_most_recent_session_id() -> str:+ session_id_toml_path = (+ Path(get_freemocap_data_folder_path()) / MOST_RECENT_SESSION_ID_FILENAME+ )session_id_dict = toml.load(str(session_id_toml_path))- return session_id_dict['session_id']+ return session_id_dict["session_id"]def os_independent_home_dir():@@ -52,42 +55,55 @@ def get_session_folder_path(session_id: str, create_folder: bool = False):return str(session_path)-def create_session_folder(session_id:str):+def create_session_folder(session_id: str):session_path = Path(get_session_folder_path(session_id, create_folder=True))- logger.info(f'Creating session folder at: {str(session_path)}')+ logger.info(f"Creating session folder at: {str(session_path)}")+def get_synchronized_videos_folder_path(session_id: str, create_folder: bool = True):- synchronized_videos_path = Path(get_session_folder_path(session_id)) / 'synchronized_videos'+ synchronized_videos_path = (+ Path(get_session_folder_path(session_id)) / "synchronized_videos"+ )if create_folder:synchronized_videos_path.mkdir(exist_ok=create_folder, parents=True)- #this now counts as a "proper" session, so save it to the toml+ # this now counts as a "proper" session, so save it to the tomlsave_most_recent_session_id_to_a_toml_in_the_freemocap_data_folder(session_id)return str(synchronized_videos_path)def get_calibration_videos_folder_path(session_id: str, create_folder: bool = True):- calibration_videos_path = Path(get_session_folder_path(session_id)) / 'calibration_videos'+ calibration_videos_path = (+ Path(get_session_folder_path(session_id)) / "calibration_videos"+ )if create_folder:calibration_videos_path.mkdir(exist_ok=create_folder, parents=True)return str(calibration_videos_path)-def get_mediapipe_annotated_videos_folder_path(session_id: str, create_folder: bool = True):- mediapipe_annotated_videos_path = Path(get_session_folder_path(session_id)) / 'mediapipe_annotated_videos'+def get_mediapipe_annotated_videos_folder_path(+ session_id: str, create_folder: bool = True+):+ mediapipe_annotated_videos_path = (+ Path(get_session_folder_path(session_id)) / "mediapipe_annotated_videos"+ )if create_folder:mediapipe_annotated_videos_path.mkdir(exist_ok=create_folder, parents=True)return str(mediapipe_annotated_videos_path)def get_session_output_data_folder_path(session_id: str, create_folder: bool = True):- output_data_folder_path = Path(get_session_folder_path(session_id)) / 'output_data_files'+ output_data_folder_path = (+ Path(get_session_folder_path(session_id)) / "output_data_files"+ )if create_folder:output_data_folder_path.mkdir(exist_ok=create_folder, parents=True)return str(output_data_folder_path)-def get_session_calibration_file_path(session_id:str)->str:++def get_session_calibration_file_path(session_id: str) -> str:calibration_file_name = f"{session_id}_camera_calibration.toml"- calibration_file_path = Path(get_session_folder_path(session_id))/calibration_file_name+ calibration_file_path = (+ Path(get_session_folder_path(session_id)) / calibration_file_name+ )return str(calibration_file_path)-diff --git a/src/config/webcam_config.py b/src/config/webcam_config.pyindex 2afb9cf6..c2558167 100644--- a/src/config/webcam_config.py+++ b/src/config/webcam_config.py@@ -16,7 +16,9 @@ class WebcamConfig(BaseModel):base_save_video_dir = os_independent_home_dir()-def webcam_config_to_qt_parameter_list(webcam_config:WebcamConfig = WebcamConfig())->List:+def webcam_config_to_qt_parameter_list(+ webcam_config: WebcamConfig = WebcamConfig(),+) -> List:"""take in a webcam_config and return a parameter_tree_group that you can feed into a pyqtgraph Parameter tree like this:@@ -29,10 +31,18 @@ def webcam_config_to_qt_parameter_list(webcam_config:WebcamConfig = WebcamConfig"""this_webcam_id = webcam_config.webcam_idparameter_list = [- {'name': 'use this cameraβ', 'type': 'bool', 'value': True},- {'name': 'exposure', 'type': 'int', 'value': webcam_config.exposure},- {'name': 'resolution_width', 'type': 'int', 'value': webcam_config.resolution_width},- {'name': 'resolution_height', 'type': 'int', 'value': webcam_config.resolution_height},+ {"name": "use this cameraβ", "type": "bool", "value": True},+ {"name": "exposure", "type": "int", "value": webcam_config.exposure},+ {+ "name": "resolution_width",+ "type": "int",+ "value": webcam_config.resolution_width,+ },+ {+ "name": "resolution_height",+ "type": "int",+ "value": webcam_config.resolution_height,+ },]- return parameter_list\ No newline at end of file+ return parameter_listdiff --git a/src/core_processor/camera_calibration/calibration_dataclasses.py b/src/core_processor/camera_calibration/calibration_dataclasses.pyindex 85a19ea8..c74da9af 100644--- a/src/core_processor/camera_calibration/calibration_dataclasses.py+++ b/src/core_processor/camera_calibration/calibration_dataclasses.py@@ -18,11 +18,15 @@ class CameraCalibrationData:def __post_init__(self):if self.camera_matrix is None:- self.camera_matrix = np.array([[float(self.image_width), 0., self.image_width / 2.],- [0., float(self.image_width), self.image_height / 2.],- [0., 0., 1.]])+ self.camera_matrix = np.array(+ [+ [float(self.image_width), 0.0, self.image_width / 2.0],+ [0.0, float(self.image_width), self.image_height / 2.0],+ [0.0, 0.0, 1.0],+ ]+ )if self.lens_distortion_coefficients is None:- self.lens_distortion_coefficients = np.zeros((self.number_of_lens_distortion_coefficients, 1))--+ self.lens_distortion_coefficients = np.zeros(+ (self.number_of_lens_distortion_coefficients, 1)+ )diff --git a/src/core_processor/camera_calibration/calibration_diagnostics_visualizer.py b/src/core_processor/camera_calibration/calibration_diagnostics_visualizer.pyindex 6a965497..f6b590d3 100644--- a/src/core_processor/camera_calibration/calibration_diagnostics_visualizer.py+++ b/src/core_processor/camera_calibration/calibration_diagnostics_visualizer.py@@ -7,8 +7,9 @@ from pyqtgraph.Qt import QtWidgetsimport cv2-from src.core_processor.camera_calibration.calibration_dataclasses import CameraCalibrationData-+from src.core_processor.camera_calibration.calibration_dataclasses import (+ CameraCalibrationData,+)class CalibrationDiagnosticsVisualizer:@@ -25,7 +26,9 @@ class CalibrationDiagnosticsVisualizer:self.dock_area = DockArea()self.main_window_widget.setCentralWidget(self.dock_area)self.main_window_widget.resize(1000, 500)- self.main_window_widget.setWindowTitle('Lens Distortion Calibration Diagnostics (pyqtgraph)')+ self.main_window_widget.setWindowTitle(+ "Lens Distortion Calibration Diagnostics (pyqtgraph)"+ )# create widget/dock for image subplot (based on `GraphicsItems - ImageItem - video` example `python -m pyqtgraph.examples`)self.image_view_widget = pg.ImageView()@@ -36,43 +39,61 @@ class CalibrationDiagnosticsVisualizer:# create widget/dock for reprojection error subplotself.reprojection_error_plot_widget = pg.PlotWidget(title="Reprojection Error")- self.reprojection_error_plot_widget.setLabel('left', "Reprojection Error", units='pixels')- self.reprojection_error_plot_widget.setLabel('bottom', "Time", units='Frame#')+ self.reprojection_error_plot_widget.setLabel(+ "left", "Reprojection Error", units="pixels"+ )+ self.reprojection_error_plot_widget.setLabel("bottom", "Time", units="Frame#")self.reprojection_error_plot_widget.setYRange(0, 2)self.reprojection_error_plot_widget.addLegend()- self.reprojection_error_all_line = self.reprojection_error_plot_widget.plot(self.reprojection_error_all,- pen=(255, 0, 128),- name="reprojection error- all")- self.reprojection_error_current_best_line = self.reprojection_error_plot_widget.plot(- self.reprojection_error_current_best,- name="current best")+ self.reprojection_error_all_line = self.reprojection_error_plot_widget.plot(+ self.reprojection_error_all,+ pen=(255, 0, 128),+ name="reprojection error- all",+ )+ self.reprojection_error_current_best_line = (+ self.reprojection_error_plot_widget.plot(+ self.reprojection_error_current_best, name="current best"+ )+ )self.reprojection_error_dock = Dock("Reprojection Error")self.reprojection_error_dock.addWidget(self.reprojection_error_plot_widget)- self.dock_area.addDock(self.reprojection_error_dock, position='right')+ self.dock_area.addDock(self.reprojection_error_dock, position="right")# create widget/dock lens distortion remapping subplot- self.image_point_remapping_plot_widget = pg.PlotWidget(title="Image Point Remapping")- self.image_point_remapping_plot_widget.setLabel('left', "Y-position", units='pixels')- self.image_point_remapping_plot_widget.setLabel('bottom', "X-position", units='pixels')+ self.image_point_remapping_plot_widget = pg.PlotWidget(+ title="Image Point Remapping"+ )+ self.image_point_remapping_plot_widget.setLabel(+ "left", "Y-position", units="pixels"+ )+ self.image_point_remapping_plot_widget.setLabel(+ "bottom", "X-position", units="pixels"+ )self.image_point_remapping_plot_widget.addLegend()- self.image_point_original_dottos = self.image_point_remapping_plot_widget.plot(self.image_point_original,- self.image_point_original,- pen=None,- symbol='o',- symbolBrush=None,- symbolPen=(150, 150, 150),- name="image points - original")- self.image_point_remapped_dottos = self.image_point_remapping_plot_widget.plot(self.image_point_remapped,- self.image_point_remapped,- symbol='+',- pen=None,- symbolBrush=None,- symbolPen=(50, 150, 210),- name="image points - remapped")+ self.image_point_original_dottos = self.image_point_remapping_plot_widget.plot(+ self.image_point_original,+ self.image_point_original,+ pen=None,+ symbol="o",+ symbolBrush=None,+ symbolPen=(150, 150, 150),+ name="image points - original",+ )+ self.image_point_remapped_dottos = self.image_point_remapping_plot_widget.plot(+ self.image_point_remapped,+ self.image_point_remapped,+ symbol="+",+ pen=None,+ symbolBrush=None,+ symbolPen=(50, 150, 210),+ name="image points - remapped",+ )self.image_point_remapping_plot_dock = Dock("Image Point Remapping")- self.image_point_remapping_plot_dock.addWidget(self.image_point_remapping_plot_widget)+ self.image_point_remapping_plot_dock.addWidget(+ self.image_point_remapping_plot_widget+ )self.dock_area.addDock(self.image_point_remapping_plot_dock)###########@@ -86,27 +107,28 @@ class CalibrationDiagnosticsVisualizer:y_axis_line_array = np.array([[0, 0, 0], [0, 1, 0]])z_axis_line_array = np.array([[0, 0, 0], [0, 0, 1]])- self.origin_x_axis_gl_lineplot_item = gl.GLLinePlotItem(pos=x_axis_line_array,- color=(1, 0, 0, 1),- width=1.,- antialias=True)- self.origin_y_axis_gl_lineplot_item = gl.GLLinePlotItem(pos=y_axis_line_array,- color=(0, 1, 0, 1),- width=1.,- antialias=True)- self.origin_z_axis_gl_lineplot_item = gl.GLLinePlotItem(pos=z_axis_line_array,- color=(0, 0, 1, 1),- width=1.,- antialias=True)+ self.origin_x_axis_gl_lineplot_item = gl.GLLinePlotItem(+ pos=x_axis_line_array, color=(1, 0, 0, 1), width=1.0, antialias=True+ )+ self.origin_y_axis_gl_lineplot_item = gl.GLLinePlotItem(+ pos=y_axis_line_array, color=(0, 1, 0, 1), width=1.0, antialias=True+ )+ self.origin_z_axis_gl_lineplot_item = gl.GLLinePlotItem(+ pos=z_axis_line_array, color=(0, 0, 1, 1), width=1.0, antialias=True+ )self.opengl_3d_plot_widget.addItem(self.origin_x_axis_gl_lineplot_item)self.opengl_3d_plot_widget.addItem(self.origin_y_axis_gl_lineplot_item)self.opengl_3d_plot_widget.addItem(self.origin_z_axis_gl_lineplot_item)- self.opengl_charuco_scatter_item = gl.GLScatterPlotItem(pos=(0, 0, 0), color=(1, 0, 1), size=1, pxMode=False)+ self.opengl_charuco_scatter_item = gl.GLScatterPlotItem(+ pos=(0, 0, 0), color=(1, 0, 1), size=1, pxMode=False+ )self.opengl_3d_plot_dock = Dock("Image Point Remapping")self.opengl_3d_plot_dock.addWidget(self.opengl_3d_plot_widget)- self.dock_area.addDock(self.opengl_3d_plot_dock, 'right', self.image_point_remapping_plot_dock)+ self.dock_area.addDock(+ self.opengl_3d_plot_dock, "right", self.image_point_remapping_plot_dock+ )#### create widget/dock for calibration text window@@ -114,7 +136,9 @@ class CalibrationDiagnosticsVisualizer:self.calibration_text_dock = Dock("Calibration Data")self.calibration_text_dock.addWidget(self.calibration_text_widget)- self.dock_area.addDock(self.calibration_text_dock, 'above', self.image_point_remapping_plot_dock)+ self.dock_area.addDock(+ self.calibration_text_dock, "above", self.image_point_remapping_plot_dock+ )self._display_main_window()@@ -124,28 +148,47 @@ class CalibrationDiagnosticsVisualizer:def update_image_subplot(self, new_image):self.image_view_widget.setImage(- cv2.cvtColor(cv2.rotate(new_image, cv2.ROTATE_90_COUNTERCLOCKWISE), cv2.COLOR_BGR2RGB))-- def update_reprojection_error_subplot(self, reprojection_error_new, reprojection_error_current_best):- self.reprojection_error_all = np.append(self.reprojection_error_all, reprojection_error_new)- self.reprojection_error_current_best = np.append(self.reprojection_error_current_best,- reprojection_error_current_best)+ cv2.cvtColor(+ cv2.rotate(new_image, cv2.ROTATE_90_COUNTERCLOCKWISE), cv2.COLOR_BGR2RGB+ )+ )++ def update_reprojection_error_subplot(+ self, reprojection_error_new, reprojection_error_current_best+ ):+ self.reprojection_error_all = np.append(+ self.reprojection_error_all, reprojection_error_new+ )+ self.reprojection_error_current_best = np.append(+ self.reprojection_error_current_best, reprojection_error_current_best+ )self.reprojection_error_all_line.setData(self.reprojection_error_all)- self.reprojection_error_current_best_line.setData(self.reprojection_error_current_best)+ self.reprojection_error_current_best_line.setData(+ self.reprojection_error_current_best+ )def update_calibration_text_overlay(self, calibration_data: CameraCalibrationData):- lens_distortion_as_str = [f"{d[0]:.3f}" for d in calibration_data.lens_distortion_coefficients]- calibration_text = f'reprojection error:{calibration_data.reprojection_error:.3f} \n {lens_distortion_as_str} \n {str(calibration_data.camera_matrix)}'+ lens_distortion_as_str = [+ f"{d[0]:.3f}" for d in calibration_data.lens_distortion_coefficients+ ]+ calibration_text = f"reprojection error:{calibration_data.reprojection_error:.3f} \n {lens_distortion_as_str} \n {str(calibration_data.camera_matrix)}"self.calibration_text_widget.setText(calibration_text)- def update_image_point_remapping_subplot(self, image_point_original_x, image_point_original_y,- image_point_remapped_x, image_point_remapped_y):- self.image_point_original_dottos.setData(x=image_point_original_x,- y=image_point_original_y)-- self.image_point_remapped_dottos.setData(x=image_point_remapped_x,- y=image_point_remapped_y)+ def update_image_point_remapping_subplot(+ self,+ image_point_original_x,+ image_point_original_y,+ image_point_remapped_x,+ image_point_remapped_y,+ ):+ self.image_point_original_dottos.setData(+ x=image_point_original_x, y=image_point_original_y+ )++ self.image_point_remapped_dottos.setData(+ x=image_point_remapped_x, y=image_point_remapped_y+ )def update_3d_subplot(self, charuco_xyz: np.ndarray = None):if charuco_xyz is None:diff --git a/src/core_processor/camera_calibration/camera_calibrator.py b/src/core_processor/camera_calibration/camera_calibrator.pyindex d3776dfb..ba39c5b6 100644--- a/src/core_processor/camera_calibration/camera_calibrator.py+++ b/src/core_processor/camera_calibration/camera_calibrator.py@@ -4,19 +4,27 @@ import cv2from src.cameras.capture.dataclasses.frame_payload import FramePayloadfrom src.cameras.capture.opencv_camera.opencv_camera import OpenCVCamera-from src.core_processor.camera_calibration.calibration_dataclasses import CameraCalibrationData-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import CharucoBoardDetector-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_image_annotator import \- annotate_image_with_charuco_data-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import CharucoViewData+from src.core_processor.camera_calibration.calibration_dataclasses import (+ CameraCalibrationData,+)+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (+ CharucoBoardDetector,+)+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_image_annotator import (+ annotate_image_with_charuco_data,+)+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import (+ CharucoViewData,+)logger = logging.getLogger(__name__)class CameraCalibrator:- def __init__(self,- charuco_board_detector: CharucoBoardDetector = CharucoBoardDetector(),- ):+ def __init__(+ self,+ charuco_board_detector: CharucoBoardDetector = CharucoBoardDetector(),+ ):self._current_calibration = Noneself._charuco_board_detector = charuco_board_detector@@ -31,13 +39,16 @@ class CameraCalibrator:def _process_incoming_frame(self, raw_frame_payload: FramePayload):- charuco_frame_payload = self._charuco_board_detector.detect_charuco_board(raw_frame_payload)+ charuco_frame_payload = self._charuco_board_detector.detect_charuco_board(+ raw_frame_payload+ )charuco_view_data = charuco_frame_payload.charuco_view_dataif self._current_calibration is None:- self._current_calibration = CameraCalibrationData(charuco_view_data.image_width,- charuco_view_data.image_height)+ self._current_calibration = CameraCalibrationData(+ charuco_view_data.image_width, charuco_view_data.image_height+ )if not charuco_view_data.full_board_found and self._current_calibration is None:return charuco_frame_payload.annotated_image, self._current_calibration@@ -49,13 +60,14 @@ class CameraCalibrator:annotate_image_with_charuco_data(raw_frame_payload.image, # this image will have stuff drawn on top of it inside this functioncharuco_view_data,- self._charuco_board_detector.number_of_charuco_corners+ self._charuco_board_detector.number_of_charuco_corners,)- undistorted_image = cv2.undistort(raw_frame_payload.image,- self._current_calibration.camera_matrix,- self._current_calibration.lens_distortion_coefficients,- )+ undistorted_image = cv2.undistort(+ raw_frame_payload.image,+ self._current_calibration.camera_matrix,+ self._current_calibration.lens_distortion_coefficients,+ )return undistorted_image, self._current_calibration@@ -78,26 +90,29 @@ class CameraCalibrator:image_height = charuco_view.image_height# flag definitions -> https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#ga3207604e4b1a1758aa66acb6ed5aa65d- flags = (cv2.CALIB_USE_INTRINSIC_GUESS +- cv2.CALIB_ZERO_TANGENT_DIST +- cv2.cv2.CALIB_FIX_ASPECT_RATIO +- cv2.CALIB_FIX_PRINCIPAL_POINT +- # cv2.CALIB_FIX_FOCAL_LENGTH- cv2.CALIB_RATIONAL_MODEL- # cv2.CALIB_THIN_PRISM_MODEL- # cv2.CALIB_TILTED_MODEL- )+ flags = (+ cv2.CALIB_USE_INTRINSIC_GUESS+ + cv2.CALIB_ZERO_TANGENT_DIST+ + cv2.cv2.CALIB_FIX_ASPECT_RATIO+ + cv2.CALIB_FIX_PRINCIPAL_POINT+ ++ # cv2.CALIB_FIX_FOCAL_LENGTH+ cv2.CALIB_RATIONAL_MODEL+ # cv2.CALIB_THIN_PRISM_MODEL+ # cv2.CALIB_TILTED_MODEL+ )# https://docs.opencv.org/4.x/d9/d6a/group__aruco.html#gacf03e5afb0bc516b73028cf209984a06- (this_reprojection_error,- this_camera_matrix,- these_lens_distortion_coefficients,- these_rotation_vectors_of_the_board,- these_translation_vectors_of_the_board,- this_lens_distortion_std_dev,- this_camera_location_std_dev,- these_reprojection_error_per_view- ) = cv2.aruco.calibrateCameraCharucoExtended(+ (+ this_reprojection_error,+ this_camera_matrix,+ these_lens_distortion_coefficients,+ these_rotation_vectors_of_the_board,+ these_translation_vectors_of_the_board,+ this_lens_distortion_std_dev,+ this_camera_location_std_dev,+ these_reprojection_error_per_view,+ ) = cv2.aruco.calibrateCameraCharucoExtended(charucoCorners=[charuco_view.charuco_corners],charucoIds=[charuco_view.charuco_ids],board=self._charuco_board_detector.cv2_aruco_charuco_board,@@ -105,7 +120,8 @@ class CameraCalibrator:cameraMatrix=self._current_calibration.camera_matrix,distCoeffs=self._current_calibration.lens_distortion_coefficients,flags=flags,- criteria=(cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT, 10000, 1e-9))+ criteria=(cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT, 10000, 1e-9),+ )if this_reprojection_error < self._current_calibration.reprojection_error:self._current_calibration = CameraCalibrationData(@@ -117,7 +133,8 @@ class CameraCalibrator:rotation_vectors_of_the_board=these_rotation_vectors_of_the_board,translation_vectors_of_the_board=these_translation_vectors_of_the_board,lens_distortion_std_dev=this_lens_distortion_std_dev,- camera_location_std_dev=this_camera_location_std_dev)+ camera_location_std_dev=this_camera_location_std_dev,+ )def reconstruct_3d_charuco(self, charuco_data_per_camera_dictionary):passdiff --git a/src/core_processor/camera_calibration/lens_distortion_calibrator.py b/src/core_processor/camera_calibration/lens_distortion_calibrator.pyindex ae0a45b9..d00815c9 100644--- a/src/core_processor/camera_calibration/lens_distortion_calibrator.py+++ b/src/core_processor/camera_calibration/lens_distortion_calibrator.py@@ -3,32 +3,44 @@ import numpy as npfrom rich import printfrom src.cameras.capture.dataclasses.frame_payload import FramePayload-from src.core_processor.camera_calibration.calibration_dataclasses import LensDistortionCalibrationData+from src.core_processor.camera_calibration.calibration_dataclasses import (+ LensDistortionCalibrationData,+)from src.pipelines.calibration_pipeline.charuco_board_detection import BoardDetector-from src.core_processor.camera_calibration.calibration_diagnostics_visualizer import CalibrationDiagnosticsVisualizer-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses import CharucoViewData-from src.pipelines.calibration_pipeline.charuco_board_detection import \- annotate_image_with_charuco_data+from src.core_processor.camera_calibration.calibration_diagnostics_visualizer import (+ CalibrationDiagnosticsVisualizer,+)+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses import (+ CharucoViewData,+)+from src.pipelines.calibration_pipeline.charuco_board_detection import (+ annotate_image_with_charuco_data,+)class LensDistortionCalibrator:- def __init__(self,- board_detector: BoardDetector = BoardDetector(),- show_calibration_diagnostics_visualizer=False, ):+ def __init__(+ self,+ board_detector: BoardDetector = BoardDetector(),+ show_calibration_diagnostics_visualizer=False,+ ):self._board_detector = board_detectorself._current_calibration = Noneself._all_charuco_views = []self._num_charuco_views = 0self._best_combo_of_charuco_views = []self._previous_best_calibrations = []- self.show_calibration_diagnostics_visualizer = show_calibration_diagnostics_visualizer+ self.show_calibration_diagnostics_visualizer = (+ show_calibration_diagnostics_visualizer+ )if self.show_calibration_diagnostics_visualizer:self.calibration_diagnostics_visualizer = CalibrationDiagnosticsVisualizer()- def process_incoming_frame(self,- raw_frame_payload: FramePayload):+ def process_incoming_frame(self, raw_frame_payload: FramePayload):- charuco_frame_payload = self._board_detector.detect_charuco_board(raw_frame_payload)+ charuco_frame_payload = self._board_detector.detect_charuco_board(+ raw_frame_payload+ )charuco_view_data = charuco_frame_payload.charuco_view_data@@ -42,29 +54,36 @@ class LensDistortionCalibrator:if self._num_charuco_views < 2:return charuco_frame_payload.annotated_image- self.estimate_lens_distortion(charuco_view_data) # this is where the magic happens+ self.estimate_lens_distortion(+ charuco_view_data+ ) # this is where the magic happensif self._current_calibration is None:- self._current_calibration = LensDistortionCalibrationData(charuco_view_data.image_width,- charuco_view_data.image_height)+ self._current_calibration = LensDistortionCalibrationData(+ charuco_view_data.image_width, charuco_view_data.image_height+ )return raw_frame_payload.image- undistorted_image_for_debug = self.undistort_image_with_invalid_pixels_as_black(raw_frame_payload.image,- self._current_calibration)+ undistorted_image_for_debug = self.undistort_image_with_invalid_pixels_as_black(+ raw_frame_payload.image, self._current_calibration+ )if charuco_view_data.some_charuco_corners_found:annotate_image_with_charuco_data(undistorted_image_for_debug, # this image will have stuff drawn on top of it inside this functioncharuco_view_data,- self._board_detector.number_of_charuco_corners+ self._board_detector.number_of_charuco_corners,)if self.show_calibration_diagnostics_visualizer:- self.calibration_diagnostics_visualizer.update_image_subplot(undistorted_image_for_debug)+ self.calibration_diagnostics_visualizer.update_image_subplot(+ undistorted_image_for_debug+ )- undistorted_image = cv2.undistort(raw_frame_payload.image,- self._current_calibration.camera_matrix,- self._current_calibration.lens_distortion_coefficients,- )+ undistorted_image = cv2.undistort(+ raw_frame_payload.image,+ self._current_calibration.camera_matrix,+ self._current_calibration.lens_distortion_coefficients,+ )return undistorted_image@@ -84,9 +103,13 @@ class LensDistortionCalibrator:print("CAMERA CALIBRATION")if self._current_calibration is None:- current_best_reprojection_error = 1e9 # put a big fake number on the first frame+ current_best_reprojection_error = (+ 1e9 # put a big fake number on the first frame+ )else:- current_best_reprojection_error = self._current_calibration.reprojection_error+ current_best_reprojection_error = (+ self._current_calibration.reprojection_error+ )image_width = new_charuco_view.image_widthimage_height = new_charuco_view.image_height@@ -97,34 +120,38 @@ class LensDistortionCalibrator:if self._num_charuco_views < num_charuco_views_per_combo:list_of_combos_of_charuco_views = [self._all_charuco_views]else:- list_of_combos_of_charuco_views = self.generate_combos_of_charuco_views(new_charuco_view,- num_combos_to_generate=10,- num_views_per_combo=num_charuco_views_per_combo,- )+ list_of_combos_of_charuco_views = self.generate_combos_of_charuco_views(+ new_charuco_view,+ num_combos_to_generate=10,+ num_views_per_combo=num_charuco_views_per_combo,+ )# flag definitions -> https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#ga3207604e4b1a1758aa66acb6ed5aa65d- flags = (cv2.CALIB_USE_INTRINSIC_GUESS +- cv2.CALIB_ZERO_TANGENT_DIST +- cv2.cv2.CALIB_FIX_ASPECT_RATIO +- cv2.CALIB_FIX_PRINCIPAL_POINT +- # cv2.CALIB_FIX_FOCAL_LENGTH- cv2.CALIB_RATIONAL_MODEL- # cv2.CALIB_THIN_PRISM_MODEL- # cv2.CALIB_TILTED_MODEL- )+ flags = (+ cv2.CALIB_USE_INTRINSIC_GUESS+ + cv2.CALIB_ZERO_TANGENT_DIST+ + cv2.cv2.CALIB_FIX_ASPECT_RATIO+ + cv2.CALIB_FIX_PRINCIPAL_POINT+ ++ # cv2.CALIB_FIX_FOCAL_LENGTH+ cv2.CALIB_RATIONAL_MODEL+ # cv2.CALIB_THIN_PRISM_MODEL+ # cv2.CALIB_TILTED_MODEL+ )for this_combo_of_charuco_views in list_of_combos_of_charuco_views:for this_charuco_view in this_combo_of_charuco_views:# https://docs.opencv.org/4.x/d9/d6a/group__aruco.html#gacf03e5afb0bc516b73028cf209984a06- (this_reprojection_error,- this_camera_matrix,- these_lens_distortion_coefficients,- these_rotation_vectors_of_the_board,- these_translation_vectors_of_the_board,- this_lens_distortion_std_dev,- this_camera_location_std_dev,- these_reprojection_error_per_view- ) = cv2.aruco.calibrateCameraCharucoExtended(+ (+ this_reprojection_error,+ this_camera_matrix,+ these_lens_distortion_coefficients,+ these_rotation_vectors_of_the_board,+ these_translation_vectors_of_the_board,+ this_lens_distortion_std_dev,+ this_camera_location_std_dev,+ these_reprojection_error_per_view,+ ) = cv2.aruco.calibrateCameraCharucoExtended(charucoCorners=[this_charuco_view.charuco_corners],charucoIds=[this_charuco_view.charuco_ids],board=self._board_detector.cv2_aruco_charuco_board,@@ -132,14 +159,22 @@ class LensDistortionCalibrator:cameraMatrix=default_calibration.camera_matrix,distCoeffs=default_calibration.lens_distortion_coefficients,flags=flags,- criteria=(cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT, 10000, 1e-9))+ criteria=(+ cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT,+ 10000,+ 1e-9,+ ),+ )if self.show_calibration_diagnostics_visualizer:- self.calibration_diagnostics_visualizer.update_reprojection_error_subplot(this_reprojection_error,- current_best_reprojection_error,- )+ self.calibration_diagnostics_visualizer.update_reprojection_error_subplot(+ this_reprojection_error,+ current_best_reprojection_error,+ )if self._current_calibration is not None:- self.calibration_diagnostics_visualizer.update_calibration_text_overlay(self._current_calibration)+ self.calibration_diagnostics_visualizer.update_calibration_text_overlay(+ self._current_calibration+ )if this_reprojection_error < current_best_reprojection_error:@@ -152,16 +187,18 @@ class LensDistortionCalibrator:rotation_vectors_of_the_board=these_rotation_vectors_of_the_board,translation_vectors_of_the_board=these_translation_vectors_of_the_board,lens_distortion_std_dev=this_lens_distortion_std_dev,- camera_location_std_dev=this_camera_location_std_dev)+ camera_location_std_dev=this_camera_location_std_dev,+ )if self.is_this_calibration_valid(this_calibration):self._current_calibration = this_calibration- self._previous_best_calibrations.append(self._current_calibration)+ self._previous_best_calibrations.append(+ self._current_calibration+ )- def generate_combos_of_charuco_views(self,- new_charuco_view,- num_combos_to_generate=10,- num_views_per_combo=5):+ def generate_combos_of_charuco_views(+ self, new_charuco_view, num_combos_to_generate=10, num_views_per_combo=5+ ):"""return -1x current best combo plus new view@@ -176,7 +213,9 @@ class LensDistortionCalibrator:# add previous best viewif self._best_combo_of_charuco_views:- list_of_combos_of_charuco_views.append(self._best_combo_of_charuco_views.copy())+ list_of_combos_of_charuco_views.append(+ self._best_combo_of_charuco_views.copy()+ )# add previous best view + new viewbest_combo_plus_new = self._best_combo_of_charuco_views.copy()@@ -188,12 +227,16 @@ class LensDistortionCalibrator:# and a some random combos - half with the new view and half without itfor this_iter in range(num_combos_to_generate):- this_random_sample_of_charuco_views = np.random.choice(num_views_per_combo,- size=min(len(self._all_charuco_views),- num_views_per_combo),- replace=False)+ this_random_sample_of_charuco_views = np.random.choice(+ num_views_per_combo,+ size=min(len(self._all_charuco_views), num_views_per_combo),+ replace=False,+ )- this_combo = [self._all_charuco_views[cc] for cc in this_random_sample_of_charuco_views]+ this_combo = [+ self._all_charuco_views[cc]+ for cc in this_random_sample_of_charuco_views+ ]if this_iter < np.ceil(num_views_per_combo / 2):this_combo.append(new_charuco_view)@@ -205,7 +248,9 @@ class LensDistortionCalibrator:def is_this_calibration_valid(self, this_calibration):is_valid = False- if not self.lens_distortion_is_monotonic(this_calibration): # not currently working - returns True every time+ if not self.lens_distortion_is_monotonic(+ this_calibration+ ): # not currently working - returns True every timereturn Falseif self.check_if_too_many_invalid_pixels(this_calibration):@@ -213,7 +258,9 @@ class LensDistortionCalibrator:return True- def lens_distortion_is_monotonic(self, this_calibration, assume_wide_angle_lens=True) -> bool:+ def lens_distortion_is_monotonic(+ self, this_calibration, assume_wide_angle_lens=True+ ) -> bool:"""check if the lens distortion coefficients are valid by checking it the resulting undistorted image is monotonically increasing along each diagonal (i.e. the image doesn't wrap around itself)@@ -240,9 +287,12 @@ class LensDistortionCalibrator:# following naming conventions of open cv docsk1 = this_calibration.lens_distortion_coefficients[0]k2 = this_calibration.lens_distortion_coefficients[1]- p1 = this_calibration.lens_distortion_coefficients[2] # I think p1 and p2 are for tangential distortion...+ p1 = this_calibration.lens_distortion_coefficients[+ 2+ ] # I think p1 and p2 are for tangential distortion...p2 = this_calibration.lens_distortion_coefficients[- 3] # ...which I might want to check on later? Currently fixed at Zero+ 3+ ] # ...which I might want to check on later? Currently fixed at Zerok3 = this_calibration.lens_distortion_coefficients[4]number_of_points = 40@@ -251,34 +301,57 @@ class LensDistortionCalibrator:original_points_xx, original_points_yy = np.meshgrid(original_x, original_y)- original_points_x = original_points_xx.flatten() # transform 2D grid with 1D array- original_points_y = original_points_yy.flatten() # transform 2D grid with 1D array+ original_points_x = (+ original_points_xx.flatten()+ ) # transform 2D grid with 1D array+ original_points_y = (+ original_points_yy.flatten()+ ) # transform 2D grid with 1D arrayoriginal_points_xy = np.vstack((original_points_x, original_points_y))- distorted_points_xy = cv2.undistortPoints(original_points_xy,- this_calibration.camera_matrix,- this_calibration.lens_distortion_coefficients)+ distorted_points_xy = cv2.undistortPoints(+ original_points_xy,+ this_calibration.camera_matrix,+ this_calibration.lens_distortion_coefficients,+ )- distorted_points_x = np.squeeze(distorted_points_xy[:, :, 0])*this_calibration.image_width+this_calibration.image_width/2- distorted_points_y = np.squeeze(distorted_points_xy[:, :, 1])*this_calibration.image_height+this_calibration.image_height/2+ distorted_points_x = (+ np.squeeze(distorted_points_xy[:, :, 0]) * this_calibration.image_width+ + this_calibration.image_width / 2+ )+ distorted_points_y = (+ np.squeeze(distorted_points_xy[:, :, 1]) * this_calibration.image_height+ + this_calibration.image_height / 2+ )if self.show_calibration_diagnostics_visualizer:- self.calibration_diagnostics_visualizer.update_image_point_remapping_subplot(original_points_x,- original_points_y,- distorted_points_x,- distorted_points_y)+ self.calibration_diagnostics_visualizer.update_image_point_remapping_subplot(+ original_points_x,+ original_points_y,+ distorted_points_x,+ distorted_points_y,+ )return True- def check_if_too_many_invalid_pixels(self, this_calibration, how_many_is_too_many=.1) -> bool:+ def check_if_too_many_invalid_pixels(+ self, this_calibration, how_many_is_too_many=0.1+ ) -> bool:"""create new camera matrix that will show full undistored image with black pixels in spots with no data.return FALSE if black pixels are greater than `how_many_is_too_many` proportion of the full image"""- dummy_image = np.ones((this_calibration.image_width, this_calibration.image_height), np.uint8) * 255+ dummy_image = (+ np.ones(+ (this_calibration.image_width, this_calibration.image_height), np.uint8+ )+ * 255+ )- undistorted_image = self.undistort_image_with_invalid_pixels_as_black(dummy_image, this_calibration)+ undistorted_image = self.undistort_image_with_invalid_pixels_as_black(+ dummy_image, this_calibration+ )num_invalid_pixels = np.sum(undistorted_image == 0)proportion_invalid_pixels = num_invalid_pixels / dummy_image.size@@ -294,15 +367,17 @@ class LensDistortionCalibrator:calibration.lens_distortion_coefficients,(calibration.image_width, calibration.image_height),1,- centerPrincipalPoint=True+ centerPrincipalPoint=True,)# https://docs.opencv.org/4.5.5/d9/d0c/group__calib3d.html#ga69f2545a8b62a6b0fc2ee060dc30559d- undistorted_image = cv2.undistort(image,- calibration.camera_matrix,- calibration.lens_distortion_coefficients,- None,- new_camera_matrix)+ undistorted_image = cv2.undistort(+ image,+ calibration.camera_matrix,+ calibration.lens_distortion_coefficients,+ None,+ new_camera_matrix,+ )return undistorted_imagediff --git a/src/core_processor/mediapipe_skeleton_detector/medaipipe_tracked_points_names_dict.py b/src/core_processor/mediapipe_skeleton_detector/medaipipe_tracked_points_names_dict.pyindex fef4281b..7e92fae0 100644--- a/src/core_processor/mediapipe_skeleton_detector/medaipipe_tracked_points_names_dict.py+++ b/src/core_processor/mediapipe_skeleton_detector/medaipipe_tracked_points_names_dict.py@@ -86,6 +86,6 @@ mediapipe_tracked_point_names_dict = {"nose_tip","mouth_center","right_ear_tragion",- "left_ear_tragion"- ]+ "left_ear_tragion",+ ],}diff --git a/src/core_processor/mediapipe_skeleton_detector/mediapipe_skeleton_detector.py b/src/core_processor/mediapipe_skeleton_detector/mediapipe_skeleton_detector.pyindex 5c3080dc..895634d1 100644--- a/src/core_processor/mediapipe_skeleton_detector/mediapipe_skeleton_detector.py+++ b/src/core_processor/mediapipe_skeleton_detector/mediapipe_skeleton_detector.py@@ -9,10 +9,14 @@ import mediapipe as mpfrom src.cameras.capture.dataclasses.frame_payload import FramePayloadfrom src.cameras.persistence.video_writer.video_recorder import VideoRecorder-from src.config.home_dir import get_session_folder_path, get_synchronized_videos_folder_path, \- get_session_output_data_folder_path-from src.core_processor.mediapipe_skeleton_detector.medaipipe_tracked_points_names_dict import \- mediapipe_tracked_point_names_dict+from src.config.home_dir import (+ get_session_folder_path,+ get_synchronized_videos_folder_path,+ get_session_output_data_folder_path,+)+from src.core_processor.mediapipe_skeleton_detector.medaipipe_tracked_points_names_dict import (+ mediapipe_tracked_point_names_dict,+)from src.gui.main.state.app_state import APP_STATElogger = logging.getLogger(__name__)@@ -39,21 +43,32 @@ class Mediapipe2dNumpyArrays:# if there's no body data, there's no hand or face data eitherreturn- if len(self.body2d_frameNumber_trackedPointNumber_XY.shape) == 3: #multiple frames- return np.hstack([self.body2d_frameNumber_trackedPointNumber_XY,- self.rightHand2d_frameNumber_trackedPointNumber_XY,- self.leftHand2d_frameNumber_trackedPointNumber_XY,- self.face2d_frameNumber_trackedPointNumber_XY])- elif len(self.body2d_frameNumber_trackedPointNumber_XY.shape) == 2: #single frame- return np.vstack([self.body2d_frameNumber_trackedPointNumber_XY,- self.rightHand2d_frameNumber_trackedPointNumber_XY,- self.leftHand2d_frameNumber_trackedPointNumber_XY,- self.face2d_frameNumber_trackedPointNumber_XY])+ if (+ len(self.body2d_frameNumber_trackedPointNumber_XY.shape) == 3+ ): # multiple frames+ return np.hstack(+ [+ self.body2d_frameNumber_trackedPointNumber_XY,+ self.rightHand2d_frameNumber_trackedPointNumber_XY,+ self.leftHand2d_frameNumber_trackedPointNumber_XY,+ self.face2d_frameNumber_trackedPointNumber_XY,+ ]+ )+ elif (+ len(self.body2d_frameNumber_trackedPointNumber_XY.shape) == 2+ ): # single frame+ return np.vstack(+ [+ self.body2d_frameNumber_trackedPointNumber_XY,+ self.rightHand2d_frameNumber_trackedPointNumber_XY,+ self.leftHand2d_frameNumber_trackedPointNumber_XY,+ self.face2d_frameNumber_trackedPointNumber_XY,+ ]+ )else:logger.error("data should have either 2 or 3 dimensions")-@dataclassclass Mediapipe2dDataPayload:raw_frame_payload: FramePayload = None@@ -66,8 +81,8 @@ class MediaPipeSkeletonDetector:def __init__(self, session_id: str = None):self._session_id = session_idself.model_complexity = 2 # can be 0,1, or 2 - higher numbers are more accurate but heavier computationally- self.min_detection_confidence = .5- self.min_tracking_confidence = .5+ self.min_detection_confidence = 0.5+ self.min_tracking_confidence = 0.5self._mediapipe_payload_list = []@@ -75,18 +90,30 @@ class MediaPipeSkeletonDetector:self._mp_drawing_styles = mp.solutions.drawing_stylesself._mp_holistic = mp.solutions.holistic- self._body_drawing_spec = self._mp_drawing.DrawingSpec(thickness=1, circle_radius=1)- self._hand_drawing_spec = self._mp_drawing.DrawingSpec(thickness=1, circle_radius=1)- self._face_drawing_spec = self._mp_drawing.DrawingSpec(thickness=1, circle_radius=1)-- self._holistic_tracker = self._mp_holistic.Holistic(model_complexity=self.model_complexity,- min_detection_confidence=self.min_detection_confidence,- min_tracking_confidence=self.min_tracking_confidence)+ self._body_drawing_spec = self._mp_drawing.DrawingSpec(+ thickness=1, circle_radius=1+ )+ self._hand_drawing_spec = self._mp_drawing.DrawingSpec(+ thickness=1, circle_radius=1+ )+ self._face_drawing_spec = self._mp_drawing.DrawingSpec(+ thickness=1, circle_radius=1+ )++ self._holistic_tracker = self._mp_holistic.Holistic(+ model_complexity=self.model_complexity,+ min_detection_confidence=self.min_detection_confidence,+ min_tracking_confidence=self.min_tracking_confidence,+ )self._mediapipe_tracked_point_names_dict = mediapipe_tracked_point_names_dictself.body_names_list = self._mediapipe_tracked_point_names_dict["body"]- self.right_hand_names_list = self._mediapipe_tracked_point_names_dict["right_hand"]- self.left_hand_names_list = self._mediapipe_tracked_point_names_dict["left_hand"]+ self.right_hand_names_list = self._mediapipe_tracked_point_names_dict[+ "right_hand"+ ]+ self.left_hand_names_list = self._mediapipe_tracked_point_names_dict[+ "left_hand"+ ]self.face_names_list = self._mediapipe_tracked_point_names_dict["face"]# TODO - build a better iterator and list of `face_marker_names` that will only pull out the face_counters & iris edges (mp.python.solutions.face_mesh_connections.FACEMESH_CONTOURS, FACE_MESH_IRISES)@@ -94,55 +121,72 @@ class MediaPipeSkeletonDetector:self.number_of_body_tracked_points = len(self.body_names_list)self.number_of_right_hand_tracked_points = len(self.right_hand_names_list)self.number_of_left_hand_tracked_points = len(self.left_hand_names_list)- self.number_of_face_tracked_points = mp.python.solutions.face_mesh.FACEMESH_NUM_LANDMARKS_WITH_IRISES-- self.number_of_tracked_points_total = self.number_of_body_tracked_points + \- self.number_of_left_hand_tracked_points + \- self.number_of_right_hand_tracked_points + \- self.number_of_face_tracked_points-- def detect_skeleton_in_image(self,- raw_image: np.ndarray = None,- raw_frame_payload: FramePayload = None,- annotated_image: np.ndarray = None) -> Mediapipe2dDataPayload:+ self.number_of_face_tracked_points = (+ mp.python.solutions.face_mesh.FACEMESH_NUM_LANDMARKS_WITH_IRISES+ )++ self.number_of_tracked_points_total = (+ self.number_of_body_tracked_points+ + self.number_of_left_hand_tracked_points+ + self.number_of_right_hand_tracked_points+ + self.number_of_face_tracked_points+ )++ def detect_skeleton_in_image(+ self,+ raw_image: np.ndarray = None,+ raw_frame_payload: FramePayload = None,+ annotated_image: np.ndarray = None,+ ) -> Mediapipe2dDataPayload:if raw_frame_payload is not None:raw_image = raw_frame_payload.image.copy()mediapipe_results = self._holistic_tracker.process(- raw_image) # <-this is where the magic happens, i.e. where the raw image is processed by a convolutional neural network to provide an estimate of joint position in pixel coordinates. Please don't forget that this is insane and should not be possible lol+ raw_image+ ) # <-this is where the magic happens, i.e. where the raw image is processed by a convolutional neural network to provide an estimate of joint position in pixel coordinates. Please don't forget that this is insane and should not be possible lolif annotated_image is None:annotated_image = raw_image.copy()annotated_image = self._annotate_image(annotated_image, mediapipe_results)- mediapipe_single_frame_npy_data = self._list_of_mediapipe_results_to_npy_arrays([mediapipe_results],- image_width=- annotated_image.shape[0],- image_height=- annotated_image.shape[1])+ mediapipe_single_frame_npy_data = self._list_of_mediapipe_results_to_npy_arrays(+ [mediapipe_results],+ image_width=annotated_image.shape[0],+ image_height=annotated_image.shape[1],+ )mediapipe_single_frame_npy_data.body2d_frameNumber_trackedPointNumber_XY = self._threshold_by_confidence(mediapipe_single_frame_npy_data.body2d_frameNumber_trackedPointNumber_XY,mediapipe_single_frame_npy_data.body2d_frameNumber_trackedPointNumber_confidence,- confidence_threshold=.5)-- return Mediapipe2dDataPayload(raw_frame_payload=raw_frame_payload,- mediapipe_results=mediapipe_results,- annotated_image=annotated_image,- pixel_data_numpy_arrays=mediapipe_single_frame_npy_data)-- def process_session_folder(self,- save_annotated_videos: bool = True):- synchronized_videos_path = Path(get_synchronized_videos_folder_path(self._session_id))+ confidence_threshold=0.5,+ )++ return Mediapipe2dDataPayload(+ raw_frame_payload=raw_frame_payload,+ mediapipe_results=mediapipe_results,+ annotated_image=annotated_image,+ pixel_data_numpy_arrays=mediapipe_single_frame_npy_data,+ )++ def process_session_folder(self, save_annotated_videos: bool = True):+ synchronized_videos_path = Path(+ get_synchronized_videos_folder_path(self._session_id)+ )logger.info(f"loading synchronized videos from: {synchronized_videos_path}")each_video_frame_width_list = []each_video_frame_height_list = []mediapipe2d_single_camera_npy_arrays_list = []- for video_number, this_synchronized_video_file_path in enumerate(synchronized_videos_path.glob('*.mp4')):- logger.info(f"Running `mediapipe` skeleton detection on video: {str(this_synchronized_video_file_path)}")- this_video_capture_object = cv2.VideoCapture(str(this_synchronized_video_file_path))+ for video_number, this_synchronized_video_file_path in enumerate(+ synchronized_videos_path.glob("*.mp4")+ ):+ logger.info(+ f"Running `mediapipe` skeleton detection on video: {str(this_synchronized_video_file_path)}"+ )+ this_video_capture_object = cv2.VideoCapture(+ str(this_synchronized_video_file_path)+ )this_video_width = this_video_capture_object.get(cv2.CAP_PROP_FRAME_WIDTH)this_video_height = this_video_capture_object.get(cv2.CAP_PROP_FRAME_HEIGHT)@@ -152,125 +196,169 @@ class MediaPipeSkeletonDetector:success, image = this_video_capture_object.read()if not success or image is None:- logger.error(f"Failed to load an image from: {str(this_synchronized_video_file_path)}")+ logger.error(+ f"Failed to load an image from: {str(this_synchronized_video_file_path)}"+ )raise Exceptionframe_number = 0while success and image is not None:- frame_number+=1- if frame_number%5==0:- print(f"frame {frame_number} out of {APP_STATE.number_of_frames_in_the_mocap_videos}")-- mediapipe2d_data_payload = self.detect_skeleton_in_image(raw_image=image)- this_video_mediapipe_results_list.append(mediapipe2d_data_payload.mediapipe_results)- annotated_image = self._annotate_image(image, mediapipe2d_data_payload.mediapipe_results)+ frame_number += 1+ if frame_number % 5 == 0:+ print(+ f"frame {frame_number} out of {APP_STATE.number_of_frames_in_the_mocap_videos}"+ )++ mediapipe2d_data_payload = self.detect_skeleton_in_image(+ raw_image=image+ )+ this_video_mediapipe_results_list.append(+ mediapipe2d_data_payload.mediapipe_results+ )+ annotated_image = self._annotate_image(+ image, mediapipe2d_data_payload.mediapipe_results+ )this_video_annotated_images_list.append(annotated_image)success, image = this_video_capture_object.read()if save_annotated_videos:- self.save_annotated_videos(this_video_annotated_images_list,- this_synchronized_video_file_path.stem,- this_video_width,- this_video_height,- )-- this_camera_mediapipe_2d_single_camera_npy_arrays = self._list_of_mediapipe_results_to_npy_arrays(- this_video_mediapipe_results_list,- image_width=this_video_width,- image_height=this_video_height)-- mediapipe2d_single_camera_npy_arrays_list.append(this_camera_mediapipe_2d_single_camera_npy_arrays)-- all_cameras_data2d_list = [m2d.all_data2d_nFrames_nTrackedPts_XY for m2d in- mediapipe2d_single_camera_npy_arrays_list]+ self.save_annotated_videos(+ this_video_annotated_images_list,+ this_synchronized_video_file_path.stem,+ this_video_width,+ this_video_height,+ )++ this_camera_mediapipe_2d_single_camera_npy_arrays = (+ self._list_of_mediapipe_results_to_npy_arrays(+ this_video_mediapipe_results_list,+ image_width=this_video_width,+ image_height=this_video_height,+ )+ )++ mediapipe2d_single_camera_npy_arrays_list.append(+ this_camera_mediapipe_2d_single_camera_npy_arrays+ )++ all_cameras_data2d_list = [+ m2d.all_data2d_nFrames_nTrackedPts_XY+ for m2d in mediapipe2d_single_camera_npy_arrays_list+ ]number_of_cameras = len(all_cameras_data2d_list)number_of_frames = all_cameras_data2d_list[0].shape[0]number_of_tracked_points = all_cameras_data2d_list[0].shape[1]- number_of_spatial_dimensions = all_cameras_data2d_list[0].shape[2] # XY, 2d data+ number_of_spatial_dimensions = all_cameras_data2d_list[0].shape[+ 2+ ] # XY, 2d dataif not number_of_spatial_dimensions == 2:- logger.error(f"this should be 2D data (XY pixel coordinates), but we founds {number_of_spatial_dimensions}")+ logger.error(+ f"this should be 2D data (XY pixel coordinates), but we founds {number_of_spatial_dimensions}"+ )raise Exception- data2d_numCams_numFrames_numTrackedPts_XY = np.empty((number_of_cameras,- number_of_frames,- number_of_tracked_points,- number_of_spatial_dimensions))+ data2d_numCams_numFrames_numTrackedPts_XY = np.empty(+ (+ number_of_cameras,+ number_of_frames,+ number_of_tracked_points,+ number_of_spatial_dimensions,+ )+ )for this_cam_num in range(number_of_cameras):- data2d_numCams_numFrames_numTrackedPts_XY[this_cam_num, :, :, :] = all_cameras_data2d_list[this_cam_num]+ data2d_numCams_numFrames_numTrackedPts_XY[+ this_cam_num, :, :, :+ ] = all_cameras_data2d_list[this_cam_num]self._save_mediapipe2d_data_to_npy(data2d_numCams_numFrames_numTrackedPts_XY)return data2d_numCams_numFrames_numTrackedPts_XYdef _save_mediapipe2d_data_to_npy(self, data2d_numCams_numFrames_numTrackedPts_XY):output_data_folder = Path(get_session_output_data_folder_path(self._session_id))- mediapipe_2dData_save_path = output_data_folder / "mediapipe_2dData_numCams_numFrames_numTrackedPoints_pixelXY.npy"+ mediapipe_2dData_save_path = (+ output_data_folder+ / "mediapipe_2dData_numCams_numFrames_numTrackedPoints_pixelXY.npy"+ )logger.info(f"saving: {mediapipe_2dData_save_path}")- np.save(str(mediapipe_2dData_save_path), data2d_numCams_numFrames_numTrackedPts_XY)-- def save_annotated_videos(self,- annotated_images_list: List[np.ndarray],- video_file_name: str,- image_width: Union[int, float],- image_height: Union[int, float],- ):+ np.save(+ str(mediapipe_2dData_save_path), data2d_numCams_numFrames_numTrackedPts_XY+ )++ def save_annotated_videos(+ self,+ annotated_images_list: List[np.ndarray],+ video_file_name: str,+ image_width: Union[int, float],+ image_height: Union[int, float],+ ):this_video_name = video_file_name + "_mediapipe.mp4"- video_recorder = VideoRecorder(this_video_name,- image_width,- image_height,- self._session_id,- mediapipe_annotated_video_bool=True)-- logger.info(f'Saving mediapipe annotated video: {this_video_name}')- video_recorder.save_image_list_to_disk(annotated_images_list, frames_per_second=30)- logger.info(f'mediapipe annotated video saved to: {video_recorder.path_to_save_video_file}')+ video_recorder = VideoRecorder(+ this_video_name,+ image_width,+ image_height,+ self._session_id,+ mediapipe_annotated_video_bool=True,+ )++ logger.info(f"Saving mediapipe annotated video: {this_video_name}")+ video_recorder.save_image_list_to_disk(+ annotated_images_list, frames_per_second=30+ )+ logger.info(+ f"mediapipe annotated video saved to: {video_recorder.path_to_save_video_file}"+ )def _annotate_image(self, image, mediapipe_results):- self._mp_drawing.draw_landmarks(image=image,- landmark_list=mediapipe_results.face_landmarks,- connections=self._mp_holistic.FACEMESH_CONTOURS,- landmark_drawing_spec=None,- connection_drawing_spec=self._mp_drawing_styles- .get_default_face_mesh_contours_style())- self._mp_drawing.draw_landmarks(image=image,- landmark_list=mediapipe_results.face_landmarks,- connections=self._mp_holistic.FACEMESH_TESSELATION,- landmark_drawing_spec=None,- connection_drawing_spec=self._mp_drawing_styles- .get_default_face_mesh_tesselation_style())- self._mp_drawing.draw_landmarks(image=image,- landmark_list=mediapipe_results.pose_landmarks,- connections=self._mp_holistic.POSE_CONNECTIONS,- landmark_drawing_spec=self._mp_drawing_styles- .get_default_pose_landmarks_style())+ self._mp_drawing.draw_landmarks(+ image=image,+ landmark_list=mediapipe_results.face_landmarks,+ connections=self._mp_holistic.FACEMESH_CONTOURS,+ landmark_drawing_spec=None,+ connection_drawing_spec=self._mp_drawing_styles.get_default_face_mesh_contours_style(),+ )+ self._mp_drawing.draw_landmarks(+ image=image,+ landmark_list=mediapipe_results.face_landmarks,+ connections=self._mp_holistic.FACEMESH_TESSELATION,+ landmark_drawing_spec=None,+ connection_drawing_spec=self._mp_drawing_styles.get_default_face_mesh_tesselation_style(),+ )+ self._mp_drawing.draw_landmarks(+ image=image,+ landmark_list=mediapipe_results.pose_landmarks,+ connections=self._mp_holistic.POSE_CONNECTIONS,+ landmark_drawing_spec=self._mp_drawing_styles.get_default_pose_landmarks_style(),+ )self._mp_drawing.draw_landmarks(image=image,landmark_list=mediapipe_results.left_hand_landmarks,connections=self._mp_holistic.HAND_CONNECTIONS,landmark_drawing_spec=None,- connection_drawing_spec=self._mp_drawing_styles- .get_default_hand_connections_style())+ connection_drawing_spec=self._mp_drawing_styles.get_default_hand_connections_style(),+ )self._mp_drawing.draw_landmarks(image=image,landmark_list=mediapipe_results.right_hand_landmarks,connections=self._mp_holistic.HAND_CONNECTIONS,landmark_drawing_spec=None,- connection_drawing_spec=self._mp_drawing_styles- .get_default_hand_connections_style())+ connection_drawing_spec=self._mp_drawing_styles.get_default_hand_connections_style(),+ )return image- def _list_of_mediapipe_results_to_npy_arrays(self,- mediapipe_results_list: List,- image_width: Union[int, float],- image_height: Union[int, float],- ) -> Mediapipe2dNumpyArrays:+ def _list_of_mediapipe_results_to_npy_arrays(+ self,+ mediapipe_results_list: List,+ image_width: Union[int, float],+ image_height: Union[int, float],+ ) -> Mediapipe2dNumpyArrays:# apparently `mediapipe_results.pose_landmarks.landmark` returns something iterable Β―\_(γ)_/Β―mediapipe_pose_landmark_iterator = mp.python.solutions.pose.PoseLandmark@@ -280,26 +368,46 @@ class MediaPipeSkeletonDetector:number_of_spatial_dimensions = 2 # this will be 2d XY pixel databody2d_frameNumber_trackedPointNumber_XY = np.zeros(- (number_of_frames, self.number_of_body_tracked_points, number_of_spatial_dimensions))+ (+ number_of_frames,+ self.number_of_body_tracked_points,+ number_of_spatial_dimensions,+ )+ )body2d_frameNumber_trackedPointNumber_XY[:] = np.nanbody2d_frameNumber_trackedPointNumber_confidence = np.zeros(- (number_of_frames, self.number_of_body_tracked_points))- body2d_frameNumber_trackedPointNumber_confidence[:] = np.nan # only body markers get a 'confidence' value-- rightHand2d_frameNumber_trackedPointNumber_XY = np.zeros((number_of_frames,- self.number_of_right_hand_tracked_points,- number_of_spatial_dimensions))+ (number_of_frames, self.number_of_body_tracked_points)+ )+ body2d_frameNumber_trackedPointNumber_confidence[+ :+ ] = np.nan # only body markers get a 'confidence' value++ rightHand2d_frameNumber_trackedPointNumber_XY = np.zeros(+ (+ number_of_frames,+ self.number_of_right_hand_tracked_points,+ number_of_spatial_dimensions,+ )+ )rightHand2d_frameNumber_trackedPointNumber_XY[:] = np.nan- leftHand2d_frameNumber_trackedPointNumber_XY = np.zeros((number_of_frames,- self.number_of_left_hand_tracked_points,- number_of_spatial_dimensions))+ leftHand2d_frameNumber_trackedPointNumber_XY = np.zeros(+ (+ number_of_frames,+ self.number_of_left_hand_tracked_points,+ number_of_spatial_dimensions,+ )+ )leftHand2d_frameNumber_trackedPointNumber_XY[:] = np.nan- face2d_frameNumber_trackedPointNumber_XY = np.zeros((number_of_frames,- self.number_of_face_tracked_points,- number_of_spatial_dimensions))+ face2d_frameNumber_trackedPointNumber_XY = np.zeros(+ (+ number_of_frames,+ self.number_of_face_tracked_points,+ number_of_spatial_dimensions,+ )+ )face2d_frameNumber_trackedPointNumber_XY[:] = np.nanall_body_tracked_points_visible_on_this_frame_bool_list = []@@ -313,83 +421,147 @@ class MediaPipeSkeletonDetector:# get the Body data (aka 'pose')if this_frame_results.pose_landmarks is not None:- for this_landmark_number, this_landmark_data in enumerate(this_frame_results.pose_landmarks.landmark):- body2d_frameNumber_trackedPointNumber_XY[this_frame_number,- this_landmark_number,- 0] = this_landmark_data.x * image_width- body2d_frameNumber_trackedPointNumber_XY[this_frame_number,- this_landmark_number,- 1] = this_landmark_data.y * image_height- body2d_frameNumber_trackedPointNumber_confidence[this_frame_number,- this_landmark_number] = this_landmark_data.visibility # mediapipe calls their 'confidence' score 'visibility'+ for this_landmark_number, this_landmark_data in enumerate(+ this_frame_results.pose_landmarks.landmark+ ):+ body2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, this_landmark_number, 0+ ] = (this_landmark_data.x * image_width)+ body2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, this_landmark_number, 1+ ] = (this_landmark_data.y * image_height)+ body2d_frameNumber_trackedPointNumber_confidence[+ this_frame_number, this_landmark_number+ ] = (+ this_landmark_data.visibility+ ) # mediapipe calls their 'confidence' score 'visibility'# get Right Hand dataif this_frame_results.right_hand_landmarks is not None:for this_landmark_number, this_landmark_data in enumerate(- this_frame_results.right_hand_landmarks.landmark):- rightHand2d_frameNumber_trackedPointNumber_XY[this_frame_number,- this_landmark_number,- 0] = this_landmark_data.x * image_width- rightHand2d_frameNumber_trackedPointNumber_XY[this_frame_number,- this_landmark_number,- 1] = this_landmark_data.y * image_height+ this_frame_results.right_hand_landmarks.landmark+ ):+ rightHand2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, this_landmark_number, 0+ ] = (this_landmark_data.x * image_width)+ rightHand2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, this_landmark_number, 1+ ] = (this_landmark_data.y * image_height)# get Left Hand dataif this_frame_results.left_hand_landmarks is not None:for this_landmark_number, this_landmark_data in enumerate(- this_frame_results.left_hand_landmarks.landmark):- leftHand2d_frameNumber_trackedPointNumber_XY[this_frame_number,- this_landmark_number,- 0] = this_landmark_data.x * image_width- leftHand2d_frameNumber_trackedPointNumber_XY[this_frame_number,- this_landmark_number,- 1] = this_landmark_data.y * image_height+ this_frame_results.left_hand_landmarks.landmark+ ):+ leftHand2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, this_landmark_number, 0+ ] = (this_landmark_data.x * image_width)+ leftHand2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, this_landmark_number, 1+ ] = (this_landmark_data.y * image_height)# get Face dataif this_frame_results.face_landmarks is not None:- for this_landmark_number, this_landmark_data in enumerate(this_frame_results.face_landmarks.landmark):- face2d_frameNumber_trackedPointNumber_XY[this_frame_number, this_landmark_number,- 0] = this_landmark_data.x * image_width- face2d_frameNumber_trackedPointNumber_XY[this_frame_number, this_landmark_number,- 1] = this_landmark_data.y * image_height+ for this_landmark_number, this_landmark_data in enumerate(+ this_frame_results.face_landmarks.landmark+ ):+ face2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, this_landmark_number, 0+ ] = (this_landmark_data.x * image_width)+ face2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, this_landmark_number, 1+ ] = (this_landmark_data.y * image_height)# check if all tracked points are visible on this frame- all_body_visible = all(sum(- np.isnan(body2d_frameNumber_trackedPointNumber_XY[this_frame_number, :, :])) == 0)- all_body_tracked_points_visible_on_this_frame_bool_list.append(all_body_visible)-- all_right_hand_visible = all(sum(- np.isnan(rightHand2d_frameNumber_trackedPointNumber_XY[this_frame_number, :, :])) == 0)- all_right_hand_points_visible_on_this_frame_bool_list.append(all_right_hand_visible)-- all_left_hand_visible = all(sum(- np.isnan(leftHand2d_frameNumber_trackedPointNumber_XY[this_frame_number, :, :])) == 0)- all_left_hand_points_visible_on_this_frame_bool_list.append(all_left_hand_visible)-- all_face_visible = all(sum(- np.isnan(face2d_frameNumber_trackedPointNumber_XY[this_frame_number, :, :])) == 0)+ all_body_visible = all(+ sum(+ np.isnan(+ body2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, :, :+ ]+ )+ )+ == 0+ )+ all_body_tracked_points_visible_on_this_frame_bool_list.append(+ all_body_visible+ )++ all_right_hand_visible = all(+ sum(+ np.isnan(+ rightHand2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, :, :+ ]+ )+ )+ == 0+ )+ all_right_hand_points_visible_on_this_frame_bool_list.append(+ all_right_hand_visible+ )++ all_left_hand_visible = all(+ sum(+ np.isnan(+ leftHand2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, :, :+ ]+ )+ )+ == 0+ )+ all_left_hand_points_visible_on_this_frame_bool_list.append(+ all_left_hand_visible+ )++ all_face_visible = all(+ sum(+ np.isnan(+ face2d_frameNumber_trackedPointNumber_XY[+ this_frame_number, :, :+ ]+ )+ )+ == 0+ )all_face_points_visible_on_this_frame_bool_list.append(all_face_visible)- all_points_visible = all([all_body_visible,- all_right_hand_visible,- all_left_hand_visible,- all_face_visible],- )+ all_points_visible = all(+ [+ all_body_visible,+ all_right_hand_visible,+ all_left_hand_visible,+ all_face_visible,+ ],+ )all_tracked_points_visible_on_this_frame_list.append(all_points_visible)return Mediapipe2dNumpyArrays(- body2d_frameNumber_trackedPointNumber_XY=np.squeeze(body2d_frameNumber_trackedPointNumber_XY),- rightHand2d_frameNumber_trackedPointNumber_XY=np.squeeze(rightHand2d_frameNumber_trackedPointNumber_XY),- leftHand2d_frameNumber_trackedPointNumber_XY=np.squeeze(leftHand2d_frameNumber_trackedPointNumber_XY),- face2d_frameNumber_trackedPointNumber_XY=np.squeeze(face2d_frameNumber_trackedPointNumber_XY),+ body2d_frameNumber_trackedPointNumber_XY=np.squeeze(+ body2d_frameNumber_trackedPointNumber_XY+ ),+ rightHand2d_frameNumber_trackedPointNumber_XY=np.squeeze(+ rightHand2d_frameNumber_trackedPointNumber_XY+ ),+ leftHand2d_frameNumber_trackedPointNumber_XY=np.squeeze(+ leftHand2d_frameNumber_trackedPointNumber_XY+ ),+ face2d_frameNumber_trackedPointNumber_XY=np.squeeze(+ face2d_frameNumber_trackedPointNumber_XY+ ),body2d_frameNumber_trackedPointNumber_confidence=np.squeeze(- body2d_frameNumber_trackedPointNumber_confidence))-- def _threshold_by_confidence(self,- data2d_trackedPoint_dim: np.ndarray,- data2d_trackedPoint_confidence: np.ndarray,- confidence_threshold: float):+ body2d_frameNumber_trackedPointNumber_confidence+ ),+ )++ def _threshold_by_confidence(+ self,+ data2d_trackedPoint_dim: np.ndarray,+ data2d_trackedPoint_confidence: np.ndarray,+ confidence_threshold: float,+ ):threshold_mask = data2d_trackedPoint_confidence < confidence_thresholddata2d_trackedPoint_dim[threshold_mask, :] = np.nandiff --git a/src/core_processor/show_cam_window.py b/src/core_processor/show_cam_window.pyindex 8a98c2ff..b89a97ec 100644--- a/src/core_processor/show_cam_window.py+++ b/src/core_processor/show_cam_window.py@@ -9,6 +9,9 @@ from src.core_processor.utils.image_fps_writer import write_fps_to_imagelogger = logging.getLogger(__name__)-def show_cam_window(webcam_id: str, image: np.array,):+def show_cam_window(+ webcam_id: str,+ image: np.array,+):cv2.imshow(webcam_id, image)return Truediff --git a/src/core_processor/timestamp_manager/timestamp_manager.py b/src/core_processor/timestamp_manager/timestamp_manager.pyindex 76760242..e05b6ae9 100644--- a/src/core_processor/timestamp_manager/timestamp_manager.py+++ b/src/core_processor/timestamp_manager/timestamp_manager.py@@ -14,17 +14,17 @@ logger = logging.getLogger(__name__)class TimestampLogger:- def __init__(self,- logger_name:str,- session_start_time_perf_counter_ns:int,- ):+ def __init__(+ self,+ logger_name: str,+ session_start_time_perf_counter_ns: int,+ ):self._logger_name = logger_nameself._session_start_time_perf_counter_ns = session_start_time_perf_counter_nsself._num_frames_processed = 0self._timestamps_perf_counter_ns = np.empty(0)self._timestamps_from_zero = np.empty(0)-@propertydef median_frames_per_second(self):if self._num_frames_processed <= 10:@@ -38,7 +38,9 @@ class TimestampLogger:@propertydef timestamps_in_seconds_from_session_start(self):"""timestamps in seconds counting up from zero (where zero is the start time of the recording session)"""- return (self._timestamps_perf_counter_ns - self._session_start_time_perf_counter_ns) / 1e9+ return (+ self._timestamps_perf_counter_ns - self._session_start_time_perf_counter_ns+ ) / 1e9@propertydef latest_timestamp_in_seconds_from_record_start(self):@@ -49,12 +51,13 @@ class TimestampLogger:except:logger.error(f"Failed to return latest timestamp")-- def log_new_timestamp_perf_counter_ns(self, timestamp:int):- self._timestamps_perf_counter_ns = np.append(self._timestamps_perf_counter_ns, timestamp)+ def log_new_timestamp_perf_counter_ns(self, timestamp: int):+ self._timestamps_perf_counter_ns = np.append(+ self._timestamps_perf_counter_ns, timestamp+ )self._num_frames_processed += 1- def log_new_timestamp_seconds_from_unspecified_zero(self, timestamp:float):+ def log_new_timestamp_seconds_from_unspecified_zero(self, timestamp: float):self._timestamps_from_zero = np.append(self._timestamps_from_zero, timestamp)self._num_frames_processed += 1@@ -62,74 +65,89 @@ class TimestampLogger:def timestamps_in_seconds_from_unspecified_zero(self):return self._timestamps_from_zero+class TimestampManager:- def __init__(self,- session_id:str,- webcam_ids: List[str],- session_start_time_unix_ns: int,- _session_start_time_perf_counter_ns:int):+ def __init__(+ self,+ session_id: str,+ webcam_ids: List[str],+ session_start_time_unix_ns: int,+ _session_start_time_perf_counter_ns: int,+ ):self._session_id = session_idself._session_start_time_unix_ns = session_start_time_unix_nsself._session_start_time_perf_counter_ns = _session_start_time_perf_counter_nsself._webcam_ids = webcam_ids- self._webcam_timestamp_loggers: Dict[str, TimestampLogger] = self._initialize_webcam_timestamp_loggers(webcam_ids)- self._main_loop_timestamp_logger = TimestampLogger(logger_name = 'main_loop',- session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns)-- self._multi_frame_timestamp_logger = TimestampLogger(logger_name = 'multi_frame',- session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns)-+ self._webcam_timestamp_loggers: Dict[+ str, TimestampLogger+ ] = self._initialize_webcam_timestamp_loggers(webcam_ids)+ self._main_loop_timestamp_logger = TimestampLogger(+ logger_name="main_loop",+ session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns,+ )+ self._multi_frame_timestamp_logger = TimestampLogger(+ logger_name="multi_frame",+ session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns,+ )- def timestamp_logger_for_webcam_id(self, webcam_id:str)->TimestampLogger:+ def timestamp_logger_for_webcam_id(self, webcam_id: str) -> TimestampLogger:return self._webcam_timestamp_loggers[webcam_id]-@propertydef multi_frame_timestamp_logger(self):return self._multi_frame_timestamp_logger---def _initialize_webcam_timestamp_loggers(self, webcam_ids: List[str]):dictionary_of_timestamp_logger_objects = {}for webcam_id in webcam_ids:- dictionary_of_timestamp_logger_objects[webcam_id] = TimestampLogger(logger_name = 'camera_'+webcam_id,- session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns)+ dictionary_of_timestamp_logger_objects[webcam_id] = TimestampLogger(+ logger_name="camera_" + webcam_id,+ session_start_time_perf_counter_ns=self._session_start_time_perf_counter_ns,+ )return dictionary_of_timestamp_logger_objects--def _get_each_frame_timestamp(self, this_multi_frame_dict) -> List:this_multi_frame_timestamps = []for this_cam_id, this_cam_frame in this_multi_frame_dict.items():- this_multi_frame_timestamps.append(this_cam_frame.timestamp_in_seconds_from_record_start)+ this_multi_frame_timestamps.append(+ this_cam_frame.timestamp_in_seconds_from_record_start+ )return this_multi_frame_timestamps- def _get_intra_frame_interval_in_seconds(self, this_multi_frame_timestamps_in_seconds) -> float:+ def _get_intra_frame_interval_in_seconds(+ self, this_multi_frame_timestamps_in_seconds+ ) -> float:timestamp_min = np.min(this_multi_frame_timestamps_in_seconds)timestamp_max = np.max(this_multi_frame_timestamps_in_seconds)- return (timestamp_max - timestamp_min)-- def verify_multi_frame_is_synchronized(self,- this_multi_frame_dict,) -> bool:-- this_multi_frame_timestamps = self._get_each_frame_timestamp(this_multi_frame_dict)- this_multiframe_timestamp_interval_in_seconds = self._get_intra_frame_interval_in_seconds(this_multi_frame_timestamps)+ return timestamp_max - timestamp_min+ def verify_multi_frame_is_synchronized(+ self,+ this_multi_frame_dict,+ ) -> bool:+ this_multi_frame_timestamps = self._get_each_frame_timestamp(+ this_multi_frame_dict+ )+ this_multiframe_timestamp_interval_in_seconds = (+ self._get_intra_frame_interval_in_seconds(this_multi_frame_timestamps)+ )- print(f"this_multiframe_timestamp_interval: {this_multiframe_timestamp_interval_in_seconds*1e3:.6f} milliseconds")+ print(+ f"this_multiframe_timestamp_interval: {this_multiframe_timestamp_interval_in_seconds*1e3:.6f} milliseconds"+ )return Truedef create_diagnostic_plots(self):"""plot some diagnostics to assess quality of camera sync"""- multi_frame_timestamp_list = self.multi_frame_timestamp_logger.timestamps_in_seconds_from_unspecified_zero+ multi_frame_timestamp_list = (+ self.multi_frame_timestamp_logger.timestamps_in_seconds_from_unspecified_zero+ )# multi_frame_interval_list = self.multi_frame_interval_listfig = plt.figure(figsize=(18, 10))@@ -175,7 +193,10 @@ class TimestampManager:ylabel="Probability",)- for this_camera_id, this_camera_timestamp_logger in self._webcam_timestamp_loggers.items():+ for (+ this_camera_id,+ this_camera_timestamp_logger,+ ) in self._webcam_timestamp_loggers.items():this_camera_timestamps = this_camera_timestamp_logger._timestamps_from_zeroax1.plot(this_camera_timestamps, label=f"Camera# {this_camera_id}")ax1.legend()@@ -186,11 +207,7 @@ class TimestampManager:alpha=0.5,)- ax4.plot(- multi_frame_timestamp_list,- color="darkslategrey",- label=f"MultiFrame"- )+ ax4.plot(multi_frame_timestamp_list, color="darkslategrey", label=f"MultiFrame")ax5.plot(np.diff(np.asarray(multi_frame_timestamp_list)),".",@@ -219,7 +236,10 @@ class TimestampManager:# )ax5.legend()- fig_save_path = Path(get_session_folder_path(self._session_id)) / "camera_timestamp_diagnostics.png"+ fig_save_path = (+ Path(get_session_folder_path(self._session_id))+ / "camera_timestamp_diagnostics.png"+ )plt.savefig(str(fig_save_path))logger.info(f"Saving diagnostic figure to - {str(fig_save_path)}")# plt.show()diff --git a/src/core_processor/utils/image_fps_writer.py b/src/core_processor/utils/image_fps_writer.pyindex 8e926a44..e60c83fe 100644--- a/src/core_processor/utils/image_fps_writer.py+++ b/src/core_processor/utils/image_fps_writer.py@@ -4,7 +4,7 @@ import cv2import numpy as np-def write_fps_to_image(image: np.array, fps_number: Union[int,float])->np.ndarray:+def write_fps_to_image(image: np.array, fps_number: Union[int, float]) -> np.ndarray:annotated_image = image.copy()cv2.putText(annotated_image,diff --git a/src/debug_plot_makers/simple_plot_3d_points.py b/src/debug_plot_makers/simple_plot_3d_points.pyindex 7ead59fd..006e9b3b 100644--- a/src/debug_plot_makers/simple_plot_3d_points.py+++ b/src/debug_plot_makers/simple_plot_3d_points.py@@ -8,8 +8,8 @@ def simple_plot_3d(data_pt_xyz):z = data_pt_xyz[:, 2]fig = plt.figure()- matplotlib.use('qtagg')- ax3d = fig.add_subplot(projection='3d')+ matplotlib.use("qtagg")+ ax3d = fig.add_subplot(projection="3d")ax3d.scatter(x, y, z)- plt.pause(.1)+ plt.pause(0.1)plt.show()diff --git a/src/experiments/cv_cam_multi_process.py b/src/experiments/cv_cam_multi_process.pyindex e7b1ce89..69e8a89b 100644--- a/src/experiments/cv_cam_multi_process.py+++ b/src/experiments/cv_cam_multi_process.py@@ -18,7 +18,6 @@ logger = logging.getLogger(__name__)class CvCamByProcess:-def __init__(self, config: WebcamConfig = WebcamConfig()):self._process: Process = Noneself._recv, self._send = multiprocessing.Pipe(duplex=False)@@ -39,10 +38,7 @@ class CvCamByProcess:def begin_frame_capture(self):if not self._is_capturing_frames:self._is_capturing_frames = True- self._process = Process(- target=self._start_frame_loop,- args=(self._send,)- )+ self._process = Process(target=self._start_frame_loop, args=(self._send,))self._process.start()def end_frame_capture(self):@@ -56,9 +52,7 @@ class CvCamByProcess:else:cap_backend = cv2.CAP_ANY- opencv_video_capture_object = cv2.VideoCapture(- config.webcam_id, cap_backend- )+ opencv_video_capture_object = cv2.VideoCapture(config.webcam_id, cap_backend)# self._apply_configuration()if not self._test_cap_connection(opencv_video_capture_object):raise Exception("Could not find camera")@@ -108,11 +102,4 @@ class CvCamByProcess:self._new_frame_ready = success- return FramePayload(success=success,- image=image,- timestamp=timestamp_ns)-----+ return FramePayload(success=success, image=image, timestamp=timestamp_ns)diff --git a/src/gui/examples/main.py b/src/gui/examples/main.pyindex 05de0631..1db19fe7 100644--- a/src/gui/examples/main.py+++ b/src/gui/examples/main.py@@ -1,17 +1,13 @@-import sys-from PyQt6.QtWidgets import (QWidget, QPushButton,- QHBoxLayout, QVBoxLayout, QApplication)+from PyQt6.QtWidgets import QWidget, QPushButton, QHBoxLayout, QVBoxLayout, QApplicationclass Example(QWidget):-def __init__(self):super().__init__()self.initUI()-def initUI(self):okButton = QPushButton("OK")@@ -29,7 +25,7 @@ class Example(QWidget):self.setLayout(vbox)self.setGeometry(300, 300, 350, 250)- self.setWindowTitle('Buttons')+ self.setWindowTitle("Buttons")self.show()@@ -40,5 +36,5 @@ def main():sys.exit(app.exec())-if __name__ == '__main__':- main()\ No newline at end of file+if __name__ == "__main__":+ main()diff --git a/src/gui/main/layout/main_window.py b/src/gui/main/layout/main_window.pyindex 33780a7e..332939ef 100644--- a/src/gui/main/layout/main_window.py+++ b/src/gui/main/layout/main_window.py@@ -1,16 +1,13 @@from PyQt6.QtCore import Qtfrom PyQt6.QtGui import QAction-from PyQt6.QtWidgets import QMainWindow, QMenuBar, QVBoxLayout, \- QWidget, QCheckBox+from PyQt6.QtWidgets import QMainWindow, QMenuBar, QVBoxLayout, QWidget, QCheckBoxfrom src.gui.main.app import get_qt_appfrom src.gui.main.qt_utils.clear_layout import clearLayoutfrom src.gui.main.state.app_state import APP_STATE-from src.gui.main.workflows.calibration_instructions import \- CalibrationInstructions+from src.gui.main.workflows.calibration_instructions import CalibrationInstructionsfrom src.gui.main.workflows.camera_configuration import CameraConfiguration-from src.gui.main.workflows.new_recording_session import \- NewRecordingSession+from src.gui.main.workflows.new_recording_session import NewRecordingSessionfrom src.gui.main.workflows.record_videos import RecordVideosfrom src.gui.main.workflows.show_cams_charuco import ShowCamsCharucofrom src.gui.main.workflows.welcome import Welcome@@ -19,7 +16,7 @@ from src.gui.main.workflows.welcome import Welcomeclass MainWindow(QMainWindow):def __init__(self):super().__init__()- self.setWindowTitle('freemocap')+ self.setWindowTitle("freemocap")self.setGeometry(0, 0, 800, 600)self._create_menu_bar()self.statusBar()@@ -27,7 +24,6 @@ class MainWindow(QMainWindow):self._show_welcome_screen()-def _create_basic_layout(self):main_layout = QVBoxLayout()widget = QWidget()@@ -43,15 +39,15 @@ class MainWindow(QMainWindow):def _add_file_menu(self, menubar: QMenuBar):exit_action = self._create_exit_action()record_action = self._create_new_recording_session_action()- file_menu = menubar.addMenu('&File')+ file_menu = menubar.addMenu("&File")file_menu.addAction(record_action)file_menu.addAction(exit_action)def _create_exit_action(self):# src/freemocap_qt_gui/conference/assets/new_recording_session.png- action = QAction('&Quit', self)- action.setShortcut('Ctrl+Q')- action.setStatusTip('Exit application')+ action = QAction("&Quit", self)+ action.setShortcut("Ctrl+Q")+ action.setStatusTip("Exit application")# On triggered, quit the applicationaction.triggered.connect(get_qt_app().quit)@@ -59,9 +55,9 @@ class MainWindow(QMainWindow):return actiondef _create_new_recording_session_action(self):- action = QAction('&New Recording Session', self)- action.setShortcut('Ctrl+N')- action.setStatusTip('Begin a new recording session')+ action = QAction("&New Recording Session", self)+ action.setShortcut("Ctrl+N")+ action.setStatusTip("Begin a new recording session")# On triggered, quit the applicationaction.triggered.connect(self._show_recording_session_screen)@@ -94,7 +90,6 @@ class MainWindow(QMainWindow):else:screen.config_accepted.clicked.connect(self._show_calibration_screen)-def _show_calibration_instructions_screen(self):clearLayout(self._main_layout)screen = CalibrationInstructions()@@ -114,4 +109,3 @@ class MainWindow(QMainWindow):screen = RecordVideos()self._main_layout.addWidget(screen)self._main_layout.setAlignment(Qt.AlignmentFlag.AlignTop)-diff --git a/src/gui/main/shared_widgets/primary_button.py b/src/gui/main/shared_widgets/primary_button.pyindex 29ab6864..ad60b23f 100644--- a/src/gui/main/shared_widgets/primary_button.py+++ b/src/gui/main/shared_widgets/primary_button.py@@ -1,7 +1,6 @@from PyQt6.QtWidgets import QPushButton-active_primary_button_style_sheet = """QPushButton {background-color: #336e6d;@@ -32,8 +31,8 @@ not_active_primary_button_style_sheet = """}"""+class PrimaryButton(QPushButton):def __init__(self, *args, **kwargs):super().__init__(*args, **kwargs)self.setStyleSheet(active_primary_button_style_sheet)-diff --git a/src/gui/main/state/app_state.py b/src/gui/main/state/app_state.pyindex afdcf39f..2c057e94 100644--- a/src/gui/main/state/app_state.py+++ b/src/gui/main/state/app_state.py@@ -6,7 +6,7 @@ from pydantic import BaseModelclass AppState(BaseModel):session_id: str = Noneselected_cameras: List[str] = []- use_previous_calibration:bool = False+ use_previous_calibration: bool = Falsenumber_of_frames_in_the_mocap_videos: int = Nonediff --git a/src/gui/main/workers/cam_charuco_frame_worker.py b/src/gui/main/workers/cam_charuco_frame_worker.pyindex 70cf908e..da0dc0b3 100644--- a/src/gui/main/workers/cam_charuco_frame_worker.py+++ b/src/gui/main/workers/cam_charuco_frame_worker.py@@ -5,14 +5,17 @@ from PyQt6.QtGui import QImagefrom src.cameras.capture.opencv_camera.opencv_camera import OpenCVCamerafrom src.config.webcam_config import WebcamConfigfrom src.gui.main.state.app_state import APP_STATE-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import \- CharucoBoardDetector+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (+ CharucoBoardDetector,+)class CamCharucoFrameWorker(QThread):ImageUpdate = pyqtSignal(QImage)- def __init__(self, cam_id=None, should_save_frames:bool=False, calibration_videos=False):+ def __init__(+ self, cam_id=None, should_save_frames: bool = False, calibration_videos=False+ ):super().__init__()self._board_thingy = CharucoBoardDetector()self._cam_id = cam_id@@ -33,9 +36,7 @@ class CamCharucoFrameWorker(QThread):def run(self):cam = OpenCVCamera(- WebcamConfig(- webcam_id=self._cam_id- ),+ WebcamConfig(webcam_id=self._cam_id),session_id=APP_STATE.session_id,)cam.connect()@@ -62,12 +63,16 @@ class CamCharucoFrameWorker(QThread):image_to_display.data,image_to_display.shape[1],image_to_display.shape[0],- QImage.Format.Format_RGB888+ QImage.Format.Format_RGB888,+ )+ converted_frame = converted_frame.scaled(+ 640, 480, Qt.AspectRatioMode.KeepAspectRatio)- converted_frame = converted_frame.scaled(640, 480, Qt.AspectRatioMode.KeepAspectRatio)self.ImageUpdate.emit(converted_frame)finally:- print(f"Closing the camera {self._cam.webcam_id_as_str}, and saving video to disk")+ print(+ f"Closing the camera {self._cam.webcam_id_as_str}, and saving video to disk"+ )self._cam.close()# if any_frames_recorded:# print(f"saving video for camera {self._cam.webcam_id_as_str}")diff --git a/src/gui/main/workers/cam_frame_worker.py b/src/gui/main/workers/cam_frame_worker.pyindex f1b8e5bb..60ec9857 100644--- a/src/gui/main/workers/cam_frame_worker.py+++ b/src/gui/main/workers/cam_frame_worker.py@@ -14,11 +14,7 @@ class CamFrameWorker(QThread):self._cam_id = cam_iddef run(self):- cam = OpenCVCamera(- WebcamConfig(- webcam_id=self._cam_id- )- )+ cam = OpenCVCamera(WebcamConfig(webcam_id=self._cam_id))cam.connect()cam.start_frame_capture_thread()self._cam = cam@@ -34,9 +30,11 @@ class CamFrameWorker(QThread):image.data,image.shape[1],image.shape[0],- QImage.Format.Format_RGB888+ QImage.Format.Format_RGB888,+ )+ converted_frame = converted_frame.scaled(+ 640, 480, Qt.AspectRatioMode.KeepAspectRatio)- converted_frame = converted_frame.scaled(640, 480, Qt.AspectRatioMode.KeepAspectRatio)self.ImageUpdate.emit(converted_frame)finally:print("Closing the camera")diff --git a/src/gui/main/workflows/available_camera.py b/src/gui/main/workflows/available_camera.pyindex e275a4df..ce8ba19c 100644--- a/src/gui/main/workflows/available_camera.py+++ b/src/gui/main/workflows/available_camera.py@@ -2,7 +2,6 @@ from PyQt6.QtWidgets import QCheckBox, QHBoxLayout, QLabel, QPushButton, QWidgetclass AvailableCamera(QWidget):-def __init__(self, webcam_id: str = "0"):super().__init__()self._webcam_id = webcam_iddiff --git a/src/gui/main/workflows/available_cameras_list.py b/src/gui/main/workflows/available_cameras_list.pyindex 0bf2f012..e4350ea6 100644--- a/src/gui/main/workflows/available_cameras_list.py+++ b/src/gui/main/workflows/available_cameras_list.py@@ -37,7 +37,7 @@ class AvailableCamerasList(QWidget):self._enable_accept_button_callback = None- #automatically detect cameras on page load+ # automatically detect cameras on page loadself._detect()@property@@ -70,8 +70,6 @@ class AvailableCamerasList(QWidget):returnself._worker.start()--def _create_available_camera_widgets(self, detected: FoundCamerasResponse):camera_widgets = []for cam in detected.cameras_found_list:@@ -95,4 +93,3 @@ class AvailableCamerasList(QWidget):def _handle_camera_preview_click(self, cam_id):self.PreviewClicked.emit(cam_id)-diff --git a/src/gui/main/workflows/calibration_instructions.py b/src/gui/main/workflows/calibration_instructions.pyindex 3597e1e3..17000c5c 100644--- a/src/gui/main/workflows/calibration_instructions.py+++ b/src/gui/main/workflows/calibration_instructions.py@@ -5,7 +5,6 @@ from src.gui.main.shared_widgets.primary_button import PrimaryButtonclass CalibrationInstructions(QWidget):-def __init__(self):super().__init__()self._continue_button = PrimaryButton("Continue")@@ -29,8 +28,12 @@ class CalibrationInstructions(QWidget):def _create_description_layout(self):description_layout = QVBoxLayout()- description_1 = QLabel("Hold up the Charuko boards so all cameras can see every square. Don't have one? Get one here ")- description_2 = QLabel("Note: You may need to adjust the cameras if you are having an issue")+ description_1 = QLabel(+ "Hold up the Charuko boards so all cameras can see every square. Don't have one? Get one here "+ )+ description_2 = QLabel(+ "Note: You may need to adjust the cameras if you are having an issue"+ )description_layout.addWidget(description_1)description_layout.addWidget(description_2)return description_layoutdiff --git a/src/gui/main/workflows/camera_configuration.py b/src/gui/main/workflows/camera_configuration.pyindex deb0da83..0fa4bb44 100644--- a/src/gui/main/workflows/camera_configuration.py+++ b/src/gui/main/workflows/camera_configuration.py@@ -4,21 +4,22 @@ from PyQt6.QtGui import QPixmapfrom PyQt6.QtWidgets import QHBoxLayout, QLabel, QPushButton, QVBoxLayout, QWidgetfrom src.gui.main.shared_widgets.page_title import PageTitle-from src.gui.main.shared_widgets.primary_button import PrimaryButton, \- not_active_primary_button_style_sheet, active_primary_button_style_sheet+from src.gui.main.shared_widgets.primary_button import (+ PrimaryButton,+ not_active_primary_button_style_sheet,+ active_primary_button_style_sheet,+)from src.gui.main.state.app_state import APP_STATEfrom src.gui.main.workers.cam_frame_worker import CamFrameWorkerfrom src.gui.main.workflows.available_cameras_list import AvailableCamerasListclass CameraConfiguration(QWidget):-def __init__(self):super().__init__()self._worker = self._init_frame_worker()self._accept_button = self._create_accept_button()-container = QVBoxLayout()# Holds the Camera Configuration Title@@ -79,7 +80,7 @@ class CameraConfiguration(QWidget):if self._worker.isRunning():self._worker.quit()while not self._worker.isFinished():- time.sleep(.1)+ time.sleep(0.1)self._worker._cam_id = cam_idself._worker.start()diff --git a/src/gui/main/workflows/new_recording_session.py b/src/gui/main/workflows/new_recording_session.pyindex 1f681848..a6f09527 100644--- a/src/gui/main/workflows/new_recording_session.py+++ b/src/gui/main/workflows/new_recording_session.py@@ -1,5 +1,12 @@-from PyQt6.QtWidgets import QHBoxLayout, QLabel, QLineEdit, QPushButton, QVBoxLayout, \- QWidget, QCheckBox+from PyQt6.QtWidgets import (+ QHBoxLayout,+ QLabel,+ QLineEdit,+ QPushButton,+ QVBoxLayout,+ QWidget,+ QCheckBox,+)from src.config.home_dir import create_session_idfrom src.gui.main.shared_widgets.page_title import PageTitle@@ -8,12 +15,13 @@ from src.gui.main.state.app_state import APP_STATEclass NewRecordingSession(QWidget):-def __init__(self):super().__init__()self._submit_button = PrimaryButton("&Start Session")- self._use_previous_calibration_checkbox = self._create_use_previous_calibration_checkbox()+ self._use_previous_calibration_checkbox = (+ self._create_use_previous_calibration_checkbox()+ )container = QVBoxLayout()@@ -40,7 +48,9 @@ class NewRecordingSession(QWidget):def _create_use_previous_calibration_checkbox(self):previous_calibration_checkbox = QCheckBox("Use Previous Calibration")previous_calibration_checkbox.setChecked(False)- previous_calibration_checkbox.stateChanged.connect(self._use_previous_calibration_changed)+ previous_calibration_checkbox.stateChanged.connect(+ self._use_previous_calibration_changed+ )return previous_calibration_checkboxdef _create_record_sesion_title(self):@@ -62,5 +72,6 @@ class NewRecordingSession(QWidget):APP_STATE.session_id = self._session_input.text()def _use_previous_calibration_changed(self):- APP_STATE.use_previous_calibration = self._use_previous_calibration_checkbox.isChecked()-+ APP_STATE.use_previous_calibration = (+ self._use_previous_calibration_checkbox.isChecked()+ )diff --git a/src/gui/main/workflows/record_instructions.py b/src/gui/main/workflows/record_instructions.pyindex 27ee492b..80d3b77f 100644--- a/src/gui/main/workflows/record_instructions.py+++ b/src/gui/main/workflows/record_instructions.py@@ -4,7 +4,6 @@ from src.gui.main.shared_widgets.primary_button import PrimaryButtonclass RecordInstructions(QWidget):-def __init__(self):super().__init__()self._record_button = PrimaryButton("Record")@@ -28,7 +27,9 @@ class RecordInstructions(QWidget):def _create_description_layout(self):description_layout = QVBoxLayout()- description_1 = QLabel("Cameras are configured & calibrated. Hit the record button when you're ready! ")+ description_1 = QLabel(+ "Cameras are configured & calibrated. Hit the record button when you're ready! "+ )description_layout.addWidget(description_1)return description_layoutdiff --git a/src/gui/main/workflows/record_videos.py b/src/gui/main/workflows/record_videos.pyindex 5cd98f84..ec9e2464 100644--- a/src/gui/main/workflows/record_videos.py+++ b/src/gui/main/workflows/record_videos.py@@ -1,20 +1,29 @@-from PyQt6.QtWidgets import QCheckBox, QHBoxLayout, QLabel, QPushButton, QVBoxLayout, QWidget+from PyQt6.QtWidgets import (+ QCheckBox,+ QHBoxLayout,+ QLabel,+ QPushButton,+ QVBoxLayout,+ QWidget,+)from src.cameras.save_synchronized_videos import save_synchronized_videos-from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import \- MediaPipeSkeletonDetector+from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import (+ MediaPipeSkeletonDetector,+)from src.gui.main.state.app_state import APP_STATEfrom src.gui.main.workflows.single_camera import SingleCamerafrom src.open_in_blender.open_session_in_blender import open_session_in_blender-from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import \- CalibrationPipelineOrchestrator-from src.pipelines.session_pipeline.session_pipeline_orchestrator import \- SessionPipelineOrchestrator, \- load_mediapipe2d_data+from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import (+ CalibrationPipelineOrchestrator,+)+from src.pipelines.session_pipeline.session_pipeline_orchestrator import (+ SessionPipelineOrchestrator,+ load_mediapipe2d_data,+)class RecordVideos(QWidget):-def __init__(self):super().__init__()# TODO: Take it in from init@@ -39,8 +48,8 @@ class RecordVideos(QWidget):# start/stop recording button layoutrecord_button_layout = QHBoxLayout()- self._start_recording_button = QPushButton('Begin Recording')- self._stop_recording_button = QPushButton('Stop Recording')+ self._start_recording_button = QPushButton("Begin Recording")+ self._stop_recording_button = QPushButton("Stop Recording")self._stop_recording_button.setEnabled(False)self._start_recording_button.clicked.connect(self._start_recording_frames)@@ -55,7 +64,9 @@ class RecordVideos(QWidget):self._detect_2d_skeletons_button = QPushButton("Detect 2d Skeletons in Videos")self._reconstruct_3d_skeletons_button = QPushButton("Reconstruct 3d Skeletons")- self._visualize_freemocap_session_button = QPushButton("Visualize Freemocap Session")+ self._visualize_freemocap_session_button = QPushButton(+ "Visualize Freemocap Session"+ )self._open_in_blender_button = QPushButton("Open Session in Blender")self._detect_2d_skeletons_button.setEnabled(False)@@ -63,9 +74,15 @@ class RecordVideos(QWidget):self._visualize_freemocap_session_button.setEnabled(False)self._detect_2d_skeletons_button.clicked.connect(self._detect_2d_skeletons)- self._reconstruct_3d_skeletons_button.clicked.connect(self._reconstruct_3d_skeletons)- self._visualize_freemocap_session_button.clicked.connect(self._visualize_freemocap_session)- self._open_in_blender_button.clicked.connect(self._create_blender_scene_from_session_data)+ self._reconstruct_3d_skeletons_button.clicked.connect(+ self._reconstruct_3d_skeletons+ )+ self._visualize_freemocap_session_button.clicked.connect(+ self._visualize_freemocap_session+ )+ self._open_in_blender_button.clicked.connect(+ self._create_blender_scene_from_session_data+ )self._process_automatically_checkbox = QCheckBox("Process Videos Automatically")self._process_automatically_checkbox.setChecked(True)@@ -111,7 +128,9 @@ class RecordVideos(QWidget):self._detect_2d_skeletons()def _detect_2d_skeletons(self):- print(f"tracking 2D mediapipe skeletons in videos from session: {APP_STATE.session_id}")+ print(+ f"tracking 2D mediapipe skeletons in videos from session: {APP_STATE.session_id}"+ )mediapipe_skeleton_detector = MediaPipeSkeletonDetector(APP_STATE.session_id)mediapipe_skeleton_detector.process_session_folder()@@ -123,19 +142,24 @@ class RecordVideos(QWidget):def _reconstruct_3d_skeletons(self):print(f"Reconstruct 3D Skeletons : {APP_STATE.session_id}")- session_orchestrator = SessionPipelineOrchestrator(session_id=APP_STATE.session_id)+ session_orchestrator = SessionPipelineOrchestrator(+ session_id=APP_STATE.session_id+ )if APP_STATE.use_previous_calibration:- session_orchestrator.anipose_camera_calibration_object = \+ session_orchestrator.anipose_camera_calibration_object = (CalibrationPipelineOrchestrator().load_most_recent_calibration()+ )else:- session_orchestrator.anipose_camera_calibration_object = \+ session_orchestrator.anipose_camera_calibration_object = (CalibrationPipelineOrchestrator().load_calibration_from_session_id(- APP_STATE.session_id)+ APP_STATE.session_id+ )+ )- session_orchestrator.mediapipe2d_numCams_numFrames_numTrackedPoints_XY = \- load_mediapipe2d_data(- APP_STATE.session_id)+ session_orchestrator.mediapipe2d_numCams_numFrames_numTrackedPoints_XY = (+ load_mediapipe2d_data(APP_STATE.session_id)+ )session_orchestrator.reconstruct3d_from_2d_data_offline()self._reconstruct_3d_skeletons_button.setEnabled(False)diff --git a/src/gui/main/workflows/show_cams_charuco.py b/src/gui/main/workflows/show_cams_charuco.pyindex 867ff2df..ff5100ae 100644--- a/src/gui/main/workflows/show_cams_charuco.py+++ b/src/gui/main/workflows/show_cams_charuco.py@@ -1,14 +1,22 @@-from PyQt6.QtWidgets import QHBoxLayout, QLabel, QVBoxLayout, QWidget, QPushButton, QCheckBox+from PyQt6.QtWidgets import (+ QHBoxLayout,+ QLabel,+ QVBoxLayout,+ QWidget,+ QPushButton,+ QCheckBox,+)from src.cameras.save_synchronized_videos import save_synchronized_videosfrom src.gui.main.state.app_state import APP_STATEfrom src.gui.main.workflows.single_camera import SingleCamera-from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import CalibrationPipelineOrchestrator+from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import (+ CalibrationPipelineOrchestrator,+)class ShowCamsCharuco(QWidget):-def __init__(self):super().__init__()# TODO: Take it in from init@@ -17,7 +25,9 @@ class ShowCamsCharuco(QWidget):self._continue_button = QPushButton("Continue")self._continue_button.setEnabled(False)- self._use_previous_calibration_checkbox = self._create_use_previous_calibration_checkbox()+ self._use_previous_calibration_checkbox = (+ self._create_use_previous_calibration_checkbox()+ )container = QVBoxLayout()@@ -40,8 +50,8 @@ class ShowCamsCharuco(QWidget):# start/stop recording button layoutrecord_button_layout = QHBoxLayout()- self._start_recording_button = QPushButton('Begin Recording')- self._stop_recording_button = QPushButton('Stop Recording')+ self._start_recording_button = QPushButton("Begin Recording")+ self._stop_recording_button = QPushButton("Stop Recording")self._stop_recording_button.setEnabled(False)self._start_recording_button.clicked.connect(self._start_recording_frames)@@ -63,7 +73,9 @@ class ShowCamsCharuco(QWidget):def _create_use_previous_calibration_checkbox(self):previous_calibration_checkbox = QCheckBox("Use Previous Calibration")previous_calibration_checkbox.setChecked(False)- previous_calibration_checkbox.stateChanged.connect(self._use_previous_calibration_changed)+ previous_calibration_checkbox.stateChanged.connect(+ self._use_previous_calibration_changed+ )return previous_calibration_checkboxdef _create_continue_button_container(self):@@ -72,8 +84,12 @@ class ShowCamsCharuco(QWidget):return continue_button_layoutdef _use_previous_calibration_changed(self):- self._continue_button.setEnabled(self._use_previous_calibration_checkbox.isChecked())- APP_STATE.use_previous_calibration = self._use_previous_calibration_checkbox.isChecked()+ self._continue_button.setEnabled(+ self._use_previous_calibration_checkbox.isChecked()+ )+ APP_STATE.use_previous_calibration = (+ self._use_previous_calibration_checkbox.isChecked()+ )def _start_recording_frames(self):self._start_recording_button.setEnabled(False)@@ -94,12 +110,12 @@ class ShowCamsCharuco(QWidget):self._continue_button.setEnabled(True)def _run_anipose_calibration(self):- print('Beginning Anipose calibration')+ print("Beginning Anipose calibration")calibration_orchestrator = CalibrationPipelineOrchestrator(APP_STATE.session_id)try:calibration_orchestrator.run_anipose_camera_calibration(- charuco_square_size=39,- pin_camera_0_to_origin=True)+ charuco_square_size=39, pin_camera_0_to_origin=True+ )except:- print('something failed in the anipose calibration')+ print("something failed in the anipose calibration")raise Exceptiondiff --git a/src/gui/main/workflows/single_camera.py b/src/gui/main/workflows/single_camera.pyindex 22977f0a..20958bdc 100644--- a/src/gui/main/workflows/single_camera.py+++ b/src/gui/main/workflows/single_camera.py@@ -56,7 +56,6 @@ class SingleCamera(QWidget):def quit(self):self._worker.quit()-def _init_frame_worker(self):worker = construct_worker(WorkerType.CHARUCO)(self._cam_id)worker.ImageUpdate.connect(self._handle_image_update)diff --git a/src/gui/main/workflows/visualize_session.py b/src/gui/main/workflows/visualize_session.pyindex 58677769..bd162811 100644--- a/src/gui/main/workflows/visualize_session.py+++ b/src/gui/main/workflows/visualize_session.py@@ -17,18 +17,17 @@ logger = logging.getLogger(__name__)class VisualizeSkeleton(QDialog):-- def __init__(self,- mediapipe_skel_fr_mar_xyz: np.ndarray,- start_frame: int = None,- end_frame: int = None,- move_data_to_origin: bool = True):+ def __init__(+ self,+ mediapipe_skel_fr_mar_xyz: np.ndarray,+ start_frame: int = None,+ end_frame: int = None,+ move_data_to_origin: bool = True,+ ):super().__init__()self.mediapipe_fr_mar_xyz = mediapipe_skel_fr_mar_xyz--if move_data_to_origin:self.move_data_to_origin()@@ -62,7 +61,7 @@ class VisualizeSkeleton(QDialog):def create_app_window(self):self.gl_view_widget = gl.GLViewWidget()- self.gl_view_widget.opts['distance'] = 2000+ self.gl_view_widget.opts["distance"] = 2000# self.gl_view_widget.setWindowTitle('FreeMoCap Skeleton')self.gl_view_widget.setGeometry(0, 110, 1920, 1080)# self.gl_view_widget.show()@@ -91,14 +90,22 @@ class VisualizeSkeleton(QDialog):self.gl_view_widget.addItem(grid_plane_z)def get_mediapipe_connections(self):- self.mediapipe_body_connections = [this_connection for this_connection in mp_holistic.POSE_CONNECTIONS]- self.mediapipe_hand_connections = [this_connection for this_connection in mp_holistic.HAND_CONNECTIONS]- self.mediapipe_face_connections = [this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION]+ self.mediapipe_body_connections = [+ this_connection for this_connection in mp_holistic.POSE_CONNECTIONS+ ]+ self.mediapipe_hand_connections = [+ this_connection for this_connection in mp_holistic.HAND_CONNECTIONS+ ]+ self.mediapipe_face_connections = [+ this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION+ ]def initialize_skel_dottos(self):self.skeleton_scatter_item = gl.GLScatterPlotItem(- pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, :, :], color=(0, 1, 1, 1), size=10,- pxMode=False+ pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, :, :],+ color=(0, 1, 1, 1),+ size=10,+ pxMode=False,)self.gl_view_widget.addItem(self.skeleton_scatter_item)@@ -106,16 +113,17 @@ class VisualizeSkeleton(QDialog):self.skeleton_connections_list = []for this_connection in self.mediapipe_body_connections:this_skel_line = gl.GLLinePlotItem(- pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, this_connection, :])+ pos=self.mediapipe_fr_mar_xyz[+ self.current_frame_number, this_connection, :+ ]+ )self.skeleton_connections_list.append(this_skel_line)self.gl_view_widget.addItem(this_skel_line)--def update(self):self.update_frame_number()if self.current_frame_number % 10 == 0:- print(f'frame number: {self.current_frame_number}')+ print(f"frame number: {self.current_frame_number}")self.update_skeleton()@@ -126,9 +134,13 @@ class VisualizeSkeleton(QDialog):)# skel lines- for this_skeleton_line_number, this_connection in enumerate(self.mediapipe_body_connections):+ for this_skeleton_line_number, this_connection in enumerate(+ self.mediapipe_body_connections+ ):self.skeleton_connections_list[this_skeleton_line_number].setData(- pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, this_connection, :]+ pos=self.mediapipe_fr_mar_xyz[+ self.current_frame_number, this_connection, :+ ])def update_frame_number(self):@@ -137,11 +149,11 @@ class VisualizeSkeleton(QDialog):self.current_frame_number = self.start_frame_numberdef start(self):- print('starting animation')+ print("starting animation")pg.exec()def start_animation(self):- print('initializing display window')+ print("initializing display window")self.initialize_display_window()timer = QtCore.QTimer()timer.timeout.connect(self.update)@@ -149,9 +161,10 @@ class VisualizeSkeleton(QDialog):# self.start()def move_data_to_origin(self):- mean_position_xyz = np.nanmedian(np.nanmedian(self.mediapipe_fr_mar_xyz, axis=0), axis=0)+ mean_position_xyz = np.nanmedian(+ np.nanmedian(self.mediapipe_fr_mar_xyz, axis=0), axis=0+ )self.mediapipe_fr_mar_xyz[:, :, 0] -= mean_position_xyz[0]self.mediapipe_fr_mar_xyz[:, :, 1] -= mean_position_xyz[1]self.mediapipe_fr_mar_xyz[:, :, 2] -= mean_position_xyz[2]-diff --git a/src/gui/main/workflows/welcome.py b/src/gui/main/workflows/welcome.pyindex 1f882310..1c103250 100644--- a/src/gui/main/workflows/welcome.py+++ b/src/gui/main/workflows/welcome.py@@ -1,12 +1,19 @@from PyQt6.QtCore import Qt-from PyQt6.QtWidgets import QHBoxLayout, QLabel, QVBoxLayout, QWidget, QGridLayout, QSpacerItem, QCheckBox+from PyQt6.QtWidgets import (+ QHBoxLayout,+ QLabel,+ QVBoxLayout,+ QWidget,+ QGridLayout,+ QSpacerItem,+ QCheckBox,+)from src.gui.main.qt_utils.set_font_size import set_font_sizefrom src.gui.main.shared_widgets.primary_button import PrimaryButtonclass Welcome(QWidget):-def __init__(self):super().__init__()diff --git a/src/open_in_blender/alpha_freemocap_blender_megascript.py b/src/open_in_blender/alpha_freemocap_blender_megascript.pyindex 2ba7d649..b518b83e 100644--- a/src/open_in_blender/alpha_freemocap_blender_megascript.py+++ b/src/open_in_blender/alpha_freemocap_blender_megascript.py@@ -4,7 +4,7 @@ import numpy as npfrom pathlib import Pathimport sys-print(' - Starting (alpha) blender megascript - ')+print(" - Starting (alpha) blender megascript - ")#########################################################################% Activate necessary addons@@ -17,7 +17,7 @@ try:##% Session path# #get session path as command line argumentargv = sys.argv- argv = argv[argv.index("--") + 1:]+ argv = argv[argv.index("--") + 1 :]session_path = argv[0]if len(argv) > 1:@@ -25,20 +25,31 @@ try:else:good_clean_frame_number = 0except:- print('we appear to be running from the Blender Scripting tab! Manually enter your `session_path` at line 23')+ print(+ "we appear to be running from the Blender Scripting tab! Manually enter your `session_path` at line 23"+ )session_path = Path(- r'C:\Users\jonma\Dropbox\FreeMoCapProject\FreeMocap_Data\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0')+ r"C:\Users\jonma\Dropbox\FreeMoCapProject\FreeMocap_Data\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0"+ )good_clean_frame_number = 3341print(str(session_path))session_path = Path(session_path)-path_to_mediapipe_npy = session_path / 'output_data_files' / 'mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy'-path_to_mediapipe_3d_reproj = session_path / 'output_data_files' / 'mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy'+path_to_mediapipe_npy = (+ session_path+ / "output_data_files"+ / "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"+)+path_to_mediapipe_3d_reproj = (+ session_path+ / "output_data_files"+ / "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"+)######################################################################## %% load mediapipe data# %%-print('loading data')+print("loading data")# %%mediapipe_skel_fr_mar_dim = np.load(str(path_to_mediapipe_npy))mediapipe_skel_fr_mar_dim = mediapipe_skel_fr_mar_dim / 1000 # convert to meters@@ -49,12 +60,30 @@ mediapipe_reproj_fr_mar = mediapipe_reproj_fr_mar / 1000 # convert to metersbody_marker_range = np.arange(0, 33)right_hand_marker_range = np.arange(33, 33 + 21)left_hand_marker_range = np.arange(33 + 21, 33 + 21 + 21)-face_marker_range = np.arange(left_hand_marker_range[-1], mediapipe_skel_fr_mar_dim.shape[1])--body_skel_fr_mar_dim = mediapipe_skel_fr_mar_dim[:, body_marker_range, :, ]-right_hand_fr_mar_dim = mediapipe_skel_fr_mar_dim[:, right_hand_marker_range, :, ]-left_hand_fr_mar_dim = mediapipe_skel_fr_mar_dim[:, left_hand_marker_range, :, ]-face_fr_mar_dim = mediapipe_skel_fr_mar_dim[:, face_marker_range, :, ]+face_marker_range = np.arange(+ left_hand_marker_range[-1], mediapipe_skel_fr_mar_dim.shape[1]+)++body_skel_fr_mar_dim = mediapipe_skel_fr_mar_dim[+ :,+ body_marker_range,+ :,+]+right_hand_fr_mar_dim = mediapipe_skel_fr_mar_dim[+ :,+ right_hand_marker_range,+ :,+]+left_hand_fr_mar_dim = mediapipe_skel_fr_mar_dim[+ :,+ left_hand_marker_range,+ :,+]+face_fr_mar_dim = mediapipe_skel_fr_mar_dim[+ :,+ face_marker_range,+ :,+]# %% Session Specific stuffnumber_of_frames = mediapipe_skel_fr_mar_dim.shape[0]@@ -71,8 +100,12 @@ if good_clean_frame_number == 0:frame_mean_reproj_error = []for this_frame in range(mediapipe_skel_fr_mar_dim.shape[0]):- frame_nan_counts.append(np.sum(np.isnan(mediapipe_skel_fr_mar_dim[this_frame, :, 0])))- frame_mean_reproj_error.append(np.nanmean(mediapipe_reproj_fr_mar[this_frame, :]))+ frame_nan_counts.append(+ np.sum(np.isnan(mediapipe_skel_fr_mar_dim[this_frame, :, 0]))+ )+ frame_mean_reproj_error.append(+ np.nanmean(mediapipe_reproj_fr_mar[this_frame, :])+ )nan_times_vis = np.array(frame_nan_counts) * np.array(frame_mean_reproj_error)num_frames = len(frame_nan_counts)@@ -80,7 +113,8 @@ if good_clean_frame_number == 0:# nan_times_vis[-int(num_frames/5):-1] = np.nanmax(nan_times_vis)good_clean_frame_number = np.nanargmin(- nan_times_vis) # the frame with the fewest nans (i.e. hopefully a frame where all tracked points are visible)+ nan_times_vis+ ) # the frame with the fewest nans (i.e. hopefully a frame where all tracked points are visible)print(f"----good_clean_frame_number: {good_clean_frame_number}----")@@ -171,14 +205,14 @@ mediapipe_tracked_point_names = [try:########################################################################%% clear scene- print('clearing scene')+ print("clearing scene")try:- bpy.ops.object.mode_set(mode='OBJECT')+ bpy.ops.object.mode_set(mode="OBJECT")except:passtry:- bpy.ops.object.select_all(action='SELECT') # select all objects+ bpy.ops.object.select_all(action="SELECT") # select all objectsbpy.ops.object.delete(use_global=True) # delete all objects from all scenesexcept:pass@@ -186,30 +220,34 @@ try:######################################################################## %% create world origin- bpy.ops.object.empty_add(type='ARROWS')+ bpy.ops.object.empty_add(type="ARROWS")world_origin_axes = bpy.context.active_object- world_origin_axes.name = 'world_origin' # will stay at origin+ world_origin_axes.name = "world_origin" # will stay at origin# inspect(world_origin_axes, methods=True)- bpy.ops.object.empty_add(type='ARROWS')+ bpy.ops.object.empty_add(type="ARROWS")freemocap_origin_axes = bpy.context.active_object- freemocap_origin_axes.name = 'freemocap_origin_axes' # will translate to put skelly on ground symmetric-ish about origin+ freemocap_origin_axes.name = "freemocap_origin_axes" # will translate to put skelly on ground symmetric-ish about origin######################################################################## %% load empties- print('loading {} empties on {} frames'.format(len(mediapipe_tracked_point_names), number_of_frames))+ print(+ "loading {} empties on {} frames".format(+ len(mediapipe_tracked_point_names), number_of_frames+ )+ )- empty_size = .01+ empty_size = 0.01for this_point_index, this_point_name in enumerate(mediapipe_tracked_point_names):- print(f'loading {this_point_name}...')- bpy.ops.object.empty_add(type='SPHERE')+ print(f"loading {this_point_name}...")+ bpy.ops.object.empty_add(type="SPHERE")this_empty = bpy.context.active_objectthis_empty.name = this_point_name- if 'face' in this_point_name:+ if "face" in this_point_name:this_empty.scale = [empty_size / 4] * 3- if 'hand' in this_point_name:+ if "hand" in this_point_name:this_empty.scale = [empty_size / 2] * 3else:this_empty.scale = [empty_size] * 3@@ -219,288 +257,306 @@ try:this_point_fr_xyz = mediapipe_skel_fr_mar_dim[:, this_point_index, :]for frame_num in range(end_frame):- this_empty.location = [this_point_fr_xyz[frame_num, 0],- this_point_fr_xyz[frame_num, 1],- this_point_fr_xyz[frame_num, 2]]+ this_empty.location = [+ this_point_fr_xyz[frame_num, 0],+ this_point_fr_xyz[frame_num, 1],+ this_point_fr_xyz[frame_num, 2],+ ]bpy.context.view_layer.update()- this_empty.keyframe_insert(data_path='location', frame=frame_num)+ this_empty.keyframe_insert(data_path="location", frame=frame_num)######################################################################## %% create virtual markersleft_ear_index = 7right_ear_index = 8#######################################################################- print('creating virtual markers')- print('head_center - midway between left and right ears')- hips_xyz = (mediapipe_skel_fr_mar_dim[:, left_ear_index, :] + mediapipe_skel_fr_mar_dim[:, right_ear_index, :]) / 2- bpy.ops.object.empty_add(type='SPHERE')+ print("creating virtual markers")+ print("head_center - midway between left and right ears")+ hips_xyz = (+ mediapipe_skel_fr_mar_dim[:, left_ear_index, :]+ + mediapipe_skel_fr_mar_dim[:, right_ear_index, :]+ ) / 2+ bpy.ops.object.empty_add(type="SPHERE")this_empty = bpy.context.active_object- this_empty.name = 'head_center'+ this_empty.name = "head_center"this_empty.scale = [empty_size] * 3this_empty.parent = freemocap_origin_axesmediapipe_tracked_point_names.append(this_empty.name)for frame_num in range(end_frame):- this_empty.location = [hips_xyz[frame_num, 0],- hips_xyz[frame_num, 1],- hips_xyz[frame_num, 2]]- this_empty.keyframe_insert(data_path='location', frame=frame_num)-- print('neck_center - midway between left and right shoulders')+ this_empty.location = [+ hips_xyz[frame_num, 0],+ hips_xyz[frame_num, 1],+ hips_xyz[frame_num, 2],+ ]+ this_empty.keyframe_insert(data_path="location", frame=frame_num)++ print("neck_center - midway between left and right shoulders")left_shoulder_index = 11right_shoulder_index = 12- neck_xyz = (mediapipe_skel_fr_mar_dim[:, left_shoulder_index, :] + mediapipe_skel_fr_mar_dim[:,- right_shoulder_index, :]) / 2- bpy.ops.object.empty_add(type='PLAIN_AXES')+ neck_xyz = (+ mediapipe_skel_fr_mar_dim[:, left_shoulder_index, :]+ + mediapipe_skel_fr_mar_dim[:, right_shoulder_index, :]+ ) / 2+ bpy.ops.object.empty_add(type="PLAIN_AXES")this_empty = bpy.context.active_object- this_empty.name = 'neck_center'+ this_empty.name = "neck_center"this_empty.scale = [empty_size] * 3this_empty.parent = freemocap_origin_axesmediapipe_tracked_point_names.append(this_empty.name)for frame_num in range(end_frame):- this_empty.location = [neck_xyz[frame_num, 0],- neck_xyz[frame_num, 1],- neck_xyz[frame_num, 2]]- this_empty.keyframe_insert(data_path='location', frame=frame_num)-- print('hip_center - midway between left and right hips')+ this_empty.location = [+ neck_xyz[frame_num, 0],+ neck_xyz[frame_num, 1],+ neck_xyz[frame_num, 2],+ ]+ this_empty.keyframe_insert(data_path="location", frame=frame_num)++ print("hip_center - midway between left and right hips")left_hip_index = 23right_hip_index = 24- hips_xyz = (mediapipe_skel_fr_mar_dim[:, left_hip_index, :] + mediapipe_skel_fr_mar_dim[:, right_hip_index, :]) / 2- bpy.ops.object.empty_add(type='PLAIN_AXES')+ hips_xyz = (+ mediapipe_skel_fr_mar_dim[:, left_hip_index, :]+ + mediapipe_skel_fr_mar_dim[:, right_hip_index, :]+ ) / 2+ bpy.ops.object.empty_add(type="PLAIN_AXES")this_empty = bpy.context.active_object- this_empty.name = 'hip_center'+ this_empty.name = "hip_center"this_empty.scale = [empty_size] * 3this_empty.parent = freemocap_origin_axesmediapipe_tracked_point_names.append(this_empty.name)rig_constraint_dict_of_dicts = {"spine": {- 'COPY_LOCATION': 'hip_center',- 'STRETCH_TO': 'chest_center',+ "COPY_LOCATION": "hip_center",+ "STRETCH_TO": "chest_center",},"spine.001": {- 'STRETCH_TO': 'neck_center',+ "STRETCH_TO": "neck_center",},"spine.002": {- 'STRETCH_TO': 'head_center',+ "STRETCH_TO": "head_center",},"spine.003": {- 'STRETCH_TO': 'nose',+ "STRETCH_TO": "nose",},"pelvis.L": {- 'STRETCH_TO': 'left_hip',+ "STRETCH_TO": "left_hip",},"thigh.L": {- 'COPY_LOCATION': 'left_hip',- 'STRETCH_TO': 'left_knee',+ "COPY_LOCATION": "left_hip",+ "STRETCH_TO": "left_knee",},"shin.L": {- 'STRETCH_TO': 'left_ankle',+ "STRETCH_TO": "left_ankle",},"foot.L": {- 'STRETCH_TO': 'left_foot_index',+ "STRETCH_TO": "left_foot_index",},"heel.02.L": {- 'STRETCH_TO': 'left_ankle',+ "STRETCH_TO": "left_ankle",},"shoulder.L": {- 'COPY_LOCATION': 'neck_center',- 'STRETCH_TO': 'left_shoulder',+ "COPY_LOCATION": "neck_center",+ "STRETCH_TO": "left_shoulder",},"upper_arm.L": {- 'STRETCH_TO': 'left_elbow',+ "STRETCH_TO": "left_elbow",},"forearm.L": {- 'STRETCH_TO': 'left_wrist',+ "STRETCH_TO": "left_wrist",},"hand.L": {- 'STRETCH_TO': 'left_index',+ "STRETCH_TO": "left_index",},"pelvis.R": {- 'STRETCH_TO': 'right_hip',+ "STRETCH_TO": "right_hip",},"thigh.R": {- 'COPY_LOCATION': 'right_hip',- 'STRETCH_TO': 'right_knee',+ "COPY_LOCATION": "right_hip",+ "STRETCH_TO": "right_knee",},"shin.R": {- 'STRETCH_TO': 'right_ankle',+ "STRETCH_TO": "right_ankle",},"foot.R": {- 'STRETCH_TO': 'right_foot_index',+ "STRETCH_TO": "right_foot_index",},"heel.02.R": {- 'STRETCH_TO': 'right_ankle',+ "STRETCH_TO": "right_ankle",},"shoulder.R": {- 'COPY_LOCATION': 'neck_center',- 'STRETCH_TO': 'right_shoulder',+ "COPY_LOCATION": "neck_center",+ "STRETCH_TO": "right_shoulder",},"upper_arm.R": {- 'STRETCH_TO': 'right_elbow',+ "STRETCH_TO": "right_elbow",},"forearm.R": {- 'STRETCH_TO': 'right_wrist',+ "STRETCH_TO": "right_wrist",},"hand.R": {- 'STRETCH_TO': 'right_index',+ "STRETCH_TO": "right_index",},"thumb.01.R": {- 'COPY_LOCATION': 'right_wrist',- 'STRETCH_TO': 'right_hand_thumb_mcp',+ "COPY_LOCATION": "right_wrist",+ "STRETCH_TO": "right_hand_thumb_mcp",},"thumb.02.R": {- 'STRETCH_TO': 'right_hand_thumb_ip',+ "STRETCH_TO": "right_hand_thumb_ip",},"thumb.02.R": {- 'STRETCH_TO': 'right_hand_thumb_tip',+ "STRETCH_TO": "right_hand_thumb_tip",},"palm.01.R": {- 'COPY_LOCATION': 'right_wrist',- 'STRETCH_TO': 'right_hand_index_finger_mcp',+ "COPY_LOCATION": "right_wrist",+ "STRETCH_TO": "right_hand_index_finger_mcp",},"f_index.01.R": {- 'STRETCH_TO': 'right_hand_index_finger_pip',+ "STRETCH_TO": "right_hand_index_finger_pip",},"f_index.02.R": {- 'STRETCH_TO': 'right_hand_index_finger_dip',+ "STRETCH_TO": "right_hand_index_finger_dip",},"f_index.03.R": {- 'STRETCH_TO': 'right_hand_index_finger_tip',+ "STRETCH_TO": "right_hand_index_finger_tip",},"palm.02.R": {- 'COPY_LOCATION': 'right_wrist',- 'STRETCH_TO': 'right_hand_middle_finger_mcp',+ "COPY_LOCATION": "right_wrist",+ "STRETCH_TO": "right_hand_middle_finger_mcp",},"f_middle.01.R": {- 'STRETCH_TO': 'right_hand_middle_finger_pip',+ "STRETCH_TO": "right_hand_middle_finger_pip",},"f_middle.02.R": {- 'STRETCH_TO': 'right_hand_middle_finger_dip',+ "STRETCH_TO": "right_hand_middle_finger_dip",},"f_middle.03.R": {- 'STRETCH_TO': 'right_hand_middle_finger_tip',+ "STRETCH_TO": "right_hand_middle_finger_tip",},"palm.03.R": {- 'COPY_LOCATION': 'right_wrist',- 'STRETCH_TO': 'right_hand_ring_finger_mcp',+ "COPY_LOCATION": "right_wrist",+ "STRETCH_TO": "right_hand_ring_finger_mcp",},"f_ring.01.R": {- 'STRETCH_TO': 'right_hand_ring_finger_pip',+ "STRETCH_TO": "right_hand_ring_finger_pip",},"f_ring.02.R": {- 'STRETCH_TO': 'right_hand_ring_finger_dip',+ "STRETCH_TO": "right_hand_ring_finger_dip",},"f_ring.03.R": {- 'STRETCH_TO': 'right_hand_ring_finger_tip',+ "STRETCH_TO": "right_hand_ring_finger_tip",},"palm.04.R": {- 'COPY_LOCATION': 'right_wrist',- 'STRETCH_TO': 'right_hand_pinky_finger_mcp',+ "COPY_LOCATION": "right_wrist",+ "STRETCH_TO": "right_hand_pinky_finger_mcp",},"f_pinky.01.R": {- 'STRETCH_TO': 'right_hand_pinky_finger_pip',+ "STRETCH_TO": "right_hand_pinky_finger_pip",},"f_pinky.02.R": {- 'STRETCH_TO': 'right_hand_pinky_finger_dip',+ "STRETCH_TO": "right_hand_pinky_finger_dip",},"f_pinky.03.R": {- 'STRETCH_TO': 'right_hand_pinky_finger_tip',+ "STRETCH_TO": "right_hand_pinky_finger_tip",},"thumb.01.L": {- 'COPY_LOCATION': 'left_wrist',- 'STRETCH_TO': 'left_hand_thumb_mcp',+ "COPY_LOCATION": "left_wrist",+ "STRETCH_TO": "left_hand_thumb_mcp",},"thumb.02.L": {- 'STRETCH_TO': 'left_hand_thumb_ip',+ "STRETCH_TO": "left_hand_thumb_ip",},"thumb.03.L": {- 'STRETCH_TO': 'left_hand_thumb_tip',+ "STRETCH_TO": "left_hand_thumb_tip",},"palm.01.L": {- 'COPY_LOCATION': 'left_wrist',- 'STRETCH_TO': 'left_hand_index_finger_mcp',+ "COPY_LOCATION": "left_wrist",+ "STRETCH_TO": "left_hand_index_finger_mcp",},"f_index.01.L": {- 'STRETCH_TO': 'left_hand_index_finger_pip',+ "STRETCH_TO": "left_hand_index_finger_pip",},"f_index.02.L": {- 'STRETCH_TO': 'left_hand_index_finger_dip',+ "STRETCH_TO": "left_hand_index_finger_dip",},"f_index.03.L": {- 'STRETCH_TO': 'left_hand_index_finger_tip',+ "STRETCH_TO": "left_hand_index_finger_tip",},"palm.02.L": {- 'COPY_LOCATION': 'left_wrist',- 'STRETCH_TO': 'left_hand_middle_finger_mcp',+ "COPY_LOCATION": "left_wrist",+ "STRETCH_TO": "left_hand_middle_finger_mcp",},"f_middle.01.L": {- 'STRETCH_TO': 'left_hand_middle_finger_pip',+ "STRETCH_TO": "left_hand_middle_finger_pip",},"f_middle.02.L": {- 'STRETCH_TO': 'left_hand_middle_finger_dip',+ "STRETCH_TO": "left_hand_middle_finger_dip",},"f_middle.03.L": {- 'STRETCH_TO': 'left_hand_middle_finger_tip',+ "STRETCH_TO": "left_hand_middle_finger_tip",},"palm.03.L": {- 'COPY_LOCATION': 'left_wrist',- 'STRETCH_TO': 'left_hand_ring_finger_mcp',+ "COPY_LOCATION": "left_wrist",+ "STRETCH_TO": "left_hand_ring_finger_mcp",},"f_ring.01.L": {- 'STRETCH_TO': 'left_hand_ring_finger_pip',+ "STRETCH_TO": "left_hand_ring_finger_pip",},"f_ring.02.L": {- 'STRETCH_TO': 'left_hand_ring_finger_dip',+ "STRETCH_TO": "left_hand_ring_finger_dip",},"f_ring.03.L": {- 'STRETCH_TO': 'left_hand_ring_finger_tip',+ "STRETCH_TO": "left_hand_ring_finger_tip",},"palm.04.L": {- 'COPY_LOCATION': 'left_wrist',- 'STRETCH_TO': 'left_hand_pinky_finger_mcp',+ "COPY_LOCATION": "left_wrist",+ "STRETCH_TO": "left_hand_pinky_finger_mcp",},"f_pinky.01.L": {- 'STRETCH_TO': 'left_hand_pinky_finger_pip',+ "STRETCH_TO": "left_hand_pinky_finger_pip",},"f_pinky.02.L": {- 'STRETCH_TO': 'left_hand_pinky_finger_dip',+ "STRETCH_TO": "left_hand_pinky_finger_dip",},"f_pinky.03.L": {- 'STRETCH_TO': 'left_hand_pinky_finger_tip',+ "STRETCH_TO": "left_hand_pinky_finger_tip",},}# loop through dictionary applying constraintstry:- bpy.ops.object.mode_set(mode='POSE')+ bpy.ops.object.mode_set(mode="POSE")except:passfor frame_num in range(end_frame):- this_empty.location = [hips_xyz[frame_num, 0],- hips_xyz[frame_num, 1],- hips_xyz[frame_num, 2]]- this_empty.keyframe_insert(data_path='location', frame=frame_num)-- print('chest_center - mean of R/L shoulders and R/L hips')+ this_empty.location = [+ hips_xyz[frame_num, 0],+ hips_xyz[frame_num, 1],+ hips_xyz[frame_num, 2],+ ]+ this_empty.keyframe_insert(data_path="location", frame=frame_num)++ print("chest_center - mean of R/L shoulders and R/L hips")chest_xyz = (neck_xyz + hips_xyz) / 2- bpy.ops.object.empty_add(type='PLAIN_AXES')+ bpy.ops.object.empty_add(type="PLAIN_AXES")this_empty = bpy.context.active_object- this_empty.name = 'chest_center'+ this_empty.name = "chest_center"this_empty.scale = [empty_size] * 3this_empty.parent = freemocap_origin_axesmediapipe_tracked_point_names.append(this_empty.name)for frame_num in range(end_frame):- this_empty.location = [chest_xyz[frame_num, 0],- chest_xyz[frame_num, 1],- chest_xyz[frame_num, 2]]- this_empty.keyframe_insert(data_path='location', frame=frame_num)+ this_empty.location = [+ chest_xyz[frame_num, 0],+ chest_xyz[frame_num, 1],+ chest_xyz[frame_num, 2],+ ]+ this_empty.keyframe_insert(data_path="location", frame=frame_num)- print('Done loading empties :D')+ print("Done loading empties :D")######################################################################## Fit Rigify metarig armatures to empties@@ -510,308 +566,301 @@ try:# each bone name has to match one in the Blender Rigify Human Metarig, or it will be ignoredmetarig_bone_empty_pairs_dict_of_dicts = {"nose.001": {- 'head': 'head_center',- 'tail': 'nose',+ "head": "head_center",+ "tail": "nose",},"nose.L.001": {- 'head': 'nose',- 'tail': 'left_eye_inner',+ "head": "nose",+ "tail": "left_eye_inner",},"eye.L": {- 'head': 'left_eye_inner',- 'tail': 'left_eye',+ "head": "left_eye_inner",+ "tail": "left_eye",},"lid.B.L": {- 'head': 'left_eye',- 'tail': 'left_eye_outer',+ "head": "left_eye",+ "tail": "left_eye_outer",},"ear.L": {- 'head': 'left_eye_outer',- 'tail': 'left_ear',+ "head": "left_eye_outer",+ "tail": "left_ear",},"nose.R.001": {- 'head': 'nose',- 'tail': 'right_eye_inner',+ "head": "nose",+ "tail": "right_eye_inner",},"eye.R": {- 'head': 'right_eye_inner',- 'tail': 'right_eye',+ "head": "right_eye_inner",+ "tail": "right_eye",},"lid.B.R": {- 'head': 'right_eye',- 'tail': 'right_eye_outer',+ "head": "right_eye",+ "tail": "right_eye_outer",},"ear.R": {- 'head': 'right_eye_outer',- 'tail': 'right_ear',+ "head": "right_eye_outer",+ "tail": "right_ear",},"lip.T.R": {- 'head': 'nose',- 'tail': 'mouth_right',+ "head": "nose",+ "tail": "mouth_right",},"lip.B.R": {- 'head': 'mouth_right',- 'tail': 'mouth_left',+ "head": "mouth_right",+ "tail": "mouth_left",},"lip.T.L": {- 'head': 'nose',- 'tail': 'mouth_left',+ "head": "nose",+ "tail": "mouth_left",},"hand.L": {- 'head': 'left_wrist',- 'tail': 'left_index',+ "head": "left_wrist",+ "tail": "left_index",},"forearm.L": {- 'head': 'left_elbow',- 'tail': 'left_wrist',+ "head": "left_elbow",+ "tail": "left_wrist",},"upper_arm.L": {- 'head': 'left_shoulder',- 'tail': 'left_elbow',+ "head": "left_shoulder",+ "tail": "left_elbow",},"shoulder.L": {- 'head': 'neck_center',- 'tail': 'left_shoulder',+ "head": "neck_center",+ "tail": "left_shoulder",},"hand.R": {- 'head': 'right_wrist',- 'tail': 'right_index',+ "head": "right_wrist",+ "tail": "right_index",},"forearm.R": {- 'head': 'right_elbow',- 'tail': 'right_wrist',+ "head": "right_elbow",+ "tail": "right_wrist",},"upper_arm.R": {- 'head': 'right_shoulder',- 'tail': 'right_elbow',+ "head": "right_shoulder",+ "tail": "right_elbow",},"shoulder.R": {- 'head': 'neck_center',- 'tail': 'right_shoulder',+ "head": "neck_center",+ "tail": "right_shoulder",},"thigh.L": {- 'head': 'left_hip',- 'tail': 'left_knee',+ "head": "left_hip",+ "tail": "left_knee",},"shin.L": {- 'head': 'left_knee',- 'tail': 'left_ankle',+ "head": "left_knee",+ "tail": "left_ankle",},"heel.02.L": {- 'head': 'left_heel',- 'tail': 'left_ankle',+ "head": "left_heel",+ "tail": "left_ankle",},"foot.L": {- 'head': 'left_ankle',- 'tail': 'left_foot_index',+ "head": "left_ankle",+ "tail": "left_foot_index",},"toe.L": {- 'head': 'left_foot_index',- 'tail': 'left_heel',+ "head": "left_foot_index",+ "tail": "left_heel",},"thigh.R": {- 'head': 'right_hip',- 'tail': 'right_knee',+ "head": "right_hip",+ "tail": "right_knee",},"shin.R": {- 'head': 'right_knee',- 'tail': 'right_ankle',+ "head": "right_knee",+ "tail": "right_ankle",},"heel.02.R": {- 'head': 'right_heel',- 'tail': 'right_ankle',+ "head": "right_heel",+ "tail": "right_ankle",},"foot.R": {- 'head': 'right_ankle',- 'tail': 'right_foot_index',+ "head": "right_ankle",+ "tail": "right_foot_index",},"toe.R": {- 'head': 'right_foot_index',- 'tail': 'right_heel',+ "head": "right_foot_index",+ "tail": "right_heel",},"pelvis.L": {- 'head': 'hip_center',- 'tail': 'left_hip',+ "head": "hip_center",+ "tail": "left_hip",},"pelvis.R": {- 'head': 'hip_center',- 'tail': 'right_hip',+ "head": "hip_center",+ "tail": "right_hip",},"spine": {- 'head': 'hip_center',- 'tail': 'chest_center',+ "head": "hip_center",+ "tail": "chest_center",},"spine.001": {- 'head': 'chest_center',- 'tail': 'neck_center',+ "head": "chest_center",+ "tail": "neck_center",},"spine.002": {- 'head': 'neck_center',- 'tail': 'head_center',+ "head": "neck_center",+ "tail": "head_center",},"spine.003": {- 'head': 'head_center',- 'tail': 'nose',+ "head": "head_center",+ "tail": "nose",},"palm.01.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_index_finger_mcp',+ "head": "right_wrist",+ "tail": "right_hand_index_finger_mcp",},"palm.02.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_middle_finger_mcp',+ "head": "right_wrist",+ "tail": "right_hand_middle_finger_mcp",},"palm.03.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_ring_finger_mcp',+ "head": "right_wrist",+ "tail": "right_hand_ring_finger_mcp",},"palm.04.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_pinky_finger_mcp',+ "head": "right_wrist",+ "tail": "right_hand_pinky_finger_mcp",},-"thumb.01.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_thumb_mcp',+ "head": "right_wrist",+ "tail": "right_hand_thumb_mcp",},"thumb.02.R": {- 'head': 'right_hand_thumb_mcp',- 'tail': 'right_hand_thumb_ip',+ "head": "right_hand_thumb_mcp",+ "tail": "right_hand_thumb_ip",},"thumb.03.R": {- 'head': 'right_hand_thumb_ip',- 'tail': 'right_hand_thumb_tip',+ "head": "right_hand_thumb_ip",+ "tail": "right_hand_thumb_tip",},-"f_index.01.R": {- 'head': 'right_hand_index_finger_mcp',- 'tail': 'right_hand_index_finger_pip',+ "head": "right_hand_index_finger_mcp",+ "tail": "right_hand_index_finger_pip",},"f_index.02.R": {- 'head': 'right_hand_index_finger_pip',- 'tail': 'right_hand_index_finger_dip',+ "head": "right_hand_index_finger_pip",+ "tail": "right_hand_index_finger_dip",},"f_index.03.R": {- 'head': 'right_hand_index_finger_dip',- 'tail': 'right_hand_index_finger_tip',+ "head": "right_hand_index_finger_dip",+ "tail": "right_hand_index_finger_tip",},-"f_middle.01.R": {- 'head': 'right_hand_middle_finger_mcp',- 'tail': 'right_hand_middle_finger_pip',+ "head": "right_hand_middle_finger_mcp",+ "tail": "right_hand_middle_finger_pip",},"f_middle.02.R": {- 'head': 'right_hand_middle_finger_pip',- 'tail': 'right_hand_middle_finger_dip',+ "head": "right_hand_middle_finger_pip",+ "tail": "right_hand_middle_finger_dip",},"f_middle.03.R": {- 'head': 'right_hand_middle_finger_dip',- 'tail': 'right_hand_middle_finger_tip',+ "head": "right_hand_middle_finger_dip",+ "tail": "right_hand_middle_finger_tip",},-"f_ring.01.R": {- 'head': 'right_hand_ring_finger_mcp',- 'tail': 'right_hand_ring_finger_pip',+ "head": "right_hand_ring_finger_mcp",+ "tail": "right_hand_ring_finger_pip",},"f_ring.02.R": {- 'head': 'right_hand_ring_finger_pip',- 'tail': 'right_hand_ring_finger_dip',+ "head": "right_hand_ring_finger_pip",+ "tail": "right_hand_ring_finger_dip",},"f_ring.03.R": {- 'head': 'right_hand_ring_finger_dip',- 'tail': 'right_hand_ring_finger_tip',+ "head": "right_hand_ring_finger_dip",+ "tail": "right_hand_ring_finger_tip",},-"f_pinky.01.R": {- 'head': 'right_hand_pinky_finger_mcp',- 'tail': 'right_hand_pinky_finger_pip',+ "head": "right_hand_pinky_finger_mcp",+ "tail": "right_hand_pinky_finger_pip",},"f_pinky.02.R": {- 'head': 'right_hand_pinky_finger_pip',- 'tail': 'right_hand_pinky_finger_dip',+ "head": "right_hand_pinky_finger_pip",+ "tail": "right_hand_pinky_finger_dip",},"f_pinky.03.R": {- 'head': 'right_hand_pinky_finger_dip',- 'tail': 'right_hand_pinky_finger_tip',+ "head": "right_hand_pinky_finger_dip",+ "tail": "right_hand_pinky_finger_tip",},"thumb.01.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_thumb_mcp',+ "head": "left_wrist",+ "tail": "left_hand_thumb_mcp",},"thumb.02.L": {- 'head': 'left_hand_thumb_mcp',- 'tail': 'left_hand_thumb_ip',+ "head": "left_hand_thumb_mcp",+ "tail": "left_hand_thumb_ip",},"thumb.03.L": {- 'head': 'left_hand_thumb_ip',- 'tail': 'left_hand_thumb_tip',+ "head": "left_hand_thumb_ip",+ "tail": "left_hand_thumb_tip",},"palm.01.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_index_finger_mcp',+ "head": "left_wrist",+ "tail": "left_hand_index_finger_mcp",},"palm.02.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_middle_finger_mcp',+ "head": "left_wrist",+ "tail": "left_hand_middle_finger_mcp",},"palm.03.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_ring_finger_mcp',+ "head": "left_wrist",+ "tail": "left_hand_ring_finger_mcp",},"palm.04.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_pinky_finger_mcp',+ "head": "left_wrist",+ "tail": "left_hand_pinky_finger_mcp",},"f_index.01.L": {- 'head': 'left_hand_index_finger_mcp',- 'tail': 'left_hand_index_finger_pip',+ "head": "left_hand_index_finger_mcp",+ "tail": "left_hand_index_finger_pip",},"f_index.02.L": {- 'head': 'left_hand_index_finger_pip',- 'tail': 'left_hand_index_finger_dip',+ "head": "left_hand_index_finger_pip",+ "tail": "left_hand_index_finger_dip",},"f_index.03.L": {- 'head': 'left_hand_index_finger_dip',- 'tail': 'left_hand_index_finger_tip',+ "head": "left_hand_index_finger_dip",+ "tail": "left_hand_index_finger_tip",},-"f_middle.01.L": {- 'head': 'left_hand_middle_finger_mcp',- 'tail': 'left_hand_middle_finger_pip',+ "head": "left_hand_middle_finger_mcp",+ "tail": "left_hand_middle_finger_pip",},"f_middle.02.L": {- 'head': 'left_hand_middle_finger_pip',- 'tail': 'left_hand_middle_finger_dip',+ "head": "left_hand_middle_finger_pip",+ "tail": "left_hand_middle_finger_dip",},"f_middle.03.L": {- 'head': 'left_hand_middle_finger_dip',- 'tail': 'left_hand_middle_finger_tip',+ "head": "left_hand_middle_finger_dip",+ "tail": "left_hand_middle_finger_tip",},"f_ring.01.L": {- 'head': 'left_hand_ring_finger_mcp',- 'tail': 'left_hand_ring_finger_pip',+ "head": "left_hand_ring_finger_mcp",+ "tail": "left_hand_ring_finger_pip",},"f_ring.02.L": {- 'head': 'left_hand_ring_finger_pip',- 'tail': 'left_hand_ring_finger_dip',+ "head": "left_hand_ring_finger_pip",+ "tail": "left_hand_ring_finger_dip",},"f_ring.03.L": {- 'head': 'left_hand_ring_finger_dip',- 'tail': 'left_hand_ring_finger_tip',+ "head": "left_hand_ring_finger_dip",+ "tail": "left_hand_ring_finger_tip",},"f_pinky.01.L": {- 'head': 'left_hand_pinky_finger_mcp',- 'tail': 'left_hand_pinky_finger_pip',+ "head": "left_hand_pinky_finger_mcp",+ "tail": "left_hand_pinky_finger_pip",},"f_pinky.02.L": {- 'head': 'left_hand_pinky_finger_pip',- 'tail': 'left_hand_pinky_finger_dip',+ "head": "left_hand_pinky_finger_pip",+ "tail": "left_hand_pinky_finger_dip",},"f_pinky.03.L": {- 'head': 'left_hand_pinky_finger_dip',- 'tail': 'left_hand_pinky_finger_tip',+ "head": "left_hand_pinky_finger_dip",+ "tail": "left_hand_pinky_finger_tip",},-}# %% create metarig@@ -821,313 +870,328 @@ try:# %% loop through dictionary - set head and tail of each bone's to the relevant empty# bpy.ops.object.posemode_toggle()try:- bpy.ops.object.mode_set(mode='EDIT')+ bpy.ops.object.mode_set(mode="EDIT")except:passfor this_bone in list(this_metarig.data.bones):this_bone_name = this_bone.name- print(f'--{this_bone_name}--')+ print(f"--{this_bone_name}--")this_edit_bone = this_metarig.data.edit_bones[- this_bone_name] # note this is using EDIT_bones, not regular bones b/c blender is a wild wacky place+ this_bone_name+ ] # note this is using EDIT_bones, not regular bones b/c blender is a wild wacky placeif this_bone_name in metarig_bone_empty_pairs_dict_of_dicts:this_bone_dict = metarig_bone_empty_pairs_dict_of_dicts[this_bone_name]- this_head_empty_name = this_bone_dict['head']- this_tail_empty_name = this_bone_dict['tail']- print('setting (bone:head->tail){}:{}->{})'.format(this_bone_name, this_head_empty_name,- this_tail_empty_name))+ this_head_empty_name = this_bone_dict["head"]+ this_tail_empty_name = this_bone_dict["tail"]+ print(+ "setting (bone:head->tail){}:{}->{})".format(+ this_bone_name, this_head_empty_name, this_tail_empty_name+ )+ )if this_tail_empty_name is not None:print(- f'setting bone: {this_bone_name}: head: {bpy.data.objects[this_head_empty_name].location}, tail: {bpy.data.objects[this_tail_empty_name].location}')- this_edit_bone.head = bpy.data.objects[this_head_empty_name].location- this_edit_bone.tail = bpy.data.objects[this_tail_empty_name].location+ f"setting bone: {this_bone_name}: head: {bpy.data.objects[this_head_empty_name].location}, tail: {bpy.data.objects[this_tail_empty_name].location}"+ )+ this_edit_bone.head = bpy.data.objects[+ this_head_empty_name+ ].location+ this_edit_bone.tail = bpy.data.objects[+ this_tail_empty_name+ ].locationelse: # remove bones with no tail (and/or head) until we figure out a better plan- print(f'removing bone: {this_bone_name}')+ print(f"removing bone: {this_bone_name}")this_metarig.data.edit_bones.remove(this_edit_bone)else:- print(f'removing bone: {this_bone_name}')+ print(f"removing bone: {this_bone_name}")this_metarig.data.edit_bones.remove(this_edit_bone)try:- bpy.ops.object.mode_set(mode='OBJECT')+ bpy.ops.object.mode_set(mode="OBJECT")except:pass########################################################### Damped track bones to empties- bpy.ops.object.mode_set(mode='OBJECT')- armature_name = 'metarig'+ bpy.ops.object.mode_set(mode="OBJECT")+ armature_name = "metarig"this_armature = bpy.data.objects[armature_name]rig_constraint_dict_of_dicts = {"nose.001": {- 'COPY_LOCATION': 'head_center',- 'DAMPED_TRACK': 'nose',+ "COPY_LOCATION": "head_center",+ "DAMPED_TRACK": "nose",},"nose.L.001": {- 'DAMPED_TRACK': 'left_eye_inner',+ "DAMPED_TRACK": "left_eye_inner",},"eye.L": {- 'DAMPED_TRACK': 'left_eye',+ "DAMPED_TRACK": "left_eye",},"lid.B.L": {- 'DAMPED_TRACK': 'left_eye_outer',+ "DAMPED_TRACK": "left_eye_outer",},"ear.L": {- 'DAMPED_TRACK': 'left_ear',+ "DAMPED_TRACK": "left_ear",},"nose.R.001": {- 'DAMPED_TRACK': 'right_eye_inner',+ "DAMPED_TRACK": "right_eye_inner",},"eye.R": {- 'DAMPED_TRACK': 'right_eye',+ "DAMPED_TRACK": "right_eye",},"lid.B.R": {- 'DAMPED_TRACK': 'right_eye_outer',+ "DAMPED_TRACK": "right_eye_outer",},"ear.R": {- 'DAMPED_TRACK': 'right_ear',+ "DAMPED_TRACK": "right_ear",},"lip.T.R": {- 'DAMPED_TRACK': 'mouth_right',+ "DAMPED_TRACK": "mouth_right",},"lip.B.R": {- 'DAMPED_TRACK': 'mouth_left',+ "DAMPED_TRACK": "mouth_left",},"lip.T.L": {- 'DAMPED_TRACK': 'nose',+ "DAMPED_TRACK": "nose",},"spine": {- 'COPY_LOCATION': 'hip_center',- 'DAMPED_TRACK': 'chest_center',+ "COPY_LOCATION": "hip_center",+ "DAMPED_TRACK": "chest_center",},"spine.001": {- 'DAMPED_TRACK': 'neck_center',+ "DAMPED_TRACK": "neck_center",},"spine.002": {- 'DAMPED_TRACK': 'head_center',+ "DAMPED_TRACK": "head_center",},"spine.003": {- 'DAMPED_TRACK': 'nose',+ "DAMPED_TRACK": "nose",},"pelvis.L": {- 'DAMPED_TRACK': 'left_hip',+ "DAMPED_TRACK": "left_hip",},"thigh.L": {- 'COPY_LOCATION': 'left_hip',- 'DAMPED_TRACK': 'left_knee',+ "COPY_LOCATION": "left_hip",+ "DAMPED_TRACK": "left_knee",},"shin.L": {- 'DAMPED_TRACK': 'left_ankle',+ "DAMPED_TRACK": "left_ankle",},"foot.L": {- 'DAMPED_TRACK': 'left_foot_index',+ "DAMPED_TRACK": "left_foot_index",},"heel.02.L": {- 'DAMPED_TRACK': 'left_ankle',+ "DAMPED_TRACK": "left_ankle",},"shoulder.L": {- 'COPY_LOCATION': 'neck_center',- 'DAMPED_TRACK': 'left_shoulder',+ "COPY_LOCATION": "neck_center",+ "DAMPED_TRACK": "left_shoulder",},"upper_arm.L": {- 'DAMPED_TRACK': 'left_elbow',+ "DAMPED_TRACK": "left_elbow",},"forearm.L": {- 'DAMPED_TRACK': 'left_wrist',+ "DAMPED_TRACK": "left_wrist",},"hand.L": {- 'DAMPED_TRACK': 'left_index',+ "DAMPED_TRACK": "left_index",},"pelvis.R": {- 'DAMPED_TRACK': 'right_hip',+ "DAMPED_TRACK": "right_hip",},"thigh.R": {- 'COPY_LOCATION': 'right_hip',- 'DAMPED_TRACK': 'right_knee',+ "COPY_LOCATION": "right_hip",+ "DAMPED_TRACK": "right_knee",},"shin.R": {- 'DAMPED_TRACK': 'right_ankle',+ "DAMPED_TRACK": "right_ankle",},"foot.R": {- 'DAMPED_TRACK': 'right_foot_index',+ "DAMPED_TRACK": "right_foot_index",},"heel.02.R": {- 'DAMPED_TRACK': 'right_ankle',+ "DAMPED_TRACK": "right_ankle",},"shoulder.R": {- 'COPY_LOCATION': 'neck_center',- 'DAMPED_TRACK': 'right_shoulder',+ "COPY_LOCATION": "neck_center",+ "DAMPED_TRACK": "right_shoulder",},"upper_arm.R": {- 'DAMPED_TRACK': 'right_elbow',+ "DAMPED_TRACK": "right_elbow",},"forearm.R": {- 'DAMPED_TRACK': 'right_wrist',+ "DAMPED_TRACK": "right_wrist",},"hand.R": {- 'DAMPED_TRACK': 'right_index',+ "DAMPED_TRACK": "right_index",},"thumb.01.R": {- 'COPY_LOCATION': 'right_wrist',- 'DAMPED_TRACK': 'right_hand_thumb_mcp',+ "COPY_LOCATION": "right_wrist",+ "DAMPED_TRACK": "right_hand_thumb_mcp",},"thumb.02.R": {- 'DAMPED_TRACK': 'right_hand_thumb_ip',+ "DAMPED_TRACK": "right_hand_thumb_ip",},"thumb.02.R": {- 'DAMPED_TRACK': 'right_hand_thumb_tip',+ "DAMPED_TRACK": "right_hand_thumb_tip",},"palm.01.R": {- 'COPY_LOCATION': 'right_wrist',- 'DAMPED_TRACK': 'right_hand_index_finger_mcp',+ "COPY_LOCATION": "right_wrist",+ "DAMPED_TRACK": "right_hand_index_finger_mcp",},"f_index.01.R": {- 'DAMPED_TRACK': 'right_hand_index_finger_pip',+ "DAMPED_TRACK": "right_hand_index_finger_pip",},"f_index.02.R": {- 'DAMPED_TRACK': 'right_hand_index_finger_dip',+ "DAMPED_TRACK": "right_hand_index_finger_dip",},"f_index.03.R": {- 'DAMPED_TRACK': 'right_hand_index_finger_tip',+ "DAMPED_TRACK": "right_hand_index_finger_tip",},"palm.02.R": {- 'COPY_LOCATION': 'right_wrist',- 'DAMPED_TRACK': 'right_hand_middle_finger_mcp',+ "COPY_LOCATION": "right_wrist",+ "DAMPED_TRACK": "right_hand_middle_finger_mcp",},"f_middle.01.R": {- 'DAMPED_TRACK': 'right_hand_middle_finger_pip',+ "DAMPED_TRACK": "right_hand_middle_finger_pip",},"f_middle.02.R": {- 'DAMPED_TRACK': 'right_hand_middle_finger_dip',+ "DAMPED_TRACK": "right_hand_middle_finger_dip",},"f_middle.03.R": {- 'DAMPED_TRACK': 'right_hand_middle_finger_tip',+ "DAMPED_TRACK": "right_hand_middle_finger_tip",},"palm.03.R": {- 'COPY_LOCATION': 'right_wrist',- 'DAMPED_TRACK': 'right_hand_ring_finger_mcp',+ "COPY_LOCATION": "right_wrist",+ "DAMPED_TRACK": "right_hand_ring_finger_mcp",},"f_ring.01.R": {- 'DAMPED_TRACK': 'right_hand_ring_finger_pip',+ "DAMPED_TRACK": "right_hand_ring_finger_pip",},"f_ring.02.R": {- 'DAMPED_TRACK': 'right_hand_ring_finger_dip',+ "DAMPED_TRACK": "right_hand_ring_finger_dip",},"f_ring.03.R": {- 'DAMPED_TRACK': 'right_hand_ring_finger_tip',+ "DAMPED_TRACK": "right_hand_ring_finger_tip",},"palm.04.R": {- 'COPY_LOCATION': 'right_wrist',- 'DAMPED_TRACK': 'right_hand_pinky_finger_mcp',+ "COPY_LOCATION": "right_wrist",+ "DAMPED_TRACK": "right_hand_pinky_finger_mcp",},"f_pinky.01.R": {- 'DAMPED_TRACK': 'right_hand_pinky_finger_pip',+ "DAMPED_TRACK": "right_hand_pinky_finger_pip",},"f_pinky.02.R": {- 'DAMPED_TRACK': 'right_hand_pinky_finger_dip',+ "DAMPED_TRACK": "right_hand_pinky_finger_dip",},"f_pinky.03.R": {- 'DAMPED_TRACK': 'right_hand_pinky_finger_tip',+ "DAMPED_TRACK": "right_hand_pinky_finger_tip",},"thumb.01.L": {- 'COPY_LOCATION': 'left_wrist',- 'DAMPED_TRACK': 'left_hand_thumb_mcp',+ "COPY_LOCATION": "left_wrist",+ "DAMPED_TRACK": "left_hand_thumb_mcp",},"thumb.02.L": {- 'DAMPED_TRACK': 'left_hand_thumb_ip',+ "DAMPED_TRACK": "left_hand_thumb_ip",},"thumb.03.L": {- 'DAMPED_TRACK': 'left_hand_thumb_tip',+ "DAMPED_TRACK": "left_hand_thumb_tip",},"palm.01.L": {- 'COPY_LOCATION': 'left_wrist',- 'DAMPED_TRACK': 'left_hand_index_finger_mcp',+ "COPY_LOCATION": "left_wrist",+ "DAMPED_TRACK": "left_hand_index_finger_mcp",},"f_index.01.L": {- 'DAMPED_TRACK': 'left_hand_index_finger_pip',+ "DAMPED_TRACK": "left_hand_index_finger_pip",},"f_index.02.L": {- 'DAMPED_TRACK': 'left_hand_index_finger_dip',+ "DAMPED_TRACK": "left_hand_index_finger_dip",},"f_index.03.L": {- 'DAMPED_TRACK': 'left_hand_index_finger_tip',+ "DAMPED_TRACK": "left_hand_index_finger_tip",},"palm.02.L": {- 'COPY_LOCATION': 'left_wrist',- 'DAMPED_TRACK': 'left_hand_middle_finger_mcp',+ "COPY_LOCATION": "left_wrist",+ "DAMPED_TRACK": "left_hand_middle_finger_mcp",},"f_middle.01.L": {- 'DAMPED_TRACK': 'left_hand_middle_finger_pip',+ "DAMPED_TRACK": "left_hand_middle_finger_pip",},"f_middle.02.L": {- 'DAMPED_TRACK': 'left_hand_middle_finger_dip',+ "DAMPED_TRACK": "left_hand_middle_finger_dip",},"f_middle.03.L": {- 'DAMPED_TRACK': 'left_hand_middle_finger_tip',+ "DAMPED_TRACK": "left_hand_middle_finger_tip",},"palm.03.L": {- 'COPY_LOCATION': 'left_wrist',- 'DAMPED_TRACK': 'left_hand_ring_finger_mcp',+ "COPY_LOCATION": "left_wrist",+ "DAMPED_TRACK": "left_hand_ring_finger_mcp",},"f_ring.01.L": {- 'DAMPED_TRACK': 'left_hand_ring_finger_pip',+ "DAMPED_TRACK": "left_hand_ring_finger_pip",},"f_ring.02.L": {- 'DAMPED_TRACK': 'left_hand_ring_finger_dip',+ "DAMPED_TRACK": "left_hand_ring_finger_dip",},"f_ring.03.L": {- 'DAMPED_TRACK': 'left_hand_ring_finger_tip',+ "DAMPED_TRACK": "left_hand_ring_finger_tip",},"palm.04.L": {- 'COPY_LOCATION': 'left_wrist',- 'DAMPED_TRACK': 'left_hand_pinky_finger_mcp',+ "COPY_LOCATION": "left_wrist",+ "DAMPED_TRACK": "left_hand_pinky_finger_mcp",},"f_pinky.01.L": {- 'DAMPED_TRACK': 'left_hand_pinky_finger_pip',+ "DAMPED_TRACK": "left_hand_pinky_finger_pip",},"f_pinky.02.L": {- 'DAMPED_TRACK': 'left_hand_pinky_finger_dip',+ "DAMPED_TRACK": "left_hand_pinky_finger_dip",},"f_pinky.03.L": {- 'DAMPED_TRACK': 'left_hand_pinky_finger_tip',+ "DAMPED_TRACK": "left_hand_pinky_finger_tip",},}# loop through dictionary applying constraintstry:- bpy.ops.object.mode_set(mode='POSE')+ bpy.ops.object.mode_set(mode="POSE")except:passfor this_bone_name, this_bone_dict in rig_constraint_dict_of_dicts.items():- print(f'---Setting constraints for bone:{this_bone_name}---')+ print(f"---Setting constraints for bone:{this_bone_name}---")- for this_constraint_name, this_constraint_target_empty_name in this_bone_dict.items():- print(f'constraint: {this_constraint_name} with target:{this_constraint_target_empty_name}')- print('grab bone')+ for (+ this_constraint_name,+ this_constraint_target_empty_name,+ ) in this_bone_dict.items():+ print(+ f"constraint: {this_constraint_name} with target:{this_constraint_target_empty_name}"+ )+ print("grab bone")this_bone = this_armature.pose.bones[this_bone_name]- print('apply bone')+ print("apply bone")this_constraint = this_bone.constraints.new(type=this_constraint_name)this_constraint.name = this_constraint_namethis_constraint.target = bpy.data.objects[- this_constraint_target_empty_name] # point constraint at relevant empty object+ this_constraint_target_empty_name+ ] # point constraint at relevant empty object################ create extra bonesextra_bone_empty_pairs_dict_of_dicts = {"hand.L": {- 'head': 'left_wrist',- 'tail': 'left_index',+ "head": "left_wrist",+ "tail": "left_index",},"forearm.L": {- 'head': 'left_elbow',- 'tail': 'left_wrist',+ "head": "left_elbow",+ "tail": "left_wrist",},}@@ -1136,327 +1200,316 @@ try:stick_figure_mesh_edge_pairs_dict_of_dicts = {"nose.001": {- 'head': 'head_center',- 'tail': 'nose',+ "head": "head_center",+ "tail": "nose",},"nose.L.001": {- 'head': 'nose',- 'tail': 'left_eye_inner',+ "head": "nose",+ "tail": "left_eye_inner",},"eye.L": {- 'head': 'left_eye_inner',- 'tail': 'left_eye',+ "head": "left_eye_inner",+ "tail": "left_eye",},"lid.B.L": {- 'head': 'left_eye',- 'tail': 'left_eye_outer',+ "head": "left_eye",+ "tail": "left_eye_outer",},"ear.L": {- 'head': 'left_eye_outer',- 'tail': 'left_ear',+ "head": "left_eye_outer",+ "tail": "left_ear",},"nose.R.001": {- 'head': 'nose',- 'tail': 'right_eye_inner',+ "head": "nose",+ "tail": "right_eye_inner",},"eye.R": {- 'head': 'right_eye_inner',- 'tail': 'right_eye',+ "head": "right_eye_inner",+ "tail": "right_eye",},"lid.B.R": {- 'head': 'right_eye',- 'tail': 'right_eye_outer',+ "head": "right_eye",+ "tail": "right_eye_outer",},"ear.R": {- 'head': 'right_eye_outer',- 'tail': 'right_ear',+ "head": "right_eye_outer",+ "tail": "right_ear",},"lip.T.R": {- 'head': 'nose',- 'tail': 'mouth_right',+ "head": "nose",+ "tail": "mouth_right",},"lip.B.R": {- 'head': 'mouth_right',- 'tail': 'mouth_left',+ "head": "mouth_right",+ "tail": "mouth_left",},"lip.T.L": {- 'head': 'nose',- 'tail': 'mouth_left',+ "head": "nose",+ "tail": "mouth_left",},"forearm.L": {- 'head': 'left_elbow',- 'tail': 'left_wrist',+ "head": "left_elbow",+ "tail": "left_wrist",},"upper_arm.L": {- 'head': 'left_shoulder',- 'tail': 'left_elbow',+ "head": "left_shoulder",+ "tail": "left_elbow",},"shoulder.L": {- 'head': 'neck_center',- 'tail': 'left_shoulder',+ "head": "neck_center",+ "tail": "left_shoulder",},"forearm.R": {- 'head': 'right_elbow',- 'tail': 'right_wrist',+ "head": "right_elbow",+ "tail": "right_wrist",},"upper_arm.R": {- 'head': 'right_shoulder',- 'tail': 'right_elbow',+ "head": "right_shoulder",+ "tail": "right_elbow",},"shoulder.R": {- 'head': 'neck_center',- 'tail': 'right_shoulder',+ "head": "neck_center",+ "tail": "right_shoulder",},"thigh.L": {- 'head': 'left_hip',- 'tail': 'left_knee',+ "head": "left_hip",+ "tail": "left_knee",},"shin.L": {- 'head': 'left_knee',- 'tail': 'left_ankle',+ "head": "left_knee",+ "tail": "left_ankle",},"heel.02.L": {- 'head': 'left_heel',- 'tail': 'left_ankle',+ "head": "left_heel",+ "tail": "left_ankle",},"foot.L": {- 'head': 'left_ankle',- 'tail': 'left_foot_index',+ "head": "left_ankle",+ "tail": "left_foot_index",},"toe.L": {- 'head': 'left_foot_index',- 'tail': 'left_heel',+ "head": "left_foot_index",+ "tail": "left_heel",},"thigh.R": {- 'head': 'right_hip',- 'tail': 'right_knee',+ "head": "right_hip",+ "tail": "right_knee",},"shin.R": {- 'head': 'right_knee',- 'tail': 'right_ankle',+ "head": "right_knee",+ "tail": "right_ankle",},"heel.02.R": {- 'head': 'right_heel',- 'tail': 'right_ankle',+ "head": "right_heel",+ "tail": "right_ankle",},"foot.R": {- 'head': 'right_ankle',- 'tail': 'right_foot_index',+ "head": "right_ankle",+ "tail": "right_foot_index",},"toe.R": {- 'head': 'right_foot_index',- 'tail': 'right_heel',+ "head": "right_foot_index",+ "tail": "right_heel",},"pelvis.L": {- 'head': 'hip_center',- 'tail': 'left_hip',+ "head": "hip_center",+ "tail": "left_hip",},"pelvis.R": {- 'head': 'hip_center',- 'tail': 'right_hip',+ "head": "hip_center",+ "tail": "right_hip",},"spine": {- 'head': 'hip_center',- 'tail': 'chest_center',+ "head": "hip_center",+ "tail": "chest_center",},"spine.001": {- 'head': 'chest_center',- 'tail': 'neck_center',+ "head": "chest_center",+ "tail": "neck_center",},"spine.002": {- 'head': 'neck_center',- 'tail': 'head_center',+ "head": "neck_center",+ "tail": "head_center",},"spine.003": {- 'head': 'head_center',- 'tail': 'nose',+ "head": "head_center",+ "tail": "nose",},"palm.01.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_index_finger_mcp',+ "head": "right_wrist",+ "tail": "right_hand_index_finger_mcp",},"palm.02.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_middle_finger_mcp',+ "head": "right_wrist",+ "tail": "right_hand_middle_finger_mcp",},"palm.03.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_ring_finger_mcp',+ "head": "right_wrist",+ "tail": "right_hand_ring_finger_mcp",},"palm.04.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_pinky_finger_mcp',+ "head": "right_wrist",+ "tail": "right_hand_pinky_finger_mcp",},-"thumb.01.R": {- 'head': 'right_wrist',- 'tail': 'right_hand_thumb_cmc',+ "head": "right_wrist",+ "tail": "right_hand_thumb_cmc",},"thumb.02.R": {- 'head': 'right_hand_thumb_cmc',- 'tail': 'right_hand_thumb_mcp',+ "head": "right_hand_thumb_cmc",+ "tail": "right_hand_thumb_mcp",},"thumb.03.R": {- 'head': 'right_hand_thumb_mcp',- 'tail': 'right_hand_thumb_ip',+ "head": "right_hand_thumb_mcp",+ "tail": "right_hand_thumb_ip",},"thumb.04.R": {- 'head': 'right_hand_thumb_ip',- 'tail': 'right_hand_thumb_tip',+ "head": "right_hand_thumb_ip",+ "tail": "right_hand_thumb_tip",},"f_index.01.R": {- 'head': 'right_hand_index_finger_mcp',- 'tail': 'right_hand_index_finger_pip',+ "head": "right_hand_index_finger_mcp",+ "tail": "right_hand_index_finger_pip",},"f_index.02.R": {- 'head': 'right_hand_index_finger_pip',- 'tail': 'right_hand_index_finger_dip',+ "head": "right_hand_index_finger_pip",+ "tail": "right_hand_index_finger_dip",},"f_index.03.R": {- 'head': 'right_hand_index_finger_dip',- 'tail': 'right_hand_index_finger_tip',+ "head": "right_hand_index_finger_dip",+ "tail": "right_hand_index_finger_tip",},"f_middle.01.R": {- 'head': 'right_hand_middle_finger_mcp',- 'tail': 'right_hand_middle_finger_pip',+ "head": "right_hand_middle_finger_mcp",+ "tail": "right_hand_middle_finger_pip",},"f_middle.02.R": {- 'head': 'right_hand_middle_finger_pip',- 'tail': 'right_hand_middle_finger_dip',+ "head": "right_hand_middle_finger_pip",+ "tail": "right_hand_middle_finger_dip",},"f_middle.03.R": {- 'head': 'right_hand_middle_finger_dip',- 'tail': 'right_hand_middle_finger_tip',+ "head": "right_hand_middle_finger_dip",+ "tail": "right_hand_middle_finger_tip",},"f_ring.01.R": {- 'head': 'right_hand_ring_finger_mcp',- 'tail': 'right_hand_ring_finger_pip',+ "head": "right_hand_ring_finger_mcp",+ "tail": "right_hand_ring_finger_pip",},"f_ring.02.R": {- 'head': 'right_hand_ring_finger_pip',- 'tail': 'right_hand_ring_finger_dip',+ "head": "right_hand_ring_finger_pip",+ "tail": "right_hand_ring_finger_dip",},"f_ring.03.R": {- 'head': 'right_hand_ring_finger_dip',- 'tail': 'right_hand_ring_finger_tip',+ "head": "right_hand_ring_finger_dip",+ "tail": "right_hand_ring_finger_tip",},-"f_pinky.01.R": {- 'head': 'right_hand_pinky_finger_mcp',- 'tail': 'right_hand_pinky_finger_pip',+ "head": "right_hand_pinky_finger_mcp",+ "tail": "right_hand_pinky_finger_pip",},"f_pinky.02.R": {- 'head': 'right_hand_pinky_finger_pip',- 'tail': 'right_hand_pinky_finger_dip',+ "head": "right_hand_pinky_finger_pip",+ "tail": "right_hand_pinky_finger_dip",},"f_pinky.03.R": {- 'head': 'right_hand_pinky_finger_dip',- 'tail': 'right_hand_pinky_finger_tip',+ "head": "right_hand_pinky_finger_dip",+ "tail": "right_hand_pinky_finger_tip",},"thumb.01.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_thumb_cmc',+ "head": "left_wrist",+ "tail": "left_hand_thumb_cmc",},"thumb.02.L": {- 'head': 'left_hand_thumb_cmc',- 'tail': 'left_hand_thumb_mcp',+ "head": "left_hand_thumb_cmc",+ "tail": "left_hand_thumb_mcp",},"thumb.03.L": {- 'head': 'left_hand_thumb_mcp',- 'tail': 'left_hand_thumb_ip',+ "head": "left_hand_thumb_mcp",+ "tail": "left_hand_thumb_ip",},"thumb.04.L": {- 'head': 'left_hand_thumb_ip',- 'tail': 'left_hand_thumb_tip',+ "head": "left_hand_thumb_ip",+ "tail": "left_hand_thumb_tip",},"palm.01.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_index_finger_mcp',+ "head": "left_wrist",+ "tail": "left_hand_index_finger_mcp",},"palm.02.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_middle_finger_mcp',+ "head": "left_wrist",+ "tail": "left_hand_middle_finger_mcp",},"palm.03.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_ring_finger_mcp',+ "head": "left_wrist",+ "tail": "left_hand_ring_finger_mcp",},"palm.04.L": {- 'head': 'left_wrist',- 'tail': 'left_hand_pinky_finger_mcp',+ "head": "left_wrist",+ "tail": "left_hand_pinky_finger_mcp",},"f_index.01.L": {- 'head': 'left_hand_index_finger_mcp',- 'tail': 'left_hand_index_finger_pip',+ "head": "left_hand_index_finger_mcp",+ "tail": "left_hand_index_finger_pip",},"f_index.02.L": {- 'head': 'left_hand_index_finger_pip',- 'tail': 'left_hand_index_finger_dip',+ "head": "left_hand_index_finger_pip",+ "tail": "left_hand_index_finger_dip",},"f_index.03.L": {- 'head': 'left_hand_index_finger_dip',- 'tail': 'left_hand_index_finger_tip',+ "head": "left_hand_index_finger_dip",+ "tail": "left_hand_index_finger_tip",},"f_middle.01.L": {- 'head': 'left_hand_middle_finger_mcp',- 'tail': 'left_hand_middle_finger_pip',+ "head": "left_hand_middle_finger_mcp",+ "tail": "left_hand_middle_finger_pip",},"f_middle.02.L": {- 'head': 'left_hand_middle_finger_pip',- 'tail': 'left_hand_middle_finger_dip',+ "head": "left_hand_middle_finger_pip",+ "tail": "left_hand_middle_finger_dip",},"f_middle.03.L": {- 'head': 'left_hand_middle_finger_dip',- 'tail': 'left_hand_middle_finger_tip',+ "head": "left_hand_middle_finger_dip",+ "tail": "left_hand_middle_finger_tip",},"f_ring.01.L": {- 'head': 'left_hand_ring_finger_mcp',- 'tail': 'left_hand_ring_finger_pip',+ "head": "left_hand_ring_finger_mcp",+ "tail": "left_hand_ring_finger_pip",},"f_ring.02.L": {- 'head': 'left_hand_ring_finger_pip',- 'tail': 'left_hand_ring_finger_dip',+ "head": "left_hand_ring_finger_pip",+ "tail": "left_hand_ring_finger_dip",},"f_ring.03.L": {- 'head': 'left_hand_ring_finger_dip',- 'tail': 'left_hand_ring_finger_tip',+ "head": "left_hand_ring_finger_dip",+ "tail": "left_hand_ring_finger_tip",},"f_pinky.01.L": {- 'head': 'left_hand_pinky_finger_mcp',- 'tail': 'left_hand_pinky_finger_pip',+ "head": "left_hand_pinky_finger_mcp",+ "tail": "left_hand_pinky_finger_pip",},"f_pinky.02.L": {- 'head': 'left_hand_pinky_finger_pip',- 'tail': 'left_hand_pinky_finger_dip',+ "head": "left_hand_pinky_finger_pip",+ "tail": "left_hand_pinky_finger_dip",},"f_pinky.03.L": {- 'head': 'left_hand_pinky_finger_dip',- 'tail': 'left_hand_pinky_finger_tip',+ "head": "left_hand_pinky_finger_dip",+ "tail": "left_hand_pinky_finger_tip",},}mediapipe_virtual_markers_weights_dict = {- 'head_center': {- 'right_ear': .5,- 'left_ear': .5- },- 'neck_center': {- 'right_shoulder': .5,- 'left_shoulder': .5- },- 'hip_center': {- 'right_hip': .5,- 'left_hip': .5- },- 'chest_center': {- 'right_shoulder': .25,- 'left_shoulder': .25,- 'right_hip': .25,- 'left_hip': .25,+ "head_center": {"right_ear": 0.5, "left_ear": 0.5},+ "neck_center": {"right_shoulder": 0.5, "left_shoulder": 0.5},+ "hip_center": {"right_hip": 0.5, "left_hip": 0.5},+ "chest_center": {+ "right_shoulder": 0.25,+ "left_shoulder": 0.25,+ "right_hip": 0.25,+ "left_hip": 0.25,},}- print('loading data as stick figure mesh \o/')+ print("loading data as stick figure mesh \o/")######################################################################## %% create virtual marker points (redundant from above, I think ? )@@ -1472,17 +1525,23 @@ try:this_weight = this_weighted_point_dict[1]these_weighted_points.append(mediapipe_skel_fr_mar_dim[- :,- mediapipe_tracked_point_names.index(this_point_name),- :- ] * this_weight+ :, mediapipe_tracked_point_names.index(this_point_name), :+ ]+ * this_weight)this_virtual_point_fr_xyz = np.sum(these_weighted_points, axis=0)- this_virtual_point_fr_mar_xyz = np.expand_dims(this_virtual_point_fr_xyz, axis=1)- mediapipe_skel_fr_mar_dim = np.append(mediapipe_skel_fr_mar_dim, this_virtual_point_fr_mar_xyz, axis=1)- virtual_point_index_dict[this_virtual_marker_name] = mediapipe_skel_fr_mar_dim.shape[1] - 1+ this_virtual_point_fr_mar_xyz = np.expand_dims(+ this_virtual_point_fr_xyz, axis=1+ )+ mediapipe_skel_fr_mar_dim = np.append(+ mediapipe_skel_fr_mar_dim, this_virtual_point_fr_mar_xyz, axis=1+ )+ virtual_point_index_dict[this_virtual_marker_name] = (+ mediapipe_skel_fr_mar_dim.shape[1] - 1+ )print(- f'Virtual marker: {this_virtual_marker_name} created at marker index {mediapipe_skel_fr_mar_dim.shape[1] - 1}')+ f"Virtual marker: {this_virtual_marker_name} created at marker index {mediapipe_skel_fr_mar_dim.shape[1] - 1}"+ )# %% define edges@@ -1490,29 +1549,34 @@ try:# regular datafor this_bone_dict in stick_figure_mesh_edge_pairs_dict_of_dicts.items():edge_name = this_bone_dict[0]- head_name = this_bone_dict[1]['head']- tail_name = this_bone_dict[1]['tail']+ head_name = this_bone_dict[1]["head"]+ tail_name = this_bone_dict[1]["tail"]try:head_index = mediapipe_tracked_point_names.index(head_name)tail_index = mediapipe_tracked_point_names.index(tail_name)- if 'center' in head_name: # virtual markers are acting weird so we're brute forcing it :-/ this will fail if there is a virtual marker without 'center' in the name....+ if (+ "center" in head_name+ ): # virtual markers are acting weird so we're brute forcing it :-/ this will fail if there is a virtual marker without 'center' in the name....head_index = virtual_point_index_dict[head_name]- if 'center' in tail_name:+ if "center" in tail_name:tail_index = virtual_point_index_dict[tail_name]this_edge = (head_index, tail_index)edges.append(this_edge)print(- f'edge created for bone: ({this_bone_dict[0]} : {head_name}-{tail_name}), indicies({this_edge[0]},{this_edge[1]})')+ f"edge created for bone: ({this_bone_dict[0]} : {head_name}-{tail_name}), indicies({this_edge[0]},{this_edge[1]})"+ )except:- print(f'edge FAILED for for bone:{this_bone_dict[0]} : {head_name}-{tail_name}')+ print(+ f"edge FAILED for for bone:{this_bone_dict[0]} : {head_name}-{tail_name}"+ )pass- print('edges created!')+ print("edges created!")# ## # ############################### # # %% Create mesh@@ -1575,31 +1639,41 @@ try:####################### Load nSynched Videostry:- print('loading videos as planes...')- annotated_videos_path = session_path / 'mediapipe_annotated_videos'+ print("loading videos as planes...")+ annotated_videos_path = session_path / "mediapipe_annotated_videos"if annotated_videos_path.is_dir():vidFolderPath = annotated_videos_pathelse:- vidFolderPath = session_path / 'synchronized_videos'+ vidFolderPath = session_path / "synchronized_videos"- world_origin = bpy.data.objects['world_origin']+ world_origin = bpy.data.objects["world_origin"]- number_of_videos = len(list(vidFolderPath.glob('*.mp4')))+ number_of_videos = len(list(vidFolderPath.glob("*.mp4")))vid_location_scale = 1- for vid_number, thisVidPath, in enumerate(vidFolderPath.glob('*.mp4')):+ for (+ vid_number,+ thisVidPath,+ ) in enumerate(vidFolderPath.glob("*.mp4")):print(thisVidPath)# use 'images as planes' add on to load in the video files as planes- bpy.ops.import_image.to_plane(files=[{"name": thisVidPath.name}], directory=str(thisVidPath.parent),- shader='EMISSION', )+ bpy.ops.import_image.to_plane(+ files=[{"name": thisVidPath.name}],+ directory=str(thisVidPath.parent),+ shader="EMISSION",+ )this_vid_as_plane = bpy.context.active_object- this_vid_as_plane.name = 'vid_' + str(vid_number)+ this_vid_as_plane.name = "vid_" + str(vid_number)vid_x = (vid_number - number_of_videos / 2) * vid_location_scale- this_vid_as_plane.location = [vid_x, vid_location_scale, vid_location_scale]+ this_vid_as_plane.location = [+ vid_x,+ vid_location_scale,+ vid_location_scale,+ ]this_vid_as_plane.rotation_euler = [np.pi / 2, 0, 0]this_vid_as_plane.scale = [vid_location_scale * 1.5] * 3this_vid_as_plane.parent = world_origin@@ -1607,16 +1681,18 @@ try:# bpy.ops.object.light_add(type='POINT', radius=1, align='WORLD')except:print(- 'Failed to load videos to Blender scene - You might need to install the "images as planes" addon to this version of Blender')+ 'Failed to load videos to Blender scene - You might need to install the "images as planes" addon to this version of Blender'+ )except Exception as e:print(e)print(- 'Something broke in the Blender Megascript - Maybe it failed to find a good clean frame and then that borked something downstream?')+ "Something broke in the Blender Megascript - Maybe it failed to find a good clean frame and then that borked something downstream?"+ )# set shading type to 'Material Preview' so you can see the videos# save .blend filesessionID = session_path.stem-blend_file_save_path = session_path / (sessionID + '.blend')+blend_file_save_path = session_path / (sessionID + ".blend")print(f"Saving Blender output file to: {str(blend_file_save_path)}")-bpy.ops.wm.save_as_mainfile(filepath=str(blend_file_save_path))\ No newline at end of file+bpy.ops.wm.save_as_mainfile(filepath=str(blend_file_save_path))diff --git a/src/open_in_blender/open_session_in_blender.py b/src/open_in_blender/open_session_in_blender.pyindex d6eb21ba..eee14b77 100644--- a/src/open_in_blender/open_session_in_blender.py+++ b/src/open_in_blender/open_session_in_blender.py@@ -8,28 +8,36 @@ from src.config.home_dir import get_session_folder_path, get_most_recent_sessionblender_exe_path = r"C:\Users\jonma\Blender Foundation\Blender 3.1\blender.exe"-def open_session_in_blender(session_id:str):+def open_session_in_blender(session_id: str):create_session(session_id)- blender_file_name = session_id + '.blend'- blender_file_path = Path(get_session_folder_path(session_id)) / blender_file_name+ blender_file_name = session_id + ".blend"+ blender_file_path = Path(get_session_folder_path(session_id)) / blender_file_nameos.startfile(str(blender_file_path))-def create_session(session_id:str, good_clean_frame_number:int = 0):++def create_session(session_id: str, good_clean_frame_number: int = 0):path_to_this_py_file = Path(__file__).parent.resolve()- freemocap_blender_megascript_path = path_to_this_py_file / 'alpha_freemocap_blender_megascript.py'+ freemocap_blender_megascript_path = (+ path_to_this_py_file / "alpha_freemocap_blender_megascript.py"+ )print(str(freemocap_blender_megascript_path))print(f"sending {session_id} data to Blender")- command_str = str(blender_exe_path) + " --background" + " --python " + str(freemocap_blender_megascript_path) + " -- " +\- get_session_folder_path(session_id) + ' ' + str(good_clean_frame_number)+ command_str = (+ str(blender_exe_path)+ + " --background"+ + " --python "+ + str(freemocap_blender_megascript_path)+ + " -- "+ + get_session_folder_path(session_id)+ + " "+ + str(good_clean_frame_number)+ )print(command_str)blender_process = subprocess.Popen(- command_str,- shell=False,- stdout=subprocess.PIPE,- stderr=subprocess.PIPE+ command_str, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)while True:output = blender_process.stdout.readline()@@ -43,5 +51,6 @@ def create_session(session_id:str, good_clean_frame_number:int = 0):print(blender_process.stderr.read().decode())print(f"done with blender stuff :D")+if __name__ == "__main__":- open_session_in_blender(get_most_recent_session_id())\ No newline at end of file+ open_session_in_blender(get_most_recent_session_id())diff --git a/src/pipelines/calibration_pipeline/anipose_camera_calibration/anipose_camera_calibration.py b/src/pipelines/calibration_pipeline/anipose_camera_calibration/anipose_camera_calibration.pyindex 7ebb7cea..9556cc87 100644--- a/src/pipelines/calibration_pipeline/anipose_camera_calibration/anipose_camera_calibration.py+++ b/src/pipelines/calibration_pipeline/anipose_camera_calibration/anipose_camera_calibration.py@@ -6,80 +6,118 @@ import numpy as npfrom cv2 import aruco_CharucoBoardfrom scipy.spatial.transform import Rotation-from src.config.home_dir import get_session_folder_path, get_freemocap_data_folder_path, \- get_calibration_videos_folder_path-from src.pipelines.calibration_pipeline.anipose_camera_calibration import freemocap_anipose+from src.config.home_dir import (+ get_session_folder_path,+ get_freemocap_data_folder_path,+ get_calibration_videos_folder_path,+)+from src.pipelines.calibration_pipeline.anipose_camera_calibration import (+ freemocap_anipose,+)from aniposelib.boards import CharucoBoard as AniposeCharucoBoard-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import \- CharucoBoardDataClass+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import (+ CharucoBoardDataClass,+)logger = logging.getLogger(__name__)class AniposeCameraCalibrator:- def __init__(self,- session_id: str,- charuco_board_object: CharucoBoardDataClass,- charuco_square_size: Union[int, float] = 1,- ):+ def __init__(+ self,+ session_id: str,+ charuco_board_object: CharucoBoardDataClass,+ charuco_square_size: Union[int, float] = 1,+ ):self._session_id = session_idself._charuco_board_object = charuco_board_objectif charuco_square_size == 1:logger.warning(- 'Charuco square size is not set, so units of 3d reconstructed data will be in units of `however_long_the_black_edge_of_the_charuco_square_was`. Please input `charuco_square_size` in millimeters (or your preferred unity of length)')+ "Charuco square size is not set, so units of 3d reconstructed data will be in units of `however_long_the_black_edge_of_the_charuco_square_was`. Please input `charuco_square_size` in millimeters (or your preferred unity of length)"+ )self._charuco_square_size = charuco_square_sizeself.get_paths_and_whatnot()self.initialize_anipose_objects()def get_paths_and_whatnot(self):self.session_folder_path = Path(get_session_folder_path(self._session_id))- calibration_videos_folder = Path(get_calibration_videos_folder_path(self._session_id))- self._list_of_video_paths = [this_video_path for this_video_path in- calibration_videos_folder.glob('*.mp4'.lower())]+ calibration_videos_folder = Path(+ get_calibration_videos_folder_path(self._session_id)+ )+ self._list_of_video_paths = [+ this_video_path+ for this_video_path in calibration_videos_folder.glob("*.mp4".lower())+ ]def initialize_anipose_objects(self):- list_of_camera_names = [this_video_path.stem for this_video_path in self._list_of_video_paths]- self._anipose_camera_group_object = freemocap_anipose.CameraGroup.from_names(list_of_camera_names)- self._anipose_camera_group_object.metadata['session_id'] = self._session_id- self._anipose_camera_group_object.metadata['charuco_square_size'] = self._charuco_square_size- self._anipose_camera_group_object.metadata['charuco_board_object'] = str(self._charuco_board_object)-- self._anipose_charuco_board = AniposeCharucoBoard(self._charuco_board_object.number_of_squares_width,- self._charuco_board_object.number_of_squares_height,- square_length=self._charuco_square_size, # mm- marker_length=self._charuco_square_size * .8,- marker_bits=4,- dict_size=250)+ list_of_camera_names = [+ this_video_path.stem for this_video_path in self._list_of_video_paths+ ]+ self._anipose_camera_group_object = freemocap_anipose.CameraGroup.from_names(+ list_of_camera_names+ )+ self._anipose_camera_group_object.metadata["session_id"] = self._session_id+ self._anipose_camera_group_object.metadata[+ "charuco_square_size"+ ] = self._charuco_square_size+ self._anipose_camera_group_object.metadata["charuco_board_object"] = str(+ self._charuco_board_object+ )++ self._anipose_charuco_board = AniposeCharucoBoard(+ self._charuco_board_object.number_of_squares_width,+ self._charuco_board_object.number_of_squares_height,+ square_length=self._charuco_square_size, # mm+ marker_length=self._charuco_square_size * 0.8,+ marker_bits=4,+ dict_size=250,+ )def calibrate_camera_capture_volume(self, pin_camera_0_to_origin: bool = False):# anipose needs this to be a list of lists (which is annoying but whatevs)- video_paths_list_of_list_of_strings = [[str(this_path)] for this_path in self._list_of_video_paths]-- error, charuco_frame_data, charuco_frame_numbers = self._anipose_camera_group_object.calibrate_videos(- video_paths_list_of_list_of_strings,- self._anipose_charuco_board)+ video_paths_list_of_list_of_strings = [+ [str(this_path)] for this_path in self._list_of_video_paths+ ]++ (+ error,+ charuco_frame_data,+ charuco_frame_numbers,+ ) = self._anipose_camera_group_object.calibrate_videos(+ video_paths_list_of_list_of_strings, self._anipose_charuco_board+ )logger.info("Anipose Calibration Successful!")if pin_camera_0_to_origin:# translate cameras so camera0 is on `0,0,0`- self._anipose_camera_group_object = self.pin_camera_zero_to_origin(self._anipose_camera_group_object)+ self._anipose_camera_group_object = self.pin_camera_zero_to_origin(+ self._anipose_camera_group_object+ )# self._anipose_camera_group_object = self.rotate_cameras_so_camera_zero_aligns_with_XYZ(self._anipose_camera_group_object)# save calibration info to filescalibration_toml_filename = f"{self._session_id}_camera_calibration.toml"- camera_calibration_toml_path = self.session_folder_path / calibration_toml_filename+ camera_calibration_toml_path = (+ self.session_folder_path / calibration_toml_filename+ )self._anipose_camera_group_object.dump(camera_calibration_toml_path)- logger.info(f"anipose camera calibration data saved to {str(camera_calibration_toml_path)}")+ logger.info(+ f"anipose camera calibration data saved to {str(camera_calibration_toml_path)}"+ )- last_successful_calibration_path = Path(get_freemocap_data_folder_path(), "last_successful_calibration.toml")+ last_successful_calibration_path = Path(+ get_freemocap_data_folder_path(), "last_successful_calibration.toml"+ )self._anipose_camera_group_object.dump(last_successful_calibration_path)- logger.info(f"anipose camera calibration data also saved to {str(last_successful_calibration_path)}")+ logger.info(+ f"anipose camera calibration data also saved to {str(last_successful_calibration_path)}"+ )return self._anipose_camera_group_object# # convert charuco data into a format that can be 3d reconstructed (effectively providing dummy data for the rest of the 3d reconstruction pipeline)@@ -101,16 +139,22 @@ class AniposeCameraCalibrator:camera_0_translation = original_translation_vectors[0, :]altered_translation_vectors = np.zeros(original_translation_vectors.shape)for this_camera_number in range(original_translation_vectors.shape[0]):- altered_translation_vectors[this_camera_number, :] = original_translation_vectors[this_camera_number,- :] - camera_0_translation+ altered_translation_vectors[this_camera_number, :] = (+ original_translation_vectors[this_camera_number, :]+ - camera_0_translation+ )_anipose_camera_group_object.set_translations(altered_translation_vectors)logger.info(f"original translation vectors: {original_translation_vectors}")- logger.info(f"altered translation vectors: {_anipose_camera_group_object.get_translations()}")+ logger.info(+ f"altered translation vectors: {_anipose_camera_group_object.get_translations()}"+ )return _anipose_camera_group_object- def rotate_cameras_so_camera_zero_aligns_with_XYZ(self, _anipose_camera_group_object):- logger.warning('this function does not work, dont use it lol ')+ def rotate_cameras_so_camera_zero_aligns_with_XYZ(+ self, _anipose_camera_group_object+ ):+ logger.warning("this function does not work, dont use it lol ")pass# original_rotations_euler = _anipose_camera_group_object.get_rotations()# original_translation_vectors = _anipose_camera_group_object.get_translations()diff --git a/src/pipelines/calibration_pipeline/anipose_camera_calibration/freemocap_anipose.py b/src/pipelines/calibration_pipeline/anipose_camera_calibration/freemocap_anipose.pyindex fc252730..bf2f081a 100644--- a/src/pipelines/calibration_pipeline/anipose_camera_calibration/freemocap_anipose.py+++ b/src/pipelines/calibration_pipeline/anipose_camera_calibration/freemocap_anipose.py@@ -15,14 +15,18 @@ from rich import printimport time-from aniposelib.boards import merge_rows, extract_points, \- extract_rtvecs, get_video_params-from aniposelib.utils import get_initial_extrinsics, make_M, get_rtvec, \- get_connections--numba_logger = logging.getLogger('numba')+from aniposelib.boards import (+ merge_rows,+ extract_points,+ extract_rtvecs,+ get_video_params,+)+from aniposelib.utils import get_initial_extrinsics, make_M, get_rtvec, get_connections++numba_logger = logging.getLogger("numba")numba_logger.setLevel(logging.WARNING)+@jit(nopython=True, parallel=True)def triangulate_simple(points, camera_mats):num_cams = len(camera_mats)@@ -30,8 +34,8 @@ def triangulate_simple(points, camera_mats):for i in range(num_cams):x, y = points[i]mat = camera_mats[i]- A[(i * 2):(i * 2 + 1)] = x * mat[2] - mat[0]- A[(i * 2 + 1):(i * 2 + 2)] = y * mat[2] - mat[1]+ A[(i * 2) : (i * 2 + 1)] = x * mat[2] - mat[0]+ A[(i * 2 + 1) : (i * 2 + 2)] = y * mat[2] - mat[1]u, s, vh = np.linalg.svd(A, full_matrices=True)p3d = vh[-1]p3d = p3d[:3] / p3d[3]@@ -69,36 +73,38 @@ def subset_extra(extra, ixs):return Nonenew_extra = {- 'objp': extra['objp'][ixs],- 'ids': extra['ids'][ixs],- 'rvecs': extra['rvecs'][:, ixs],- 'tvecs': extra['tvecs'][:, ixs]+ "objp": extra["objp"][ixs],+ "ids": extra["ids"][ixs],+ "rvecs": extra["rvecs"][:, ixs],+ "tvecs": extra["tvecs"][:, ixs],}return new_extradef resample_points_extra(imgp, extra, n_samp=25):n_cams, n_points, _ = imgp.shape- ids = remap_ids(extra['ids'])+ ids = remap_ids(extra["ids"])n_ids = np.max(ids) + 1good = ~np.isnan(imgp[:, :, 0])ixs = np.arange(n_points)- cam_counts = np.zeros((n_ids, n_cams), dtype='int64')+ cam_counts = np.zeros((n_ids, n_cams), dtype="int64")for idnum in range(n_ids):cam_counts[idnum] = np.sum(good[:, ids == idnum], axis=1)cam_counts_random = cam_counts + np.random.random(size=cam_counts.shape)best_boards = np.argsort(-cam_counts_random, axis=0)- cam_totals = np.zeros(n_cams, dtype='int64')+ cam_totals = np.zeros(n_cams, dtype="int64")include = set()for cam_num in range(n_cams):for board_id in best_boards[:, cam_num]:include.update(ixs[ids == board_id])cam_totals += cam_counts[board_id]- if cam_totals[cam_num] >= n_samp or \- cam_counts_random[board_id, cam_num] < 1:+ if (+ cam_totals[cam_num] >= n_samp+ or cam_counts_random[board_id, cam_num] < 1+ ):breakfinal_ixs = sorted(include)@@ -125,7 +131,7 @@ def resample_points(imgp, extra=None, n_samp=25):n_good = np.sum(subset)if n_good > 0:## pick points, prioritizing points seen by more cameras- arr = np.copy(num_cams[subset]).astype('float64')+ arr = np.copy(num_cams[subset]).astype("float64")arr += np.random.random(size=arr.shape)picked_ix = np.argsort(-arr)[:n_samp]picked = ixs[subset][picked_ix]@@ -139,7 +145,7 @@ def resample_points(imgp, extra=None, n_samp=25):def medfilt_data(values, size=15):padsize = size + 5- vpad = np.pad(values, (padsize, padsize), mode='reflect')+ vpad = np.pad(values, (padsize, padsize), mode="reflect")vpadf = signal.medfilt(vpad, kernel_size=size)return vpadf[padsize:-padsize]@@ -171,29 +177,31 @@ def transform_points(points, rvecs, tvecs):Rodrigues' rotation formula is used."""theta = np.linalg.norm(rvecs, axis=1)[:, np.newaxis]- with np.errstate(invalid='ignore'):+ with np.errstate(invalid="ignore"):v = rvecs / thetav = np.nan_to_num(v)dot = np.sum(points * v, axis=1)[:, np.newaxis]cos_theta = np.cos(theta)sin_theta = np.sin(theta)- rotated = cos_theta * points + \- sin_theta * np.cross(v, points) + \- dot * (1 - cos_theta) * v+ rotated = (+ cos_theta * points + sin_theta * np.cross(v, points) + dot * (1 - cos_theta) * v+ )return rotated + tvecsclass Camera:- def __init__(self,- matrix=np.eye(3),- dist=np.zeros(5),- size=None,- rvec=np.zeros(3),- tvec=np.zeros(3),- name=None,- extra_dist=False):+ def __init__(+ self,+ matrix=np.eye(3),+ dist=np.zeros(5),+ size=None,+ rvec=np.zeros(3),+ tvec=np.zeros(3),+ name=None,+ extra_dist=False,+ ):self.set_camera_matrix(matrix)self.set_distortions(dist)@@ -205,21 +213,21 @@ class Camera:def get_dict(self):return {- 'name': self.get_name(),- 'size': list(self.get_size()),- 'matrix': self.get_camera_matrix().tolist(),- 'distortions': self.get_distortions().tolist(),- 'rotation': self.get_rotation().tolist(),- 'translation': self.get_translation().tolist(),+ "name": self.get_name(),+ "size": list(self.get_size()),+ "matrix": self.get_camera_matrix().tolist(),+ "distortions": self.get_distortions().tolist(),+ "rotation": self.get_rotation().tolist(),+ "translation": self.get_translation().tolist(),}def load_dict(self, d):- self.set_camera_matrix(d['matrix'])- self.set_rotation(d['rotation'])- self.set_translation(d['translation'])- self.set_distortions(d['distortions'])- self.set_name(d['name'])- self.set_size(d['size'])+ self.set_camera_matrix(d["matrix"])+ self.set_rotation(d["rotation"])+ self.set_translation(d["translation"])+ self.set_distortions(d["distortions"])+ self.set_name(d["name"])+ self.set_size(d["size"])def from_dict(d):cam = Camera()@@ -233,7 +241,7 @@ class Camera:return self.distdef set_camera_matrix(self, matrix):- self.matrix = np.array(matrix, dtype='float64')+ self.matrix = np.array(matrix, dtype="float64")def set_focal_length(self, fx, fy=None):if fy is None:@@ -250,16 +258,16 @@ class Camera:return (fx + fy) / 2.0def set_distortions(self, dist):- self.dist = np.array(dist, dtype='float64').ravel()+ self.dist = np.array(dist, dtype="float64").ravel()def set_rotation(self, rvec):- self.rvec = np.array(rvec, dtype='float64').ravel()+ self.rvec = np.array(rvec, dtype="float64").ravel()def get_rotation(self):return self.rvecdef set_translation(self, tvec):- self.tvec = np.array(tvec, dtype='float64').ravel()+ self.tvec = np.array(tvec, dtype="float64").ravel()def get_translation(self):return self.tvec@@ -292,7 +300,7 @@ class Camera:self.set_camera_matrix(new_matrix)def get_params(self):- params = np.zeros(8 + self.extra_dist, dtype='float64')+ params = np.zeros(8 + self.extra_dist, dtype="float64")params[0:3] = self.get_rotation()params[3:6] = self.get_translation()params[6] = self.get_focal_length()@@ -307,7 +315,7 @@ class Camera:self.set_translation(params[3:6])self.set_focal_length(params[6])- dist = np.zeros(5, dtype='float64')+ dist = np.zeros(5, dtype="float64")dist[0] = params[7]if self.extra_dist:dist[1] = params[8]@@ -317,24 +325,32 @@ class Camera:shape = points.shapepoints = points.reshape(-1, 1, 2)new_points = np.dstack([points, np.ones((points.shape[0], 1, 1))])- out, _ = cv2.projectPoints(new_points, np.zeros(3), np.zeros(3),- self.matrix.astype('float64'),- self.dist.astype('float64'))+ out, _ = cv2.projectPoints(+ new_points,+ np.zeros(3),+ np.zeros(3),+ self.matrix.astype("float64"),+ self.dist.astype("float64"),+ )return out.reshape(shape)def undistort_points(self, points):shape = points.shapepoints = points.reshape(-1, 1, 2)- out = cv2.undistortPoints(points,- self.matrix.astype('float64'),- self.dist.astype('float64'))+ out = cv2.undistortPoints(+ points, self.matrix.astype("float64"), self.dist.astype("float64")+ )return out.reshape(shape)def project(self, points):points = points.reshape(-1, 1, 3)- out, _ = cv2.projectPoints(points, self.rvec, self.tvec,- self.matrix.astype('float64'),- self.dist.astype('float64'))+ out, _ = cv2.projectPoints(+ points,+ self.rvec,+ self.tvec,+ self.matrix.astype("float64"),+ self.dist.astype("float64"),+ )return outdef reprojection_error(self, p3d, p2d):@@ -342,25 +358,28 @@ class Camera:return p2d - projdef copy(self):- return \- Camera(matrix=self.get_camera_matrix().copy(),- dist=self.get_distortions().copy(),- size=self.get_size(),- rvec=self.get_rotation().copy(),- tvec=self.get_translation().copy(),- name=self.get_name(),- extra_dist=self.extra_dist)+ return Camera(+ matrix=self.get_camera_matrix().copy(),+ dist=self.get_distortions().copy(),+ size=self.get_size(),+ rvec=self.get_rotation().copy(),+ tvec=self.get_translation().copy(),+ name=self.get_name(),+ extra_dist=self.extra_dist,+ )class FisheyeCamera(Camera):- def __init__(self,- matrix=np.eye(3),- dist=np.zeros(4),- size=None,- rvec=np.zeros(3),- tvec=np.zeros(3),- name=None,- extra_dist=False):+ def __init__(+ self,+ matrix=np.eye(3),+ dist=np.zeros(4),+ size=None,+ rvec=np.zeros(3),+ tvec=np.zeros(3),+ name=None,+ extra_dist=False,+ ):self.set_camera_matrix(matrix)self.set_distortions(dist)self.set_size(size)@@ -376,33 +395,41 @@ class FisheyeCamera(Camera):def get_dict(self):d = super().get_dict()- d['fisheye'] = True+ d["fisheye"] = Truereturn ddef distort_points(self, points):shape = points.shapepoints = points.reshape(-1, 1, 2)new_points = np.dstack([points, np.ones((points.shape[0], 1, 1))])- out, _ = cv2.fisheye.projectPoints(new_points,- np.zeros(3), np.zeros(3),- self.matrix.astype('float64'),- self.dist.astype('float64'))+ out, _ = cv2.fisheye.projectPoints(+ new_points,+ np.zeros(3),+ np.zeros(3),+ self.matrix.astype("float64"),+ self.dist.astype("float64"),+ )return out.reshape(shape)def undistort_points(self, points):shape = points.shapepoints = points.reshape(-1, 1, 2)- out = cv2.fisheye.undistortPoints(points.astype('float64'),- self.matrix.astype('float64'),- self.dist.astype('float64'))+ out = cv2.fisheye.undistortPoints(+ points.astype("float64"),+ self.matrix.astype("float64"),+ self.dist.astype("float64"),+ )return out.reshape(shape)def project(self, points):points = points.reshape(-1, 1, 3)- out, _ = cv2.fisheye.projectPoints(points,- self.rvec, self.tvec,- self.matrix.astype('float64'),- self.dist.astype('float64'))+ out, _ = cv2.fisheye.projectPoints(+ points,+ self.rvec,+ self.tvec,+ self.matrix.astype("float64"),+ self.dist.astype("float64"),+ )return outdef set_params(self, params):@@ -410,7 +437,7 @@ class FisheyeCamera(Camera):self.set_translation(params[3:6])self.set_focal_length(params[6])- dist = np.zeros(4, dtype='float64')+ dist = np.zeros(4, dtype="float64")dist[0] = params[7]if self.extra_dist:dist[1] = params[8]@@ -419,7 +446,7 @@ class FisheyeCamera(Camera):self.set_distortions(dist)def get_params(self):- params = np.zeros(8 + self.extra_dist, dtype='float64')+ params = np.zeros(8 + self.extra_dist, dtype="float64")params[0:3] = self.get_rotation()params[3:6] = self.get_translation()params[6] = self.get_focal_length()@@ -439,7 +466,8 @@ class FisheyeCamera(Camera):rvec=self.get_rotation().copy(),tvec=self.get_translation().copy(),name=self.get_name(),- extra_dist=self.extra_dist)+ extra_dist=self.extra_dist,+ )class CameraGroup:@@ -458,9 +486,8 @@ class CameraGroup:for name in names:if name not in cur_names_dict:raise IndexError(- "name {} not part of camera names: {}".format(- name, cur_names- ))+ "name {} not part of camera names: {}".format(name, cur_names)+ )indices.append(cur_names_dict[name])return self.subset_cameras(indices)@@ -471,7 +498,7 @@ class CameraGroup:n_points = points.shape[0]n_cams = len(self.cameras)- out = np.empty((n_cams, n_points, 2), dtype='float64')+ out = np.empty((n_cams, n_points, 2), dtype="float64")for cnum, cam in enumerate(self.cameras):out[cnum] = cam.project(points).reshape(n_points, 2)@@ -481,11 +508,12 @@ class CameraGroup:"""Given an CxNx2 array, this returns an Nx3 array of points,where N is the number of points and C is the number of cameras"""- assert points.shape[0] == len(self.cameras), \- "Invalid points shape, first dim should be equal to" \+ assert points.shape[0] == len(self.cameras), (+ "Invalid points shape, first dim should be equal to"" number of cameras ({}), but shape is {}".format(len(self.cameras), points.shape)+ )one_point = Falseif len(points.shape) == 2:@@ -523,8 +551,9 @@ class CameraGroup:return out- def triangulate_possible(self, points, undistort=True,- min_cams=2, progress=False, threshold=0.5):+ def triangulate_possible(+ self, points, undistort=True, min_cams=2, progress=False, threshold=0.5+ ):"""Given an CxNxPx2 array, this returns an Nx3 array of pointsby triangulating all possible points and picking the ones withbest reprojection error@@ -534,21 +563,22 @@ class CameraGroup:P: number of possible options per point"""- assert points.shape[0] == len(self.cameras), \- "Invalid points shape, first dim should be equal to" \+ assert points.shape[0] == len(self.cameras), (+ "Invalid points shape, first dim should be equal to"" number of cameras ({}), but shape is {}".format(len(self.cameras), points.shape)+ )n_cams, n_points, n_possible, _ = points.shape- cam_nums, point_nums, possible_nums = np.where(- ~np.isnan(points[:, :, :, 0]))+ cam_nums, point_nums, possible_nums = np.where(~np.isnan(points[:, :, :, 0]))all_iters = defaultdict(dict)- for cam_num, point_num, possible_num in zip(cam_nums, point_nums,- possible_nums):+ for cam_num, point_num, possible_num in zip(+ cam_nums, point_nums, possible_nums+ ):if cam_num not in all_iters[point_num]:all_iters[point_num][cam_num] = []all_iters[point_num][cam_num].append((cam_num, possible_num))@@ -557,10 +587,10 @@ class CameraGroup:for cam_num in all_iters[point_num].keys():all_iters[point_num][cam_num].append(None)- out = np.full((n_points, 3), np.nan, dtype='float64')- picked_vals = np.zeros((n_cams, n_points, n_possible), dtype='bool')- errors = np.zeros(n_points, dtype='float64')- points_2d = np.full((n_cams, n_points, 2), np.nan, dtype='float64')+ out = np.full((n_points, 3), np.nan, dtype="float64")+ picked_vals = np.zeros((n_cams, n_points, n_possible), dtype="bool")+ errors = np.zeros(n_points, dtype="float64")+ points_2d = np.full((n_cams, n_points, 2), np.nan, dtype="float64")if progress:iterator = trange(n_points, ncols=70)@@ -589,46 +619,46 @@ class CameraGroup:if err < best_error:best_point = {- 'error': err,- 'point': p3d[:3],- 'points': pts,- 'picked': picked,- 'joint_ix': point_ix+ "error": err,+ "point": p3d[:3],+ "points": pts,+ "picked": picked,+ "joint_ix": point_ix,}best_error = errif best_error < threshold:breakif best_point is not None:- out[point_ix] = best_point['point']- picked = best_point['picked']+ out[point_ix] = best_point["point"]+ picked = best_point["picked"]cnums = [p[0] for p in picked]xnums = [p[1] for p in picked]picked_vals[cnums, point_ix, xnums] = True- errors[point_ix] = best_point['error']- points_2d[cnums, point_ix] = best_point['points']+ errors[point_ix] = best_point["error"]+ points_2d[cnums, point_ix] = best_point["points"]# return out, picked_vals, points_2d, errors #original code from OG anipose- return out #simplify output so that `triangulate_ransac` can be used exactly the same way as `triangulate`+ return out # simplify output so that `triangulate_ransac` can be used exactly the same way as `triangulate`def triangulate_ransac(self, points, undistort=True, min_cams=2, progress=False):"""Given an CxNx2 array, this returns an Nx3 array of points,where N is the number of points and C is the number of cameras"""- assert points.shape[0] == len(self.cameras), \- "Invalid points shape, first dim should be equal to" \+ assert points.shape[0] == len(self.cameras), (+ "Invalid points shape, first dim should be equal to"" number of cameras ({}), but shape is {}".format(len(self.cameras), points.shape)+ )n_cams, n_points, _ = points.shapepoints_ransac = points.reshape(n_cams, n_points, 1, 2)- return self.triangulate_possible(points_ransac,- undistort=undistort,- min_cams=min_cams,- progress=progress)+ return self.triangulate_possible(+ points_ransac, undistort=undistort, min_cams=min_cams, progress=progress+ )@jit(parallel=True, forceobj=True)def reprojection_error(self, p3ds, p2ds, mean=False):@@ -644,9 +674,12 @@ class CameraGroup:one_point = Truen_cams, n_points, _ = p2ds.shape- assert p3ds.shape == (n_points, 3), \- "shapes of 2D and 3D points are not consistent: " \- "2D={}, 3D={}".format(p2ds.shape, p3ds.shape)+ assert p3ds.shape == (+ n_points,+ 3,+ ), "shapes of 2D and 3D points are not consistent: " "2D={}, 3D={}".format(+ p2ds.shape, p3ds.shape+ )errors = np.empty((n_cams, n_points, 2))@@ -657,7 +690,7 @@ class CameraGroup:errors_norm = np.linalg.norm(errors, axis=2)good = ~np.isnan(errors_norm)errors_norm[~good] = 0- denom = np.sum(good, axis=0).astype('float64')+ denom = np.sum(good, axis=0).astype("float64")denom[denom < 1.5] = np.nanerrors = np.sum(errors_norm, axis=0) / denom@@ -669,12 +702,20 @@ class CameraGroup:return errors- def bundle_adjust_iter(self, p2ds, extra=None,- n_iters=10, start_mu=15, end_mu=1,- max_nfev=200, ftol=1e-4,- n_samp_iter=100, n_samp_full=1000,- error_threshold=0.3,- verbose=False):+ def bundle_adjust_iter(+ self,+ p2ds,+ extra=None,+ n_iters=10,+ start_mu=15,+ end_mu=1,+ max_nfev=200,+ ftol=1e-4,+ n_samp_iter=100,+ n_samp_full=1000,+ error_threshold=0.3,+ verbose=False,+ ):"""Given an CxNx2 array of 2D points,where N is the number of points and C is the number of cameras,this performs iterative bundle adjustsment to fine-tune the parameters of the cameras.@@ -684,30 +725,29 @@ class CameraGroup:"""error_list = []- assert p2ds.shape[0] == len(self.cameras), \- "Invalid points shape, first dim should be equal to" \+ assert p2ds.shape[0] == len(self.cameras), (+ "Invalid points shape, first dim should be equal to"" number of cameras ({}), but shape is {}".format(len(self.cameras), p2ds.shape)+ )p2ds_full = p2dsextra_full = extra- p2ds, extra = resample_points(p2ds_full, extra_full,- n_samp=n_samp_full)+ p2ds, extra = resample_points(p2ds_full, extra_full, n_samp=n_samp_full)error = self.average_error(p2ds, median=True)if verbose:- print('error: ', error)+ print("error: ", error)mus = np.exp(np.linspace(np.log(start_mu), np.log(end_mu), num=n_iters))if verbose:- print('n_samples: {}'.format(n_samp_iter))+ print("n_samples: {}".format(n_samp_iter))for i in range(n_iters):- p2ds, extra = resample_points(p2ds_full, extra_full,- n_samp=n_samp_full)+ p2ds, extra = resample_points(p2ds_full, extra_full, n_samp=n_samp_full)p3ds = self.triangulate(p2ds)errors_full = self.reprojection_error(p3ds, p2ds, mean=False)errors_norm = self.reprojection_error(p3ds, p2ds, mean=True)@@ -724,7 +764,8 @@ class CameraGroup:good = errors_norm < muextra_good = subset_extra(extra, good)p2ds_samp, extra_samp = resample_points(- p2ds[:, good], extra_good, n_samp=n_samp_iter)+ p2ds[:, good], extra_good, n_samp=n_samp_iter+ )error = np.median(errors_norm)error_list.append(error)@@ -734,16 +775,23 @@ class CameraGroup:if verbose:print(error_dict)- print('error: {:.2f}, mu: {:.1f}, ratio: {:.3f}'.format(error, mu, np.mean(good)))- print(f'previous error scores: [magenta] {error_list}[/magenta]')-- self.bundle_adjust(p2ds_samp, extra_samp,- loss='linear', ftol=ftol,- max_nfev=max_nfev,- verbose=verbose)+ print(+ "error: {:.2f}, mu: {:.1f}, ratio: {:.3f}".format(+ error, mu, np.mean(good)+ )+ )+ print(f"previous error scores: [magenta] {error_list}[/magenta]")++ self.bundle_adjust(+ p2ds_samp,+ extra_samp,+ loss="linear",+ ftol=ftol,+ max_nfev=max_nfev,+ verbose=verbose,+ )- p2ds, extra = resample_points(p2ds_full, extra_full,- n_samp=n_samp_full)+ p2ds, extra = resample_points(p2ds_full, extra_full, n_samp=n_samp_full)p3ds = self.triangulate(p2ds)errors_full = self.reprojection_error(p3ds, p2ds, mean=False)errors_norm = self.reprojection_error(p3ds, p2ds, mean=True)@@ -761,10 +809,14 @@ class CameraGroup:good = errors_norm < muextra_good = subset_extra(extra, good)- self.bundle_adjust(p2ds[:, good], extra_good,- loss='linear',- ftol=ftol, max_nfev=max(200, max_nfev),- verbose=verbose)+ self.bundle_adjust(+ p2ds[:, good],+ extra_good,+ loss="linear",+ ftol=ftol,+ max_nfev=max(200, max_nfev),+ verbose=verbose,+ )error = self.average_error(p2ds, median=True)@@ -775,30 +827,35 @@ class CameraGroup:print(error_dict)if verbose:- print('error: ', error)+ print("error: ", error)return error- def bundle_adjust(self, p2ds, extra=None,- loss='linear',- threshold=50,- ftol=1e-4,- max_nfev=1000,- weights=None,- start_params=None,- verbose=True):+ def bundle_adjust(+ self,+ p2ds,+ extra=None,+ loss="linear",+ threshold=50,+ ftol=1e-4,+ max_nfev=1000,+ weights=None,+ start_params=None,+ verbose=True,+ ):"""Given an CxNx2 array of 2D points,where N is the number of points and C is the number of cameras,this performs bundle adjustsment to fine-tune the parameters of the cameras"""- assert p2ds.shape[0] == len(self.cameras), \- "Invalid points shape, first dim should be equal to" \+ assert p2ds.shape[0] == len(self.cameras), (+ "Invalid points shape, first dim should be equal to"" number of cameras ({}), but shape is {}".format(len(self.cameras), p2ds.shape)+ )if extra is not None:- extra['ids_map'] = remap_ids(extra['ids'])+ extra["ids_map"] = remap_ids(extra["ids"])x0, n_cam_params = self._initialize_params_bundle(p2ds, extra)@@ -811,18 +868,20 @@ class CameraGroup:jac_sparse = self._jac_sparsity_bundle(p2ds, n_cam_params, extra)f_scale = threshold- opt = optimize.least_squares(error_fun,- x0,- jac_sparsity=jac_sparse,- f_scale=f_scale,- x_scale='jac',- loss=loss,- ftol=ftol,- method='trf',- tr_solver='lsmr',- verbose=2 * verbose,- max_nfev=max_nfev,- args=(p2ds, n_cam_params, extra))+ opt = optimize.least_squares(+ error_fun,+ x0,+ jac_sparsity=jac_sparse,+ f_scale=f_scale,+ x_scale="jac",+ loss=loss,+ ftol=ftol,+ method="trf",+ tr_solver="lsmr",+ verbose=2 * verbose,+ max_nfev=max_nfev,+ args=(p2ds, n_cam_params, extra),+ )best_params = opt.xfor i, cam in enumerate(self.cameras):@@ -848,18 +907,18 @@ class CameraGroup:n_cams = len(self.cameras)sub = n_cam_params * n_camsn3d = p2ds.shape[1] * 3- p3ds_test = params[sub:sub + n3d].reshape(-1, 3)+ p3ds_test = params[sub : sub + n3d].reshape(-1, 3)errors = self.reprojection_error(p3ds_test, p2ds)errors_reproj = errors[good]if extra is not None:- ids = extra['ids_map']- objp = extra['objp']+ ids = extra["ids_map"]+ objp = extra["objp"]min_scale = np.min(objp[objp > 0])n_boards = int(np.max(ids)) + 1a = sub + n3d- rvecs = params[a:a + n_boards * 3].reshape(-1, 3)- tvecs = params[a + n_boards * 3:a + n_boards * 6].reshape(-1, 3)+ rvecs = params[a : a + n_boards * 3].reshape(-1, 3)+ tvecs = params[a + n_boards * 3 : a + n_boards * 6].reshape(-1, 3)expected = transform_points(objp, rvecs[ids], tvecs[ids])errors_obj = 2 * (p3ds_test - expected).ravel() / min_scaleelse:@@ -872,8 +931,8 @@ class CameraGroup:where N is the number of points and C is the number of cameras,compute the sparsity structure of the jacobian for bundle adjustment"""- point_indices = np.zeros(p2ds.shape, dtype='int32')- cam_indices = np.zeros(p2ds.shape, dtype='int32')+ point_indices = np.zeros(p2ds.shape, dtype="int32")+ cam_indices = np.zeros(p2ds.shape, dtype="int32")for i in range(p2ds.shape[1]):point_indices[:, i] = i@@ -884,7 +943,7 @@ class CameraGroup:good = ~np.isnan(p2ds)if extra is not None:- ids = extra['ids_map']+ ids = extra["ids_map"]n_boards = int(np.max(ids)) + 1total_board_params = n_boards * (3 + 3) # rvecs + tvecselse:@@ -902,7 +961,7 @@ class CameraGroup:else:n_errors = n_good_values- A_sparse = dok_matrix((n_errors, n_params), dtype='int16')+ A_sparse = dok_matrix((n_errors, n_params), dtype="int16")cam_indices_good = cam_indices[good]point_indices_good = point_indices[good]@@ -929,15 +988,21 @@ class CameraGroup:## update board rotation and translation based on error from expectedfor i in range(3):for j in range(3):- A_sparse[n_good_values + point_ix * 3 + i,- total_params_reproj + ids * 3 + j] = 1- A_sparse[n_good_values + point_ix * 3 + i,- total_params_reproj + n_boards * 3 + ids * 3 + j] = 1+ A_sparse[+ n_good_values + point_ix * 3 + i,+ total_params_reproj + ids * 3 + j,+ ] = 1+ A_sparse[+ n_good_values + point_ix * 3 + i,+ total_params_reproj + n_boards * 3 + ids * 3 + j,+ ] = 1## update point position based on error from expectedfor i in range(3):- A_sparse[n_good_values + point_ix * 3 + i,- n_cams * n_cam_params + point_ix * 3 + i] = 1+ A_sparse[+ n_good_values + point_ix * 3 + i,+ n_cams * n_cam_params + point_ix * 3 + i,+ ] = 1return A_sparse@@ -952,59 +1017,67 @@ class CameraGroup:total_cam_params = len(cam_params)n_cams, n_points, _ = p2ds.shape- assert n_cams == len(self.cameras), \- "number of cameras in CameraGroup does not " \+ assert n_cams == len(self.cameras), (+ "number of cameras in CameraGroup does not ""match number of cameras in 2D points given"+ )p3ds = self.triangulate(p2ds)if extra is not None:- ids = extra['ids_map']+ ids = extra["ids_map"]n_boards = int(np.max(ids[~np.isnan(ids)])) + 1total_board_params = n_boards * (3 + 3) # rvecs + tvecs# initialize to 0- rvecs = np.zeros((n_boards, 3), dtype='float64')- tvecs = np.zeros((n_boards, 3), dtype='float64')+ rvecs = np.zeros((n_boards, 3), dtype="float64")+ tvecs = np.zeros((n_boards, 3), dtype="float64")- if 'rvecs' in extra and 'tvecs' in extra:- rvecs_all = extra['rvecs']- tvecs_all = extra['tvecs']+ if "rvecs" in extra and "tvecs" in extra:+ rvecs_all = extra["rvecs"]+ tvecs_all = extra["tvecs"]for board_num in range(n_boards):point_id = np.where(ids == board_num)[0][0]cam_ids_possible = np.where(~np.isnan(p2ds[:, point_id, 0]))[0]cam_id = np.random.choice(cam_ids_possible)M_cam = self.cameras[cam_id].get_extrinsics_mat()- M_board_cam = make_M(rvecs_all[cam_id, point_id],- tvecs_all[cam_id, point_id])+ M_board_cam = make_M(+ rvecs_all[cam_id, point_id], tvecs_all[cam_id, point_id]+ )M_board = np.matmul(np.linalg.inv(M_cam), M_board_cam)rvec, tvec = get_rtvec(M_board)rvecs[board_num] = rvectvecs[board_num] = tvec-else:total_board_params = 0x0 = np.zeros(total_cam_params + p3ds.size + total_board_params)x0[:total_cam_params] = cam_params- x0[total_cam_params:total_cam_params + p3ds.size] = p3ds.ravel()+ x0[total_cam_params : total_cam_params + p3ds.size] = p3ds.ravel()if extra is not None:start_board = total_cam_params + p3ds.size- x0[start_board:start_board + n_boards * 3] = rvecs.ravel()- x0[start_board + n_boards * 3:start_board + n_boards * 6] = \- tvecs.ravel()+ x0[start_board : start_board + n_boards * 3] = rvecs.ravel()+ x0[start_board + n_boards * 3 : start_board + n_boards * 6] = tvecs.ravel()return x0, n_cam_params- def optim_points(self, points, p3ds,- constraints=[],- constraints_weak=[],- scale_smooth=4,- scale_length=2, scale_length_weak=0.5,- reproj_error_threshold=15, reproj_loss='soft_l1',- n_deriv_smooth=1, scores=None, verbose=False):+ def optim_points(+ self,+ points,+ p3ds,+ constraints=[],+ constraints_weak=[],+ scale_smooth=4,+ scale_length=2,+ scale_length_weak=0.5,+ reproj_error_threshold=15,+ reproj_loss="soft_l1",+ n_deriv_smooth=1,+ scores=None,+ verbose=False,+ ):"""Take in an array of 2D points of shape CxNxJx2,an array of 3D points of shape NxJx3,@@ -1021,11 +1094,12 @@ class CameraGroup:(meaning that lengths of segments 0->1, 1->2, 2->3 are all constant)"""- assert points.shape[0] == len(self.cameras), \- "Invalid points shape, first dim should be equal to" \+ assert points.shape[0] == len(self.cameras), (+ "Invalid points shape, first dim should be equal to"" number of cameras ({}), but shape is {}".format(len(self.cameras), points.shape)+ )n_cams, n_frames, n_joints, _ = points.shapeconstraints = np.array(constraints)@@ -1041,45 +1115,60 @@ class CameraGroup:t1 = time.time()x0 = self._initialize_params_triangulation(- p3ds_intp, constraints, constraints_weak)+ p3ds_intp, constraints, constraints_weak+ )x0[~np.isfinite(x0)] = 0jac = self._jac_sparsity_triangulation(- points, constraints, constraints_weak, n_deriv_smooth)-- opt2 = optimize.least_squares(self._error_fun_triangulation,- x0=x0, jac_sparsity=jac,- loss='linear',- ftol=1e-3,- verbose=2 * verbose,- args=(points,- constraints,- constraints_weak,- scores,- scale_smooth_full,- scale_length,- scale_length_weak,- reproj_error_threshold,- reproj_loss,- n_deriv_smooth))-- p3ds_new2 = opt2.x[:p3ds.size].reshape(p3ds.shape)+ points, constraints, constraints_weak, n_deriv_smooth+ )++ opt2 = optimize.least_squares(+ self._error_fun_triangulation,+ x0=x0,+ jac_sparsity=jac,+ loss="linear",+ ftol=1e-3,+ verbose=2 * verbose,+ args=(+ points,+ constraints,+ constraints_weak,+ scores,+ scale_smooth_full,+ scale_length,+ scale_length_weak,+ reproj_error_threshold,+ reproj_loss,+ n_deriv_smooth,+ ),+ )++ p3ds_new2 = opt2.x[: p3ds.size].reshape(p3ds.shape)t2 = time.time()if verbose:- print('optimization took {:.2f} seconds'.format(t2 - t1))+ print("optimization took {:.2f} seconds".format(t2 - t1))return p3ds_new2- def optim_points_possible(self, points, p3ds,- constraints=[],- constraints_weak=[],- scale_smooth=4,- scale_length=2, scale_length_weak=0.5,- reproj_error_threshold=15, reproj_loss='soft_l1',- n_deriv_smooth=1, scores=None, verbose=False):+ def optim_points_possible(+ self,+ points,+ p3ds,+ constraints=[],+ constraints_weak=[],+ scale_smooth=4,+ scale_length=2,+ scale_length_weak=0.5,+ reproj_error_threshold=15,+ reproj_loss="soft_l1",+ n_deriv_smooth=1,+ scores=None,+ verbose=False,+ ):"""Take in an array of 2D points of shape CxNxJxPx2,an array of 3D points of shape NxJx3,@@ -1097,11 +1186,12 @@ class CameraGroup:(meaning that lengths of segments 0->1, 1->2, 2->3 are all constant)"""- assert points.shape[0] == len(self.cameras), \- "Invalid points shape, first dim should be equal to" \+ assert points.shape[0] == len(self.cameras), (+ "Invalid points shape, first dim should be equal to"" number of cameras ({}), but shape is {}".format(len(self.cameras), points.shape)+ )n_cams, n_frames, n_joints, n_possible, _ = points.shapeconstraints = np.array(constraints)@@ -1117,44 +1207,54 @@ class CameraGroup:t1 = time.time()x0 = self._initialize_params_triangulation_possible(- p3ds_intp, points, constraints=constraints, constraints_weak=constraints_weak)+ p3ds_intp,+ points,+ constraints=constraints,+ constraints_weak=constraints_weak,+ )- print('getting jacobian...')+ print("getting jacobian...")jac = self._jac_sparsity_triangulation_possible(points,constraints=constraints,constraints_weak=constraints_weak,- n_deriv_smooth=n_deriv_smooth)+ n_deriv_smooth=n_deriv_smooth,+ )beta = 5- print('starting optimization...')- opt2 = optimize.least_squares(self._error_fun_triangulation_possible,- x0=x0, jac_sparsity=jac,- loss='linear',- ftol=1e-3,- verbose=2 * verbose,- args=(points,- beta,- constraints,- constraints_weak,- scores,- scale_smooth_full,- scale_length,- scale_length_weak,- reproj_error_threshold,- reproj_loss,- n_deriv_smooth))+ print("starting optimization...")+ opt2 = optimize.least_squares(+ self._error_fun_triangulation_possible,+ x0=x0,+ jac_sparsity=jac,+ loss="linear",+ ftol=1e-3,+ verbose=2 * verbose,+ args=(+ points,+ beta,+ constraints,+ constraints_weak,+ scores,+ scale_smooth_full,+ scale_length,+ scale_length_weak,+ reproj_error_threshold,+ reproj_loss,+ n_deriv_smooth,+ ),+ )params = opt2.x- p3ds_new2 = params[:p3ds.size].reshape(p3ds.shape)+ p3ds_new2 = params[: p3ds.size].reshape(p3ds.shape)bad = np.isnan(points[:, :, :, :, 0])all_bad = np.all(bad, axis=3)n_params_norm = p3ds.size + len(constraints) + len(constraints_weak)- alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype='float64')+ alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype="float64")alphas[~bad] = params[n_params_norm:]alphas_exp = np.exp(beta * alphas)@@ -1167,12 +1267,13 @@ class CameraGroup:t2 = time.time()if verbose:- print('optimization took {:.2f} seconds'.format(t2 - t1))+ print("optimization took {:.2f} seconds".format(t2 - t1))return p3ds_new2, alphas_norm- def triangulate_optim(self, points, init_ransac=False, init_progress=False,- **kwargs):+ def triangulate_optim(+ self, points, init_ransac=False, init_progress=False, **kwargs+ ):"""Take in an array of 2D points of shape CxNxJx2, and an array of constraints of shape Kx2, whereC: number of camera@@ -1188,11 +1289,12 @@ class CameraGroup:"""- assert points.shape[0] == len(self.cameras), \- "Invalid points shape, first dim should be equal to" \+ assert points.shape[0] == len(self.cameras), (+ "Invalid points shape, first dim should be equal to"" number of cameras ({}), but shape is {}".format(len(self.cameras), points.shape)+ )n_cams, n_frames, n_joints, _ = points.shape# constraints = np.array(constraints)@@ -1200,7 +1302,9 @@ class CameraGroup:points_shaped = points.reshape(n_cams, n_frames * n_joints, 2)if init_ransac:- p3ds, picked, p2ds, errors = self.triangulate_ransac(points_shaped, progress=init_progress)+ p3ds, picked, p2ds, errors = self.triangulate_ransac(+ points_shaped, progress=init_progress+ )points = p2ds.reshape(points.shape)else:p3ds = self.triangulate(points_shaped, progress=init_progress)@@ -1214,16 +1318,20 @@ class CameraGroup:return self.optim_points(points, p3ds, **kwargs)@jit(forceobj=True, parallel=True)- def _error_fun_triangulation(self, params, p2ds,- constraints=[],- constraints_weak=[],- scores=None,- scale_smooth=10000,- scale_length=1,- scale_length_weak=0.2,- reproj_error_threshold=100,- reproj_loss='soft_l1',- n_deriv_smooth=1):+ def _error_fun_triangulation(+ self,+ params,+ p2ds,+ constraints=[],+ constraints_weak=[],+ scores=None,+ scale_smooth=10000,+ scale_length=1,+ scale_length_weak=0.2,+ reproj_error_threshold=100,+ reproj_loss="soft_l1",+ n_deriv_smooth=1,+ ):n_cams, n_frames, n_joints, _ = p2ds.shapen_3d = n_frames * n_joints * 3@@ -1232,8 +1340,8 @@ class CameraGroup:# load paramsp3ds = params[:n_3d].reshape((n_frames, n_joints, 3))- joint_lengths = np.array(params[n_3d:n_3d + n_constraints])- joint_lengths_weak = np.array(params[n_3d + n_constraints:])+ joint_lengths = np.array(params[n_3d : n_3d + n_constraints])+ joint_lengths_weak = np.array(params[n_3d + n_constraints :])# reprojection errorsp3ds_flat = p3ds.reshape(-1, 3)@@ -1246,40 +1354,39 @@ class CameraGroup:rp = reproj_error_thresholderrors_reproj = np.abs(errors_reproj)- if reproj_loss == 'huber':+ if reproj_loss == "huber":bad = errors_reproj > rperrors_reproj[bad] = rp * (2 * np.sqrt(errors_reproj[bad] / rp) - 1)- elif reproj_loss == 'linear':+ elif reproj_loss == "linear":pass- elif reproj_loss == 'soft_l1':+ elif reproj_loss == "soft_l1":errors_reproj = rp * 2 * (np.sqrt(1 + errors_reproj / rp) - 1)# temporal constrainterrors_smooth = np.diff(p3ds, n=n_deriv_smooth, axis=0).ravel() * scale_smooth# joint length constraint- errors_lengths = np.empty((n_constraints, n_frames), dtype='float64')+ errors_lengths = np.empty((n_constraints, n_frames), dtype="float64")for cix, (a, b) in enumerate(constraints):lengths = np.linalg.norm(p3ds[:, a] - p3ds[:, b], axis=1)expected = joint_lengths[cix]errors_lengths[cix] = 100 * (lengths - expected) / expectederrors_lengths = errors_lengths.ravel() * scale_length- errors_lengths_weak = np.empty((n_constraints_weak, n_frames), dtype='float64')+ errors_lengths_weak = np.empty((n_constraints_weak, n_frames), dtype="float64")for cix, (a, b) in enumerate(constraints_weak):lengths = np.linalg.norm(p3ds[:, a] - p3ds[:, b], axis=1)expected = joint_lengths_weak[cix]errors_lengths_weak[cix] = 100 * (lengths - expected) / expectederrors_lengths_weak = errors_lengths_weak.ravel() * scale_length_weak- return np.hstack([errors_reproj, errors_smooth,- errors_lengths, errors_lengths_weak])+ return np.hstack(+ [errors_reproj, errors_smooth, errors_lengths, errors_lengths_weak]+ )- def _error_fun_triangulation_possible(self, params, p2ds,- beta=2,- constraints=[],- constraints_weak=[],- *args):+ def _error_fun_triangulation_possible(+ self, params, p2ds, beta=2, constraints=[], constraints_weak=[], *args+ ):# extract alphas from end of params# soft argmax for picking the appropriate points from p2ds# pass the points to error_fun_triangulate_possible for residuals@@ -1297,7 +1404,7 @@ class CameraGroup:bad = np.isnan(p2ds[:, :, :, :, 0])all_bad = np.all(bad, axis=3)- alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype='float64')+ alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype="float64")alphas[~bad] = params[n_params_norm:]params_rest = np.array(params[:n_params_norm])@@ -1314,19 +1421,20 @@ class CameraGroup:p2ds_adj = np.sum(alphas_norm[:, :, :, :, None] * p2ds_test, axis=3)p2ds_adj[all_bad] = np.nan- errors = self._error_fun_triangulation(params_rest, p2ds_adj,- constraints, constraints_weak, *args)+ errors = self._error_fun_triangulation(+ params_rest, p2ds_adj, constraints, constraints_weak, *args+ )alphas_test = alphas_norm[~all_bad]errors_alphas = (1 - np.std(alphas_test, axis=1)) * 10return np.hstack([errors, errors_alphas])- def _initialize_params_triangulation(self, p3ds,- constraints=[],- constraints_weak=[]):- joint_lengths = np.empty(len(constraints), dtype='float64')- joint_lengths_weak = np.empty(len(constraints_weak), dtype='float64')+ def _initialize_params_triangulation(+ self, p3ds, constraints=[], constraints_weak=[]+ ):+ joint_lengths = np.empty(len(constraints), dtype="float64")+ joint_lengths_weak = np.empty(len(constraints_weak), dtype="float64")for cix, (a, b) in enumerate(constraints):lengths = np.linalg.norm(p3ds[:, a] - p3ds[:, b], axis=1)@@ -1357,7 +1465,7 @@ class CameraGroup:n_cams, n_frames, n_joints, n_possible, _ = p2ds.shapegood = ~np.isnan(p2ds[:, :, :, :, 0])- alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype='float64')+ alphas = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype="float64")alphas[:, :, :, 0] = 0params = self._initialize_params_triangulation(p3ds, **kwargs)@@ -1365,22 +1473,20 @@ class CameraGroup:return params_full- def _jac_sparsity_triangulation(self, p2ds,- constraints=[],- constraints_weak=[],- n_deriv_smooth=1):+ def _jac_sparsity_triangulation(+ self, p2ds, constraints=[], constraints_weak=[], n_deriv_smooth=1+ ):n_cams, n_frames, n_joints, _ = p2ds.shapen_constraints = len(constraints)n_constraints_weak = len(constraints_weak)p2ds_flat = p2ds.reshape((n_cams, -1, 2))- point_indices = np.zeros(p2ds_flat.shape, dtype='int32')+ point_indices = np.zeros(p2ds_flat.shape, dtype="int32")for i in range(p2ds_flat.shape[1]):point_indices[:, i] = i- point_indices_3d = np.arange(n_frames * n_joints) \- .reshape((n_frames, n_joints))+ point_indices_3d = np.arange(n_frames * n_joints).reshape((n_frames, n_joints))good = ~np.isnan(p2ds_flat)n_errors_reproj = np.sum(good)@@ -1388,15 +1494,16 @@ class CameraGroup:n_errors_lengths = n_constraints * n_framesn_errors_lengths_weak = n_constraints_weak * n_frames- n_errors = n_errors_reproj + n_errors_smooth + \- n_errors_lengths + n_errors_lengths_weak+ n_errors = (+ n_errors_reproj + n_errors_smooth + n_errors_lengths + n_errors_lengths_weak+ )n_3d = n_frames * n_joints * 3n_params = n_3d + n_constraints + n_constraints_weakpoint_indices_good = point_indices[good]- A_sparse = dok_matrix((n_errors, n_params), dtype='int16')+ A_sparse = dok_matrix((n_errors, n_params), dtype="int16")# constraints for reprojection errorsix_reproj = np.arange(n_errors_reproj)@@ -1463,17 +1570,22 @@ class CameraGroup:n_errors, n_params = A_sparse.shape- B_sparse = dok_matrix((n_errors + n_errors_alphas, n_params + n_alphas), dtype='int16')+ B_sparse = dok_matrix(+ (n_errors + n_errors_alphas, n_params + n_alphas), dtype="int16"+ )for r, c in zip(*A_sparse.nonzero()):B_sparse[r, c] = A_sparse[r, c]- point_indices_2d = np.arange(n_cams * n_frames * n_joints) \- .reshape(n_cams, n_frames, n_joints)+ point_indices_2d = np.arange(n_cams * n_frames * n_joints).reshape(+ n_cams, n_frames, n_joints+ )point_indices_2d_rep = np.repeat(point_indices_2d[:, :, :, None], 2, axis=3)point_indices_2d_good = point_indices_2d_rep[~np.isnan(p2ds)]point_indices_good = point_indices_2d[any_good]- alpha_indices = np.zeros((n_cams, n_frames, n_joints, n_possible), dtype='int64')+ alpha_indices = np.zeros(+ (n_cams, n_frames, n_joints, n_possible), dtype="int64"+ )for pnum in range(n_possible):alpha_indices[:, :, :, pnum] = point_indices_2d@@ -1485,8 +1597,7 @@ class CameraGroup:point_indices_2d_good_find[p].append(ix)for ix, alpha_index in enumerate(alpha_indices_good):- B_sparse[point_indices_2d_good_find[alpha_index],- n_params + ix] = 1+ B_sparse[point_indices_2d_good_find[alpha_index], n_params + ix] = 1# alphas should change according to the alpha errorspoint_indices_good_find = dict()@@ -1542,23 +1653,34 @@ class CameraGroup:else:return np.mean(errors)- def calibrate_rows(self, all_rows, board,- init_intrinsics=True, init_extrinsics=True, verbose=True,- **kwargs):- assert len(all_rows) == len(self.cameras), \- "Number of camera detections does not match number of cameras"+ def calibrate_rows(+ self,+ all_rows,+ board,+ init_intrinsics=True,+ init_extrinsics=True,+ verbose=True,+ **kwargs,+ ):+ assert len(all_rows) == len(+ self.cameras+ ), "Number of camera detections does not match number of cameras"for rows, camera in zip(all_rows, self.cameras):size = camera.get_size()- assert size is not None, \- "Camera with name {} has no specified frame size".format(camera.get_name())-+ assert (+ size is not None+ ), "Camera with name {} has no specified frame size".format(+ camera.get_name()+ )if init_intrinsics:objp, imgp = board.get_all_calibration_points(rows)mixed = [(o, i) for (o, i) in zip(objp, imgp) if len(o) >= 7]- assert len(objp) != 0 and len(imgp) != 0, "No Charuco board points detected"+ assert (+ len(objp) != 0 and len(imgp) != 0+ ), "No Charuco board points detected"objp, imgp = zip(*mixed)matrix = cv2.initCameraMatrix2D(objp, imgp, tuple(size))camera.set_camera_matrix(matrix)@@ -1566,7 +1688,7 @@ class CameraGroup:for i, (row, cam) in enumerate(zip(all_rows, self.cameras)):all_rows[i] = board.estimate_pose_rows(cam, row)- charuco_frames = [f['framenum'][1] for f in all_rows[0]]+ charuco_frames = [f["framenum"][1] for f in all_rows[0]]merged = merge_rows(all_rows)imgp, extra = extract_points(merged, board, min_cameras=2)@@ -1588,9 +1710,11 @@ class CameraGroup:for cix, (cam, cam_videos) in enumerate(zip(self.cameras, videos)):rows_cam = []for vnum, vidname in enumerate(cam_videos):- if verbose: print(vidname)+ if verbose:+ print(vidname)rows = board.detect_video(vidname, prefix=vnum, progress=verbose)- if verbose: print("{} boards detected".format(len(rows)))+ if verbose:+ print("{} boards detected".format(len(rows)))rows_cam.extend(rows)all_rows.append(rows_cam)@@ -1601,12 +1725,18 @@ class CameraGroup:rows_cam = []for vnum, vidname in enumerate(cam_videos):params = get_video_params(vidname)- size = (params['width'], params['height'])+ size = (params["width"], params["height"])cam.set_size(size)- def calibrate_videos(self, videos, board,- init_intrinsics=True, init_extrinsics=True, verbose=True,- **kwargs):+ def calibrate_videos(+ self,+ videos,+ board,+ init_intrinsics=True,+ init_extrinsics=True,+ verbose=True,+ **kwargs,+ ):"""Takes as input a list of list of video filenames, one list of each camera.Also takes a board which specifies what should be detected in the videos"""@@ -1614,10 +1744,14 @@ class CameraGroup:if init_extrinsics:self.set_camera_sizes_videos(videos)- error, merged, charuco_frames = self.calibrate_rows(all_rows, board,- init_intrinsics=init_intrinsics,- init_extrinsics=init_extrinsics,- verbose=verbose, **kwargs)+ error, merged, charuco_frames = self.calibrate_rows(+ all_rows,+ board,+ init_intrinsics=init_intrinsics,+ init_extrinsics=init_extrinsics,+ verbose=verbose,+ **kwargs,+ )return error, merged, charuco_framesdef get_dicts(self):@@ -1629,7 +1763,7 @@ class CameraGroup:def from_dicts(arr):cameras = []for d in arr:- if 'fisheye' in d and d['fisheye']:+ if "fisheye" in d and d["fisheye"]:cam = FisheyeCamera.from_dict(d)else:cam = Camera.from_dict(d)@@ -1653,20 +1787,20 @@ class CameraGroup:def dump(self, fname):dicts = self.get_dicts()- names = ['cam_{}'.format(i) for i in range(len(dicts))]+ names = ["cam_{}".format(i) for i in range(len(dicts))]master_dict = dict(zip(names, dicts))- master_dict['metadata'] = self.metadata- with open(fname, 'w') as f:+ master_dict["metadata"] = self.metadata+ with open(fname, "w") as f:toml.dump(master_dict, f)@staticmethoddef load(fname):master_dict = toml.load(fname)keys = sorted(master_dict.keys())- items = [master_dict[k] for k in keys if k != 'metadata']+ items = [master_dict[k] for k in keys if k != "metadata"]cgroup = CameraGroup.from_dicts(items)- if 'metadata' in master_dict:- cgroup.metadata = master_dict['metadata']+ if "metadata" in master_dict:+ cgroup.metadata = master_dict["metadata"]return cgroupdef resize_cameras(self, scale):diff --git a/src/pipelines/calibration_pipeline/calibration_pipeline_orchestrator.py b/src/pipelines/calibration_pipeline/calibration_pipeline_orchestrator.pyindex ea34efa3..2af920c0 100644--- a/src/pipelines/calibration_pipeline/calibration_pipeline_orchestrator.py+++ b/src/pipelines/calibration_pipeline/calibration_pipeline_orchestrator.py@@ -7,31 +7,45 @@ from typing import Unionimport cv2from src.cameras.multicam_manager.cv_camera_manager import OpenCVCameraManager-from src.config.home_dir import create_session_id, get_session_folder_path, get_freemocap_data_folder_path, \- get_session_output_data_folder_path, get_session_calibration_file_path+from src.config.home_dir import (+ create_session_id,+ get_session_folder_path,+ get_freemocap_data_folder_path,+ get_session_output_data_folder_path,+ get_session_calibration_file_path,+)from src.core_processor.timestamp_manager.timestamp_manager import TimestampManagerfrom src.core_processor.show_cam_window import show_cam_windowfrom src.core_processor.utils.image_fps_writer import write_fps_to_image-from src.pipelines.calibration_pipeline.anipose_camera_calibration import freemocap_anipose-from src.pipelines.calibration_pipeline.anipose_camera_calibration.anipose_camera_calibration import \- AniposeCameraCalibrator-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import CharucoBoardDetector+from src.pipelines.calibration_pipeline.anipose_camera_calibration import (+ freemocap_anipose,+)+from src.pipelines.calibration_pipeline.anipose_camera_calibration.anipose_camera_calibration import (+ AniposeCameraCalibrator,+)+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (+ CharucoBoardDetector,+)from src.qt_visualizer_and_gui.qt_visualizer_and_gui import QTVisualizerAndGuilogger = logging.getLogger(__name__)class CalibrationPipelineOrchestrator:-- def __init__(self,- session_id: str = None,- expected_framerate: Union[int, float] = None,- ):+ def __init__(+ self,+ session_id: str = None,+ expected_framerate: Union[int, float] = None,+ ):if session_id is None:- self._session_id = create_session_id(string_tag='calibration') #create a 'calibration only' session+ self._session_id = create_session_id(+ string_tag="calibration"+ ) # create a 'calibration only' sessionelse:- self._session_id = session_id #add calibration folder to "full" session folder+ self._session_id = (+ session_id # add calibration folder to "full" session folder+ )self._calibration_start_time_unix_ns = time.time_ns()self._charuco_board_detector = CharucoBoardDetector()@@ -48,65 +62,87 @@ class CalibrationPipelineOrchestrator:return get_session_folder_path(self._session_id)def record_videos(- self,- show_visualizer_gui=False,- show_camera_views_in_windows=True,- save_video_in_frame_loop=False,-+ self,+ show_visualizer_gui=False,+ show_camera_views_in_windows=True,+ save_video_in_frame_loop=False,):"""open all cameras and start recording, detect charuco boards until the user closes the windows, then send recorded videos to `aniposelib` for processingproduces `[session_id]_calibration.toml` file that can be used to estiamte 3d trajectorires from synchronized images"""with self._open_cv_camera_manager.start_capture_session_all_cams(- calibration_videos=True,+ calibration_videos=True,) as connected_cameras_dict:timestamp_manager = self._open_cv_camera_manager.timestamp_managertry:if show_visualizer_gui:self._visualizer_gui = QTVisualizerAndGui()- self._visualizer_gui.setup_and_launch(self._open_cv_camera_manager.available_webcam_ids)+ self._visualizer_gui.setup_and_launch(+ self._open_cv_camera_manager.available_webcam_ids+ )should_continue = Truewhile should_continue:- timestamp_manager.log_new_timestamp_for_main_loop_perf_coutner_ns(time.perf_counter_ns())+ timestamp_manager.log_new_timestamp_for_main_loop_perf_coutner_ns(+ time.perf_counter_ns()+ )if not self._open_cv_camera_manager.new_multi_frame_ready():continue- this_multi_frame_payload = self._open_cv_camera_manager.latest_multi_frame+ this_multi_frame_payload = (+ self._open_cv_camera_manager.latest_multi_frame+ )- for this_webcam_id, this_open_cv_camera in connected_cameras_dict.items():+ for (+ this_webcam_id,+ this_open_cv_camera,+ ) in connected_cameras_dict.items():- this_cam_latest_frame = this_multi_frame_payload.frames_dict[this_webcam_id]+ this_cam_latest_frame = this_multi_frame_payload.frames_dict[+ this_webcam_id+ ]if this_cam_latest_frame is None:continue-# save frame to video fileif save_video_in_frame_loop:# save this frame straight to the video file (no risk of memory overflow, but can't handle higher numbers of cameras)- this_open_cv_camera.video_recorder.save_frame_payload_to_video_file(this_cam_latest_frame)+ this_open_cv_camera.video_recorder.save_frame_payload_to_video_file(+ this_cam_latest_frame+ )else:# can handle large numbers of cameras, but will eventually fill up RAM and cause a crash- this_open_cv_camera.video_recorder.append_frame_payload_to_list(this_cam_latest_frame)+ this_open_cv_camera.video_recorder.append_frame_payload_to_list(+ this_cam_latest_frame+ )# detect charuco board- this_charuco_frame_payload = self._charuco_board_detector.detect_charuco_board(- this_cam_latest_frame)+ this_charuco_frame_payload = (+ self._charuco_board_detector.detect_charuco_board(+ this_cam_latest_frame+ )+ )if show_camera_views_in_windows:should_continue = show_cam_window(- this_webcam_id, this_charuco_frame_payload.annotated_image, timestamp_manager+ this_webcam_id,+ this_charuco_frame_payload.annotated_image,+ timestamp_manager,)if show_visualizer_gui:write_fps_to_image(this_charuco_frame_payload.annotated_image,- timestamp_manager.median_frames_per_second_for_webcam(this_webcam_id),+ timestamp_manager.median_frames_per_second_for_webcam(+ this_webcam_id+ ),+ )+ self._visualizer_gui.update_camera_view_image(+ this_webcam_id,+ this_charuco_frame_payload.annotated_image,)- self._visualizer_gui.update_camera_view_image(this_webcam_id,- this_charuco_frame_payload.annotated_image)# exit loop when user presses ESC keyexit_key = cv2.waitKey(1)@@ -124,50 +160,68 @@ class CalibrationPipelineOrchestrator:cv2.destroyAllWindows()for this_open_cv_camera in connected_cameras_dict.values():if not save_video_in_frame_loop:- this_open_cv_camera.video_recorder.save_list_of_frames_to_video_file(this_open_cv_camera.video_recorder.frame_list)+ this_open_cv_camera.video_recorder.save_list_of_frames_to_video_file(+ this_open_cv_camera.video_recorder.frame_list+ )this_open_cv_camera.video_recorder.close()if show_visualizer_gui:self._visualizer_gui.close()- def run_anipose_camera_calibration(self,- charuco_square_size: Union[int, float] = 1,- pin_camera_0_to_origin: bool = False,- ):- anipose_camera_calibrator = AniposeCameraCalibrator(self.session_id,- self._charuco_board_detector.charuco_board_data_class_object,- charuco_square_size=charuco_square_size)- return anipose_camera_calibrator.calibrate_camera_capture_volume(pin_camera_0_to_origin=pin_camera_0_to_origin)+ def run_anipose_camera_calibration(+ self,+ charuco_square_size: Union[int, float] = 1,+ pin_camera_0_to_origin: bool = False,+ ):+ anipose_camera_calibrator = AniposeCameraCalibrator(+ self.session_id,+ self._charuco_board_detector.charuco_board_data_class_object,+ charuco_square_size=charuco_square_size,+ )+ return anipose_camera_calibrator.calibrate_camera_capture_volume(+ pin_camera_0_to_origin=pin_camera_0_to_origin+ )def load_most_recent_calibration(self):- last_successful_calibration_path = Path(get_freemocap_data_folder_path(), "last_successful_calibration.toml")- logger.info(f"loading `most recent calibration from:{str(last_successful_calibration_path)}")+ last_successful_calibration_path = Path(+ get_freemocap_data_folder_path(), "last_successful_calibration.toml"+ )+ logger.info(+ f"loading `most recent calibration from:{str(last_successful_calibration_path)}"+ )return freemocap_anipose.CameraGroup.load(str(last_successful_calibration_path))- def load_calibration_from_session_id(self, session_id:str):+ def load_calibration_from_session_id(self, session_id: str):session_calibration_file_path = get_session_calibration_file_path(session_id)- logger.info(f"loading camera calibration file from:{str(session_calibration_file_path)}")+ logger.info(+ f"loading camera calibration file from:{str(session_calibration_file_path)}"+ )if Path(session_calibration_file_path).is_file():- return freemocap_anipose.CameraGroup.load(str(session_calibration_file_path))+ return freemocap_anipose.CameraGroup.load(+ str(session_calibration_file_path)+ )return self.load_most_recent_calibration()if __name__ == "__main__":- print('running `calibration_pipeline` as a `__main__` file')+ print("running `calibration_pipeline` as a `__main__` file")record_new = Trueif record_new:- session_id = create_session_id('calibration')+ session_id = create_session_id("calibration")session_path = Path(get_session_folder_path(session_id))- logger.info(f'Creating `calibration only` session folder at: {str(session_path)}')+ logger.info(+ f"Creating `calibration only` session folder at: {str(session_path)}"+ )session_path.mkdir(parents=True, exist_ok=False)calibration_orchestrator = CalibrationPipelineOrchestrator(session_id)- calibration_orchestrator.record_videos(show_visualizer_gui=False,- save_video_in_frame_loop=False,- show_camera_views_in_windows=True,- )+ calibration_orchestrator.record_videos(+ show_visualizer_gui=False,+ save_video_in_frame_loop=False,+ show_camera_views_in_windows=True,+ )else:session_id = "session_06-05-2022-10_31_34_calibration"diff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_board_detector.py b/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_board_detector.pyindex 891e867b..d5ec696e 100644--- a/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_board_detector.py+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_board_detector.py@@ -5,10 +5,15 @@ import cv2import numpy as npfrom src.cameras.capture.dataclasses.frame_payload import FramePayload-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import CharucoBoardDataClass-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_frame_payload import \- CharucoFramePayload-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import CharucoViewData+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import (+ CharucoBoardDataClass,+)+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_frame_payload import (+ CharucoFramePayload,+)+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import (+ CharucoViewData,+)from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_image_annotator import (annotate_image_with_charuco_data,@@ -20,30 +25,38 @@ logger = logging.getLogger(__name__)class CharucoBoardDetector:def __init__(self):self.charuco_board_data_class_object = CharucoBoardDataClass()- self.cv2_aruco_charuco_board = self.charuco_board_data_class_object.charuco_board+ self.cv2_aruco_charuco_board = (+ self.charuco_board_data_class_object.charuco_board+ )self.aruco_marker_dict = self.charuco_board_data_class_object.aruco_marker_dict- self.number_of_charuco_corners = self.charuco_board_data_class_object.number_of_charuco_corners+ self.number_of_charuco_corners = (+ self.charuco_board_data_class_object.number_of_charuco_corners+ )- def detect_charuco_board(self, raw_frame_payload: FramePayload) -> CharucoFramePayload:+ def detect_charuco_board(+ self, raw_frame_payload: FramePayload+ ) -> CharucoFramePayload:if raw_frame_payload.image is None:logger.error("Frame is empty")return CharucoFramePayload()- charuco_view_data = self.detect_charuco_board_in_an_image(raw_frame_payload.image)+ charuco_view_data = self.detect_charuco_board_in_an_image(+ raw_frame_payload.image+ )annotated_image = raw_frame_payload.image.copy()if charuco_view_data.some_charuco_corners_found:annotate_image_with_charuco_data(annotated_image, # this image will have stuff drawn on top of it inside this functioncharuco_view_data,- self.number_of_charuco_corners+ self.number_of_charuco_corners,)return CharucoFramePayload(annotated_image=annotated_image,raw_frame_payload=raw_frame_payload,- charuco_view_data=charuco_view_data+ charuco_view_data=charuco_view_data,)def detect_charuco_board_in_an_image(self, raw_image) -> CharucoViewData:@@ -61,13 +74,19 @@ class CharucoBoardDetector:charuco_ids = []# SUB PIXEL CORNER DETECTION CRITERION- termination_criteria_threshold = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.00001)+ termination_criteria_threshold = (+ cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER,+ 100,+ 0.00001,+ )grayscale_image = cv2.cvtColor(raw_image, cv2.COLOR_BGR2GRAY)- aruco_square_corners, aruco_square_ids, rejected_image_points = cv2.aruco.detectMarkers(- grayscale_image, self.aruco_marker_dict- )+ (+ aruco_square_corners,+ aruco_square_ids,+ rejected_image_points,+ ) = cv2.aruco.detectMarkers(grayscale_image, self.aruco_marker_dict)full_board_found = Falseany_markers_found = False@@ -79,16 +98,24 @@ class CharucoBoardDetector:# https://docs.opencv.org/4.x/dd/d1a/group__imgproc__feature.html#ga354e0d7c86d0d9da75de9b9701a9a87efor this_corner in aruco_square_corners:cv2.cornerSubPix(- grayscale_image, this_corner, winSize=(3, 3), zeroZone=(-1, -1),- criteria=termination_criteria_threshold+ grayscale_image,+ this_corner,+ winSize=(3, 3),+ zeroZone=(-1, -1),+ criteria=termination_criteria_threshold,)- results = cv2.aruco.interpolateCornersCharuco(aruco_square_corners,- aruco_square_ids,- grayscale_image,- self.cv2_aruco_charuco_board- )+ results = cv2.aruco.interpolateCornersCharuco(+ aruco_square_corners,+ aruco_square_ids,+ grayscale_image,+ self.cv2_aruco_charuco_board,+ )- if results[1] is not None and results[2] is not None and len(results[1]) > 3:+ if (+ results[1] is not None+ and results[2] is not None+ and len(results[1]) > 3+ ):some_charuco_corners_found = Truecharuco_corners = results[1]@@ -97,37 +124,51 @@ class CharucoBoardDetector:if len(charuco_ids) == self.number_of_charuco_corners:full_board_found = True- return CharucoViewData(charuco_board_object=self.charuco_board_data_class_object,- full_board_found=full_board_found,- some_charuco_corners_found=some_charuco_corners_found,- any_markers_found=any_markers_found,- charuco_corners=charuco_corners,- charuco_ids=charuco_ids,- aruco_square_corners=aruco_square_corners,- aruco_square_ids=aruco_square_ids,- image_width=image_width,- image_height=image_height)+ return CharucoViewData(+ charuco_board_object=self.charuco_board_data_class_object,+ full_board_found=full_board_found,+ some_charuco_corners_found=some_charuco_corners_found,+ any_markers_found=any_markers_found,+ charuco_corners=charuco_corners,+ charuco_ids=charuco_ids,+ aruco_square_corners=aruco_square_corners,+ aruco_square_ids=aruco_square_ids,+ image_width=image_width,+ image_height=image_height,+ )def format_charuco2d_data(self, this_multi_frame_charuco_data) -> Dict:number_of_tracked_points = this_multi_frame_charuco_data[- 0].charuco_view_data.charuco_board_object.number_of_charuco_corners+ 0+ ].charuco_view_data.charuco_board_object.number_of_charuco_cornerscharuco2d_data_per_cam_dict = {}- base_charuco_data_npy_with_nans_for_missing_data_xy = np.zeros((number_of_tracked_points, 2))+ base_charuco_data_npy_with_nans_for_missing_data_xy = np.zeros(+ (number_of_tracked_points, 2)+ )base_charuco_data_npy_with_nans_for_missing_data_xy[:] = np.nanfor this_cam_data in this_multi_frame_charuco_data:if this_cam_data.charuco_view_data.some_charuco_corners_found:- this_frame_charuco_data_xy = base_charuco_data_npy_with_nans_for_missing_data_xy.copy()- charuco_ids_in_this_frame_idx = this_cam_data.charuco_view_data.charuco_ids- charuco_corners_in_this_frame_xy = this_cam_data.charuco_view_data.charuco_corners- this_frame_charuco_data_xy[charuco_ids_in_this_frame_idx, :] = charuco_corners_in_this_frame_xy+ this_frame_charuco_data_xy = (+ base_charuco_data_npy_with_nans_for_missing_data_xy.copy()+ )+ charuco_ids_in_this_frame_idx = (+ this_cam_data.charuco_view_data.charuco_ids+ )+ charuco_corners_in_this_frame_xy = (+ this_cam_data.charuco_view_data.charuco_corners+ )+ this_frame_charuco_data_xy[+ charuco_ids_in_this_frame_idx, :+ ] = charuco_corners_in_this_frame_xythis_webcam_id = this_cam_data.raw_frame_payload.webcam_idcharuco2d_data_per_cam_dict[this_webcam_id] = this_frame_charuco_data_xyreturn charuco2d_data_per_cam_dict+if __name__ == "__main__":pass# ## Jon, Run me to easily start a charuco board detect rundiff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_image_annotator.py b/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_image_annotator.pyindex 5dcef17b..ee958720 100644--- a/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_image_annotator.py+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/charuco_image_annotator.py@@ -3,13 +3,15 @@ from typing import Listimport cv2import numpy as np-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import CharucoViewData+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import (+ CharucoViewData,+)def _board_text(- charuco_ids: List[int],- num_charuco_corners: int,- full_board_detected_bool: bool,+ charuco_ids: List[int],+ num_charuco_corners: int,+ full_board_detected_bool: bool,):num_ids_as_str = str(len(charuco_ids))num_corners_as_str = str(num_charuco_corners)@@ -21,19 +23,17 @@ def _board_text(def annotate_image_with_charuco_data(- image,- charuco_view_data: CharucoViewData,- number_of_charuco_corners: int) -> bool:+ image, charuco_view_data: CharucoViewData, number_of_charuco_corners: int+) -> bool:annotated_image = cv2.aruco.drawDetectedCornersCharuco(image,np.array(charuco_view_data.charuco_corners),np.array(charuco_view_data.charuco_ids),- (0, 255, 125, 255)+ (0, 255, 125, 255),) # yellow? I# think cv2 uses BGR instead of RGB?-current_cam_corner_count_str = _board_text(charuco_view_data.charuco_ids,number_of_charuco_corners,@@ -72,5 +72,9 @@ def annotate_image_with_charuco_data(if charuco_view_data.full_board_found:cv2.polylines(- annotated_image, np.int32([charuco_view_data.charuco_corners]), False, (0, 100, 255), 2+ annotated_image,+ np.int32([charuco_view_data.charuco_corners]),+ False,+ (0, 100, 255),+ 2,)diff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_board_definition.py b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_board_definition.pyindex 1a88ec0e..0de571a0 100644--- a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_board_definition.py+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_board_definition.py@@ -11,16 +11,17 @@ class CharucoBoardDataClass:number_of_squares_width: int = 7number_of_squares_height: int = 5black_square_side_length: int = 1- aruco_marker_length_proportional: float = .8+ aruco_marker_length_proportional: float = 0.8def __post_init__(self):- self.charuco_board = cv2.aruco.CharucoBoard_create(self.number_of_squares_width,- self.number_of_squares_height,- self.black_square_side_length,- self.aruco_marker_length_proportional,- self.aruco_marker_dict)-- self.number_of_charuco_corners = (self.number_of_squares_width - 1) * (self.number_of_squares_height - 1)---+ self.charuco_board = cv2.aruco.CharucoBoard_create(+ self.number_of_squares_width,+ self.number_of_squares_height,+ self.black_square_side_length,+ self.aruco_marker_length_proportional,+ self.aruco_marker_dict,+ )++ self.number_of_charuco_corners = (self.number_of_squares_width - 1) * (+ self.number_of_squares_height - 1+ )diff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_frame_payload.py b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_frame_payload.pyindex 8cc07b49..dee283e1 100644--- a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_frame_payload.py+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_frame_payload.py@@ -3,8 +3,9 @@ from dataclasses import dataclassimport numpy as npfrom src.cameras.capture.dataclasses.frame_payload import FramePayload-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import \- CharucoViewData+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_view_data import (+ CharucoViewData,+)@dataclassdiff --git a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_view_data.py b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_view_data.pyindex 46b7db8d..f7767413 100644--- a/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_view_data.py+++ b/src/pipelines/calibration_pipeline/charuco_board_detection/dataclasses/charuco_view_data.py@@ -3,14 +3,18 @@ from dataclasses import dataclassimport numpy as np-from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import CharucoBoardDataClass+from src.pipelines.calibration_pipeline.charuco_board_detection.dataclasses.charuco_board_definition import (+ CharucoBoardDataClass,+)logger = logging.getLogger(__name__)@dataclassclass CharucoViewData:- charuco_board_object: CharucoBoardDataClass = None # the charuco board definition this 'view' was trying to detect+ charuco_board_object: CharucoBoardDataClass = (+ None # the charuco board definition this 'view' was trying to detect+ )charuco_corners: np.ndarray = Nonecharuco_ids: np.ndarray = Nonearuco_square_corners: np.ndarray = None@@ -26,4 +30,5 @@ class CharucoViewData:if self.some_charuco_corners_found:if self.charuco_board_object is None:logger.warning(- 'If there is any data in this view point, you should also include a copy of the charcuo_board that was being detected')+ "If there is any data in this view point, you should also include a copy of the charcuo_board that was being detected"+ )diff --git a/src/pipelines/session_pipeline/data_classes/data3d_full_session_data.py b/src/pipelines/session_pipeline/data_classes/data3d_full_session_data.pyindex 00976d66..5435d3ea 100644--- a/src/pipelines/session_pipeline/data_classes/data3d_full_session_data.py+++ b/src/pipelines/session_pipeline/data_classes/data3d_full_session_data.py@@ -10,6 +10,6 @@ class TweakedModel(BaseModel):class Data3dFullSessionPayload(TweakedModel):- has_data: bool = True,+ has_data: bool = (True,)data3d_numFrames_numTrackedPoints_XYZ: np.ndarray = Nonedata3d_numFrames_numTrackedPoint_reprojectionError: np.ndarray = Nonediff --git a/src/pipelines/session_pipeline/data_classes/data_3d_single_frame_payload.py b/src/pipelines/session_pipeline/data_classes/data_3d_single_frame_payload.pyindex ef58ac57..e8388045 100644--- a/src/pipelines/session_pipeline/data_classes/data_3d_single_frame_payload.py+++ b/src/pipelines/session_pipeline/data_classes/data_3d_single_frame_payload.py@@ -10,6 +10,6 @@ class TweakedModel(BaseModel):class Data3dMultiFramePayload(TweakedModel):- has_data: bool = True,+ has_data: bool = (True,)data3d_trackedPointNum_xyz: np.ndarray = Nonedata3d_trackedPointNum_reprojectionError: np.ndarray = Nonediff --git a/src/pipelines/session_pipeline/session_pipeline_orchestrator.py b/src/pipelines/session_pipeline/session_pipeline_orchestrator.pyindex 2e8e719c..9fd2b37e 100644--- a/src/pipelines/session_pipeline/session_pipeline_orchestrator.py+++ b/src/pipelines/session_pipeline/session_pipeline_orchestrator.py@@ -9,22 +9,31 @@ import numpy as npfrom src.cameras.persistence.video_writer.save_options_dataclass import SaveOptionsfrom src.config.data_paths import freemocap_data_path-from src.config.home_dir import create_session_folder, create_session_id, \- get_most_recent_session_id, get_session_output_data_folder_path-from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import \- MediaPipeSkeletonDetector, \- Mediapipe2dDataPayload+from src.config.home_dir import (+ create_session_folder,+ create_session_id,+ get_most_recent_session_id,+ get_session_output_data_folder_path,+)+from src.core_processor.mediapipe_skeleton_detector.mediapipe_skeleton_detector import (+ MediaPipeSkeletonDetector,+ Mediapipe2dDataPayload,+)from src.core_processor.show_cam_window import show_cam_windowfrom src.core_processor.timestamp_manager.timestamp_manager import TimestampManagerfrom src.core_processor.utils.image_fps_writer import write_fps_to_image-from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import \- CalibrationPipelineOrchestrator-from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import \- CharucoBoardDetector-from src.pipelines.session_pipeline.data_classes.data3d_full_session_data import \- Data3dFullSessionPayload-from src.pipelines.session_pipeline.data_classes.data_3d_single_frame_payload import \- Data3dMultiFramePayload+from src.pipelines.calibration_pipeline.calibration_pipeline_orchestrator import (+ CalibrationPipelineOrchestrator,+)+from src.pipelines.calibration_pipeline.charuco_board_detection.charuco_board_detector import (+ CharucoBoardDetector,+)+from src.pipelines.session_pipeline.data_classes.data3d_full_session_data import (+ Data3dFullSessionPayload,+)+from src.pipelines.session_pipeline.data_classes.data_3d_single_frame_payload import (+ Data3dMultiFramePayload,+)from src.qt_visualizer_and_gui.qt_visualizer_and_gui import QTVisualizerAndGuilogger = logging.getLogger(__name__)@@ -35,8 +44,9 @@ def get_canonical_time_str():class SessionPipelineOrchestrator:-- def __init__(self, session_id: str = None, expected_framerate: Union[int, float] = None):+ def __init__(+ self, session_id: str = None, expected_framerate: Union[int, float] = None+ ):if session_id is not None:self._session_id = session_id@@ -71,40 +81,50 @@ class SessionPipelineOrchestrator:return self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY@mediapipe2d_numCams_numFrames_numTrackedPoints_XY.setter- def mediapipe2d_numCams_numFrames_numTrackedPoints_XY(self,- mediapipe2d_numCams_numFrames_numTrackedPoints_XY: np.ndarray):- self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY = \+ def mediapipe2d_numCams_numFrames_numTrackedPoints_XY(+ self, mediapipe2d_numCams_numFrames_numTrackedPoints_XY: np.ndarray+ ):+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY = (mediapipe2d_numCams_numFrames_numTrackedPoints_XY+ )@propertydef session_folder_path(self):return freemocap_data_path / self._session_id- def calibrate_camera_capture_volume(self, use_most_recent_calibration: bool = False,+ def calibrate_camera_capture_volume(+ self,+ use_most_recent_calibration: bool = False,load_calibration_from_session_id: str = None,charuco_square_size: int = 1,- pin_camera_0_to_origin: bool = False):+ pin_camera_0_to_origin: bool = False,+ ):if use_most_recent_calibration:- self._anipose_camera_calibration_object = CalibrationPipelineOrchestrator(-- ).load_most_recent_calibration()+ self._anipose_camera_calibration_object = (+ CalibrationPipelineOrchestrator().load_most_recent_calibration()+ )returnelif load_calibration_from_session_id is not None:calibration_orchestrator = CalibrationPipelineOrchestrator(- load_calibration_from_session_id)+ load_calibration_from_session_id+ )else: # create new calibration- calibration_orchestrator = CalibrationPipelineOrchestrator(self.session_id,- expected_framerate=self.expected_framerate)- calibration_orchestrator.record_videos(show_visualizer_gui=False,- save_video_in_frame_loop=False,- show_camera_views_in_windows=True,- )-- self._anipose_camera_calibration_object = \+ calibration_orchestrator = CalibrationPipelineOrchestrator(+ self.session_id, expected_framerate=self.expected_framerate+ )+ calibration_orchestrator.record_videos(+ show_visualizer_gui=False,+ save_video_in_frame_loop=False,+ show_camera_views_in_windows=True,+ )++ self._anipose_camera_calibration_object = (calibration_orchestrator.run_anipose_camera_calibration(- charuco_square_size=charuco_square_size,- pin_camera_0_to_origin=pin_camera_0_to_origin)+ charuco_square_size=charuco_square_size,+ pin_camera_0_to_origin=pin_camera_0_to_origin,+ )+ )def record_new_session(self,@@ -117,23 +137,24 @@ class SessionPipelineOrchestrator:):if reconstruct_3d:try:- self.anipose_camera_calibration_object = CalibrationPipelineOrchestrator(-- ).load_calibration_from_session_id(- self.session_id)+ self.anipose_camera_calibration_object = (+ CalibrationPipelineOrchestrator().load_calibration_from_session_id(+ self.session_id+ )+ )except:logger.info(f"No calibration found for session_id: {self.session_id}, loading most recent "- f"calibration")- self.anipose_camera_calibration_object = CalibrationPipelineOrchestrator(-- ).load_most_recent_calibration()+ f"calibration"+ )+ self.anipose_camera_calibration_object = (+ CalibrationPipelineOrchestrator().load_most_recent_calibration()+ )dictionary_of_charuco_frame_payloads_on_this_multi_frame = {}dictionary_of_mediapipe_payloads_on_this_multi_frame = {}- with self._open_cv_camera_manager.start_capture_session_all_cams() as \- connected_cameras_dict:+ with self._open_cv_camera_manager.start_capture_session_all_cams() as connected_cameras_dict:self._number_of_cameras = len(connected_cameras_dict)@@ -142,46 +163,77 @@ class SessionPipelineOrchestrator:try:if show_visualizer_gui:self._visualizer_gui.setup_and_launch(- self._open_cv_camera_manager.available_webcam_ids)+ self._open_cv_camera_manager.available_webcam_ids+ )should_continue = Truewhile should_continue: # BIG FRAME LOOP STARTS HEREif reconstruct_3d:if detect_charuco:- if len(dictionary_of_charuco_frame_payloads_on_this_multi_frame) > 0:+ if (+ len(+ dictionary_of_charuco_frame_payloads_on_this_multi_frame+ )+ > 0+ ):charuco3d_multi_frame_payload = self._reconstruct_3d_charuco(- dictionary_of_charuco_frame_payloads_on_this_multi_frame)- if show_visualizer_gui and charuco3d_multi_frame_payload is not \- None:+ dictionary_of_charuco_frame_payloads_on_this_multi_frame+ )+ if (+ show_visualizer_gui+ and charuco3d_multi_frame_payload is not None+ ):self._visualizer_gui.update_charuco_3d_dottos(- charuco3d_multi_frame_payload)- dictionary_of_charuco_frame_payloads_on_this_multi_frame = {}+ charuco3d_multi_frame_payload+ )+ dictionary_of_charuco_frame_payloads_on_this_multi_frame = (+ {}+ )if detect_mediapipe:- if len(dictionary_of_mediapipe_payloads_on_this_multi_frame) > 0:+ if (+ len(+ dictionary_of_mediapipe_payloads_on_this_multi_frame+ )+ > 0+ ):mediapipe3d_multi_frame_payload = self._reconstruct_3d_mediapipe(- dictionary_of_mediapipe_payloads_on_this_multi_frame)- if show_visualizer_gui and mediapipe3d_multi_frame_payload is not\- None:+ dictionary_of_mediapipe_payloads_on_this_multi_frame+ )+ if (+ show_visualizer_gui+ and mediapipe3d_multi_frame_payload is not None+ ):self._visualizer_gui.update_mediapipe3d_skeleton(- mediapipe3d_multi_frame_payload)- dictionary_of_mediapipe_payloads_on_this_multi_frame = {}+ mediapipe3d_multi_frame_payload+ )+ dictionary_of_mediapipe_payloads_on_this_multi_frame = (+ {}+ )timestamp_manager.log_new_timestamp_for_main_loop_perf_coutner_ns(- time.perf_counter_ns())+ time.perf_counter_ns()+ )if not self._open_cv_camera_manager.new_multi_frame_ready():continue- this_multi_frame_payload = self._open_cv_camera_manager.latest_multi_frame+ this_multi_frame_payload = (+ self._open_cv_camera_manager.latest_multi_frame+ )if this_multi_frame_payload is None:continue- for this_webcam_id, this_open_cv_camera in connected_cameras_dict.items():+ for (+ this_webcam_id,+ this_open_cv_camera,+ ) in connected_cameras_dict.items():- this_cam_latest_frame = this_multi_frame_payload.frames_dict[this_webcam_id]+ this_cam_latest_frame = this_multi_frame_payload.frames_dict[+ this_webcam_id+ ]if this_cam_latest_frame is None:continue@@ -192,7 +244,8 @@ class SessionPipelineOrchestrator:create_session_folder(self.session_id)print(f"Recording: {this_open_cv_camera.webcam_id_as_str}, "- f"Frame# {this_open_cv_camera.latest_frame_number}")+ f"Frame# {this_open_cv_camera.latest_frame_number}"+ )# # save this frame straight to the video file (no risk of memory# overflow, but can't handle higher numbers of cameras)# this_open_cv_camera.video_recorder.save_frame_to_video_file(@@ -201,30 +254,39 @@ class SessionPipelineOrchestrator:# can handle large numbers of cameras, but will eventually fill up# RAM and cause a crashthis_open_cv_camera.video_recorder.append_frame_payload_to_list(- this_cam_latest_frame)+ this_cam_latest_frame+ )if detect_charuco:# detect charuco board- this_charuco_frame_payload = \+ this_charuco_frame_payload = (self._charuco_board_detector.detect_charuco_board(- this_cam_latest_frame)- image_to_display = this_charuco_frame_payload.annotated_image+ this_cam_latest_frame+ )+ )+ image_to_display = (+ this_charuco_frame_payload.annotated_image+ )if reconstruct_3d:dictionary_of_charuco_frame_payloads_on_this_multi_frame[- this_webcam_id] = this_charuco_frame_payload+ this_webcam_id+ ] = this_charuco_frame_payloadif detect_mediapipe:# detect mediapipe skeleton- this_mediapipe_frame_payload = \- self._mediapipe_skeleton_detector.detect_skeleton_in_image(+ this_mediapipe_frame_payload = self._mediapipe_skeleton_detector.detect_skeleton_in_image(raw_frame_payload=this_cam_latest_frame,- annotated_image=image_to_display)- image_to_display = this_mediapipe_frame_payload.annotated_image+ annotated_image=image_to_display,+ )+ image_to_display = (+ this_mediapipe_frame_payload.annotated_image+ )if reconstruct_3d:dictionary_of_mediapipe_payloads_on_this_multi_frame[- this_webcam_id] = this_mediapipe_frame_payload+ this_webcam_id+ ] = this_mediapipe_frame_payloadif show_camera_views_in_windows:should_continue = show_cam_window(@@ -235,10 +297,12 @@ class SessionPipelineOrchestrator:write_fps_to_image(image_to_display,timestamp_manager.median_frames_per_second_for_webcam(- this_webcam_id),+ this_webcam_id+ ),+ )+ self._visualizer_gui.update_camera_view_image(+ this_webcam_id, image_to_display)- self._visualizer_gui.update_camera_view_image(this_webcam_id,- image_to_display)if show_camera_views_in_windows:should_continue = show_cam_window(@@ -249,10 +313,12 @@ class SessionPipelineOrchestrator:write_fps_to_image(image_to_display,timestamp_manager.median_frames_per_second_for_webcam(- this_webcam_id),+ this_webcam_id+ ),+ )+ self._visualizer_gui.update_camera_view_image(+ this_webcam_id, image_to_display)- self._visualizer_gui.update_camera_view_image(this_webcam_id,- image_to_display)# self._visualizer_gui.update_timestamp_plots(timestamp_manager)if self._visualizer_gui.shut_it_down:@@ -261,7 +327,8 @@ class SessionPipelineOrchestrator:# save frames if 'record' button is pressedthis_open_cv_camera.record_frames(- self._visualizer_gui.record_button_pressed)+ self._visualizer_gui.record_button_pressed+ )# exit loop when user presses ESC keyexit_key = cv2.waitKey(1)@@ -269,17 +336,19 @@ class SessionPipelineOrchestrator:logger.info("ESC has been pressed.")should_continue = False-except:logger.error("Printing traceback")traceback.print_exc()finally:for this_open_cv_camera in connected_cameras_dict.values():this_open_cv_camera.video_recorder.save_list_of_frames_to_video_file(- this_open_cv_camera.frame_list)+ this_open_cv_camera.frame_list+ )if show_camera_views_in_windows:- logger.info(f"Destroy window {this_open_cv_camera.webcam_id_as_str}")+ logger.info(+ f"Destroy window {this_open_cv_camera.webcam_id_as_str}"+ )cv2.destroyWindow(this_open_cv_camera.webcam_id_as_str)if show_visualizer_gui:@@ -287,81 +356,114 @@ class SessionPipelineOrchestrator:timestamp_manager.create_diagnostic_plots()- def _reconstruct_3d_charuco(self, dictionary_of_charuco_payloads_on_this_multiframe) -> \- Union[None, Data3dMultiFramePayload]:+ def _reconstruct_3d_charuco(+ self, dictionary_of_charuco_payloads_on_this_multiframe+ ) -> Union[None, Data3dMultiFramePayload]:is_there_any_charuco_data = []- for this_cam_charuco_data in dictionary_of_charuco_payloads_on_this_multiframe.values():+ for (+ this_cam_charuco_data+ ) in dictionary_of_charuco_payloads_on_this_multiframe.values():is_there_any_charuco_data.append(- this_cam_charuco_data.charuco_view_data.some_charuco_corners_found)+ this_cam_charuco_data.charuco_view_data.some_charuco_corners_found+ )min_cameras_to_reconstruct = 2- if sum(- is_there_any_charuco_data) >= min_cameras_to_reconstruct: # if at least 2 cameras+ if (+ sum(is_there_any_charuco_data) >= min_cameras_to_reconstruct+ ): # if at least 2 cameras# have any data, then we can try to make some 3d dottoscharuco2d_data_per_cam_dict = self._format_charuco2d_data(- dictionary_of_charuco_payloads_on_this_multiframe)- charuco_3d_data_payload = self._triangulate_2d_data(charuco2d_data_per_cam_dict,- self._charuco_board_detector.number_of_charuco_corners)+ dictionary_of_charuco_payloads_on_this_multiframe+ )+ charuco_3d_data_payload = self._triangulate_2d_data(+ charuco2d_data_per_cam_dict,+ self._charuco_board_detector.number_of_charuco_corners,+ )x = charuco_3d_data_payload.data3d_trackedPointNum_xyz[:, 0]return charuco_3d_data_payload- def _format_charuco2d_data(self, dictionary_of_charuco_payloads_on_this_multiframe) -> Dict:+ def _format_charuco2d_data(+ self, dictionary_of_charuco_payloads_on_this_multiframe+ ) -> Dict:- number_of_tracked_points = self._charuco_board_detector.number_of_charuco_corners+ number_of_tracked_points = (+ self._charuco_board_detector.number_of_charuco_corners+ )base_charuco_data_npy_with_nans_for_missing_data_xy = np.zeros(- (number_of_tracked_points, 2))+ (number_of_tracked_points, 2)+ )base_charuco_data_npy_with_nans_for_missing_data_xy[:] = np.nancharuco2d_data_per_cam_dict = {}for this_cam_data in dictionary_of_charuco_payloads_on_this_multiframe.values():- this_frame_charuco_data_xy = base_charuco_data_npy_with_nans_for_missing_data_xy.copy()+ this_frame_charuco_data_xy = (+ base_charuco_data_npy_with_nans_for_missing_data_xy.copy()+ )charuco_ids_in_this_frame_idx = this_cam_data.charuco_view_data.charuco_ids- charuco_corners_in_this_frame_xy = this_cam_data.charuco_view_data.charuco_corners+ charuco_corners_in_this_frame_xy = (+ this_cam_data.charuco_view_data.charuco_corners+ )if charuco_ids_in_this_frame_idx.__class__ != list:- this_frame_charuco_data_xy[charuco_ids_in_this_frame_idx,- :] = charuco_corners_in_this_frame_xy+ this_frame_charuco_data_xy[+ charuco_ids_in_this_frame_idx, :+ ] = charuco_corners_in_this_frame_xythis_webcam_id = this_cam_data.raw_frame_payload.webcam_idcharuco2d_data_per_cam_dict[this_webcam_id] = this_frame_charuco_data_xyreturn charuco2d_data_per_cam_dict- def _reconstruct_3d_mediapipe(self, dictionary_of_mediapipe_payloads_on_this_multi_frame) -> \- Union[None, Data3dMultiFramePayload]:+ def _reconstruct_3d_mediapipe(+ self, dictionary_of_mediapipe_payloads_on_this_multi_frame+ ) -> Union[None, Data3dMultiFramePayload]:is_there_any_mediapipe_data = []- for this_cam_medipipe_data in dictionary_of_mediapipe_payloads_on_this_multi_frame.values():+ for (+ this_cam_medipipe_data+ ) in dictionary_of_mediapipe_payloads_on_this_multi_frame.values():is_there_any_mediapipe_data.append(- this_cam_medipipe_data.pixel_data_numpy_arrays.has_data)+ this_cam_medipipe_data.pixel_data_numpy_arrays.has_data+ )min_cameras_to_reconstruct = 2- if sum(- is_there_any_mediapipe_data) >= min_cameras_to_reconstruct: # if at least 2 cameras+ if (+ sum(is_there_any_mediapipe_data) >= min_cameras_to_reconstruct+ ): # if at least 2 cameras# have any data, then we can try to make some 3d dottosmediapipe2d_data_per_cam_dict = {}- for this_webcam_id, this_mediapipe_payload in \- dictionary_of_mediapipe_payloads_on_this_multi_frame.items():+ for (+ this_webcam_id,+ this_mediapipe_payload,+ ) in dictionary_of_mediapipe_payloads_on_this_multi_frame.items():self._threshold_2d_data_by_confidence(- this_mediapipe_payload.pixel_data_numpy_arrays, .5)+ this_mediapipe_payload.pixel_data_numpy_arrays, 0.5+ )mediapipe2d_data_per_cam_dict[- this_webcam_id] = \+ this_webcam_id+ ] = (this_mediapipe_payload.pixel_data_numpy_arrays.all_data2d_nFrames_nTrackedPts_XY+ )- mediapipe_3d_data_payload = self._triangulate_2d_data(mediapipe2d_data_per_cam_dict,- self._mediapipe_skeleton_detector.number_of_tracked_points_total)+ mediapipe_3d_data_payload = self._triangulate_2d_data(+ mediapipe2d_data_per_cam_dict,+ self._mediapipe_skeleton_detector.number_of_tracked_points_total,+ )return mediapipe_3d_data_payload- def _triangulate_2d_data(self, data2d_per_cam_dict: Dict,- number_of_tracked_points: int) -> Data3dMultiFramePayload:+ def _triangulate_2d_data(+ self, data2d_per_cam_dict: Dict, number_of_tracked_points: int+ ) -> Data3dMultiFramePayload:- each_cam2d_data_list = [this_cam_charuco2d_frame_xy for this_cam_charuco2d_frame_xy in- data2d_per_cam_dict.values()]+ each_cam2d_data_list = [+ this_cam_charuco2d_frame_xy+ for this_cam_charuco2d_frame_xy in data2d_per_cam_dict.values()+ ]data2d_camNum_trackedPointNum_xy = np.asarray(each_cam2d_data_list)@@ -369,10 +471,11 @@ class SessionPipelineOrchestrator:# become a 3d estimate. Hurray! :`D# simple triangulation- data3d_trackedPointNum_xyz = self._anipose_camera_calibration_object.triangulate(- data2d_camNum_trackedPointNum_xy,- progress=False,- undistort=True)+ data3d_trackedPointNum_xyz = (+ self._anipose_camera_calibration_object.triangulate(+ data2d_camNum_trackedPointNum_xy, progress=False, undistort=True+ )+ )# # triangulation with ransac (to find the best combo of cameras to minimize reprojection# error, I think?@@ -390,87 +493,122 @@ class SessionPipelineOrchestrator:# they choose to call it) for thresholding, but it's problematic because of the# neural_networks' propensity to apply high confidence to bonkers estimates, which leads# to SPOOKY GHOST SKELETONS! :O- data3d_trackedPointNum_reprojectionError = \+ data3d_trackedPointNum_reprojectionError = (self._anipose_camera_calibration_object.reprojection_error(- data3d_trackedPointNum_xyz,- data2d_camNum_trackedPointNum_xy,- mean=True)+ data3d_trackedPointNum_xyz, data2d_camNum_trackedPointNum_xy, mean=True+ )+ )- return Data3dMultiFramePayload(has_data=True,- data3d_trackedPointNum_xyz=data3d_trackedPointNum_xyz,- data3d_trackedPointNum_reprojectionError=data3d_trackedPointNum_reprojectionError, )+ return Data3dMultiFramePayload(+ has_data=True,+ data3d_trackedPointNum_xyz=data3d_trackedPointNum_xyz,+ data3d_trackedPointNum_reprojectionError=data3d_trackedPointNum_reprojectionError,+ )def mediapipe_track_skeletons_offline(self):- self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY = \+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY = (self._mediapipe_skeleton_detector.process_session_folder(- save_annotated_videos=True)+ save_annotated_videos=True+ )+ )def reconstruct3d_from_2d_data_offline(self):- number_of_cameras = self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[0]- number_of_frames = self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[1]- number_of_tracked_points = self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[2]- number_of_spatial_dimensions = \- self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[3]+ number_of_cameras = (+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[0]+ )+ number_of_frames = (+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[1]+ )+ number_of_tracked_points = (+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[2]+ )+ number_of_spatial_dimensions = (+ self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.shape[3]+ )if not number_of_spatial_dimensions == 2:logger.error(f"This is supposed to be 2D data but, number_of_spatial_dimensio"- f"ns: {number_of_spatial_dimensions}")+ f"ns: {number_of_spatial_dimensions}"+ )raise Exception# reshape data to collapse across 'frames' so it becomes [number_of_cameras,# number_of_2d_points(numFrames*numPoints), XY]data2d_flat = self._mediapipe2d_numCams_numFrames_numTrackedPoints_XY.reshape(- number_of_cameras,- -1,- 2)-- logger.info(f"Reconstructing 3d points from 2d points with shape: "- f"number_of_cameras: {number_of_cameras},"- f" number_of_frames: {number_of_frames}, "- f" number_of_tracked_points: {number_of_tracked_points},"- f" number_of_spatial_dimensions: {number_of_spatial_dimensions}")-- data3d_flat = self.anipose_camera_calibration_object.triangulate(data2d_flat, progress=True)-- data3d_reprojectionError_flat = self.anipose_camera_calibration_object.reprojection_error(- data3d_flat,- data2d_flat,- mean=True)-- data3d_numFrames_numTrackedPoints_XYZ = data3d_flat.reshape(number_of_frames,- number_of_tracked_points, 3)- data3d_numFrames_numTrackedPoints_reprojectionError = data3d_reprojectionError_flat.reshape(- number_of_frames,- number_of_tracked_points)-- self._save_mediapipe3d_data_to_npy(data3d_numFrames_numTrackedPoints_XYZ,- data3d_numFrames_numTrackedPoints_reprojectionError)+ number_of_cameras, -1, 2+ )++ logger.info(+ f"Reconstructing 3d points from 2d points with shape: "+ f"number_of_cameras: {number_of_cameras},"+ f" number_of_frames: {number_of_frames}, "+ f" number_of_tracked_points: {number_of_tracked_points},"+ f" number_of_spatial_dimensions: {number_of_spatial_dimensions}"+ )++ data3d_flat = self.anipose_camera_calibration_object.triangulate(+ data2d_flat, progress=True+ )++ data3d_reprojectionError_flat = (+ self.anipose_camera_calibration_object.reprojection_error(+ data3d_flat, data2d_flat, mean=True+ )+ )++ data3d_numFrames_numTrackedPoints_XYZ = data3d_flat.reshape(+ number_of_frames, number_of_tracked_points, 3+ )+ data3d_numFrames_numTrackedPoints_reprojectionError = (+ data3d_reprojectionError_flat.reshape(+ number_of_frames, number_of_tracked_points+ )+ )++ self._save_mediapipe3d_data_to_npy(+ data3d_numFrames_numTrackedPoints_XYZ,+ data3d_numFrames_numTrackedPoints_reprojectionError,+ )self._mediapipe3d_full_session_payload = Data3dFullSessionPayload(data3d_numFrames_numTrackedPoints_XYZ=data3d_numFrames_numTrackedPoints_XYZ,- data3d_numFrames_numTrackedPoint_reprojectionError- =data3d_numFrames_numTrackedPoints_reprojectionError)+ data3d_numFrames_numTrackedPoint_reprojectionError=data3d_numFrames_numTrackedPoints_reprojectionError,+ )- def _save_mediapipe3d_data_to_npy(self, data3d_numFrames_numTrackedPoints_XYZ: np.ndarray,- data3d_numFrames_numTrackedPoints_reprojectionError: np.ndarray):+ def _save_mediapipe3d_data_to_npy(+ self,+ data3d_numFrames_numTrackedPoints_XYZ: np.ndarray,+ data3d_numFrames_numTrackedPoints_reprojectionError: np.ndarray,+ ):output_data_folder = Path(get_session_output_data_folder_path(self._session_id))# save spatial XYZ data- self._mediapipe_3dData_save_path = output_data_folder / \- "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"+ self._mediapipe_3dData_save_path = (+ output_data_folder+ / "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"+ )logger.info(f"saving: {self._mediapipe_3dData_save_path}")- np.save(str(self._mediapipe_3dData_save_path), data3d_numFrames_numTrackedPoints_XYZ)+ np.save(+ str(self._mediapipe_3dData_save_path), data3d_numFrames_numTrackedPoints_XYZ+ )# save reprojection error- self._mediapipe_reprojection_error_save_path = output_data_folder / \- "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"+ self._mediapipe_reprojection_error_save_path = (+ output_data_folder+ / "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"+ )logger.info(f"saving: {self._mediapipe_reprojection_error_save_path}")- np.save(str(self._mediapipe_reprojection_error_save_path),- data3d_numFrames_numTrackedPoints_reprojectionError)+ np.save(+ str(self._mediapipe_reprojection_error_save_path),+ data3d_numFrames_numTrackedPoints_reprojectionError,+ )- def _threshold_2d_data_by_confidence(self, mediapipe2d_data_payload: Mediapipe2dDataPayload,- confidence_threshold: float):+ def _threshold_2d_data_by_confidence(+ self,+ mediapipe2d_data_payload: Mediapipe2dDataPayload,+ confidence_threshold: float,+ ):pass@@ -478,50 +616,64 @@ def load_mediapipe3d_skeleton_data(session_id: str = None):if session_id is None:session_id = get_most_recent_session_id()output_data_folder = Path(get_session_output_data_folder_path(session_id))- mediapipe3d_xyz_file_path = output_data_folder / \- "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"+ mediapipe3d_xyz_file_path = (+ output_data_folder+ / "mediapipe_3dData_numFrames_numTrackedPoints_spatialXYZ.npy"+ )logger.info(f"loading: {mediapipe3d_xyz_file_path}")- mediapipe3d_skeleton_nFrames_nTrajectories_xyz = np.load(str(mediapipe3d_xyz_file_path))-- mediapipe_reprojection_error_file_path = output_data_folder / \- "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"+ mediapipe3d_skeleton_nFrames_nTrajectories_xyz = np.load(+ str(mediapipe3d_xyz_file_path)+ )++ mediapipe_reprojection_error_file_path = (+ output_data_folder+ / "mediapipe_3dData_numFrames_numTrackedPoints_reprojectionError.npy"+ )logger.info(f"loading: {mediapipe_reprojection_error_file_path}")mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError = np.load(- str(mediapipe_reprojection_error_file_path))+ str(mediapipe_reprojection_error_file_path)+ )return Data3dFullSessionPayload(data3d_numFrames_numTrackedPoints_XYZ=mediapipe3d_skeleton_nFrames_nTrajectories_xyz,- data3d_numFrames_numTrackedPoint_reprojectionError- =mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError)+ data3d_numFrames_numTrackedPoint_reprojectionError=mediapipe3d_skeleton_nFrames_nTrajectories_reprojectionError,+ )def load_mediapipe2d_data(session_id: str = None):if session_id is None:session_id = get_most_recent_session_id()output_data_folder = Path(get_session_output_data_folder_path(session_id))- mediapipe2d_xy_file_path = output_data_folder / \- "mediapipe_2dData_numCams_numFrames_numTrackedPoints_pixelXY.npy"+ mediapipe2d_xy_file_path = (+ output_data_folder+ / "mediapipe_2dData_numCams_numFrames_numTrackedPoints_pixelXY.npy"+ )logger.info(f"loading: {mediapipe2d_xy_file_path}")- mediapipe2d_numCams_numFrames_numTrackedPoints_pixelXY = np.load(mediapipe2d_xy_file_path)+ mediapipe2d_numCams_numFrames_numTrackedPoints_pixelXY = np.load(+ mediapipe2d_xy_file_path+ )# TODO - create a dataclass for 2d mediapipe data, including things split up by body, hands,# and face (and maybe the raw `mediapipe results` data things if that's not too much data)return mediapipe2d_numCams_numFrames_numTrackedPoints_pixelXYif __name__ == "__main__":- print('running `session_pipeline_orchestrator` as `__main__')+ print("running `session_pipeline_orchestrator` as `__main__")length_of_one_edge_of_a_black_square_on_the_charuco_board_in_mm = 39expected_framerate = 25calibrate_cameras = True- this_session_orchestrator = SessionPipelineOrchestrator(expected_framerate=expected_framerate)+ this_session_orchestrator = SessionPipelineOrchestrator(+ expected_framerate=expected_framerate+ )if calibrate_cameras:this_session_orchestrator.calibrate_camera_capture_volume(charuco_square_size=length_of_one_edge_of_a_black_square_on_the_charuco_board_in_mm,- pin_camera_0_to_origin=True)+ pin_camera_0_to_origin=True,+ )this_session_orchestrator.record_new_session()diff --git a/src/qt_visualizer_and_gui/qt_visualizer_and_gui.py b/src/qt_visualizer_and_gui/qt_visualizer_and_gui.pyindex ddbed52e..47415c39 100644--- a/src/qt_visualizer_and_gui/qt_visualizer_and_gui.py+++ b/src/qt_visualizer_and_gui/qt_visualizer_and_gui.py@@ -12,7 +12,9 @@ from pyqtgraph.Qt import QtWidgetsfrom mediapipe.python.solutions import holistic as mp_holisticfrom src.core_processor.timestamp_manager.timestamp_manager import TimestampManager-from src.pipelines.session_pipeline.data_classes.data_3d_single_frame_payload import Data3dMultiFramePayload+from src.pipelines.session_pipeline.data_classes.data_3d_single_frame_payload import (+ Data3dMultiFramePayload,+)logger = logging.getLogger(__name__)@@ -22,8 +24,8 @@ class QTVisualizerAndGui:# https://pyqtgraph.readthedocs.io/en/latest/config_options.htmlself._skeleton_connections_list = Noneself._mediapipe_skeleton_initialized = False- self.pyqtgraph_app = pg.mkQApp('freemocap! :D')- pg.setConfigOptions(imageAxisOrder='row-major')+ self.pyqtgraph_app = pg.mkQApp("freemocap! :D")+ pg.setConfigOptions(imageAxisOrder="row-major")self._dict_of_camera_image_item_widgets = {}self._number_of_cameras = None@@ -54,14 +56,14 @@ class QTVisualizerAndGui:self._main_window_widget.close()def setup_and_launch(self, webcam_ids_list):- logger.info('setting up QT Visualizer and GUI')+ logger.info("setting up QT Visualizer and GUI")self._number_of_cameras = len(webcam_ids_list)self._webcam_ids_list = webcam_ids_listself._setup_main_window()self._setup_control_panel()self._setup_camera_views_dock()self._setup_3d_viewport()- logger.info('launching QT Visualizer and GUI window')+ logger.info("launching QT Visualizer and GUI window")self._main_window_widget.show()@@ -70,11 +72,15 @@ class QTVisualizerAndGui:returntry:- camera_image_item_widget = self._dict_of_camera_image_item_widgets[webcam_id]+ camera_image_item_widget = self._dict_of_camera_image_item_widgets[+ webcam_id+ ]except Exception as e:- logger.warning(f'Could not find ViewBoxWidget for camera {webcam_id}')+ logger.warning(f"Could not find ViewBoxWidget for camera {webcam_id}")raise e- camera_image_item_widget.setImage(cv2.cvtColor(image_to_display, cv2.COLOR_BGR2RGB))+ camera_image_item_widget.setImage(+ cv2.cvtColor(image_to_display, cv2.COLOR_BGR2RGB)+ )def _setup_main_window(self, window_width: int = 1000, window_height: int = 1000):"""@@ -86,22 +92,22 @@ class QTVisualizerAndGui:self._main_dock_area = DockArea()self._main_window_widget.setCentralWidget(self._main_dock_area)- self._main_window_widget.setWindowTitle('Main Window ;D')+ self._main_window_widget.setWindowTitle("Main Window ;D")def _setup_control_panel(self):- self._control_panel_dock = Dock('Control Panel', size=(1, 1))+ self._control_panel_dock = Dock("Control Panel", size=(1, 1))control_panel_layout_widget = pg.LayoutWidget()self._control_panel_dock.addWidget(control_panel_layout_widget)label = QtWidgets.QLabel("Blah Blah Blah put words here ")control_panel_layout_widget.addWidget(label, row=0, col=0)- self._pause_button = QtWidgets.QPushButton('Pause')+ self._pause_button = QtWidgets.QPushButton("Pause")self._pause_button.setEnabled(True)self._pause_button.clicked.connect(self._pause)control_panel_layout_widget.addWidget(self._pause_button, row=1, col=0)- self._play_button = QtWidgets.QPushButton('Resume')+ self._play_button = QtWidgets.QPushButton("Resume")self._play_button.setEnabled(False)self._play_button.clicked.connect(self._play)control_panel_layout_widget.addWidget(self._play_button, row=2, col=0)@@ -111,26 +117,30 @@ class QTVisualizerAndGui:# self._reset_calibration_button.clicked.connect(self._reset_calibration)# control_panel_layout_widget.addWidget(self._reset_calibration_button, row=3, col=0)#- self._start_record_button = QtWidgets.QPushButton('Start Recording Video')+ self._start_record_button = QtWidgets.QPushButton("Start Recording Video")self._start_record_button.setEnabled(True)self._start_record_button.clicked.connect(self._record_button_pressed)control_panel_layout_widget.addWidget(self._start_record_button, row=3, col=0)- self._close_button = QtWidgets.QPushButton('Close All')+ self._close_button = QtWidgets.QPushButton("Close All")self._close_button.setEnabled(True)self._close_button.clicked.connect(self._close_button_pressed)control_panel_layout_widget.addWidget(self._close_button, row=4, col=0)- self._main_dock_area.addDock(self._control_panel_dock, position='left')+ self._main_dock_area.addDock(self._control_panel_dock, position="left")def _setup_camera_views_dock(self):self._camera_graphics_layout_window = pg.GraphicsLayoutWidget()- self._camera_views_dock = Dock('Camera Views')+ self._camera_views_dock = Dock("Camera Views")self._camera_views_dock.addWidget(self._camera_graphics_layout_window)- self._main_dock_area.addDock(self._camera_views_dock, 'right', self._control_panel_dock)+ self._main_dock_area.addDock(+ self._camera_views_dock, "right", self._control_panel_dock+ )for this_webcam_id in self._webcam_ids_list:- self._dict_of_camera_image_item_widgets[this_webcam_id] = self._create_camera_view_widget()+ self._dict_of_camera_image_item_widgets[+ this_webcam_id+ ] = self._create_camera_view_widget()def _create_camera_view_widget(self):camera_view_box_widget = pg.ViewBox(invertY=True, lockAspect=True)@@ -159,13 +169,15 @@ class QTVisualizerAndGui:self._start_record_button.setText("Recording! Click Again to stop")else:self._record_video_bool = False- self._start_record_button.setText("Click to record more, or click Close All to stop")+ self._start_record_button.setText(+ "Click to record more, or click Close All to stop"+ )def _setup_3d_viewport(self):self.opengl_3d_plot_widget = gl.GLViewWidget()# self.opengl_3d_plot_widget.opts['center'] = (0,0,0)- self.opengl_3d_plot_widget.opts['distance'] = 2e3+ self.opengl_3d_plot_widget.opts["distance"] = 2e3# self.opengl_3d_plot_widget.opts['azimuth'] = 0# self.opengl_3d_plot_widget.opts['elevation'] = 0@@ -176,32 +188,30 @@ class QTVisualizerAndGui:y_axis_line_array = np.array([[0, 0, 0], [0, 100, 0]])z_axis_line_array = np.array([[0, 0, 0], [0, 0, 100]])- self.origin_x_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(pos=x_axis_line_array,- color=(1, 0, 0, 1),- width=1.,- antialias=True)- self.origin_y_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(pos=y_axis_line_array,- color=(0, 1, 0, 1),- width=1.,- antialias=True)- self.origin_z_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(pos=z_axis_line_array,- color=(0, 0, 1, 1),- width=1.,- antialias=True)+ self.origin_x_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(+ pos=x_axis_line_array, color=(1, 0, 0, 1), width=1.0, antialias=True+ )+ self.origin_y_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(+ pos=y_axis_line_array, color=(0, 1, 0, 1), width=1.0, antialias=True+ )+ self.origin_z_axis_gl_lineplot_item = pg.opengl.GLLinePlotItem(+ pos=z_axis_line_array, color=(0, 0, 1, 1), width=1.0, antialias=True+ )self.opengl_3d_plot_widget.addItem(self.origin_x_axis_gl_lineplot_item)self.opengl_3d_plot_widget.addItem(self.origin_y_axis_gl_lineplot_item)self.opengl_3d_plot_widget.addItem(self.origin_z_axis_gl_lineplot_item)- self.opengl_charuco_scatter_item = gl.GLScatterPlotItem(pos=(0, 0, 0),- color=(1, 0, 1, 1),- size=10,- pxMode=False)+ self.opengl_charuco_scatter_item = gl.GLScatterPlotItem(+ pos=(0, 0, 0), color=(1, 0, 1, 1), size=10, pxMode=False+ )self.opengl_3d_plot_widget.addItem(self.opengl_charuco_scatter_item)self.opengl_3d_plot_dock = Dock("3d View Port")self.opengl_3d_plot_dock.addWidget(self.opengl_3d_plot_widget)- self._main_dock_area.addDock(self.opengl_3d_plot_dock, 'bottom', self._camera_views_dock)+ self._main_dock_area.addDock(+ self.opengl_3d_plot_dock, "bottom", self._camera_views_dock+ )def create_grid_planes(self):grid_scale = 2e3@@ -229,9 +239,7 @@ class QTVisualizerAndGui:def initialize_charuco_dottos(self, number_of_charuco_corners: int):dummy_charuco_points = np.zeros((number_of_charuco_corners, 3))self._charuco_scatter_item = gl.GLScatterPlotItem(- pos=dummy_charuco_points,- color=(1, 1, 0, 1),- size=20+ pos=dummy_charuco_points, color=(1, 1, 0, 1), size=20)self.opengl_3d_plot_widget.addItem(self._charuco_scatter_item)@@ -246,14 +254,19 @@ class QTVisualizerAndGui:)def get_mediapipe_connections(self):- self.mediapipe_body_connections = [this_connection for this_connection in mp_holistic.POSE_CONNECTIONS]- self.mediapipe_hand_connections = [this_connection for this_connection in mp_holistic.HAND_CONNECTIONS]- self.mediapipe_face_connections = [this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION]+ self.mediapipe_body_connections = [+ this_connection for this_connection in mp_holistic.POSE_CONNECTIONS+ ]+ self.mediapipe_hand_connections = [+ this_connection for this_connection in mp_holistic.HAND_CONNECTIONS+ ]+ self.mediapipe_face_connections = [+ this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION+ ]def initialize_skel_dottos(self, mediapipe_trackedPoint_xyz: np.ndarray):self.skeleton_scatter_item = gl.GLScatterPlotItem(- pos=mediapipe_trackedPoint_xyz, color=(0, 1, 1, 1), size=10,- pxMode=False+ pos=mediapipe_trackedPoint_xyz, color=(0, 1, 1, 1), size=10, pxMode=False)self.opengl_3d_plot_widget.addItem(self.skeleton_scatter_item)@@ -261,12 +274,15 @@ class QTVisualizerAndGui:self._skeleton_connections_list = []for this_connection in self.mediapipe_body_connections:this_skel_line = gl.GLLinePlotItem(- pos=mediapipe_trackedPoint_xyz[this_connection, :])+ pos=mediapipe_trackedPoint_xyz[this_connection, :]+ )self._skeleton_connections_list.append(this_skel_line)self.opengl_3d_plot_widget.addItem(this_skel_line)def update_mediapipe3d_skeleton(self, mediapipe3d_multi_frame_payload):- mediapipe3d_trackedPoint_xyz = mediapipe3d_multi_frame_payload.data3d_trackedPointNum_xyz+ mediapipe3d_trackedPoint_xyz = (+ mediapipe3d_multi_frame_payload.data3d_trackedPointNum_xyz+ )if not self._mediapipe_skeleton_initialized:self.get_mediapipe_connections()self.initialize_skel_dottos(mediapipe3d_trackedPoint_xyz)@@ -274,17 +290,16 @@ class QTVisualizerAndGui:self._mediapipe_skeleton_initialized = True# skel dottos- self.skeleton_scatter_item.setData(- pos=mediapipe3d_trackedPoint_xyz- )+ self.skeleton_scatter_item.setData(pos=mediapipe3d_trackedPoint_xyz)- #skel lines- for this_skeleton_line_number, this_connection in enumerate(self.mediapipe_body_connections):+ # skel lines+ for this_skeleton_line_number, this_connection in enumerate(+ self.mediapipe_body_connections+ ):self._skeleton_connections_list[this_skeleton_line_number].setData(pos=mediapipe3d_trackedPoint_xyz[this_connection, :])-if __name__ == "__main__":passdiff --git a/src/tests/video_writer/test_video_writer.py b/src/tests/video_writer/test_video_writer.pyindex 4a1625ce..dab82fe9 100644--- a/src/tests/video_writer/test_video_writer.py+++ b/src/tests/video_writer/test_video_writer.py@@ -21,7 +21,7 @@ class VideoWriterTestCase(TestCase):def test_write_on_video_writer(self):example_payload = FramePayload(image=np.random.randint(0, 4, (720, 1280, 3)),- timestamp_in_seconds_from_record_start=time.time_ns()+ timestamp_in_seconds_from_record_start=time.time_ns(),)vw = VideoRecorder("nothing", 100, 100, "nothing")vw.append_frame_payload_to_list(example_payload)diff --git a/src/utils/get_canonical_time_str.py b/src/utils/get_canonical_time_str.pyindex 2072c067..bd35e428 100644--- a/src/utils/get_canonical_time_str.py+++ b/src/utils/get_canonical_time_str.py@@ -1,4 +1 @@import time---