Free Motion Capture for Everyone πβ¨
TypeScript
copying my laser_skeleton code from the `main` branch.
Jonf627d0b0d752
diff --git a/jon_scratch/calculate_head_rotation_matrix.ipynb b/jon_scratch/calculate_head_rotation_matrix.ipynbnew file mode 100644index 00000000..a066aa25--- /dev/null+++ b/jon_scratch/calculate_head_rotation_matrix.ipynb@@ -0,0 +1,700 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "# Calculate head rotation matricies\n",+ "\n",+ "calculate orthonormal basis vectors of the head on each frame, stack them on top of each other to make 3x3 rotation matricies, because apparently that's how that works?!\n",+ "\n",+ "1 - calculate `head_center_xyz` as mean of left and right ear markers\n",+ "2 - define x+ as vector between `head_center_xyz` and `nose_xyz`- normalize to get `x_hat`\n",+ "3 - define y+ as vector between `head_center_xyz` and `left_ear_xyz` - normalize to get `y_hat_temp` (will be replaced later, to ensure orthogonal \\)\n",+ "4 - define z+ as cross product of `x_hat` and `y_hat`\n",+ "5 - replace `y_hat_temp` with `z_hat` crossed wtih `x_hat` (Pretty sure that makes it right handed...)\n",+ "\n",+ "Stack x_hat, y_hat, and z_hat on top of each other, and that's your head rotation matrix on that frame\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "install requirements to Python kernel (I think a 'kernel' is roughly equivalent to a conda environment, i.e. telling the system which version of `python.exe` you want to use to evaluate this Python code )"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Requirement already satisfied: numpy in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (1.22.3)\n",+ "Requirement already satisfied: matplotlib in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (3.5.1)\n",+ "Requirement already satisfied: cycler>=0.10 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from matplotlib) (0.11.0)\n",+ "Requirement already satisfied: pyparsing>=2.2.1 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from matplotlib) (3.0.8)\n",+ "Requirement already satisfied: numpy>=1.17 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from matplotlib) (1.22.3)\n",+ "Requirement already satisfied: fonttools>=4.22.0 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from matplotlib) (4.33.2)\n",+ "Requirement already satisfied: pillow>=6.2.0 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from matplotlib) (9.1.0)\n",+ "Requirement already satisfied: packaging>=20.0 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from matplotlib) (21.3)\n",+ "Requirement already satisfied: kiwisolver>=1.0.1 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from matplotlib) (1.4.2)\n",+ "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from matplotlib) (2.8.2)\n",+ "Requirement already satisfied: six>=1.5 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\n",+ "Requirement already satisfied: pyqt5 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (5.15.6)\n",+ "Requirement already satisfied: PyQt5-Qt5>=5.15.2 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from pyqt5) (5.15.2)\n",+ "Requirement already satisfied: PyQt5-sip<13,>=12.8 in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (from pyqt5) (12.10.1)\n",+ "Requirement already satisfied: keyboard in c:\\users\\jonma\\miniconda3\\envs\\humon-env\\lib\\site-packages (0.13.5)\n"+ ]+ }+ ],+ "source": [+ "#I should specify the versions to ensure future stability\n",+ "!pip install numpy\n",+ "!pip install matplotlib\n",+ "!pip install pyqt5\n",+ "!pip install keyboard"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "import numpy as np\n",+ "from pathlib import Path\n",+ "\n",+ "import matplotlib.pyplot as plt\n",+ "import keyboard"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "debug = True\n",+ "# %matplotlib widget\n",+ "import matplotlib.pyplot as plt\n",+ "# %matplotlib widget\n",+ "\n",+ "%matplotlib qt \n",+ "#pops out a window for matplot lib plots. It'd be cool to figure out how to play animations inline"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "### get yr paths right\n",+ "session_id = 'sesh_2022-02-15_11_54_28_pupil_maybe'\n",+ "\n",+ "freemocap_data_folder = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data')\n",+ "\n",+ "session_folder_path = freemocap_data_folder / session_id\n",+ "\n",+ "data_arrays_path = session_folder_path / 'DataArrays'\n",+ "\n",+ "mediapipe_skeleton_data_path = data_arrays_path / 'mediaPipeSkel_3d_smoothed.npy'"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "\n",+ " ### load in data\n",+ "\n",+ "- full body 3d kinematic data - `mediapipe_skeleton_frame_mar_xyz`\n",+ " - type: numpy array\n",+ " - dimensions: [frame_number, tracked_point_number, dimension (xyz)]\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "mediapipe_skeleton_fr_mar_xyz = np.load(mediapipe_skeleton_data_path)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "nose_id= 0\n",+ "right_ear_id = 8\n",+ "left_ear_id = 7\n",+ "\n",+ "nose_xyz = mediapipe_skeleton_fr_mar_xyz[:, nose_id, :]\n",+ "right_ear_xyz = mediapipe_skeleton_fr_mar_xyz[:, right_ear_id, :]\n",+ "left_ear_xyz = mediapipe_skeleton_fr_mar_xyz[:, left_ear_id, :]\n",+ "\n",+ "head_center_xyz = (right_ear_xyz + left_ear_xyz) / 2 "+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "fig = plt.figure(num=123)\n",+ "ax = fig.add_subplot(111)\n",+ "ax.plot(nose_xyz[:,0], label='nose_x')\n",+ "ax.plot(nose_xyz[:,1], label='nose_y')\n",+ "ax.plot(nose_xyz[:,2], label='nose_z')\n",+ "ax.plot(right_ear_xyz[:,0], label='right_ear_x')\n",+ "ax.plot(right_ear_xyz[:,1], label='right_ear_y')\n",+ "ax.plot(right_ear_xyz[:,2], label='right_ear_z')\n",+ "ax.plot(left_ear_xyz[:,0], label='left_ear_x')\n",+ "ax.plot(left_ear_xyz[:,1], label='left_ear_y')\n",+ "ax.plot(left_ear_xyz[:,2], label='left_ear_z')\n",+ "ax.plot(head_center_xyz[:,0], label='head_center_x')\n",+ "ax.plot(head_center_xyz[:,1], label='head_center_y')\n",+ "ax.plot(head_center_xyz[:,2], label='head_center_z')\n",+ "ax.legend()\n",+ "plt.show()\n",+ "plt.pause(.1)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "zero everything out, aka pin it to the origin, aka \"subtract the position of the thing you want to be the origin from everything else\", aka \"do the thing that makes the X,Y,Z coordinates of the thing that should be (0,0,0) equal to (0,0,0)\""+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "data": {+ "text/plain": [+ "array([[ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " ...,\n",+ " [ 12.47374782, 69.0853272 , -22.37502357],\n",+ " [ 14.10629544, 60.84139366, -6.04679083],\n",+ " [ 25.76475466, 63.75538109, -21.2651289 ]])"+ ]+ },+ "execution_count": 8,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "z_nose_xyz = nose_xyz - head_center_xyz\n",+ "z_nose_x = z_nose_xyz[:,0]\n",+ "z_nose_y = z_nose_xyz[:,1]\n",+ "z_nose_z = z_nose_xyz[:,2]\n",+ "\n",+ "z_left_ear_xyz = left_ear_xyz - head_center_xyz\n",+ "z_left_ear_x = z_left_ear_xyz[:,0]\n",+ "z_left_ear_y = z_left_ear_xyz[:,1]\n",+ "z_left_ear_z = z_left_ear_xyz[:,2]\n",+ "\n",+ "z_nose_xyz"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "Normalize so they are unit vectors"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "z_norm_nose_xyz = z_nose_xyz / np.linalg.norm(z_nose_xyz, axis=1)[:,None]\n",+ "z_norm_nose_x = z_norm_nose_xyz[:,0]\n",+ "z_norm_nose_y = z_norm_nose_xyz[:,1]\n",+ "z_norm_nose_z = z_norm_nose_xyz[:,2]\n",+ "\n",+ "z_norm_left_ear_xyz = z_left_ear_xyz / np.linalg.norm(z_left_ear_xyz, axis=1)[:,None]\n",+ "z_norm_left_ear_x = z_norm_left_ear_xyz[:,0]\n",+ "z_norm_left_ear_y = z_norm_left_ear_xyz[:,1]\n",+ "z_norm_left_ear_z = z_norm_left_ear_xyz[:,2]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "define x_hat as the zero'd out nose vector (i.e. the +X axis of the head starts at the head center and goes out the person's nose)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 10,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "data": {+ "text/plain": [+ "array([[ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " ...,\n",+ " [ 0.16929195, 0.93761632, -0.30367067],\n",+ " [ 0.22481111, 0.96962534, -0.09636731],\n",+ " [ 0.35795511, 0.8857668 , -0.29544087]])"+ ]+ },+ "execution_count": 10,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "x_hat_xyz = z_norm_nose_xyz\n",+ "x_hat_x = x_hat_xyz[:,0]\n",+ "x_hat_y = x_hat_xyz[:,1]\n",+ "x_hat_z = x_hat_xyz[:,2]\n",+ "\n",+ "x_hat_xyz"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "define z as x cross y"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 11,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "data": {+ "text/plain": [+ "array([[ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " ...,\n",+ " [ 0.07844496, -0.31995954, -0.9441781 ],\n",+ " [ 0.23993877, -0.15094018, -0.95898199],\n",+ " [ 0.15626882, -0.36877272, -0.91628966]])"+ ]+ },+ "execution_count": 11,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "z_hat_xyz = np.cross(x_hat_xyz, z_norm_left_ear_xyz) #create `z_hat` a vector perpendicular to `x_hat` and the vector from the nose to the left ear(i.e. pseudo y_hat)\n",+ "z_hat_xyz = z_hat_xyz / np.linalg.norm(z_hat_xyz, axis=1)[:,None] #not sure if this needs normalizing, but it couldn't hurt\n",+ "z_hat_x = z_hat_xyz[:,0]\n",+ "z_hat_y = z_hat_xyz[:,1]\n",+ "z_hat_z = z_hat_xyz[:,2]\n",+ "\n",+ "z_hat_xyz"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "define y_hat as z_hat cross with x_hat (this ensures that they are all orthogonal, where as the original vector to the left ear might not have been )"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 12,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "data": {+ "text/plain": [+ "array([[ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " ...,\n",+ " [ 0.98243912, -0.13602032, 0.12771785],\n",+ " [ 0.94439894, -0.19246755, 0.26658374],\n",+ " [ 0.9205695 , -0.28182237, 0.27042181]])"+ ]+ },+ "execution_count": 12,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "y_hat_xyz = np.cross(z_hat_xyz, z_norm_nose_xyz)\n",+ "y_hat_xyz = y_hat_xyz / np.linalg.norm(y_hat_xyz, axis=1)[:,None] #not sure if this needs normalizing, but it couldn't hurt\n",+ "y_hat_x = y_hat_xyz[:,0]\n",+ "y_hat_y = y_hat_xyz[:,1]\n",+ "y_hat_z = y_hat_xyz[:,2]\n",+ "\n",+ "y_hat_xyz"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "create head rotation matricies by stacking x_hat_xyz, y_hat_xyz, and z_hat_xyz"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 13,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "4916\n",+ "[[[nan nan nan]\n",+ " [nan nan nan]\n",+ " [nan nan nan]]]\n"+ ]+ }+ ],+ "source": [+ "head_rotation_matricies = [np.dstack((this_x_hat_xyz, this_y_hat_xyz, this_z_hat_xyz))\n",+ " for this_x_hat_xyz, this_y_hat_xyz, this_z_hat_xyz in zip(x_hat_xyz, y_hat_xyz, z_hat_xyz)]\n",+ "\n",+ "print(len(head_rotation_matricies))\n",+ "print(head_rotation_matricies[0])"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "make a point in head space and rotate it with the head rotation matricies, as a test of these strange mathemagics"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 14,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "data": {+ "text/plain": [+ "array([[ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " [ nan, nan, nan],\n",+ " ...,\n",+ " [ 1.23017602, 0.48163646, -1.12013092],\n",+ " [ 1.40914882, 0.62621761, -0.78876556],\n",+ " [ 1.43479342, 0.23517171, -0.94130872]])"+ ]+ },+ "execution_count": 14,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "test_point_xyz_og = np.array((1,1,1))\n",+ "test_point_xyz_rot = np.empty(z_hat_xyz.shape)\n",+ "\n",+ "\n",+ "for frame_number in range(len(head_rotation_matricies)):\n",+ " test_point_xyz_rot[frame_number,:] = head_rotation_matricies[frame_number] @ test_point_xyz_og\n",+ "\n",+ "\n",+ "test_point_xyz_rot"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "Visualize results! Press ESC to quit"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 15,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "You Pressed Escape!\n"+ ]+ }+ ],+ "source": [+ "if debug:\n",+ " plt.close('all')\n",+ " fig = plt.figure(num=124)\n",+ " fig.suptitle('Press ESC to Quit')\n",+ " ax = fig.add_subplot(111, projection='3d')\n",+ "\n",+ " ax_range = 2\n",+ "\n",+ "\n",+ " for frame_number in range(1200, z_nose_y.shape[0]):\n",+ " ax.clear()\n",+ "\n",+ "\n",+ " #plot head orthonormal basis vectors\n",+ " ax.plot(x_hat_x[frame_number-30:frame_number],\n",+ " x_hat_y[frame_number-30:frame_number],\n",+ " x_hat_z[frame_number-30:frame_number], 'r-')\n",+ "\n",+ " ax.plot(y_hat_x[frame_number-30:frame_number],\n",+ " y_hat_y[frame_number-30:frame_number],\n",+ " y_hat_z[frame_number-30:frame_number], 'g-')\n",+ "\n",+ " ax.plot(z_hat_x[frame_number-30:frame_number],\n",+ " z_hat_y[frame_number-30:frame_number],\n",+ " z_hat_z[frame_number-30:frame_number], 'b-')\n",+ " \n",+ " ax.plot([0, x_hat_x[frame_number]],\n",+ " [0, x_hat_y[frame_number]], \n",+ " [0, x_hat_z[frame_number]], 'r-o',label='x_hat')\n",+ "\n",+ " ax.plot([0, y_hat_x[frame_number]],\n",+ " [0, y_hat_y[frame_number]],\n",+ " [0, y_hat_z[frame_number]], 'g-o',label='y_hat')\n",+ " \n",+ " ax.plot([0, z_hat_x[frame_number]],\n",+ " [0, z_hat_y[frame_number]],\n",+ " [0, z_hat_z[frame_number]], 'b-o',label='z_hat')\n",+ " # original nose and ear directions (ear should be slightly diff from y_hat)]\\\n",+ " ax.plot([0, z_nose_x[frame_number]],\n",+ " [0, z_nose_y[frame_number]], \n",+ " [0, z_nose_z[frame_number]], 'r:o',label='z_nose')\n",+ "\n",+ " ax.plot([0, z_left_ear_x[frame_number]],\n",+ " [0, z_left_ear_y[frame_number]],\n",+ " [0, z_left_ear_z[frame_number]], 'g:o',label='z_left_ear')\n",+ "\n",+ " # plot test point rotated with head rotation matricies\n",+ "\n",+ " ax.plot(test_point_xyz_rot[frame_number-30:frame_number, 0],\n",+ " test_point_xyz_rot[frame_number-30:frame_number, 1],\n",+ " test_point_xyz_rot[frame_number-30:frame_number, 2], 'k-')\n",+ " \n",+ " ax.plot([0, test_point_xyz_rot[frame_number,0]],\n",+ " [0, test_point_xyz_rot[frame_number,1]],\n",+ " [0, test_point_xyz_rot[frame_number,2]], 'k-o',label='test point')\n",+ "\n",+ "\n",+ "\n",+ "\n",+ " ax.set_xlim([-ax_range, ax_range])\n",+ " ax.set_ylim([-ax_range, ax_range])\n",+ " ax.set_zlim([-ax_range, ax_range])\n",+ " ax.legend()\n",+ " plt.pause(.01)\n",+ " if keyboard.is_pressed('esc'): \n",+ " print('You Pressed Escape!')\n",+ " break # finishing the loop\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 16,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "data": {+ "text/plain": [+ "(4916, 3, 3)"+ ]+ },+ "execution_count": 16,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "head_rotation_matricies_fr_row_col = np.squeeze(np.asarray(head_rotation_matricies)) #convert list to array and give a friendly squeeze to pop the extra 'singleton' dimension\n",+ "head_rotation_matricies_fr_row_col.shape"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 17,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "save_head_rot_path = data_arrays_path / 'mediaPipeSkel_3d_head_rotation_matricies_fr_row_col.npy'\n",+ "np.save(save_head_rot_path, head_rotation_matricies_fr_row_col)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": []+ }+ ],+ "metadata": {+ "interpreter": {+ "hash": "4ef2e72e5b895021b427c707c9f05b4880b5a5a00e886b84de197a42e54c7f32"+ },+ "kernelspec": {+ "display_name": "Python 3.9.12 ('humon-env')",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.9.12"+ },+ "orig_nbformat": 4+ },+ "nbformat": 4,+ "nbformat_minor": 2+}diff --git a/jon_scratch/create_retina_centered_video_from_pupil_labs_data.ipynb b/jon_scratch/create_retina_centered_video_from_pupil_labs_data.ipynbnew file mode 100644index 00000000..6753d12e--- /dev/null+++ b/jon_scratch/create_retina_centered_video_from_pupil_labs_data.ipynb@@ -0,0 +1,618 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# a notebook to make a quick and dirty retina centered video from standard Pupil Player export video\n",+ "\n",+ "It'll make one retina centered video from a standard Pupil Payer export video\n",+ "\n",+ "and also a video of the gaze trace\n",+ "\n",+ "I'll put them together in Adobe Premier or, via this script. We'll see lol"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import cv2\n",+ "from pathlib import Path\n",+ "import matplotlib.pyplot as plt\n",+ "import pandas as pd\n",+ "import numpy as np\n",+ "from PIL import Image\n",+ "from scipy import signal\n",+ "from scipy.signal import savgol_filter\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# !pip install pyqt5\n",+ "%matplotlib qt"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Getcher paths straight"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "#freemocap session\n",+ "session_id = 'sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0'\n",+ "freemocap_data_folder = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data')\n",+ "session_folder_path = freemocap_data_folder / session_id\n",+ "pupil_data_path = session_folder_path / 'pupil_000'\n",+ "pupil_data_exports_path = pupil_data_path / 'exports' / '000'\n",+ "pupil_export_video_path = pupil_data_exports_path / 'world.mp4'\n",+ "pupil_world_video_path = pupil_data_path / 'world.mp4'\n",+ "\n",+ "# #argp pilot_0\n",+ "# pupil_data_path = Path(r'H:\\Other computers\\My Computer_MocapComputer\\Wirth_ARGP\\ARGP_Main\\data\\2022_04_29\\pupil_data')\n",+ "# pupil_data_exports_path = pupil_data_path / 'exports' / 'Wirth_Pilot_ARGP_2022-04-09_export'\n",+ "# pupil_export_video_path = pupil_data_exports_path / 'argp_pilot_May2022.mp4'\n",+ "\n",+ "\n",+ "pupil_world_video_path = pupil_data_path / 'world.mp4'\n",+ "\n",+ "\n",+ "gaze_positions_path = pupil_data_exports_path / 'gaze_positions.csv'\n",+ "\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "load stuff "+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "#open cv video capture object\n",+ "pupil_export_video_cap_object = cv2.VideoCapture(str(pupil_export_video_path))\n",+ "\n",+ "#pupil labs - gaze data\n",+ "pupil_dataframe = pd.read_csv(gaze_positions_path)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "get some info from the video cap"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "video_width = int(pupil_export_video_cap_object.get(cv2.CAP_PROP_FRAME_WIDTH))\n",+ "video_height = int(pupil_export_video_cap_object.get(cv2.CAP_PROP_FRAME_HEIGHT))\n",+ "video_framerate = int(pupil_export_video_cap_object.get(cv2.CAP_PROP_FPS))\n",+ "print(f'video_width : {video_width }')\n",+ "print(f'video_height: {video_height}')\n",+ "print(f'video_framerate: {video_framerate}')\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "gaze locations"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "norm_pos_x = pupil_dataframe.norm_pos_x\n",+ "norm_pos_y = pupil_dataframe.norm_pos_y\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# fig = plt.Figure(123)\n",+ "# fig.plot(norm_pos_x, label = 'norm_pos_x')\n",+ "# fig.plot(norm_pos_y, label ='norm_pos_x')"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "\n",+ "gaze_on_screen_x = norm_pos_x * video_width\n",+ "gaze_on_screen_y = norm_pos_y * video_height\n",+ "\n",+ "world_camera_frame_index_all = pupil_dataframe.world_index"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "zero-lag 4th order butterworth filter with a 7Hz cutoff"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "\n",+ "\n",+ "order = 5\n",+ "cutoff=10**-1\n",+ "b, a = signal.butter(order, cutoff)\n",+ "gaze_on_screen_x_filtered = signal.filtfilt(b, a, gaze_on_screen_x)\n",+ "gaze_on_screen_y_filtered = signal.filtfilt(b, a, gaze_on_screen_y)\n",+ "\n",+ "print(f\"nans in gaze_on_screen_x: {np.isnan(gaze_on_screen_x).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y: {np.isnan(gaze_on_screen_y).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_x_filtered: {np.isnan(gaze_on_screen_x_filtered).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y_filtered: {np.isnan(gaze_on_screen_y_filtered).sum()}\")\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "gaze_on_screen_x_filtered"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "\n",+ "plt.cla()\n",+ "fig2 = plt.Figure()\n",+ "ax = plt.gca()\n",+ "ax.plot(gaze_on_screen_x, label = 'gaze_on_screen_x_raw')\n",+ "ax.plot(gaze_on_screen_y, label = 'gaze_on_screen_y_raw')\n",+ "ax.plot(gaze_on_screen_x_filtered, label = 'gaze_on_screen_x_filtered')\n",+ "ax.plot(gaze_on_screen_y_filtered, label = 'gaze_on_screen_y_filtered')\n",+ "ax.set_ylim((0-100, video_width+100))\n",+ "ax.legend()\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "squash data down so each frame is the average of all data recorded on that frame"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "number_of_frames = np.max(world_camera_frame_index_all)\n",+ "number_of_frames\n",+ "\n",+ "gaze_on_screen_x_downsampled = []\n",+ "gaze_on_screen_y_downsampled = []\n",+ "\n",+ "for this_world_frame_index in range(number_of_frames):\n",+ " this_frame_data = world_camera_frame_index_all == this_world_frame_index\n",+ " this_frame_x_data = gaze_on_screen_x_filtered[this_frame_data]\n",+ " this_frame_y_data = gaze_on_screen_y_filtered[this_frame_data]\n",+ " gaze_on_screen_x_downsampled.append(np.nanmedian(this_frame_x_data))\n",+ " gaze_on_screen_y_downsampled.append(np.nanmedian(this_frame_y_data))"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "\n",+ "print(f\"nans in gaze_on_screen_x: {np.isnan(gaze_on_screen_x).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y: {np.isnan(gaze_on_screen_y).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_x_filtered: {np.isnan(gaze_on_screen_x_filtered).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y_filtered: {np.isnan(gaze_on_screen_y_filtered).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_x_downsampled: {np.isnan(gaze_on_screen_x_downsampled).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y_downsampled: {np.isnan(gaze_on_screen_y_downsampled).sum()}\")\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "\n",+ "# order = 4\n",+ "# cutoff=10**-1\n",+ "# b, a = signal.butter(order, cutoff)\n",+ "# gaze_on_screen_x_filtered = signal.filtfilt(b, a, gaze_on_screen_x_downsampled)\n",+ "# gaze_on_screen_y_filtered = signal.filtfilt(b, a, gaze_on_screen_y_downsampled)\n",+ "\n",+ "print(f\"nans in gaze_on_screen_x: {np.isnan(gaze_on_screen_x).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y: {np.isnan(gaze_on_screen_y).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_x_filtered: {np.isnan(gaze_on_screen_x_filtered).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y_filtered: {np.isnan(gaze_on_screen_y_filtered).sum()}\")\n",+ "\n",+ "# plt.close('all')\n",+ "# fig2 = plt.Figure()\n",+ "# ax = plt.gca()\n",+ "# # ax.plot(gaze_on_screen_x_downsampled, label = 'gaze_on_screen_x_downsampled')\n",+ "# # ax.plot(gaze_on_screen_y_downsampled, label = 'gaze_on_screen_y_downsampled')\n",+ "# ax.plot(gaze_on_screen_x_filtered, label = 'gaze_on_screen_x_filtered')\n",+ "# ax.plot(gaze_on_screen_y_filtered, label = 'gaze_on_screen_y_filtered')\n",+ "# ax.set_ylim((0-100, video_width+100))\n",+ "# ax.legend()\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "\n",+ "#smoooooooooth, just a bit\n",+ "smoothWinLength = 5\n",+ "smoothOrder = 3\n",+ "\n",+ "gaze_on_screen_x_downsampled_smoothed = savgol_filter(gaze_on_screen_x_downsampled, smoothWinLength, smoothOrder)\n",+ "gaze_on_screen_y_downsampled_smoothed = savgol_filter(gaze_on_screen_y_downsampled, smoothWinLength, smoothOrder)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "print(f\"nans in gaze_on_screen_x_downsampled: {np.isnan(gaze_on_screen_x_downsampled).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y_downsampled: {np.isnan(gaze_on_screen_y_downsampled).sum()}\")\n",+ "\n",+ "gaze_x_df = pd.DataFrame(gaze_on_screen_x_downsampled)\n",+ "gaze_x_df.interpolate(method = 'linear', inplace = True)\n",+ "gaze_on_screen_x_downsampled_gapfilled = gaze_x_df.to_numpy()\n",+ "\n",+ "gaze_y_df = pd.DataFrame(gaze_on_screen_y_downsampled)\n",+ "gaze_y_df.interpolate(method = 'linear', inplace = True)\n",+ "gaze_on_screen_y_downsampled_gapfilled = gaze_y_df.to_numpy()\n",+ "\n",+ "print(f\"nans in gaze_on_screen_x_downsampled_gapfilled: {np.isnan(gaze_on_screen_x_downsampled_gapfilled).sum()}\")\n",+ "print(f\"nans in gaze_on_screen_y_downsampled_gapfilled: {np.isnan(gaze_on_screen_y_downsampled_gapfilled).sum()}\")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "plt.close(\"all\")\n",+ "fig2 = plt.Figure()\n",+ "ax = plt.gca()\n",+ "ax.plot(gaze_on_screen_x_downsampled, '.-', label = 'gaze_on_screen_x_downsampled')\n",+ "ax.plot(gaze_on_screen_y_downsampled, '.-',label = 'gaze_on_screen_y_downsampled')\n",+ "ax.plot(gaze_on_screen_x_downsampled_smoothed, '.-', label = 'gaze_on_screen_x_downsampled_smoothed')\n",+ "ax.plot(gaze_on_screen_y_downsampled_smoothed, '.-',label = 'gaze_on_screen_y_downsampled_smoothed')\n",+ "ax.plot(gaze_on_screen_x_downsampled_gapfilled, '.-', label = 'gaze_on_screen_x_downsampled_gapfilled')\n",+ "ax.plot(gaze_on_screen_y_downsampled_gapfilled, '.-',label = 'gaze_on_screen_y_downsampled_gapfilled')\n",+ "\n",+ "ax.set_ylim((0-100, video_width+100))\n",+ "ax.legend(loc='upper left')\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "\n",+ "\n",+ "\n",+ "big_blank_image = np.ones((int(video_height)*2, int(video_width)*2, 3), dtype=np.uint8)\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# pupil_world_video_cap_object = cv2.VideoCapture(str(pupil_world_video_path))\n",+ "\n",+ "# frame_num = 0\n",+ "# success = True\n",+ "# while success: \n",+ "# success, image_raw = pupil_world_video_cap_object.read()\n",+ "# if not success:\n",+ "# break\n",+ "# frame_num += 1\n",+ "# print(f\"frame_num: {frame_num}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "create circular mask\n",+ "\n",+ "from =- https://stackoverflow.com/questions/61516526/how-to-use-opencv-to-crop-circular-image"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "#define circles\n",+ "radius = video_height #pixels\n",+ "x_center = video_width\n",+ "y_center = video_height\n",+ "\n",+ "# draw filled circles in white on black background as masks\n",+ "circular_mask = np.zeros_like(big_blank_image)\n",+ "circular_mask = cv2.circle(circular_mask, (x_center,y_center), radius, (255,255,255), -1)\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "video_framerate"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "retinal_video_save_path = pupil_data_exports_path / 'retinal_aligned_video.mp4'\n",+ "\n",+ "make_retinal_video = False\n",+ "if make_retinal_video:\n",+ " pupil_world_video_cap_object = cv2.VideoCapture(str(pupil_world_video_path))\n",+ "\n",+ " this_retinal_aligned_image = big_blank_image.copy()\n",+ " this_retinal_aligned_image[:, video_width-video_height:(video_width-video_height)+2*video_height,:]\n",+ "\n",+ " retinal_video_writer = cv2.VideoWriter(str(retinal_video_save_path),\n",+ " cv2.VideoWriter_fourcc(*\"MP4V\"),\n",+ " video_framerate, \n",+ " (this_retinal_aligned_image.shape[0], this_retinal_aligned_image.shape[0]))\n",+ "\n",+ " success = True\n",+ " for frame_number in range(gaze_on_screen_x_downsampled_gapfilled.shape[0]):\n",+ " success, image= pupil_world_video_cap_object.read()\n",+ " if not success:\n",+ " print('failed to read frame')\n",+ " break\n",+ "\n",+ " # image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",+ " this_retinal_aligned_image = big_blank_image.copy()\n",+ "\n",+ " #pull out gaze data to align image to retinal coordinates\n",+ " gx = int(gaze_on_screen_x_downsampled_gapfilled[frame_number])\n",+ " gy = int(gaze_on_screen_y_downsampled_gapfilled[frame_number])\n",+ "\n",+ " bottom_edge_row = int(video_height)*2-(int(video_height)-gy)\n",+ " top_edge_row = int(video_height) -(int(video_height)-gy)\n",+ " left_col = (int(video_width)-gx)\n",+ " right_col = int(video_width)+(int(video_width)-gx)\n",+ "\n",+ " cols = np.arange(top_edge_row,bottom_edge_row)\n",+ "\n",+ " #if things are going off screen, use whatever gaze data worked most recently\n",+ " if top_edge_row < 0:\n",+ " gy = previous_gy\n",+ " top_edge_row = int(video_height) -(int(video_height)-gy)\n",+ " bottom_edge_row = int(video_height)*2-(int(video_height)-gy)\n",+ "\n",+ " if bottom_edge_row > int(video_height)*2:\n",+ " gy = previous_gy\n",+ " top_edge_row = int(video_height) -(int(video_height)-gy)\n",+ " bottom_edge_row = int(video_height)*2-(int(video_height)-gy)\n",+ " \n",+ " if left_col < 0:\n",+ " gx = previous_gx\n",+ " right_col = int(video_width)+(int(video_width)-gx)\n",+ " left_col = (int(video_width)-gx)\n",+ " \n",+ " if right_col > int(video_width)*2:\n",+ " gx = previous_gx\n",+ " right_col = int(video_width)+(int(video_width)-gx)\n",+ " left_col = (int(video_width)-gx)\n",+ "\n",+ " previous_gx = gx \n",+ " previous_gy = gy\n",+ " \n",+ " this_retinal_aligned_image[top_edge_row:bottom_edge_row, left_col:right_col,:] = image\n",+ "\n",+ " this_retinal_aligned_image[circular_mask==0] = 0\n",+ "\n",+ "\n",+ " this_retinal_aligned_image = this_retinal_aligned_image[:, video_width-video_height:(video_width-video_height)+2*video_height,:]\n",+ "\n",+ " this_retinal_aligned_image = cv2.line(this_retinal_aligned_image, \n",+ " (int(video_height), 0), \n",+ " (int(video_height), int(video_height)*2), \n",+ " (0,0,0,255), \n",+ " 5)\n",+ " this_retinal_aligned_image = cv2.line(this_retinal_aligned_image, \n",+ " (0, int(video_height)), \n",+ " (int(video_width)*2, \n",+ " int(video_height)), \n",+ " (0,0,0,255), \n",+ " 5)\n",+ " this_retinal_aligned_image = cv2.line(this_retinal_aligned_image, \n",+ " (int(video_height), 0), \n",+ " (int(video_height), int(video_height)*2), \n",+ " (255,255,255,255), \n",+ " 1,)\n",+ " this_retinal_aligned_image = cv2.line(this_retinal_aligned_image, \n",+ " (0, int(video_height)), \n",+ " (int(video_width)*2, \n",+ " int(video_height)), \n",+ " (255,255,255,255), \n",+ " 1)\n",+ " this_retinal_aligned_image = cv2.circle(this_retinal_aligned_image, \n",+ " (int(video_height), int(video_height)),\n",+ " 20,\n",+ " (0,0,0,255), \n",+ " 5)\n",+ " this_retinal_aligned_image = cv2.circle(this_retinal_aligned_image, \n",+ " (int(video_height), int(video_height)),\n",+ " 20,\n",+ " (255,255,255,255), \n",+ " 1)\n",+ "\n",+ " # plt.cla()\n",+ " # plt.imshow(this_retinal_aligned_image[:,:,:3]) \n",+ " # plt.show()\n",+ " # plt.pause(0.1)\n",+ "\n",+ " retinal_video_writer.write(this_retinal_aligned_image) \n",+ "\n",+ " if frame_number%100 == 0:\n",+ " print(f'frame_number: {frame_number}')\n",+ "\n",+ " retinal_video_writer.release()\n",+ " print('done!')"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "make Pupil Exports + Retinal + Gaze Trace Vid"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 36,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "frame_number: 1\n"+ ]+ }+ ],+ "source": [+ "plt.close('all')\n",+ "retinal_video_save_path = pupil_data_exports_path / 'retinal_aligned_video.mp4'\n",+ "\n",+ "retinal_vid_cap = cv2.VideoCapture(str(retinal_video_save_path))\n",+ "pupil_export_vid_cap = cv2.VideoCapture(str(pupil_export_video_path))\n",+ "\n",+ "combo_video_save_path = pupil_data_exports_path / 'combo_retinal_world_video.mp4'\n",+ "\n",+ "\n",+ "combo_video_writer = cv2.VideoWriter(str(combo_video_save_path),\n",+ " cv2.VideoWriter_fourcc(*\"MP4V\"),\n",+ " video_framerate, \n",+ " (video_height,2000))\n",+ "\n",+ "pup_success = True\n",+ "ret_success = True\n",+ "frame_number = 0\n",+ "while pup_success and ret_success:\n",+ " frame_number += 1\n",+ " print(f'frame_number: {frame_number}')\n",+ " ret_success, ret_image = retinal_vid_cap.read()\n",+ " if not ret_success:\n",+ " print('ret - failed to read frame')\n",+ " break\n",+ "\n",+ " pup_success, pup_image = pupil_export_vid_cap.read()\n",+ " if not pup_success:\n",+ " print('pup - failed to read frame')\n",+ " break\n",+ " \n",+ " ret_image = cv2.cvtColor(ret_image, cv2.COLOR_BGR2RGB)\n",+ " pup_image = cv2.cvtColor(pup_image, cv2.COLOR_BGR2RGB)\n",+ "\n",+ " shrink_ret_image = cv2.resize(ret_image, (0,0), fx=0.5, fy=0.5)\n",+ " combo_image = np.hstack((shrink_ret_image, pup_image))\n",+ "\n",+ " combo_video_writer.write(combo_image) \n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": []+ }+ ],+ "metadata": {+ "interpreter": {+ "hash": "122c9931a57dd91c579842728ee2ee1c7ffefd770186c077f442b22e7e57f48e"+ },+ "kernelspec": {+ "display_name": "Python 3.7.13 ('freemocap-env')",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.7.13"+ },+ "orig_nbformat": 4+ },+ "nbformat": 4,+ "nbformat_minor": 2+}diff --git a/jon_scratch/freemocap_plotly.py b/jon_scratch/freemocap_plotly.pynew file mode 100644index 00000000..7cf531e0--- /dev/null+++ b/jon_scratch/freemocap_plotly.py@@ -0,0 +1,112 @@+# Import data+import time+import numpy as np+import plotly.graph_objects as go+from skimage import io+from pathlib import Path+## load in freemocap data+session_id = 'sesh_2022-02-15_11_54_28_pupil_maybe'+freemocap_data_folder = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data')+session_folder_path = freemocap_data_folder / session_id+data_arrays_path = session_folder_path / 'DataArrays'+mediapipe_skeleton_data_path = data_arrays_path / 'mediaPipeSkel_3d_smoothed.npy'+mediapipe_skeleton_fr_mar_xyz = np.load(mediapipe_skeleton_data_path)++##+vol = io.imread("https://s3.amazonaws.com/assets.datacamp.com/blog_assets/attention-mri.tif")+volume = vol.T+r, c = volume[0].shape++# Define frames+number_of_frames = mediapipe_skeleton_fr_mar_xyz.shape[0]++fig = go.Figure(frames=[+ go.Frame(++ data=go.Scatter3d(+ x=mediapipe_skeleton_fr_mar_xyz[this_frame_number, :, 0],+ y=mediapipe_skeleton_fr_mar_xyz[this_frame_number, :, 1],+ z=mediapipe_skeleton_fr_mar_xyz[this_frame_number, :, 2],+ mode='markers',+ marker=dict(+ size=12,+ color='red', # set color to an array/list of desired values+ colorscale='Viridis', # choose a colorscale+ opacity=0.8+ )+ ),++ name=str(this_frame_number) # you need to name the frame for the animation to behave properly+ )+ for this_frame_number in range(number_of_frames)])++# Add data to be displayed before animation starts+fig.add_trace(go.Surface(+ z=6.7 * np.ones((r, c)),+ surfacecolor=np.flipud(volume[67]),+ colorscale='Gray',+ cmin=0, cmax=200,+ colorbar=dict(thickness=20, ticklen=4)+))+++def frame_args(duration):+ return {+ "frame": {"duration": duration},+ "mode": "immediate",+ "fromcurrent": True,+ "transition": {"duration": duration, "easing": "linear"},+ }+++sliders = [+ {+ "pad": {"b": 10, "t": 60},+ "len": 0.9,+ "x": 0.1,+ "y": 0,+ "steps": [+ {+ "args": [[f.name], frame_args(0)],+ "label": str(k),+ "method": "animate",+ }+ for k, f in enumerate(fig.frames)+ ],+ }+]++# Layout+fig.update_layout(+ title='Slices in volumetric data',+ width=600,+ height=600,+ scene=dict(+ zaxis=dict(range=[-2e3, 2e3], autorange=False),+ aspectratio=dict(x=1, y=1, z=1),+ ),+ updatemenus=[+ {+ "buttons": [+ {+ "args": [None, frame_args(0)],+ "label": "▶", # play symbol+ "method": "animate",+ },+ {+ "args": [[None], frame_args(0)],+ "label": "◼", # pause symbol+ "method": "animate",+ },+ ],+ "direction": "left",+ "pad": {"r": 10, "t": 70},+ "type": "buttons",+ "x": 0.1,+ "y": 0,+ }+ ],+ sliders=sliders+)++fig.show()diff --git a/jon_scratch/karls_pupil_freemocap_code.ipynb b/jon_scratch/karls_pupil_freemocap_code.ipynbnew file mode 100644index 00000000..a613c97b--- /dev/null+++ b/jon_scratch/karls_pupil_freemocap_code.ipynb@@ -0,0 +1,1256 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "\n",+ "# this is Jon's attempt to put Karl's `alignPupilFMC.py` into an jupyter notebook "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "### import stuff"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "!pip install scipy\n",+ "!pip install pandas\n",+ "!pip install numpy\n",+ "!pip install pyqt5"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "import numpy as np\n",+ "import pandas as pd\n",+ "import json\n",+ "from pathlib import Path\n",+ "\n",+ "from scipy import optimize\n",+ "from scipy.spatial.transform import Rotation\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "debug = True\n",+ "if debug:\n",+ " %matplotlib qt\n",+ " # %matplotlib \n",+ " import matplotlib.pyplot as plt\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "### get yr paths right"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "session_id = 'sesh_2022-02-15_11_54_28_pupil_maybe'\n",+ "freemocap_data_folder = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data')\n",+ "\n",+ "session_folder_path = freemocap_data_folder / session_id\n",+ "\n",+ "mediapipe_skeleton_data_path = session_folder_path / 'DataArrays'/ 'mediaPipeSkel_3d_smoothed.npy'\n",+ "mediapipe_head_rotations_path = session_folder_path / 'DataArrays'/ 'mediaPipeSkel_3d_head_rotation_matricies_fr_row_col.npy'\n",+ "\n",+ "pupil_data_path = session_folder_path / 'pupil_000'\n",+ "pupil_data_exports_path = pupil_data_path / 'exports' / '000'\n",+ "pupil_positions_path = pupil_data_exports_path / 'pupil_positions.csv'\n",+ "pupil_recording_info_path = pupil_data_path / 'info.player.json'\n",+ "\n",+ "freemocap_unix_timestamp_path = session_folder_path /'unix_synced_timestamps.csv'\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ " ### load in data\n",+ "\n",+ "- full body 3d kinematic data - `mediapipe_skeleton_frame_mar_xyz`\n",+ " - type: numpy array\n",+ " - dimensions: [frame_number, tracked_point_number, dimension (xyz)]\n",+ "- pupil data from `pupil_positions.csv` in the **pupil exports** folder\n",+ " - definitions in `pupil_gaze_positions_info.txt`\n",+ "- pupil start time from `info.player.json` for sync\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "mediapipe_skeleton_fr_mar_xyz = np.load(mediapipe_skeleton_data_path)\n",+ "mediapipe_head_rotation_matricies_fr_row_col = np.load(mediapipe_head_rotations_path)\n",+ "pupil_dataframe = pd.read_csv(pupil_positions_path)\n",+ "pupil_recording_info_json = json.load(open(pupil_recording_info_path))\n",+ "freemocap_unix_timestamps_dataframe = pd.read_csv(freemocap_unix_timestamp_path)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "### pull out relevant pupil data"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "data for both eyes"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "timestamps = np.array(pupil_dataframe['pupil_timestamp'])\n",+ "eye_theta = np.array(pupil_dataframe['theta'])\n",+ "eye_phi = np.array(pupil_dataframe['phi'])\n",+ "sphere_center_x = np.array(pupil_dataframe['sphere_center_x'])\n",+ "sphere_center_y = np.array(pupil_dataframe['sphere_center_y'])\n",+ "sphere_center_z = np.array(pupil_dataframe['sphere_center_z'])\n",+ "pupil_center_normal_x = np.array(pupil_dataframe['circle_3d_normal_x'])\n",+ "pupil_center_normal_y = np.array(pupil_dataframe['circle_3d_normal_y'])\n",+ "pupil_center_normal_z = np.array(pupil_dataframe['circle_3d_normal_z'])\n",+ "\n",+ "eye_d = pupil_dataframe['eye_id']\n",+ "method = pupil_dataframe['method']\n",+ "# tracking_method_name = 'pye3d 0.3.0 real-time'\n",+ "\n",+ "\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "get data for the right eye\n",+ "\n",+ "pull out data according to `eye_d` (right_eye == 0) and tracking method (because pupil interleaves 2d and 3d data)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "right_eye_d = 0 \n",+ "\n",+ "#timestamps\n",+ "r_eye_timestamps_og = timestamps[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n",+ "\n",+ "r_eye_theta = eye_theta[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n",+ "r_eye_phi = eye_phi[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n",+ "\n",+ "r_eye_sphere_center_x = sphere_center_x[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n",+ "r_eye_sphere_center_y = sphere_center_y[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n",+ "r_eye_sphere_center_z = sphere_center_z[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n",+ "\n",+ "r_eye_pupil_center_normal_x = pupil_center_normal_x[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n",+ "r_eye_pupil_center_normal_y = pupil_center_normal_y[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n",+ "r_eye_pupil_center_normal_z = pupil_center_normal_z[np.logical_and(eye_d==right_eye_d, method!='2d c++')]\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "Get data for the left eye\n",+ "\n",+ "pull out data according to `eye_d` (left eye == 1) and tracking method (because pupil interleaves 2d and 3d data)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "left_eye_d = 1 \n",+ "\n",+ "#timestamps\n",+ "l_eye_timestamps_og = timestamps[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n",+ "\n",+ "l_eye_theta = eye_theta[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n",+ "l_eye_phi = eye_phi[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n",+ "\n",+ "l_eye_sphere_center_x = sphere_center_x[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n",+ "l_eye_sphere_center_y = sphere_center_y[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n",+ "l_eye_sphere_center_z = sphere_center_z[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n",+ "\n",+ "l_eye_pupil_center_normal_x = pupil_center_normal_x[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n",+ "l_eye_pupil_center_normal_y = pupil_center_normal_y[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n",+ "l_eye_pupil_center_normal_z = pupil_center_normal_z[np.logical_and(eye_d==left_eye_d, method!='2d c++')]\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "if debug:\n",+ " fig = plt.figure(num=2134,figsize=(10,10))\n",+ " \n",+ " ax1 = fig.add_subplot(321)\n",+ " ax1.plot(r_eye_timestamps_og, r_eye_theta, '.-', label='r eye theta')\n",+ " ax1.plot(r_eye_timestamps_og, r_eye_phi, '.-', label='r eye phi')\n",+ "\n",+ " ax1.legend(loc='upper right')\n",+ " \n",+ " ax3 = fig.add_subplot(323)\n",+ " ax3.plot(r_eye_timestamps_og, r_eye_pupil_center_normal_x, '.-', label='r_eye_pupil_center_normal_x')\n",+ " ax3.plot(r_eye_timestamps_og, r_eye_pupil_center_normal_y, '.-', label='r_eye_pupil_center_normal_y')\n",+ " ax3.plot(r_eye_timestamps_og, r_eye_pupil_center_normal_z, '.-', label='r_eye_pupil_center_normal_z')\n",+ " ax3.legend(loc='upper right')\n",+ "\n",+ " ax5 = fig.add_subplot(325)\n",+ " ax5.plot(r_eye_timestamps_og, r_eye_sphere_center_x, '.-', label='r_eye_sphere_center_x')\n",+ " ax5.plot(r_eye_timestamps_og, r_eye_sphere_center_y, '.-', label='r_eye_sphere_center_y')\n",+ " ax5.plot(r_eye_timestamps_og, r_eye_sphere_center_z, '.-', label='r_eye_sphere_center_z')\n",+ " ax5.legend(loc='upper right')\n",+ "\n",+ " \n",+ " ax2 = fig.add_subplot(322)\n",+ " ax2.plot(l_eye_timestamps_og, l_eye_theta, '.-', label='l eye theta')\n",+ " ax2.plot(l_eye_timestamps_og, l_eye_phi, '.-', label='l eye phi')\n",+ " ax2.legend(loc='upper right')\n",+ " \n",+ " ax4 = fig.add_subplot(324)\n",+ " ax4.plot(l_eye_timestamps_og, l_eye_pupil_center_normal_x, '.-', label='l_eye_pupil_center_normal_x')\n",+ " ax4.plot(l_eye_timestamps_og, l_eye_pupil_center_normal_y, '.-', label='l_eye_pupil_center_normal_y')\n",+ " ax4.plot(l_eye_timestamps_og, l_eye_pupil_center_normal_z, '.-', label='l_eye_pupil_center_normal_z')\n",+ " ax4.legend(loc='upper right')\n",+ "\n",+ " ax6 = fig.add_subplot(326)\n",+ " ax6.plot(l_eye_timestamps_og, l_eye_sphere_center_x, '.-', label='l_eye_sphere_center_x')\n",+ " ax6.plot(l_eye_timestamps_og, l_eye_sphere_center_y, '.-', label='l_eye_sphere_center_y')\n",+ " ax6.plot(l_eye_timestamps_og, l_eye_sphere_center_z, '.-', label='l_eye_sphere_center_z')\n",+ "\n",+ " plt.pause(.1)\n",+ " plt.show()\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "### create XYZ gaze array\n",+ "\n",+ "\n",+ "subtract eyeball center from pupil center to re-base the gaze vector to the origin "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "skipping 'remove nans' step for now, but i might need to add it later? "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "### synchronize pupil and freemocap timestamps "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "#### convert pupil timestamps into unix time"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "r_eye_timestamps = r_eye_timestamps_og - pupil_recording_info_json['start_time_synced_s'] +pupil_recording_info_json['start_time_system_s'] \n",+ "l_eye_timestamps = l_eye_timestamps_og - pupil_recording_info_json['start_time_synced_s'] +pupil_recording_info_json['start_time_system_s'] \n",+ "\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "calculate freemocap unix timestamps (as mean of each camera's timestamp on each frame)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "each_cam_timestamps_list = []\n",+ "\n",+ "camera_names = [column_name for column_name in list(freemocap_unix_timestamps_dataframe.columns) if 'Cam' in column_name]\n",+ "\n",+ "for this_camera_name in camera_names:\n",+ " each_cam_timestamps_list.append(np.array(freemocap_unix_timestamps_dataframe[this_camera_name]))\n",+ " \n",+ "cameras_timestamp_array = np.stack(each_cam_timestamps_list,axis=1)\n",+ "cameras_timestamp_array[cameras_timestamp_array==-1]=np.nan #replace -1 with nan\n",+ "\n",+ "freemocap_timestamps = np.nanmean(cameras_timestamp_array,axis=1)\n",+ "\n",+ "freemocap_frame_durations = np.diff(freemocap_timestamps)\n",+ "freemocap_frames_per_second = np.nanmean(freemocap_frame_durations**-1)\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "if debug:\n",+ "\n",+ " timestamp_diff_on_each_frame_rel_cam0 = np.diff(cameras_timestamp_array, axis=1)\n",+ " frame_duration = np.diff(freemocap_timestamps)\n",+ "\n",+ " fig = plt.figure(234899234)\n",+ " ax1 = fig.add_subplot(411)\n",+ " ax1.plot(cameras_timestamp_array, '.-',label='each cam')\n",+ " ax1.plot(freemocap_timestamps, 'r-',label='freemocap_timestamps')\n",+ " ax1.set_title('cameras_timestamps')\n",+ " ax1.set_xlabel('frame number')\n",+ " ax1.set_ylabel('timestamp')\n",+ " ax1.legend(loc='upper left')\n",+ " \n",+ " ax2 = fig.add_subplot(412)\n",+ " for this_cam_num in range(timestamp_diff_on_each_frame_rel_cam0.shape[1]):\n",+ " ax2.hist(timestamp_diff_on_each_frame_rel_cam0[:,this_cam_num], label=f'camera {this_cam_num}', alpha=0.5)\n",+ " ax2.set_xlim(-.1,.1)\n",+ " ax2.set_xlabel('timestamp diff on each frame (s)')\n",+ " ax2.legend()\n",+ "\n",+ " ax3 = fig.add_subplot(413)\n",+ " ax3.plot(freemocap_frame_durations, '.-',label='frame duration')\n",+ " ax3.plot([0, len(freemocap_frame_durations)],\n",+ " [np.nanmean(freemocap_frame_durations), np.nanmean(freemocap_frame_durations)],\n",+ " 'r-',\n",+ " label=f'mean frame duration (ms):{str(np.nanmean(freemocap_frame_durations)*1000)[:5]}',\n",+ " )\n",+ " ax3.legend(loc='upper left')\n",+ " ax3.set_xlabel('frame number')\n",+ " ax3.set_ylabel('frame duration (s)')\n",+ "\n",+ " ax4 = fig.add_subplot(414)\n",+ " ax4.hist(freemocap_frame_durations**-1, label='frame duration')\n",+ " ax4.legend(loc='upper left')\n",+ " ax4.set_xlabel('frame rate (frames per second')\n",+ " ax4.set_xlim(0,50)\n",+ " \n",+ " plt.pause(.1)\n",+ " plt.show()"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "print(f'freemocap_timestamps[-1]: {freemocap_timestamps[-1]}')\n",+ "print(f'r_eye_timestamps[-1]: {r_eye_timestamps[-1]}')\n",+ "\n",+ "np.min((freemocap_timestamps[-1],r_eye_timestamps[-1]))"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "#### align freemocap and pupil timestamps and clip the starts and ends of the various data traces so that everything covers the same timespacn"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "#find start and end frames shared by all datastreams\n",+ "start_time_unix = np.max((freemocap_timestamps[0],r_eye_timestamps[0], l_eye_timestamps[0]))\n",+ "end_time_unix = np.min((freemocap_timestamps[-1],r_eye_timestamps[-1], l_eye_timestamps[-1]))\n",+ "\n",+ "#freemocap\n",+ "if any(freemocap_timestamps>=start_time_unix):\n",+ " freemocap_start = np.where(freemocap_timestamps>=start_time_unix)[0][0]\n",+ "else:\n",+ " freemocap_start = 0\n",+ "\n",+ "if any(freemocap_timestamps<=end_time_unix):\n",+ " freemocap_end = np.where(freemocap_timestamps<=end_time_unix)[0][-1]\n",+ "else:\n",+ " freemocap_end = len(freemocap_timestamps)\n",+ "\n",+ "#right eye\n",+ "if any(r_eye_timestamps>=start_time_unix):\n",+ " r_eye_start = np.where(r_eye_timestamps>=start_time_unix)[0][0]\n",+ "else:\n",+ " r_eye_start = 0\n",+ " \n",+ "if any(r_eye_timestamps<=end_time_unix):\n",+ " r_eye_end = np.where(r_eye_timestamps<=end_time_unix)[0][-1]\n",+ "else:\n",+ " r_eye_end = len(r_eye_timestamps)\n",+ "\n",+ "#left eye\n",+ "if any(l_eye_timestamps>=start_time_unix):\n",+ " l_eye_start = np.where(l_eye_timestamps>=start_time_unix)[0][0]\n",+ "else:\n",+ " l_eye_start = 0\n",+ " \n",+ "if any(l_eye_timestamps<=end_time_unix):\n",+ " l_eye_end = np.where(l_eye_timestamps<=end_time_unix)[0][-1]\n",+ "else:\n",+ " l_eye_end = len(l_eye_timestamps)\n",+ "\n",+ "#rebase time onto freemocap's framerate (b/c it's slower than pupil) <- sloppy, assumes mocap slower than eye tracker, which is untrue for say GoPros\n",+ "new_timestamps = np.arange(start_time_unix,end_time_unix,1/freemocap_frames_per_second) \n",+ "\n",+ "\n",+ "freemocap_timestamps = freemocap_timestamps[freemocap_start:freemocap_end]\n",+ "r_eye_timestamps = r_eye_timestamps[r_eye_start:r_eye_end]\n",+ "l_eye_timestamps = l_eye_timestamps[l_eye_start:l_eye_end]\n",+ "\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "clip out portions of each eye's data stream that correspond to the shared time period\n",+ "\n",+ "(I appear to be skiping the freemocap data, which might cause fail on other recordings?)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "r_eye_sphere_center_x = r_eye_sphere_center_x[r_eye_start:r_eye_end]\n",+ "r_eye_sphere_center_y = r_eye_sphere_center_y[r_eye_start:r_eye_end]\n",+ "r_eye_sphere_center_z = r_eye_sphere_center_z[r_eye_start:r_eye_end]\n",+ "\n",+ "r_eye_pupil_center_normal_x = r_eye_pupil_center_normal_x[r_eye_start:r_eye_end]\n",+ "r_eye_pupil_center_normal_y = r_eye_pupil_center_normal_y[r_eye_start:r_eye_end]\n",+ "r_eye_pupil_center_normal_z = r_eye_pupil_center_normal_z[r_eye_start:r_eye_end]"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "l_eye_sphere_center_x = l_eye_sphere_center_x[l_eye_start:l_eye_end]\n",+ "l_eye_sphere_center_y = l_eye_sphere_center_y[l_eye_start:l_eye_end]\n",+ "l_eye_sphere_center_z = l_eye_sphere_center_z[l_eye_start:l_eye_end]\n",+ "\n",+ "l_eye_pupil_center_normal_x = l_eye_pupil_center_normal_x[l_eye_start:l_eye_end]\n",+ "l_eye_pupil_center_normal_y = l_eye_pupil_center_normal_y[l_eye_start:l_eye_end]\n",+ "l_eye_pupil_center_normal_z = l_eye_pupil_center_normal_z[l_eye_start:l_eye_end]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "create r_eye_gaze_xyz : the direction the eye is pointing *in head centered coordinates*"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "r_eye_gaze_x = r_eye_pupil_center_normal_x\n",+ "r_eye_gaze_y = r_eye_pupil_center_normal_y\n",+ "r_eye_gaze_z = r_eye_pupil_center_normal_z\n",+ "\n",+ "r_eye_gaze_xyz_og = np.vstack((r_eye_gaze_x, r_eye_gaze_y, r_eye_gaze_z)).T\n",+ "\n",+ "print(f'r_eye_gaze_xyz_og.shape: {r_eye_gaze_xyz_og.shape}')"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "l_eye_gaze_x = l_eye_pupil_center_normal_x\n",+ "l_eye_gaze_y = l_eye_pupil_center_normal_y\n",+ "l_eye_gaze_z = l_eye_pupil_center_normal_z\n",+ "\n",+ "l_eye_gaze_xyz_og = np.vstack((l_eye_gaze_x, l_eye_gaze_y, l_eye_gaze_z)).T\n",+ "\n",+ "print(f'l_eye_gaze_xyz_og.shape: {l_eye_gaze_xyz_og.shape}')"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "#### resample gaze and skeleton data so that each has the same number of frames\n",+ "\n",+ "resample pupil data onto freemocap timestamps"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "r_eye_gaze_xyz = np.zeros((len(freemocap_timestamps),3))\n",+ "l_eye_gaze_xyz = np.zeros((len(freemocap_timestamps),3))\n",+ "\n",+ "for this_dimension in range(3): #loop over the X Y and Z simensions and resample to new timestamps\n",+ " r_eye_gaze_xyz[:,this_dimension] = np.interp(freemocap_timestamps,\n",+ " r_eye_timestamps,\n",+ " r_eye_gaze_xyz_og[:,this_dimension])\n",+ " l_eye_gaze_xyz[:,this_dimension] = np.interp(freemocap_timestamps,\n",+ " l_eye_timestamps,\n",+ " l_eye_gaze_xyz_og[:,this_dimension])\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "r_eye_gaze_xyz = r_eye_gaze_xyz/np.linalg.norm(r_eye_gaze_xyz,axis=1,ord=2)[:,None]\n",+ "l_eye_gaze_xyz = l_eye_gaze_xyz/np.linalg.norm(l_eye_gaze_xyz,axis=1,ord=2)[:,None]\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "# snip off the first frame of the freemocap data becuase there is a -> BUG <- such that there are one fewer timestamp than there are recorded frames. \n",+ "\n",+ "V V V V V V"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "if r_eye_gaze_xyz.shape[0] != mediapipe_skeleton_fr_mar_xyz.shape[0]:\n",+ " mediapipe_skeleton_fr_mar_xyz = np.delete(mediapipe_skeleton_fr_mar_xyz, 0, axis=0)\n",+ " mediapipe_head_rotation_matricies_fr_row_col = np.delete(mediapipe_head_rotation_matricies_fr_row_col, 0, axis=0)\n",+ "# DOESN'T BOTHER ME AT ALL HAHA"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "These should all have the exact same number of frame"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "print(f'r_eye_gaze_xyz.shape: {r_eye_gaze_xyz.shape}')\n",+ "print(f'l_eye_gaze_xyz.shape: {l_eye_gaze_xyz.shape}')\n",+ "print(f'mediapipe_skeleton_fr_mar_xyz.shape: {mediapipe_skeleton_fr_mar_xyz.shape}')\n",+ "print(f'mediapipe_head_rotation_matricies_fr_row_col.shape: {mediapipe_head_rotation_matricies_fr_row_col.shape}')"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "if debug:\n",+ " fig = plt.figure(num=65341,figsize=(10,20))\n",+ " ax1 = fig.add_subplot(211)\n",+ " ax1.plot(freemocap_timestamps, r_eye_gaze_xyz[:,0], '.-', label='r_eye_gaze_x')\n",+ " ax1.plot(freemocap_timestamps, r_eye_gaze_xyz[:,1], '.-', label='r_eye_gaze_y')\n",+ " ax1.plot(freemocap_timestamps, r_eye_gaze_xyz[:,2], '.-', label='r_eye_gaze_z')\n",+ " ax1.legend(loc='upper left')\n",+ " \n",+ " ax2 = fig.add_subplot(212)\n",+ " ax2.plot(freemocap_timestamps, l_eye_gaze_xyz[:,0], '.-', label='_eye_gaze_x')\n",+ " ax2.plot(freemocap_timestamps, l_eye_gaze_xyz[:,1], '.-', label='_eye_gaze_y')\n",+ " ax2.plot(freemocap_timestamps, l_eye_gaze_xyz[:,2], '.-', label='_eye_gaze_z')\n",+ " ax2.legend(loc='upper left')\n",+ " plt.show()\n",+ "\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "normalize gaze vector to unit length"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "save out `npy` of gaze_xyz data"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "Smooth a bit with savitsky golay filter"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "from scipy.signal import savgol_filter\n",+ "\n",+ "smoothWinLength = 5\n",+ "smoothOrder = 3\n",+ "for dim in range(r_eye_gaze_xyz.shape[1]):\n",+ " r_eye_gaze_xyz[:,dim] = savgol_filter(r_eye_gaze_xyz[:,dim], smoothWinLength, smoothOrder)\n",+ " l_eye_gaze_xyz[:,dim] = savgol_filter(l_eye_gaze_xyz[:,dim], smoothWinLength, smoothOrder)\n",+ "\n",+ "\n",+ "# #reorient gaze (I think because it comes in in image coordiates? not really sure )\n",+ "# r_eye_gaze_xyz[:,1] = -r_eye_gaze_xyz[:,1]\n",+ "# r_eye_gaze_xyz[:,(0,2)] = r_eye_gaze_xyz[:,(2,0)]\n",+ "\n",+ "# l_eye_gaze_xyz[:,1] = -l_eye_gaze_xyz[:,1]\n",+ "# l_eye_gaze_xyz[:,(0,2)] = l_eye_gaze_xyz[:,(2,0)]"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "r_gaze_xyz_npy_save_path = pupil_data_exports_path / 'right_eye_gaze_xyz_synchronized_w_freemocap_uncalibrated.npy'\n",+ "np.save(r_gaze_xyz_npy_save_path, r_eye_gaze_xyz)\n",+ "\n",+ "l_gaze_xyz_npy_save_path = pupil_data_exports_path / 'left_eye_gaze_xyz_synchronized_w_freemocap_uncalibrated.npy'\n",+ "np.save(l_gaze_xyz_npy_save_path, l_eye_gaze_xyz)\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "# VOR Calibration"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "# prep data for VOR calibration\n",+ "\n",+ "we'll be finding a rotational offset such that gaze_xyz rotated-by-the offset-then-rotated-by-head-orientation will align with the fixation point\n",+ "\n",+ " so we want to put the fixation point (in this case the right index finger in a eyeball-centered reference frame (i.e. where `r_eye_xyz` is at `(0,0,0)`)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "vor_frame_start =1000\n",+ "vor_frame_end = 1400\n",+ "\n",+ "right_index_fingertip_idx = 41 # pretty sure this is right?\n",+ "r_eye_idx = 5\n",+ "\n",+ "fixation_point_xyz = np.squeeze(mediapipe_skeleton_fr_mar_xyz[:, right_index_fingertip_idx, :])\n",+ "r_eyeball_center_xyz = np.squeeze(mediapipe_skeleton_fr_mar_xyz[:, r_eye_idx, :])\n",+ "#make sure everyone is the right size and shape\n",+ "print(f'head_rotation_martices_fr_row_col.shape: {mediapipe_head_rotation_matricies_fr_row_col.shape}')\n",+ "print(f'r_eye_gaze_xyz.shape: {r_eye_gaze_xyz.shape}')\n",+ "print(f'fixation_point_xyz.shape: {fixation_point_xyz.shape}')\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "fixation_point_during_vor_xyz = mediapipe_skeleton_fr_mar_xyz[vor_frame_start:vor_frame_end, right_index_fingertip_idx, :]\n",+ "r_eyeball_center_during_vor_xyz = mediapipe_skeleton_fr_mar_xyz[vor_frame_start:vor_frame_end, r_eye_idx, :]\n",+ "\n",+ "fixation_point_in_r_eyeball_coordinates_fr_xyz = r_eyeball_center_during_vor_xyz - fixation_point_during_vor_xyz #Subtract the-thing-that-will-be-at-origin from the-other-things to put other-things in ## thing-that-will-be-at-origin-centered-reference-frame\n",+ "fixation_point_distance_xyz = np.squeeze(np.linalg.norm(fixation_point_in_r_eyeball_coordinates_fr_xyz,axis=1,ord=2)[:,None])\n",+ "\n",+ "\n",+ "\n",+ "r_gaze_during_vor_fr_xyz = r_eye_gaze_xyz[vor_frame_start:vor_frame_end]\n",+ "r_gaze_during_vor_fr_xyz[:,0] = r_gaze_during_vor_fr_xyz[:,0] * fixation_point_distance_xyz #scale gaze vector to reach fixation point\n",+ "r_gaze_during_vor_fr_xyz[:,1] = r_gaze_during_vor_fr_xyz[:,1] * fixation_point_distance_xyz #scale gaze vector to reach fixation point\n",+ "r_gaze_during_vor_fr_xyz[:,2] = r_gaze_during_vor_fr_xyz[:,2] * fixation_point_distance_xyz #scale gaze vector to reach fixation point\n",+ "r_gaze_during_vor_fr_xyz"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "head_rotation_matrices_during_vor_fr_row_col = mediapipe_head_rotation_matricies_fr_row_col[vor_frame_start:vor_frame_end, :, :]\n",+ "\n",+ "#make sure everyone is the right size and shape\n",+ "print(f'head_rotation_matrices_during_vor_fr_row_col.shape: {head_rotation_matrices_during_vor_fr_row_col.shape}')\n",+ "print(f'r_gaze_during_vor_xyz.shape: {r_gaze_during_vor_fr_xyz.shape}')\n",+ "print(f'fixation_point_in_r_eyeball_coordinates_xyz.shape: {fixation_point_in_r_eyeball_coordinates_fr_xyz.shape}')\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "put mediapipe skel into eyeball centered coordinates (for debugging)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "skel_during_vor_fr_mar_dim = mediapipe_skeleton_fr_mar_xyz[vor_frame_start:vor_frame_end,:,:]\n",+ "\n",+ "for this_marker_number in range(skel_during_vor_fr_mar_dim.shape[1]):\n",+ " skel_during_vor_fr_mar_dim[:,this_marker_number,:] = r_eyeball_center_during_vor_xyz - skel_during_vor_fr_mar_dim[:,this_marker_number,:]\n",+ "\n",+ "\n",+ "print(f'skel_during_vor_fr_mar_dim.shape: {skel_during_vor_fr_mar_dim.shape}')"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "# define error functions and run optimization\n",+ "\n",+ "according to lines 58:68 of - [https://github.com/trentwirth/ARGP_Matlab_LaserSkeleton/blob/main/Toolbox/vorPupilAlignErrFun_eyeCam.m](https://github.com/trentwirth/ARGP_Matlab_LaserSkeleton/blob/main/Toolbox/vorPupilAlignErrFun_eyeCam.m)\n",+ "```\n",+ "\n",+ "%%\n",+ "%%%%%%% This part's important - Rotate gaze vector by [this guess at the proper alignment matrix], prior to resituating the origin on on the eyeball\n",+ "%%%%\n",+ "\n",+ "\n",+ "for rr = 1:length(gazeXYZ)\n",+ " \n",+ " thisET_frame_unrot = camRotMatGuess * [gazeXYZ(rr,1); gazeXYZ(rr,2); gazeXYZ(rr,3)];\n",+ " thisETframe = headRotMat_row_col_fr(:,:,rr) * thisET_frame_unrot;\n",+ " \n",+ " headOrVec(rr,:) = headRotMat_row_col_fr(:,:,rr) * [2e3; 0;0];\n",+ " \n",+ " gazeXYZ(rr,:) = thisETframe;\n",+ " \n",+ "end\n",+ "\n",+ "```\n",+ "\n",+ "we want to rotate by rotation_guess THEN by head rotation \n",+ "\n",+ "(which makes sense, since we're using the optimizer to find the rotational offset between the eye camera reference frame and the eye-in-head reference frame)\n",+ "\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "### optimization error animation function \n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "plt.close('all')\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "if debug:\n",+ " def plot_optimization_error(error,gaze_xyz, gaze_rotated_by_guess_then_head_rotation_xyz, mean_fixation_point_xyz, skel_during_vor_fr_mar_dim):\n",+ " figure_number=13451\n",+ "\n",+ " if not plt.fignum_exists(figure_number):\n",+ " fig = plt.figure(figure_number)\n",+ " ax = fig.add_subplot(111, projection='3d')\n",+ " else:\n",+ " fig = plt.gcf()\n",+ " ax = plt.gca()\n",+ "\n",+ "\n",+ " fig.suptitle(f'error: {error}')\n",+ " ax_range = 1e3\n",+ " ax.clear() \n",+ "\n",+ " ax.plot(0,0,0, 'mo',label='origin')\n",+ "\n",+ " ax.plot(gaze_xyz[:,0],\n",+ " gaze_xyz[:,1], \n",+ " gaze_xyz[:,2], 'k-o',label='original gaze xyz')\n",+ "\n",+ " gaze_rot_xyz =np.array(gaze_rotated_by_guess_then_head_rotation_xyz)\n",+ " ax.plot(gaze_rot_xyz[:,0],\n",+ " gaze_rot_xyz[:,1], \n",+ " gaze_rot_xyz[:,2], 'r-o',label='gaze_rotated_by_guess_then_head_rotation_xyz')\n",+ "\n",+ " ax.plot(mean_fixation_point_xyz[0],\n",+ " mean_fixation_point_xyz[1], \n",+ " mean_fixation_point_xyz[2], 'b-o',label='mean_fixation_point_xyz')\n",+ "\n",+ " ax.set_xlim([-ax_range, ax_range])\n",+ " ax.set_ylim([-ax_range, ax_range])\n",+ " ax.set_zlim([-ax_range, ax_range])\n",+ " ax.legend()\n",+ " plt.pause(.01)\n",+ "\n",+ "\n",+ "\n",+ "def get_error_between_two_rotation_matricies(euler_angle_guess,\n",+ " gaze_xyz,\n",+ " fixation_point_in_eye_coordinates_xyz,\n",+ " head_rotation_maticies_fr_row_col,\n",+ " skel_during_vor_fr_mar_dim):\n",+ " #convert euler angles to rotation matrix\n",+ " rotation_matrix_guess = Rotation.from_euler('XYZ',euler_angle_guess).as_matrix()\n",+ "\n",+ " #rotate gaze by rotation guess\n",+ " gaze_rotated_by_guess = [rotation_matrix_guess @ gaze_xyz[this_frame_number,:] for this_frame_number in range(gaze_xyz.shape[0])]\n",+ "\n",+ " #...then rotate THAT by head_rotation matrix\n",+ " gaze_rotated_by_guess_then_head_rotation_xyz =[head_rotation_maticies_fr_row_col[this_frame_number,:,:] @ gaze_rotated_by_guess[this_frame_number]\n",+ " for this_frame_number in range(gaze_xyz.shape[0])]\n",+ "\n",+ " mean_fixation_point_xyz = np.mean(fixation_point_in_eye_coordinates_xyz, axis=0)\n",+ " #define error as difference between the fixation point and that rotated gaze estimate (these are both normalized, I think)\n",+ " error_per_frame = gaze_rotated_by_guess_then_head_rotation_xyz - mean_fixation_point_xyz\n",+ " error = np.nanmean(np.nansum(error_per_frame**2)/error_per_frame.shape[0])\n",+ " \n",+ " plot_optimization_error(error,gaze_xyz, gaze_rotated_by_guess_then_head_rotation_xyz, mean_fixation_point_xyz, skel_during_vor_fr_mar_dim)\n",+ "\n",+ " return error\n",+ "\n",+ "\n",+ "def get_optimal_rotation_matrix_to_align_gaze_with_target(gaze_xyz,\n",+ " fixation_point_in_eye_coordinates_xyz,\n",+ " head_rotation_matricies_fr_row_col,\n",+ " skel_during_vor_fr_mar_dim):\n",+ " euler_angles = optimize.least_squares(get_error_between_two_rotation_matricies,\n",+ " [0,0,0],\n",+ " args=(gaze_xyz,fixation_point_in_eye_coordinates_xyz, head_rotation_matricies_fr_row_col, skel_during_vor_fr_mar_dim),\n",+ " gtol=1e-10,\n",+ " verbose=2).x\n",+ " return Rotation.from_euler('XYZ',euler_angles).as_matrix()\n",+ "\n",+ "\n",+ "######################################\n",+ "\n",+ "\n",+ "rotation_matrix_to_align_pupil_data_to_eye_in_head_coordinates = get_optimal_rotation_matrix_to_align_gaze_with_target(r_gaze_during_vor_fr_xyz,\n",+ " fixation_point_in_r_eyeball_coordinates_fr_xyz,\n",+ " head_rotation_matrices_during_vor_fr_row_col,\n",+ " skel_during_vor_fr_mar_dim)\n",+ "\n",+ "r_eye_gaze_xyz_calibrated = [rotation_matrix_to_align_pupil_data_to_eye_in_head_coordinates @ r_eye_gaze_xyz[this_frame_number,:] for this_frame_number in range(r_eye_gaze_xyz.shape[0])]\n",+ "\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "gaze_xyz_npy_save_path = pupil_data_exports_path / 'right_eye_gaze_xyz.npy'\n",+ "np.save(gaze_xyz_npy_save_path, r_eye_gaze_xyz_calibrated)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "import plotly.offline as py\n",+ "import plotly.express as px \n",+ "import plotly.graph_objects as go\n",+ "\n",+ "fig = go.Figure()\n",+ "\n",+ "fig.add_trace(go.Scatter(x=np.arange(r_eye_gaze_xyz.shape[0]), y=r_eye_gaze_xyz[:,0], mode='lines+markers'))\n",+ "fig.add_trace(go.Scatter(x=np.arange(r_eye_gaze_xyz.shape[0]), y=r_eye_gaze_xyz[:,1], mode='lines+markers'))\n",+ "fig.add_trace(go.Scatter(x=np.arange(r_eye_gaze_xyz.shape[0]), y=r_eye_gaze_xyz[:,2], mode='lines+markers'))\n",+ "fig.show()"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "\n",+ "data_path = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data/')\n",+ "session_path = data_path / 'sesh_2022-02-15_11_54_28_pupil_maybe'\n",+ "\n",+ "qt_gl_laser_skeleton = QT_GL_LaserSkeleton(session_path)\n",+ "qt_gl_laser_skeleton.start_animation()\n"+ ]+ }+ ],+ "metadata": {+ "interpreter": {+ "hash": "4ef2e72e5b895021b427c707c9f05b4880b5a5a00e886b84de197a42e54c7f32"+ },+ "kernelspec": {+ "display_name": "Python 3.9.12 ('humon-env')",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.9.12"+ },+ "orig_nbformat": 4+ },+ "nbformat": 4,+ "nbformat_minor": 2+}diff --git a/jon_scratch/matplotlib_laser_skeleton.py b/jon_scratch/matplotlib_laser_skeleton.pynew file mode 100644index 00000000..27d59552--- /dev/null+++ b/jon_scratch/matplotlib_laser_skeleton.py@@ -0,0 +1,1039 @@+"""+============+ββββββ ββ βββββ ββ ββ βββββββ ββ ββ βββββββ ββ βββββββ ββββββββ ββββββ βββ ββ βββββ βββ ββ ββ βββ βββ βββββ ββββββββ ββ ββββββ βββ ββ+ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββββ ββ ββ ββ ββββ ββ ββ ββββ ββββ ββ ββ ββ ββ ββ ββ ββββ ββ+ββββββ ββ βββββββ ββββ βββββββ βββββ βββββ ββ βββββ ββ ββ ββ ββ ββ ββ βββββββ ββ ββ ββ ββ ββ ββββ ββ βββββββ ββ ββ ββ ββ ββ ββ ββ+ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ββ βββββββ ββ ββ ββ βββββββ ββ ββ βββββββ βββββββ βββββββ ββ ββββββ ββ ββββ ββ ββ ββ ββββ ββ ββ ββ ββ ββ ββ ββ ββββββ ββ ββββ+============+Font - ANSI Regular - https://patorjk.com/software/taag/#p=display&f=ANSI%20Regular&t=Play%20Skeleton%20Animation+============+Originally based on tutorial from - https://matplotlib.org/2.1.2/gallery/animation/simple_3danim.html++Create a matplotlib animation from a FreeMoCap recording session.+"""+++################################################################################################################+################################################################################################################+###+### ββ βββ βββ ββββββ ββββββ ββββββ ββββββββ+### ββ ββββ ββββ ββ ββ ββ ββ ββ ββ ββ+### ββ ββ ββββ ββ ββββββ ββ ββ ββββββ ββ+### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+### ββ ββ ββ ββ ββββββ ββ ββ ββ+###+################################################################################################################+################################################################################################################+++import numpy as np++import matplotlib as mpl+import matplotlib.pyplot as plt+import mpl_toolkits.mplot3d.axes3d as p3+import matplotlib.animation as animation+from matplotlib.widgets import Slider+import cv2+from scipy.signal import savgol_filter+import moviepy.editor as mp++import os+import copy+from pathlib import Path++#RICH CONSOLE STUFF+from rich import pretty+pretty.install() #makes all print statement output pretty+from rich import inspect+from rich.console import Console+console = Console()+from rich.traceback import install as rich_traceback_install+from rich.markdown import Markdown++#colors from Taylor Davis branding -+humon_dark = np.array([37, 67, 66])/255+humon_green = np.array([53, 93, 95])/255+humon_blue = np.array([14, 90, 253])/255+humon_red = np.array([217, 61, 67])/255+++def PlaySkeletonAnimation(+ session=None,+ vidType=1,+ startFrame=0,+ azimuth=-90,+ elevation=-70,+ useOpenPose=True,+ useMediaPipe=False,+ useDLC=False,+ recordVid = True,+ showAnimation =True,+ ):+++++ ###+ ###+ ### ββ ββ ββββββ ββββββ βββββ ββββββββ βββββββ βββββββ ββ ββββββ ββ ββ ββββββ βββββββ+ ### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββ ββ ββββββ ββ ββ βββββββ ββ βββββ βββββ ββ ββ βββ ββ ββ ββββββ βββββ+ ### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββββββ ββ ββββββ ββ ββ ββ βββββββ ββ ββ ββββββ ββββββ ββ ββ βββββββ+ ###+ def update_figure(frameNum):+ """+ Called by matplotlib animator for each frame.+ """++ if useOpenPose:+ op_skel_dottos = matplotlib_artist_objs['op_skel_dottos']+ op_skel_trajectories = figure_data['openPose_skel_trajectories|mar|fr_dim']++ if useMediaPipe:+ mp_skel_dottos = matplotlib_artist_objs['mp_skel_dottos']+ mp_skel_trajectories = figure_data['mediaPipe_skel_trajectories|mar|fr_dim']++ charuco_dottos = matplotlib_artist_objs['charuco_dottos']+ charuco_trajectories = figure_data['charuco_trajectories|mar|fr_dim']++ if useDLC:+ dlc_dottos = matplotlib_artist_objs['dlc_dottos']+ dlc_trajectories = figure_data['dlc_trajectories|mar|fr_dim']++ #function to update the lines for each 3d body segment+ def update_3d_skel_segments(key, data_fr_mar_xyz):+ """+ updates the Artist of each body segment with the current frame data.+ """+ split = key.split('_')++ if split[0] == 'op':+ dict_of_segments_idxs = dict_of_openPoseSegmentIdx_dicts[key]+ if split[0] == 'mp':+ dict_of_segments_idxs = dict_of_mediaPipeSegmentIdx_dicts[key]+ elif split[0] == 'dlc':+ dict_of_segments_idxs = dict_of_dlcSegmentIdx_dicts[key]++ try:+ dict_of_artists = matplotlib_artist_objs[key]+ except:+ print('KeyError:', key)+++ for thisItem in dict_of_segments_idxs.items():+ segName = thisItem[0]+ segArtist = dict_of_artists[segName]+ segArtist.set_data((data_fr_mar_xyz[frameNum, dict_of_segments_idxs[segName], 0], data_fr_mar_xyz[frameNum, dict_of_segments_idxs[segName], 1]))+ segArtist.set_3d_properties(data_fr_mar_xyz[frameNum, dict_of_segments_idxs[segName], 2])++ if useOpenPose:+ update_3d_skel_segments('op_body', openPose_skel_fr_mar_xyz)+ update_3d_skel_segments('op_rHand', openPose_skel_fr_mar_xyz)+ update_3d_skel_segments('op_lHand', openPose_skel_fr_mar_xyz)+ update_3d_skel_segments('op_face', openPose_skel_fr_mar_xyz)++ if useMediaPipe:+ update_3d_skel_segments('mp_body', mediaPipe_skel_fr_mar_xyz)+ update_3d_skel_segments('mp_rHand', mediaPipe_skel_fr_mar_xyz)+ update_3d_skel_segments('mp_lHand', mediaPipe_skel_fr_mar_xyz)++ if useDLC:+ update_3d_skel_segments('dlc_pinkBall', dlc_fr_mar_xyz)+ update_3d_skel_segments('dlc_redBall', dlc_fr_mar_xyz)+ update_3d_skel_segments('dlc_greenBall', dlc_fr_mar_xyz)+ update_3d_skel_segments('dlc_wobbleBoard', dlc_fr_mar_xyz)+ update_3d_skel_segments('dlc_wobbleWheel', dlc_fr_mar_xyz)++ for bb in range(3):+ thisTailArtist = matplotlib_artist_objs['ball_tails'][bb]++ thisTailArtist.set_data((dlc_trajectories[bb][frameNum-ballTailLen:frameNum+1, 0],+ dlc_trajectories[bb][frameNum-ballTailLen:frameNum+1, 1]))++ thisTailArtist.set_3d_properties(dlc_trajectories[bb][frameNum-ballTailLen:frameNum+1, 2])+ #dlc data+ marNum = -1+ for thisDotto, thisTraj in zip(dlc_dottos,dlc_trajectories):+ marNum+=1+ # NOTE: there is no .set_data() for 3 dim data...+ thisDotto.set_data(thisTraj[ frameNum, 0:2])+ thisDotto.set_3d_properties(thisTraj[ frameNum, 2])++ #Plots the dots!+ #openpose data+ if useOpenPose:+ marNum = -1+ for thisSkelDotto, thisTraj in zip(op_skel_dottos,op_skel_trajectories):+ marNum+=1+ # NOTE: there is no .set_data() for 3 dim data...+ thisSkelDotto.set_data(thisTraj[ frameNum, 0:2])+ thisSkelDotto.set_3d_properties(thisTraj[ frameNum, 2])++ #mediapipe data+ if useMediaPipe:+ marNum = -1+ for thisSkelDotto, thisTraj in zip(mp_skel_dottos,mp_skel_trajectories):+ marNum+=1+ # NOTE: there is no .set_data() for 3 dim data...+ thisSkelDotto.set_data(thisTraj[ frameNum, 0:2])+ thisSkelDotto.set_3d_properties(thisTraj[ frameNum, 2])++++ #charuco data+ for thisDotto, thisTraj in zip(charuco_dottos,charuco_trajectories):++ if frameNum<charuco_trajectories[0].shape[0]:+ thisDotto.set_data(thisTraj[ frameNum, 0:2])+ thisDotto.set_3d_properties(thisTraj[ frameNum, 2])+ else:+ thisDotto.set_color('none')+++++ # function to update the lines for each body segment+ def update_2d_skel_segments(vidNum,key):+ """+ updates the Artist of each body segment with the current frame data.+ """+ split = key.split('_')+ if split[0] == 'op':+ dict_of_segments_idxs = dict_of_openPoseSegmentIdx_dicts[key]+ dict_of_artists = thisVidOpenPoseArtist_dict[key]+ elif split[0] == 'mp':+ dict_of_segments_idxs = dict_of_mediaPipeSegmentIdx_dicts[key]+ dict_of_artists = thisVidMediaPipeArtist_dict[key]+++ for thisItem in dict_of_segments_idxs.items():+ segName = thisItem[0]+ segArtist = dict_of_artists[segName]++ if split[0] == 'op':+ xData = openPose_nCams_nFrames_nImgPts_XYC[vidNum, frameNum, dict_of_segments_idxs[segName],0]+ yData = openPose_nCams_nFrames_nImgPts_XYC[vidNum, frameNum, dict_of_segments_idxs[segName],1]+ elif split[0] == 'mp':+ xData = mediaPipe_nCams_nFrames_nImgPts_XYC[vidNum, frameNum, dict_of_segments_idxs[segName],0]+ yData = mediaPipe_nCams_nFrames_nImgPts_XYC[vidNum, frameNum, dict_of_segments_idxs[segName],1]++ xDataMasked = np.ma.masked_array(xData, mask=(xData==0))+ yDataMasked = np.ma.masked_array(yData, mask=(xData==0))+ segArtist.set_data(xDataMasked,yDataMasked)+++ #update video frames+ for vidNum, thisVidArtist in enumerate(vidAristList):+ vidCapObjList[vidNum].set(cv2.CAP_PROP_POS_FRAMES, frameNum)+ success, image = vidCapObjList[vidNum].read()++ if success:+ thisVidArtist.set_array(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))+ else:+ vidCapObjList[vidNum].set(cv2.CAP_PROP_POS_FRAMES, 0)+ success, image = vidCapObjList[vidNum].read()++ if useOpenPose:+ thisVidOpenPoseArtist_dict = list_of_vidOpenPoseArtistdicts[vidNum]+ update_2d_skel_segments(vidNum,'op_body')+ update_2d_skel_segments(vidNum,'op_rHand')+ update_2d_skel_segments(vidNum,'op_lHand')+ update_2d_skel_segments(vidNum,'op_face')++ if useMediaPipe:+ thisVidMediaPipeArtist_dict = list_of_vidMediaPipeArtistdicts[vidNum]+ update_2d_skel_segments(vidNum,'mp_body')+ update_2d_skel_segments(vidNum,'mp_rHand')+ update_2d_skel_segments(vidNum,'mp_lHand')++ if useDLC: #JSM NOTE - this one won't work, we'll need to fix it the next time we get DLC data+ thisVidDLCArtist[0].set_data( dlcData_nCams_nFrames_nImgPts_XYC[vidNum,frameNum,:,0],+ dlcData_nCams_nFrames_nImgPts_XYC[vidNum,frameNum,:,1],)+++ #shift timeseries xlims along with the frameNum+ xTimeSeriesAx.set_xlim([(frameNum/fps)-timeRange, (frameNum/fps)+timeRange])+ yTimeSeriesAx.set_xlim([(frameNum/fps)-timeRange, (frameNum/fps)+timeRange])+ for thisArtistKey in xCurrTimeArtists:++ xCurrTimeArtists[thisArtistKey][0].set_xdata(frameNum/fps)+ yCurrTimeArtists[thisArtistKey][0].set_xdata(frameNum/fps)+ if not thisArtistKey == 'blackLine':+ xCurrTimeArtists[thisArtistKey][0].set_ydata(xCurrTimeYdata[thisArtistKey][frameNum] )+ yCurrTimeArtists[thisArtistKey][0].set_ydata(yCurrTimeYdata[thisArtistKey][frameNum] )++ yCurrTimeArtists['frameNumber'].set_x(frameNum/fps+3/fps)+ yCurrTimeArtists['frameNumber'].set_text('Frame# '+ str(frameNum))+ # if recordVid:+ # thisFramePath = str(animationFramePath) + '/'+ str(session.sessionID) + "_frame_" + str(frameNum).zfill(6) + ".png"+ # plt.savefig(thisFramePath)+ # # fig.suptitle("nSessionID: {}, Frame: {} of {}".format(session.sessionID, frameNum, numFrames), fontsize=10)+ # # animSlider.set_val(val=frameNum)++ ####+ #### ββββββ ββββββ βββββββ βββββ ββββββββ βββββββ βββββββ ββ ββββββ ββ ββ ββββββ βββββββ+ #### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ #### ββ ββββββ βββββ βββββββ ββ βββββ βββββ ββ ββ βββ ββ ββ ββββββ βββββ+ #### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ #### ββββββ ββ ββ βββββββ ββ ββ ββ βββββββ ββ ββ ββββββ ββββββ ββ ββ βββββββ+ ####+++ fig = plt.figure(dpi=200)+ plt.ion()+++ ###+ ### ββββββ ββββββ βββββββ βββββ ββββββββ βββββββ ββββββ ββββββ βββββ ββ ββ ββ βββββββ+ ### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββ ββββββ βββββ βββββββ ββ βββββ βββββ βββββ ββ ββ βββββββ βββ ββ βββββββ+ ### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββββββ ββ ββ βββββββ ββ ββ ββ βββββββ ββββββ ββββββ ββ ββ ββ ββ ββ βββββββ+ ###+ ###++ # ax3d = p3.Axes3D(fig)+ ax3d = fig.add_subplot(projection='3d')+ ax3d.set_position([-.065, .28, .7, .7]) # [left, bottom, width, height])+ ax3d.set_xticklabels([])+ ax3d.set_yticklabels([])+ ax3d.set_zticklabels([])++ # ax3d.set_xlabel('X')+ # ax3d.set_ylabel('Y')+ # ax3d.set_zlabel('Z')+ # ax3d.set_axis_off()+++ ax3d.tick_params(length=0) # WHY DOESNT THIS WORK? I HATE THOSE TICK MARKS >:(++ smoothData=True+ smoothWinLength = 5+ smoothOrder = 3++ if useOpenPose:+ try:+ openPose_skel_fr_mar_xyz = np.load(session.dataArrayPath / 'openPoseSkel_3d.npy')+ openPose_nCams_nFrames_nImgPts_XYC = np.load(session.dataArrayPath / 'openPoseData_2d.npy')+ except:+ print('No openPose data found.')++ if smoothData:+ for dim in range(openPose_skel_fr_mar_xyz.shape[2]):+ for mm in range(openPose_skel_fr_mar_xyz.shape[1]):+ openPose_skel_fr_mar_xyz[:,mm,dim] = savgol_filter(openPose_skel_fr_mar_xyz[:,mm,dim], smoothWinLength, smoothOrder)++ if useMediaPipe:+ try:+ mediaPipe_skel_fr_mar_xyz = np.load(session.dataArrayPath / 'mediaPipeSkel_3d.npy')+ mediaPipe_nCams_nFrames_nImgPts_XYC = np.load(session.dataArrayPath / 'mediaPipeData_2d.npy')+ except:+ print('No mediaPipe data found.')++ if smoothData:+ for dim in range(mediaPipe_skel_fr_mar_xyz.shape[2]):+ for mm in range(mediaPipe_skel_fr_mar_xyz.shape[1]):+ mediaPipe_skel_fr_mar_xyz[:,mm,dim] = savgol_filter(mediaPipe_skel_fr_mar_xyz[:,mm,dim], smoothWinLength, smoothOrder)++ try:+ charuco_fr_mar_xyz = np.load(session.dataArrayPath / 'charuco_3d_points.npy')+ except:+ console.warning('No charuco data found')+++++ figure_data = dict()+ if useOpenPose:+ figure_data['openPose_skel_trajectories|mar|fr_dim'] = [openPose_skel_fr_mar_xyz[:,markerNum,:] for markerNum in range(openPose_skel_fr_mar_xyz.shape[1])]+ figure_data['openPose_skel_fr_mar_xyz'] = openPose_skel_fr_mar_xyz++ dict_of_openPoseSegmentIdx_dicts, dict_of_op_skel_lineColor = formatOpenPoseStickIndices() #these will help us draw body and hands stick figures+++ if useMediaPipe:+ mediaPipe_trajectories = [mediaPipe_skel_fr_mar_xyz[:,markerNum,:] for markerNum in range(mediaPipe_skel_fr_mar_xyz.shape[1])]+ figure_data['mediaPipe_skel_trajectories|mar|fr_dim'] = [mediaPipe_skel_fr_mar_xyz[:,markerNum,:] for markerNum in range(mediaPipe_skel_fr_mar_xyz.shape[1])]+ figure_data['mediaPipe_skel_fr_mar_xyz'] = mediaPipe_skel_fr_mar_xyz++ dict_of_mediaPipeSegmentIdx_dicts, dict_of_mp_skel_lineColor = formatMediaPipeStickIndices() #these will help us draw body and hands stick figures+++ charuco_trajectories = [charuco_fr_mar_xyz[:,markerNum,:] for markerNum in range(charuco_fr_mar_xyz.shape[1])]+ figure_data['charuco_trajectories|mar|fr_dim'] = charuco_trajectories+ figure_data['charuco_fr_mar_xyz'] = charuco_fr_mar_xyz+++ if useDLC:+ dlc_fr_mar_xyz = np.load(session.dataArrayPath / "deepLabCut_3d.npy")+ dlcData_nCams_nFrames_nImgPts_XYC = np.load(session.dataArrayPath / "deepLabCutData_2d.npy")++ for mm in range(dlc_fr_mar_xyz.shape[1]):+ for dim in range(dlc_fr_mar_xyz.shape[2]):+ dlc_fr_mar_xyz[:,mm,dim] = savgol_filter(dlc_fr_mar_xyz[:,mm,dim], 11, 3)++ ballTailLen = 6+ if ballTailLen > startFrame:+ startFrame = ballTailLen+1 #gotta do this, or the tail will index negative frames (without requiring annoying checks later)++ dlc_trajectories = [dlc_fr_mar_xyz[:,markerNum,:] for markerNum in range(dlc_fr_mar_xyz.shape[1])]+ figure_data['dlc_trajectories|mar|fr_dim'] = dlc_trajectories+ figure_data['dlc_fr_mar_xyz'] = dlc_fr_mar_xyz+++++ def build_3d_segment_artist_dict(data_fr_mar_xyz,+ dict_of_list_of_segment_idxs,+ segColor = 'k',+ lineWidth = 1,+ lineStyle = '-',+ markerType = None,+ marSize = 12,+ markerEdgeColor = 'k',):+ """+ Builds a dictionary of line artists for each 3D body segment.+ """+ segNames = list(dict_of_list_of_segment_idxs)++++ dict_of_artist_objects = dict()+ for segNum, segName in enumerate(segNames):++ #determine color of segment, based on class of 'segColor' input+ if isinstance(segColor, str):+ thisRGBA = segColor+ elif isinstance(segColor, np.ndarray):+ thisRGBA = segColor+ elif isinstance(segColor, dict):+ thisRGBA = segColor[segName]+ elif isinstance(segColor, list):+ try:+ thisRGBA = segColor[segNum]+ except:+ print('Not enough colors provided, using Black instead')+ thisRGBA = 'k'+ else:+ thisRGBA = 'k'++ if isinstance(segName, str):+ idxsOG = dict_of_list_of_segment_idxs[segName]+ else:+ idxsOG++ if isinstance(idxsOG, int) or isinstance(idxsOG, float):+ idxs = [idxsOG]+ elif isinstance(idxsOG, dict):+ idxs = idxsOG[0]+ else:+ idxs = idxsOG.copy()+++ dict_of_artist_objects[segName] = ax3d.plot(+ data_fr_mar_xyz[startFrame, idxs ,0],+ data_fr_mar_xyz[startFrame, idxs ,1],+ data_fr_mar_xyz[startFrame, idxs ,2],+ linestyle=lineStyle,+ linewidth=lineWidth,+ markerSize = marSize,+ marker = markerType,+ color = thisRGBA,+ markeredgecolor = markerEdgeColor,+ )[0]+ return dict_of_artist_objects++++ matplotlib_artist_objs = dict()++ if useOpenPose:+ matplotlib_artist_objs['op_body'] = build_3d_segment_artist_dict(openPose_skel_fr_mar_xyz, dict_of_openPoseSegmentIdx_dicts['op_body'], segColor = dict_of_op_skel_lineColor)+ matplotlib_artist_objs['op_rHand'] = build_3d_segment_artist_dict(openPose_skel_fr_mar_xyz, dict_of_openPoseSegmentIdx_dicts['op_rHand'], segColor=np.append(humon_red, 1), markerType='.', markerEdgeColor = humon_red, lineWidth=1, marSize = 2)+ matplotlib_artist_objs['op_lHand'] = build_3d_segment_artist_dict(openPose_skel_fr_mar_xyz, dict_of_openPoseSegmentIdx_dicts['op_lHand'], segColor=np.append(humon_blue, 1), markerType='.', markerEdgeColor = humon_blue, lineWidth=1, marSize = 2)+ matplotlib_artist_objs['op_face'] = build_3d_segment_artist_dict(openPose_skel_fr_mar_xyz, dict_of_openPoseSegmentIdx_dicts['op_face'], segColor='k', lineWidth=.5)++ if useMediaPipe:+ matplotlib_artist_objs['mp_body'] = build_3d_segment_artist_dict(mediaPipe_skel_fr_mar_xyz, dict_of_mediaPipeSegmentIdx_dicts['mp_body'], segColor = dict_of_mp_skel_lineColor)+ matplotlib_artist_objs['mp_rHand'] = build_3d_segment_artist_dict(mediaPipe_skel_fr_mar_xyz, dict_of_mediaPipeSegmentIdx_dicts['mp_rHand'], segColor=np.append(humon_red, 1), markerType='.', markerEdgeColor = humon_red, lineWidth=1, marSize = 2)+ matplotlib_artist_objs['mp_lHand'] = build_3d_segment_artist_dict(mediaPipe_skel_fr_mar_xyz, dict_of_mediaPipeSegmentIdx_dicts['mp_lHand'], segColor=np.append(humon_blue, 1), markerType='.', markerEdgeColor = humon_blue, lineWidth=1, marSize = 2)++ if useDLC:+ dict_of_dlcSegmentIdx_dicts = dict()+ dict_of_dlcSegmentIdx_dicts['pinkBall'] = {'pinkBall': [0]}+ dict_of_dlcSegmentIdx_dicts['greenBall'] = {'greenBall': [1]}+ dict_of_dlcSegmentIdx_dicts['redBall'] = {'redBall': [2]}+ dict_of_dlcSegmentIdx_dicts['wobbleBoard'] = {'wobbleboard':[3,4,5,6,3,7,5,4,7,6]}+ dict_of_dlcSegmentIdx_dicts['wobbleWheel'] = {'wobbleWheel':[8,9]}+++ ballColor = ['darkviolet', 'forestgreen', 'xkcd:goldenrod']+ matplotlib_artist_objs['pinkBall'] = build_3d_segment_artist_dict(dlc_fr_mar_xyz, dict_of_dlcSegmentIdx_dicts['pinkBall'], segColor=ballColor[0], markerType = 'o', marSize = 6, markerEdgeColor = 'indigo')+ matplotlib_artist_objs['redBall'] = build_3d_segment_artist_dict(dlc_fr_mar_xyz, dict_of_dlcSegmentIdx_dicts['redBall'], segColor=ballColor[2], markerType = 'o', marSize = 6, markerEdgeColor ='orangered')+ matplotlib_artist_objs['greenBall'] = build_3d_segment_artist_dict(dlc_fr_mar_xyz, dict_of_dlcSegmentIdx_dicts['greenBall'], segColor=ballColor[1], markerType = 'o', marSize = 6, markerEdgeColor ='darkgreen')++ matplotlib_artist_objs['wobbleBoard'] = build_3d_segment_artist_dict(dlc_fr_mar_xyz, dict_of_dlcSegmentIdx_dicts['wobbleBoard'], segColor='k', lineWidth=1)+ matplotlib_artist_objs['wobbleWheel'] = build_3d_segment_artist_dict(dlc_fr_mar_xyz, dict_of_dlcSegmentIdx_dicts['wobbleWheel'], segColor='k', lineWidth=1)++ if useOpenPose:+ op_skel_dottos = []+ for mm in range(67): #getcher dottos off my face!+ thisTraj = openPose_skel_fr_mar_xyz[:, mm, :]+ if mm==15:+ col = 'r'+ markerSize = 2+ elif mm == 16:+ col = 'b'+ markerSize = 2+ else:+ col = 'k'+ markerSize = 1++ thisDotto =ax3d.plot(thisTraj[0, 0:1], thisTraj[1, 0:1], thisTraj[2, 0:1][0],'.', markersize=markerSize, color = col)+ op_skel_dottos.append(thisDotto[0])++ matplotlib_artist_objs['op_skel_dottos'] = op_skel_dottos++ if useMediaPipe:+ matplotlib_artist_objs['mp_skel_dottos'] = [ax3d.plot(thisTraj[0, 0:1], thisTraj[1, 0:1], thisTraj[2, 0:1],'m.', markersize=1)[0] for thisTraj in mediaPipe_trajectories]++ matplotlib_artist_objs['charuco_dottos'] = [ax3d.plot(thisTraj[0, 0:1], thisTraj[1, 0:1], thisTraj[2, 0:1],'m.', markersize=1)[0] for thisTraj in charuco_trajectories]+++ if useDLC:+ matplotlib_artist_objs['dlc_dottos'] = [ax3d.plot(thisTraj[0, 0:1], thisTraj[1, 0:1], thisTraj[2, 0:1],'b.', markersize=1)[0] for thisTraj in dlc_trajectories]++ matplotlib_artist_objs['ball_tails'] = [ax3d.plot(+ dlc_trajectories[bb][startFrame-ballTailLen:startFrame, 0],+ dlc_trajectories[bb][startFrame-ballTailLen:startFrame, 1],+ dlc_trajectories[bb][startFrame-ballTailLen:startFrame, 2],+ '-', color=ballColor[bb])[0] for bb in range(3)]+++ if useOpenPose:+ numFrames = openPose_skel_fr_mar_xyz.shape[0]+ mx = np.nanmean(openPose_skel_fr_mar_xyz[int(numFrames/2),:,0])+ my = np.nanmean(openPose_skel_fr_mar_xyz[int(numFrames/2),:,1])+ mz = np.nanmean(openPose_skel_fr_mar_xyz[int(numFrames/2),:,2])++ if useMediaPipe:+ numFrames = mediaPipe_skel_fr_mar_xyz.shape[0]+ mx = np.nanmean(mediaPipe_skel_fr_mar_xyz[int(numFrames/2),:,0])+ my = np.nanmean(mediaPipe_skel_fr_mar_xyz[int(numFrames/2),:,1])+ mz = np.nanmean(mediaPipe_skel_fr_mar_xyz[int(numFrames/2),:,2])+++ if np.isnan(mx) or np.isnan(my) or np.isnan(mz):+ mx = 0+ my = 0+ mz = 0+ # groundX = np.arange(mx-100,mx+100)+ # groundZ = np.arange(my-100,my+100)+ # groundXX, groundZZ = np.meshgrid(groundX, groundZ)+ # groundYY = np.zeros_like(groundXX)+ # groundMesh = ax3d.plot_surface(groundXX, groundZZ, groundYY, color='k', alpha=.5)++ axRange = 1000#session.board.square_length * 10++ # Setting the axes properties+ ax3d.set_xlim3d([mx-axRange, mx+axRange])+ ax3d.set_ylim3d([my-axRange, my+axRange])+ ax3d.set_zlim3d([mz-axRange, mz+axRange])++ tit1 = fig.text(.5, .96, "The FreeMoCap Project", fontsize=15, horizontalalignment='center')+ tit2 = fig.text(.5, .92, "Session: {}".format(session.sessionID), fontsize=8, horizontalalignment='center')++++ ax3d.view_init(azim=azimuth, elev=elevation)++++ ###+ ###+ ### βββ βββ βββββ ββ ββ βββββββ ββ ββ ββ ββββββ βββββββ ββββββ βββββ ββ ββ βββββββ βββββββ+ ### ββββ ββββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββ ββββ ββ βββββββ βββββ βββββ ββ ββ ββ ββ ββ βββββ ββ ββ βββββββ βββ βββββ βββββββ+ ### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββ ββ ββ ββ ββ ββ βββββββ ββββ ββ ββββββ βββββββ ββββββ ββ ββ ββ ββ βββββββ βββββββ+ ###+ ###+++ syncedVidPathListAll = list(sorted(session.syncedVidPath.glob('*.mp4')))+ numVids = len(syncedVidPathListAll)++ syncedVidPathList = syncedVidPathListAll.copy()++ vidAxesList = []+ vidAristList = []+ vidCapObjList = []++ list_of_vidOpenPoseArtistdicts = []+ list_of_vidMediaPipeArtistdicts = []++ if useDLC: vidDLCArtistList = []++ def build_2d_segment_artist_dict(vidNum, data_nCams_nFrames_nImgPts_XYC, dict_of_list_of_segment_idxs, segColor = 'k', lineWidth = 1, lineStyle = '-'):+ """+ Builds a dictionary of line artists for each body 2d segment.+ """+ segNames = list(dict_of_list_of_segment_idxs)++ dict_of_artist_objects = dict()+ for segNum, segName in enumerate(segNames):++ #determine color of segment, based on class of 'segColor' input+ if isinstance(segColor, str):+ thisRGBA = segColor+ elif isinstance(segColor, np.ndarray):+ thisRGBA = segColor+ elif isinstance(segColor, dict):+ thisRGBA = segColor[segName].copy()+ thisRGBA[-1] = .75+ elif isinstance(segColor, list):+ try:+ thisRGBA = segColor[segNum]+ except:+ print('Not enough colors provided, using Black instead')+ thisRGBA = 'k'+ else:+ thisRGBA = 'k'++ xData = data_nCams_nFrames_nImgPts_XYC[vidNum, startFrame, dict_of_list_of_segment_idxs[segName],0]+ yData = data_nCams_nFrames_nImgPts_XYC[vidNum, startFrame, dict_of_list_of_segment_idxs[segName],1]++ # #make NaN's invisible (i thought they already would be but???!!!!)+ # thisRGBAall = np.tile(thisRGBA,(len(xData),1))+ # thisRGBAall[np.isnan(xData),3] = 0++ xDataMasked = np.ma.masked_where(xData, np.isnan(xData))+ yDataMasked = np.ma.masked_where(yData, np.isnan(yData))++ dict_of_artist_objects[segName] = thisVidAxis.plot(+ xDataMasked,+ yDataMasked,+ linestyle=lineStyle,+ linewidth=lineWidth,+ color = thisRGBA,+ )[0]+ return dict_of_artist_objects++ # vidImageGrid = ImageGrid(fig, [.45, 0.05, .54, .9], nrows_ncols = (2,2), axes_pad = 0.1) # https://matplotlib.org/stable/gallery/axes_grid1/simple_axesgrid.html+ if numVids < 4:+ numRows = numVids+ numCols = 1+ else:+ numRows = int(np.ceil(numVids/2))+ numCols = 2+++ vidAxGridSpec = fig.add_gridspec(numRows, numCols, left=.55, bottom=.025, right=.9, top=.9, wspace=.1, hspace=.1)+++ for thisVidNum, thisVidPath in enumerate(syncedVidPathList):+ #make subplot for figure (and set position)++ if numVids < 4:+ thisVidAxis = fig.add_subplot(vidAxGridSpec[thisVidNum])+ elif numVids %2 > 0: #if odd number videos plot the first vid across 2 spots+ if thisVidNum==0:+ thisVidAxis = fig.add_subplot(vidAxGridSpec[0,:])+ else:+ thisVidAxis = fig.add_subplot(vidAxGridSpec[thisVidNum+1])+ else:+ thisVidAxis = fig.add_subplot(vidAxGridSpec[thisVidNum])+++ thisVidAxis.set_axis_off()++ vidAxesList.append(thisVidAxis)++ #create video capture object+ thisVidCap = cv2.VideoCapture(str(thisVidPath))+++ #create artist object for each video+ success, image = thisVidCap.read()++ assert success==True, "{} - failed to load an image".format(thisVidPath.stem) #make sure we have a frame++ vidAristList.append(thisVidAxis.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB)))+ vidCapObjList.append(thisVidCap)++++ if useOpenPose:+ vidOpenPoseArtist_dict = dict()+ vidOpenPoseArtist_dict['op_body'] = build_2d_segment_artist_dict(thisVidNum, openPose_nCams_nFrames_nImgPts_XYC, dict_of_openPoseSegmentIdx_dicts['op_body'], segColor = dict_of_op_skel_lineColor)+ vidOpenPoseArtist_dict['op_rHand'] = build_2d_segment_artist_dict(thisVidNum, openPose_nCams_nFrames_nImgPts_XYC, dict_of_openPoseSegmentIdx_dicts['op_rHand'], segColor=np.append(humon_red, .75), lineWidth=.5)+ vidOpenPoseArtist_dict['op_lHand'] = build_2d_segment_artist_dict(thisVidNum, openPose_nCams_nFrames_nImgPts_XYC, dict_of_openPoseSegmentIdx_dicts['op_lHand'], segColor=np.append(humon_blue, .75), lineWidth=.5)+ vidOpenPoseArtist_dict['op_face'] = build_2d_segment_artist_dict(thisVidNum, openPose_nCams_nFrames_nImgPts_XYC, dict_of_openPoseSegmentIdx_dicts['op_face'], segColor = np.array([1.,1.,1.,1]), lineWidth=.25)+ list_of_vidOpenPoseArtistdicts.append(vidOpenPoseArtist_dict)++ if useMediaPipe:+ vidMediaPipeArtist_dict = dict()+ vidMediaPipeArtist_dict['mp_body'] = build_2d_segment_artist_dict(thisVidNum, mediaPipe_nCams_nFrames_nImgPts_XYC, dict_of_mediaPipeSegmentIdx_dicts['mp_body'], segColor = 'g')+ vidMediaPipeArtist_dict['mp_rHand'] = build_2d_segment_artist_dict(thisVidNum, mediaPipe_nCams_nFrames_nImgPts_XYC, dict_of_mediaPipeSegmentIdx_dicts['mp_rHand'], segColor=np.append(humon_red, .75), lineWidth=.5)+ vidMediaPipeArtist_dict['mp_lHand'] = build_2d_segment_artist_dict(thisVidNum, mediaPipe_nCams_nFrames_nImgPts_XYC, dict_of_mediaPipeSegmentIdx_dicts['mp_lHand'], segColor=np.append(humon_blue, .75), lineWidth=.5)+ list_of_vidMediaPipeArtistdicts.append(vidMediaPipeArtist_dict)+++ if useDLC:++ vidDLCArtistList.append(thisVidAxis.plot(+ dlcData_nCams_nFrames_nImgPts_XYC[thisVidNum,startFrame,:,0],+ dlcData_nCams_nFrames_nImgPts_XYC[thisVidNum,startFrame,:,1],+ linestyle='none',+ marker = '.',+ markersize = 1,+ color='w',+ markerfacecolor='none' ))++ ###+ ###+ ### βββ βββ βββββ ββ ββ βββββββ ββ ββ βββ ββ βββββββ ββββββ ββ ββββββ ββββββββ βββββ ββ ββ βββββββ βββββββ+ ### ββββ ββββ ββ ββ ββ ββ ββ ββ ββ ββββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββ ββββ ββ βββββββ βββββ βββββ ββ ββ ββ ββ ββ βββββ ββββββ ββ ββ ββ ββ βββββββ βββ βββββ βββββββ+ ### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββ ββ ββ ββ ββ ββ βββββββ βββββββ ββ ββ ββββ βββββββ ββ βββββββ ββββββ ββ ββ ββ ββ ββ βββββββ βββββββ+ ###+ ###+++ fps=30 #NOTE - This should be saved in the session Class some how+ time = np.arange(0, numFrames)/fps+ if useMediaPipe:+ rHandIdx = 20+ lHandIdx = 19+ rHandX = mediaPipe_skel_fr_mar_xyz[:,rHandIdx,0]/1000 #convert mm to meters+ rHandY = -mediaPipe_skel_fr_mar_xyz[:,rHandIdx,1]/1000+ lHandX = mediaPipe_skel_fr_mar_xyz[:,lHandIdx,0]/1000+ lHandY = -mediaPipe_skel_fr_mar_xyz[:,lHandIdx,1]/1000++ if useOpenPose:+ rHandIdx = 4+ lHandIdx = 7+ rHandX = openPose_skel_fr_mar_xyz[:,rHandIdx,0]/1000 #convert mm to meters+ rHandY = -openPose_skel_fr_mar_xyz[:,rHandIdx,1]/1000+ lHandX = openPose_skel_fr_mar_xyz[:,lHandIdx,0]/1000+ lHandY = -openPose_skel_fr_mar_xyz[:,lHandIdx,1]/1000+++ linePlotWidth = .45+ linePlotHeight = .14++ yTimeSeriesAx = fig.add_subplot(position=[.07, .225, linePlotWidth, linePlotHeight])+ yTimeSeriesAx.set_title('Hand Position vs Time', fontsize = 7, pad=2)++ yTimeSeriesAx.plot(time, rHandY, color=humon_red, linewidth=.75, label='Right Hand')+ yTimeSeriesAx.plot(time, lHandY, color=humon_blue, linewidth=.75, label='Left Hand')++ timeRange = 3+ ylimRange = .6++ yCurrTimeArtists = dict()+ yCurrTimeYdata = dict()+ yCurrTimeArtists['blackLine'] = yTimeSeriesAx.plot([startFrame/fps, startFrame/fps], [-ylimRange, ylimRange*2], color='k', linewidth=1)++ yCurrTimeArtists['rHandDot'] = yTimeSeriesAx.plot([startFrame/fps], [rHandY[startFrame]], markeredgecolor=humon_red, markerfacecolor = 'k',marker='o', markersize=3)+ yCurrTimeYdata['rHandDot'] = rHandY+ yCurrTimeArtists['lHandDot'] = yTimeSeriesAx.plot([startFrame/fps], [lHandY[startFrame]], markeredgecolor=humon_blue, markerfacecolor = 'k', marker='o', markersize=3)+ yCurrTimeYdata['lHandDot'] = lHandY+++ yCurrTimeArtists['frameNumber'] = yTimeSeriesAx.text(startFrame/fps+3/fps, ylimRange*.7, "Frame# " + str(startFrame), fontsize=6)++ yTimeSeriesAx.tick_params(labelsize=6, direction='in', width=.5)+ yTimeSeriesAx.tick_params( pad=2)+++ yTimeSeriesAx.set_ylabel('Vertical (m)', fontsize=7, labelpad=3)+ yTimeSeriesAx.set_ylim([-ylimRange, ylimRange])+ yTimeSeriesAx.set_xlim([(startFrame/fps)-timeRange, (startFrame/fps)+timeRange])++++ for axis in ['top','bottom','left','right']:+ yTimeSeriesAx.spines[axis].set_linewidth(0.5)++++ yTimeSeriesAx.legend(loc='upper left', fontsize=6)++++ xTimeSeriesAx = fig.add_subplot(position=[.07, 0.05, linePlotWidth, linePlotHeight])++ xTimeSeriesAx.plot(time, rHandX, color=humon_red, linewidth=.75, label='Right Hand')+ xTimeSeriesAx.plot(time, lHandX, color=humon_blue, linewidth=.75, label='Left Hand')++++ xCurrTimeArtists = dict()+ xCurrTimeYdata = dict()+ xCurrTimeArtists['blackLine'] = xTimeSeriesAx.plot([startFrame/fps, startFrame/fps], [-ylimRange, ylimRange], color='k', linewidth=1)+ xCurrTimeArtists['rHandDot'] = xTimeSeriesAx.plot([startFrame/fps], [rHandX[startFrame]], markeredgecolor=humon_red, markerfacecolor = 'k',marker='o', markersize=3)+ xCurrTimeYdata['rHandDot'] = rHandX+ xCurrTimeArtists['lHandDot'] = xTimeSeriesAx.plot([startFrame/fps], [lHandX[startFrame]], markeredgecolor=humon_blue, markerfacecolor = 'k', marker='o', markersize=3)+ xCurrTimeYdata['lHandDot'] = lHandX++++ xTimeSeriesAx.tick_params(labelsize=6, direction='in', width=.5)+ xTimeSeriesAx.tick_params( pad=2)++ xTimeSeriesAx.set_ylabel('Left/Right (m)', fontsize=7, labelpad=0)+ xTimeSeriesAx.set_xlabel('Time(sec)', fontsize=8, labelpad=0)++ xTimeSeriesAx.set_ylim([-ylimRange, ylimRange])++ xTimeSeriesAx.set_xlim([(startFrame/fps)-timeRange, (startFrame/fps)+timeRange])+++ for axis in ['top','bottom','left','right']:+ xTimeSeriesAx.spines[axis].set_linewidth(0.5)++++ # #slider+ # axControls = fig.add_subplot(2,1,2)+ # axControls.set_position([0.25, 0.01, 0.6, 0.05])+ # animSlider = Slider(+ # ax=axControls,+ # label="FrameNum",+ # valmin=10,+ # valmax=numFrames,+ # valinit=10,+ # orientation="horizontal"+ # )++ # thisVidAxis.text(-565,1440, 'Music - artist: Neon Exdeath, song: Meowmaline, album: Jewel Tones',color=humon_green, fontsize=6)+ thisVidAxis.text(0,thisVidCap.get(cv2.CAP_PROP_FRAME_HEIGHT)*1.06, 'github.com/jonmatthis/freemocap || freemocap.org',color=humon_green, fontsize=6)++ logoAx = fig.add_subplot(position=[.85, .85, .15, .15])+ logoIm = cv2.imread(r'logo\fmc-logo-black-border-white-bkgd.png') #JSM NOTE - THis doesn't work, but SOMETIMES IT DOES?!+ if logoIm is not None: logoAx.imshow(cv2.cvtColor(logoIm, cv2.COLOR_BGR2RGB))+ logoAx.axis('off')++ # if recordVid:+ # animationFramePath = session.sessionPath / "animationFrames"+ # animationFramePath.mkdir(parents=True, exist_ok=True)++ # Creating the Animation object+ line_animation = animation.FuncAnimation(fig, update_figure, range(startFrame,numFrames), fargs=(),+ interval=1, blit=False)++++ if recordVid:+ # vidSavePath = '{}_animVid.mp4'.format(str(session.sessionPath / session.sessionID))+ gifSavePath = '{}_animVid.gif'.format(str(session.sessionPath / session.sessionID)) #NOTE - saving as a gif first then converting to MP4, because saving directly to MP4 requires users to install ffmpeg (independently of the pip install)+ with console.status('Saving animation for {}'.format(session.sessionID)):+ # Writer = animation.writers['ffmpeg']+ Writer = animation.writers['pillow']+ writer = Writer(fps=fps, metadata=dict(artist='FreeMoCap'))#, bitrate=1800)+ line_animation.save(gifSavePath, writer = writer)+ # Writer = animation.FFMpegWriter(fps=30, metadata=dict(artist='FreeMoCap', comment=session.sessionID), bitrate=1800)+ # vidSavePath = '{}_outVid.mp4'.format(str(session.sessionPath / session.sessionID))+ # Writer.saving(fig = fig, outfile=vidSavePath, dpi=150)++ gif_filepath = mp.VideoFileClip(gifSavePath)+ gif_filepath.write_videofile(gifSavePath.replace('.gif','.mp4'))+ os.remove(gifSavePath)+++ try:+ if showAnimation:+ with console.status('Playing Skeleton animation! Close the `matplotlib` window to continue...'):+ plt.pause(0.1)+ plt.draw()+ except:+ pass+++++++ ###+ ###+ ### βββββββ ββββββ ββββββ βββ βββ βββββ ββββββββ ββββββ ββββββ βββββββ βββ ββ ββββββ ββββββ βββββββ βββββββ βββββββ ββ ββ βββββββ ββ+ ### ββ ββ ββ ββ ββ ββββ ββββ ββ ββ ββ ββ ββ ββ ββ ββ ββββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### βββββ ββ ββ ββββββ ββ ββββ ββ βββββββ ββ ββ ββ ββββββ βββββ ββ ββ ββ ββββββ ββ ββ βββββββ βββββ βββββββ βββββ βββββ ββ+ ### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+ ### ββ ββββββ ββ ββ ββ ββ ββ ββ ββ ββββββ ββ βββββββ ββ ββββ ββ ββββββ βββββββ βββββββ βββββββ ββ ββ βββββββ βββββββ+ ###+ ###+def formatOpenPoseStickIndices():+ """+ generate dictionary of arrays, each containing the 'connect-the-dots' order to draw a given body segment++ returns:+ openPoseBodySegmentIds= a dictionary of arrays containing indices of individual body segments (Note, a lot of markerless mocap comp sci types like to say 'pose' instead of 'body'. They also use 'pose' to refer to camera 6 DoF position sometimes. Comp sci is frustrating like that lol)+ openPoseHandIds = a dictionary of arrays containing indices of individual hand segments, along with offset to know where to start in the 'skel_fr_mar_xyz.shape[1]' part of the array+ dict_of_op_skel_lineColor = a dictionary of arrays, each containing the color (RGBA) to use for a given body segment+ """+ dict_of_openPoseSegmentIdx_dicts = dict()++ #make body dictionary+ openPoseBodySegmentIds = dict()+ openPoseBodySegmentIds['op_head'] = [17, 15, 0, 1,0, 16, 18, ]+ openPoseBodySegmentIds['op_spine'] = [1,8,5,1, 2, 12, 8, 9, 5, 1, 2, 8]+ openPoseBodySegmentIds['op_rArm'] = [1, 2, 3, 4, ]+ openPoseBodySegmentIds['op_lArm'] = [1, 5, 6, 7, ]+ openPoseBodySegmentIds['op_rLeg'] = [8, 9, 10, 11, 22, 23, 11, 24, ]+ openPoseBodySegmentIds['op_lLeg'] = [8,12, 13, 14, 19, 20, 14, 21,]+ dict_of_openPoseSegmentIdx_dicts['op_body'] = openPoseBodySegmentIds+++ #make colors dictionary+ openPoseBodyColor = np.array([50, 89, 0])/255+ openPoseRightColor = np.array([230, 50, 0])/255+ openPoseLeftColor = np.array([0, 50, 230])/255++ dict_of_op_skel_lineColor = dict()+ dict_of_op_skel_lineColor['op_head'] = np.append(openPoseBodyColor, 0.5)+ dict_of_op_skel_lineColor['op_spine'] = np.append(openPoseBodyColor, 1)+ dict_of_op_skel_lineColor['op_rArm'] = np.append(openPoseRightColor, 1)+ dict_of_op_skel_lineColor['op_lArm'] = np.append(openPoseLeftColor, 1)+ dict_of_op_skel_lineColor['op_rLeg'] = np.append(openPoseRightColor, 1)+ dict_of_op_skel_lineColor['op_lLeg'] = np.append(openPoseLeftColor, 1)+++ # Make some handy maps ;D+ openPoseHandIds = dict()+ rHandIDstart = 25+ lHandIDstart = rHandIDstart + 21++ openPoseHandIds['op_thumb'] = np.array([0, 1, 2, 3, 4, ])+ openPoseHandIds['op_index'] = np.array([0, 5, 6, 7, 8, ])+ openPoseHandIds['op_bird']= np.array([0, 9, 10, 11, 12, ])+ openPoseHandIds['op_ring']= np.array([0, 13, 14, 15, 16, ])+ openPoseHandIds['op_pinky'] = np.array([0, 17, 18, 19, 20, ])+++ rHand_dict = copy.deepcopy(openPoseHandIds.copy()) #copy.deepcopy() is necessary to make sure the dicts are independent of each other+ lHand_dict = copy.deepcopy(rHand_dict)++ for key in rHand_dict:+ rHand_dict[key] += rHandIDstart+ lHand_dict[key] += lHandIDstart++ dict_of_openPoseSegmentIdx_dicts['op_rHand'] = rHand_dict+ dict_of_openPoseSegmentIdx_dicts['op_lHand'] = lHand_dict+++ #how to face --> :D <--+ openPoseFaceIDs = dict()+ faceIDStart = 67+ #define face parts+ openPoseFaceIDs['op_jaw'] = np.arange(0,16) + faceIDStart+ openPoseFaceIDs['op_rBrow'] = np.arange(17,21) + faceIDStart+ openPoseFaceIDs['op_lBrow'] = np.arange(22,26) + faceIDStart+ openPoseFaceIDs['op_noseRidge'] = np.arange(27,30) + faceIDStart+ openPoseFaceIDs['op_noseBot'] = np.arange(31,35) + faceIDStart+ openPoseFaceIDs['op_rEye'] = np.concatenate((np.arange(36,41), [36])) + faceIDStart+ openPoseFaceIDs['op_lEye'] = np.concatenate((np.arange(42,47), [42])) + faceIDStart+ openPoseFaceIDs['op_upperLip'] = np.concatenate((np.arange(48,54), np.flip(np.arange(60, 64)), [48])) + faceIDStart+ openPoseFaceIDs['op_lowerLip'] = np.concatenate(([60], np.arange(64,67), np.arange(54, 59), [48], [60])) + faceIDStart+ openPoseFaceIDs['op_rPupil'] = np.array([68]) + faceIDStart+ openPoseFaceIDs['op_lPupil'] = np.array([69]) + faceIDStart #nice++ dict_of_openPoseSegmentIdx_dicts['op_face'] = openPoseFaceIDs++ return dict_of_openPoseSegmentIdx_dicts, dict_of_op_skel_lineColor++++###+### βββββββ ββββββ ββββββ βββ βββ βββββ ββββββββ βββ βββ βββββββ ββββββ ββ βββββ ββββββ ββ ββββββ βββββββ βββββββ ββ ββ βββββββ ββ+### ββ ββ ββ ββ ββ ββββ ββββ ββ ββ ββ ββββ ββββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+### βββββ ββ ββ ββββββ ββ ββββ ββ βββββββ ββ ββ ββββ ββ βββββ ββ ββ ββ βββββββ ββββββ ββ ββββββ βββββ βββββββ βββββ βββββ ββ+### ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ ββ+### ββ ββββββ ββ ββ ββ ββ ββ ββ ββ ββ ββ βββββββ ββββββ ββ ββ ββ ββ ββ ββ βββββββ βββββββ ββ ββ βββββββ βββββββ+###+###+def formatMediaPipeStickIndices():+ """+ generate dictionary of arrays, each containing the 'connect-the-dots' order to draw a given body segment++ returns:+ mediaPipeBodySegmentIds= a dictionary of arrays containing indices of individual body segments (Note, a lot of markerless mocap comp sci types like to say 'pose' instead of 'body'. They also use 'pose' to refer to camera 6 DoF position sometimes. Comp sci is frustrating like that lol)+ mediaPipeHandIds = a dictionary of arrays containing indices of individual hand segments, along with offset to know where to start in the 'skel_fr_mar_xyz.shape[1]' part of the array+ dict_of_mp_skel_lineColor = a dictionary of arrays, each containing the color (RGBA) to use for a given body segment+ """+ dict_of_mediaPipeSegmentIdx_dicts = dict()++ #make body dictionary+ mediaPipeBodySegmentIds = dict()+ mediaPipeBodySegmentIds['mp_head'] = [8, 6, 5, 4, 0, 10, 9, 0, 1, 2, 3, 7 ]+ mediaPipeBodySegmentIds['mp_tors'] = [12, 11, 24, 23, 12]+ mediaPipeBodySegmentIds['mp_rArm'] = [12, 14, 16, 18, 20, 16, 22 ]+ mediaPipeBodySegmentIds['mp_lArm'] = [11, 13, 15, 17, 19, 15, 21]+ mediaPipeBodySegmentIds['mp_rLeg'] = [24, 26, 28, 30, 32, 28, ]+ mediaPipeBodySegmentIds['mp_lLeg'] = [23, 25, 27, 29, 31, 27, ]+ dict_of_mediaPipeSegmentIdx_dicts['mp_body'] = mediaPipeBodySegmentIds++ #make colors dictionary+ mediaPipeBodyColor = np.array([125,106,0])/255+ mediaPipeRightColor = np.array([230, 0, 169])/255+ mediaPipeLeftColor = np.array([0, 176, 176])/255+ dict_of_mp_skel_lineColor = dict()++ dict_of_mp_skel_lineColor['mp_head'] = np.append(mediaPipeBodyColor, .5)+ dict_of_mp_skel_lineColor['mp_tors'] = np.append(mediaPipeBodyColor, 1)+ dict_of_mp_skel_lineColor['mp_rArm'] = np.append(mediaPipeRightColor, 1)+ dict_of_mp_skel_lineColor['mp_lArm'] = np.append(mediaPipeLeftColor, 1)+ dict_of_mp_skel_lineColor['mp_rLeg'] = np.append(mediaPipeRightColor, 1)+ dict_of_mp_skel_lineColor['mp_lLeg'] = np.append(mediaPipeLeftColor, 1)+++ # Make some handy maps ;D+ mediaPipeHandIds = dict()+ rHandIDstart = 33+ lHandIDstart = rHandIDstart + 21++ mediaPipeHandIds['mp_thumb'] = np.array([0, 1, 2, 3, 4, ])+ mediaPipeHandIds['mp_index'] = np.array([0, 5, 6, 7, 8, ])+ mediaPipeHandIds['mp_bird']= np.array([0, 9, 10, 11, 12, ])+ mediaPipeHandIds['mp_ring']= np.array([0, 13, 14, 15, 16, ])+ mediaPipeHandIds['mp_pinky'] = np.array([0, 17, 18, 19, 20, ])+++ rHand_dict = copy.deepcopy(mediaPipeHandIds.copy()) #copy.deepcopy() is necessary to make sure the dicts are independent of each other+ lHand_dict = copy.deepcopy(rHand_dict)++ for key in rHand_dict:+ rHand_dict[key] += rHandIDstart+ lHand_dict[key] += lHandIDstart++ dict_of_mediaPipeSegmentIdx_dicts['mp_rHand'] = rHand_dict+ dict_of_mediaPipeSegmentIdx_dicts['mp_lHand'] = lHand_dict+++ # #how to face --> :D <--+ # mediaPipeFaceIDs = dict()+ # faceIDStart = 67+ # #define face parts+ # mediaPipeFaceIDs['mp_jaw'] = np.arange(0,16) + faceIDStart+ # mediaPipeFaceIDs['mp_rBrow'] = np.arange(17,21) + faceIDStart+ # mediaPipeFaceIDs['mp_lBrow'] = np.arange(22,26) + faceIDStart+ # mediaPipeFaceIDs['mp_noseRidge'] = np.arange(27,30) + faceIDStart+ # mediaPipeFaceIDs['mp_noseBot'] = np.arange(31,35) + faceIDStart+ # mediaPipeFaceIDs['mp_rEye'] = np.concatenate((np.arange(36,41), [36])) + faceIDStart+ # mediaPipeFaceIDs['mp_lEye'] = np.concatenate((np.arange(42,47), [42])) + faceIDStart+ # mediaPipeFaceIDs['mp_upperLip'] = np.concatenate((np.arange(48,54), np.flip(np.arange(60, 64)), [48])) + faceIDStart+ # mediaPipeFaceIDs['mp_lowerLip'] = np.concatenate(([60], np.arange(64,67), np.arange(54, 59), [48], [60])) + faceIDStart+ # mediaPipeFaceIDs['mp_rPupil'] = np.array([68]) + faceIDStart+ # mediaPipeFaceIDs['mp_lPupil'] = np.array([69]) + faceIDStart #nice++ # dict_of_mediaPipeSegmentIdx_dicts['mp_face'] = mediaPipeFaceIDs++ return dict_of_mediaPipeSegmentIdx_dicts, dict_of_mp_skel_lineColor+++if __name__ == '__main__':+ MatplotlibLaserSkeleton()diff --git a/jon_scratch/matplotlib_laser_skeleton/__init__.py b/jon_scratch/matplotlib_laser_skeleton/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/jon_scratch/matplotlib_laser_skeleton/matplotlib_laser_skeleton.py b/jon_scratch/matplotlib_laser_skeleton/matplotlib_laser_skeleton.pynew file mode 100644index 00000000..715328c8--- /dev/null+++ b/jon_scratch/matplotlib_laser_skeleton/matplotlib_laser_skeleton.py@@ -0,0 +1,68 @@+from pathlib import Path++import keyboard+import matplotlib.pyplot as plt+import matplotlib+import numpy as np+from mediapipe.python.solutions import holistic as mp_holistic++matplotlib.use('qtagg')++mediapipe_body_pose_connections = [this_connection for this_connection in mp_holistic.POSE_CONNECTIONS]+++class MatplotlibLaserSkeleton:+ def __init__(self, file_path):+ self.fig = None+ self.ax3d = None+ self.mediapipe_all_fr_mar_xyz = None+ self.file_path = Path(file_path)+ self.load_mediapipe_data()++ def load_mediapipe_data(self):+ mediapipe_data_path = self.file_path / 'DataArrays' / 'mediaPipeSkel_3d.npy'+ self.mediapipe_all_fr_mar_xyz = np.load(str(mediapipe_data_path))++ def play_laser_skeleton_animation(self, start_frame: int = 0, end_frame: int = None):+ if end_frame is None:+ end_frame = self.mediapipe_all_fr_mar_xyz.shape[0]++ self.create_figure()++ for frame_number in range(start_frame, end_frame):+ plt.cla()+ self.plot_mediapipe_skeleton(frame_number)+ plt.show()+ plt.pause(0.001)+ if keyboard.is_pressed('esc'):+ print('You Pressed Escape!')+ plt.close('all')+ break # finishing the loop++ def create_figure(self):+ plt.ion()+ self.fig = plt.figure()+ self.ax3d = self.fig.add_subplot(111, projection='3d')+ self.set_axis_limits()++ def set_axis_limits(self):+ median_position = np.nanmedian(self.mediapipe_all_fr_mar_xyz, axis=0)+ median_position = np.nanmedian(median_position, axis=0)+ stddev_position = np.nanstd(median_position, axis=0)+ axis_range = 1000+ self.ax3d.set_xlim(median_position[0] - axis_range, median_position[0] + axis_range)+ self.ax3d.set_ylim(median_position[1] - axis_range, median_position[1] + axis_range)+ self.ax3d.set_zlim(median_position[2] - axis_range, median_position[2] + axis_range)++ def plot_mediapipe_skeleton(self, frame_number: int):+ print(f'Plotting frame {frame_number}')+ self.ax3d.plot(self.mediapipe_all_fr_mar_xyz[frame_number, :, 0],+ self.mediapipe_all_fr_mar_xyz[frame_number, :, 1],+ self.mediapipe_all_fr_mar_xyz[frame_number, :, 2],+ color='k', marker='o', linestyle='none', markersize=1)+++if __name__ == '__main__':+ session_path = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data/sesh_2022-02-15_11_54_28_pupil_maybe')+ matplotlib_laser_skeleton = MatplotlibLaserSkeleton(session_path)+ matplotlib_laser_skeleton.play_laser_skeleton_animation(start_frame=1000, end_frame=1200)diff --git a/jon_scratch/pupil_calibration_pipeline/__init__.py b/jon_scratch/pupil_calibration_pipeline/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/jon_scratch/pupil_calibration_pipeline/data_classes/__init__.py b/jon_scratch/pupil_calibration_pipeline/data_classes/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/jon_scratch/pupil_calibration_pipeline/data_classes/freemocap_session_data_class.py b/jon_scratch/pupil_calibration_pipeline/data_classes/freemocap_session_data_class.pynew file mode 100644index 00000000..703763e6--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/data_classes/freemocap_session_data_class.py@@ -0,0 +1,23 @@+from dataclasses import dataclass+from typing import List++import numpy as np++from jon_scratch.pupil_calibration_pipeline.data_classes.pupil_dataclass_and_handler import PupilLabsDataClass+from jon_scratch.pupil_calibration_pipeline.data_classes.rotation_data_class import RotationDataClass+++@dataclass+class FreemocapSessionDataClass:+ session_id: str = None+ timestamps: np.ndarray = None+ mediapipe_skel_fr_mar_dim: np.ndarray = None+ right_eye_pupil_labs_data: PupilLabsDataClass = None+ left_eye_pupil_labs_data: PupilLabsDataClass = None+ head_rotation_data: RotationDataClass = None+ right_eye_socket_rotation_data: RotationDataClass = None+ left_eye_socket_rotation_data: RotationDataClass = None+ right_gaze_vector_endpoint_fr_xyz: np.ndarray = None+ left_gaze_vector_endpoint_fr_xyz: np.ndarray = None++diff --git a/jon_scratch/pupil_calibration_pipeline/data_classes/pupil_dataclass_and_handler.py b/jon_scratch/pupil_calibration_pipeline/data_classes/pupil_dataclass_and_handler.pynew file mode 100644index 00000000..43c0c35a--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/data_classes/pupil_dataclass_and_handler.py@@ -0,0 +1,90 @@+from dataclasses import dataclass+from pathlib import Path+from typing import Union++import numpy as np+import pandas as pd+++@dataclass+class PupilLabsDataClass:+ def __init__(self,+ timestamps=None,+ theta=None,+ phi=None,+ pupil_center_normal_x=None,+ pupil_center_normal_y=None,+ pupil_center_normal_z=None,+ eye_d=None,+ method=None):+ self.timestamps = timestamps+ self.theta = theta+ self.phi = phi+ self.pupil_center_normal_x = pupil_center_normal_x+ self.pupil_center_normal_y = pupil_center_normal_y+ self.pupil_center_normal_z = pupil_center_normal_z+ self.eye_d = eye_d+ self.method = method+++class PupilDataHandler:+ """+ Class for handling data from Pupil Labs eye tracker+ """++ def __init__(self, pupil_dataframe: pd.DataFrame = None):+ if pupil_dataframe is not None:+ self.load_from_dataframe(pupil_dataframe)++ def load_from_file(self, file_path: Union[str, Path]):+ pupil_dataframe = pd.read_csv(file_path)+ self.load_from_dataframe(pupil_dataframe)++ def load_from_dataframe(self, pupil_dataframe: pd.DataFrame):+ pupil_dataframe = pupil_dataframe++ # resituate pupil data so that the pupil normal direction is situated on the origin (instead of the arbitrary eye-camera-based eyeball sphere center)+ # ... I think... I need to check this. Their docs are a bit unclear about the origin of the pupil normal direction.+ # pupil_center_normal_x = np.array(pupil_dataframe['sphere_center_x']) - np.array(pupil_dataframe['circle_3d_normal_x'])+ # pupil_center_normal_y = np.array(pupil_dataframe['sphere_center_y']) - np.array(pupil_dataframe['circle_3d_normal_y'])+ # pupil_center_normal_z = np.array(pupil_dataframe['sphere_center_z']) - np.array(pupil_dataframe['circle_3d_normal_z'])+ pupil_center_normal_x = np.array(pupil_dataframe['circle_3d_normal_x'])+ pupil_center_normal_y = np.array(pupil_dataframe['circle_3d_normal_y'])+ pupil_center_normal_z = np.array(pupil_dataframe['circle_3d_normal_z'])++ self.pupil_data = PupilLabsDataClass(timestamps=np.array(pupil_dataframe['pupil_timestamp']),+ theta=np.array(pupil_dataframe['theta']),+ phi=np.array(pupil_dataframe['phi']),+ pupil_center_normal_x=pupil_center_normal_x,+ pupil_center_normal_y=pupil_center_normal_y,+ pupil_center_normal_z=pupil_center_normal_z,+ eye_d=pupil_dataframe['eye_id'],+ method=pupil_dataframe['method'])++ def convert_to_unix_timestamps(self, pupil_recording_info_json: dict):+ self.pupil_data.timestamps = self.pupil_data.timestamps - pupil_recording_info_json['start_time_synced_s'] + \+ pupil_recording_info_json[+ 'start_time_system_s']++ def get_eye_data(self, eye_str: str):+ """+ get data for the right eye++ pull out data according to `eye_d` (right_eye == 0, left_eye==1) and tracking method (because pupil interleaves 2d and 3d data)+ """+ if eye_str == 'right':+ eye_d = 0++ if eye_str == 'left':+ eye_d = 1++ this_eye_logical_indicies = np.logical_and(self.pupil_data.eye_d == eye_d,+ self.pupil_data.method != '2d c++')+ return PupilLabsDataClass(+ timestamps=self.pupil_data.timestamps[this_eye_logical_indicies],+ theta=self.pupil_data.theta[this_eye_logical_indicies],+ phi=self.pupil_data.phi[this_eye_logical_indicies],+ pupil_center_normal_x=self.pupil_data.pupil_center_normal_x[this_eye_logical_indicies],+ pupil_center_normal_y=self.pupil_data.pupil_center_normal_y[this_eye_logical_indicies],+ pupil_center_normal_z=self.pupil_data.pupil_center_normal_z[this_eye_logical_indicies],+ )diff --git a/jon_scratch/pupil_calibration_pipeline/data_classes/rotation_data_class.py b/jon_scratch/pupil_calibration_pipeline/data_classes/rotation_data_class.pynew file mode 100644index 00000000..421f3019--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/data_classes/rotation_data_class.py@@ -0,0 +1,13 @@+from dataclasses import dataclass+from typing import List++import numpy as np+++@dataclass+class RotationDataClass:+ rotation_matricies: List[np.ndarray]+ local_origin_fr_xyz: np.ndarray = np.array([0, 0, 0])+ x_hat_norm_fr_xyz: np.ndarray = np.array([1, 0, 0])+ y_hat_norm_fr_xyz: np.ndarray = np.array([0, 1, 0])+ z_hat_norm_fr_xyz: np.ndarray = np.array([0, 0, 1])\ No newline at end of filediff --git a/jon_scratch/pupil_calibration_pipeline/load_pupil_data_blender_script.py b/jon_scratch/pupil_calibration_pipeline/load_pupil_data_blender_script.pynew file mode 100644index 00000000..777f6310--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/load_pupil_data_blender_script.py@@ -0,0 +1,83 @@+import bpy+import numpy as np+import sys+from pathlib import Path++print("loading gaze data as empties")++def load_gaze_data(gaze_fr_xyz:np.ndarray, eye_empty_name:str):++ this_eyeball_empty = bpy.data.objects[eye_empty_name]++ bpy.ops.object.empty_add(type='SPHERE')+ this_gaze_empty = bpy.context.active_object+ print(f'loading {eye_empty_name} gaze data...')+ for frame_num in range(r_gaze_eye_in_head_xyz.shape[0]):++ this_gaze_empty.location = (gaze_fr_xyz[frame_num,0],+ gaze_fr_xyz[frame_num,1],+ gaze_fr_xyz[frame_num,2])+ this_gaze_empty.scale = (0.01, 0.01, 0.01)++ this_gaze_empty.name = eye_empty_name + '_gaze_empty'++ this_gaze_empty.keyframe_insert(data_path='location',frame=frame_num)++ bpy.ops.object.armature_add(enter_editmode=False, align='WORLD', location=(0, 0, 0), scale=(1, 1, 1))+ this_gaze_armature = bpy.context.active_object+ this_gaze_armature.name = eye_empty_name + '_gaze_armature'++ this_gaze_bone_name = eye_empty_name + '_gaze_bone'+ this_gaze_bone = this_gaze_armature.data.bones[0]+ this_gaze_bone.name = this_gaze_bone_name++ # #create gaze bone+ # this_gaze_edit_bone= this_gaze_armature.data.edit_bones[0] # note this is using EDIT_bones, not regular bones b/c blender is a wild wacky place+ # this_gaze_edit_bone.tail = this_eyeball_empty.location+ # this_gaze_edit_bone.head = this_gaze_empty.location++ #track gaze bone to gaze empty+ this_gaze_pose_bone = this_gaze_armature.pose.bones[0]++ this_gaze_constraint = this_gaze_pose_bone.constraints.new('COPY_LOCATION')+ this_gaze_constraint.target = this_eyeball_empty+ this_gaze_constraint = this_gaze_pose_bone.constraints.new('DAMPED_TRACK')+ this_gaze_constraint.target = this_gaze_empty++try:+ ##% Session path+ # #get session path as command line argument+ argv = sys.argv+ argv = argv[argv.index("--") + 1:]+ session_path = argv[0]++ if len(argv)>1:+ good_clean_frame_number = int(argv[1])+ else:+ good_clean_frame_number = 0+except:+ print('we appear to be running from the Blender Scripting tab! Manually enter your `session_path` at line 23')+ session_path = Path(r'C:\Users\jonma\Dropbox\FreeMoCapProject\FreeMocap_Data\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0')+++data_arrays_path = session_path / 'DataArrays'++path_to_right_eye_gaze_xyz_npy = data_arrays_path / 'right_eye_gaze_fr_xyz.npy'+path_to_left_eye_gaze_xyz_npy = data_arrays_path / 'left_eye_gaze_fr_xyz.npy'+++# right eye+print(f'loading right_eye_data...')+r_gaze_eye_in_head_xyz = np.load(path_to_right_eye_gaze_xyz_npy)+r_gaze_eye_in_head_xyz *= .001 #convert to meters+load_gaze_data(r_gaze_eye_in_head_xyz, 'right_eye')++# right eye+print(f'loading left_eye_data...')+l_gaze_eye_in_head_xyz = np.load(path_to_left_eye_gaze_xyz_npy)+l_gaze_eye_in_head_xyz *= .001 #convert to meters+load_gaze_data(l_gaze_eye_in_head_xyz, 'left_eye')++++print('done!')\ No newline at end of filediff --git a/jon_scratch/pupil_calibration_pipeline/pupil_freemocap_calibration_pipeline_orchestrator.py b/jon_scratch/pupil_calibration_pipeline/pupil_freemocap_calibration_pipeline_orchestrator.pynew file mode 100644index 00000000..908c9439--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/pupil_freemocap_calibration_pipeline_orchestrator.py@@ -0,0 +1,157 @@+import copy+import logging+from pathlib import Path+from typing import Union+import numpy as np+from rich import inspect+from jon_scratch.pupil_calibration_pipeline.data_classes.freemocap_session_data_class import FreemocapSessionDataClass+from jon_scratch.pupil_calibration_pipeline.rotation_matrix_calculator import RotationMatrixCalculator+from jon_scratch.pupil_calibration_pipeline.pupil_freemocap_synchronizer import PupilFreemocapSynchronizer+from jon_scratch.pupil_calibration_pipeline.qt_gl_laser_skeleton_visualizer import QtGlLaserSkeletonVisualizer+from jon_scratch.pupil_calibration_pipeline.session_data_loader import SessionDataLoader+from jon_scratch.pupil_calibration_pipeline.vor_calibrator import VorCalibrator++logger = logging.getLogger(__name__)+logger.setLevel(logging.INFO)+++class PupilFreemocapCalibrationPipelineOrchestrator:+ session_data_loader: SessionDataLoader = None+ raw_session_data = FreemocapSessionDataClass()+ vor_frame_start: int = None,+ vor_frame_end: int = None++ def __init__(self, session_path: Union[Path, str], debug: bool = False, vor_frame_start: int = None,+ vor_frame_end: int = None):+ logger.info(f'initializing pupil-freemocap calibration pipeline orchestrator for session: {session_path}')+ self.session_path = session_path+ self.session_id = self.session_path.stem+ self.raw_session_data.session_id = self.session_id+ self.session_data_loader = SessionDataLoader(self.session_path)+ self.debug = debug++ if vor_frame_start is not None:+ self.vor_frame_start = vor_frame_start+ if vor_frame_end is not None:+ self.vor_frame_end = vor_frame_end++ def run(self):+ logger.info(f'loading session data from {self.session_data_loader.session_path}')++ ####+ # load raw freemocap data+ ####+ self.raw_session_data.timestamps = self.session_data_loader.load_freemocap_unix_timestamps()+ logger.info(+ f'self.raw_session_data.freemocap_timestamps.shape: {self.raw_session_data.timestamps.shape}')++ self.raw_session_data.mediapipe_skel_fr_mar_dim = self.session_data_loader.load_mediapipe_data()+ logger.info(+ f'self.raw_session_data.mediapipe_skel_fr_mar_dim.shape: {self.raw_session_data.mediapipe_skel_fr_mar_dim.shape}')++ ####+ # load pupil data+ ####+ pupil_data_handler = self.session_data_loader.load_pupil_data()+ self.raw_session_data.right_eye_pupil_labs_data = pupil_data_handler.get_eye_data('right')+ self.raw_session_data.left_eye_pupil_labs_data = pupil_data_handler.get_eye_data('left')++ ####+ # Synchronize pupil data with freemocap data - results in synchronized_session_data (each stream has exactly the same number of frames)+ ####+ synchronized_session_data = PupilFreemocapSynchronizer(self.raw_session_data).synchronize(+ vor_frame_start=self.vor_frame_start,+ vor_frame_end=self.vor_frame_end,+ debug=False, )++ logger.info(+ 'synchronization complete - I should add a test to make sure everything has the same number of frames')++ ####+ # Calculate Head Rotation matrix for each frame (gaze data will be rotated by head_rot, then calibrated_offset_rot)+ ####+ rotation_matrix_calculator = RotationMatrixCalculator(synchronized_session_data.mediapipe_skel_fr_mar_dim)++ synchronized_session_data.head_rotation_data = rotation_matrix_calculator.calculate_head_rotation_matricies(+ debug=False)++ synchronized_session_data.right_eye_socket_rotation_data = rotation_matrix_calculator.calculate_eye_rotation_matricies(+ eye='right',+ debug=False,+ )++ synchronized_session_data.left_eye_socket_rotation_data = rotation_matrix_calculator.calculate_eye_rotation_matricies(+ 'left',+ debug=False,+ )++ logger.info(+ f'len(synchronized_session_data.head_rotation_data.head_rotation_matricies): {len(synchronized_session_data.head_rotation_data.rotation_matricies)}')++ ####+ # Perform Vestibular-Ocular-Reflex based calibration (see methods from (Matthis et al, 2018 and 2022) for deetos)+ ####+ vor_calibrator = VorCalibrator(synchronized_session_data.mediapipe_skel_fr_mar_dim.copy(),+ vor_start_frame=self.vor_frame_start,+ vor_end_frame=self.vor_frame_end,+ debug=False)+ right_index_fingertip_idx = 41 # pretty sure this is right?+ fixation_point_fr_xyz = synchronized_session_data.mediapipe_skel_fr_mar_dim[+ self.vor_frame_start:self.vor_frame_end,+ right_index_fingertip_idx,+ :]+ # right eye+ synchronized_session_data.right_gaze_vector_endpoint_fr_xyz = vor_calibrator.calibrate(+ copy.deepcopy(synchronized_session_data.right_eye_pupil_labs_data),+ copy.deepcopy(synchronized_session_data.right_eye_socket_rotation_data),+ copy.deepcopy(synchronized_session_data.head_rotation_data),+ fixation_point_fr_xyz)+ # left eye+ synchronized_session_data.left_gaze_vector_endpoint_fr_xyz = vor_calibrator.calibrate(+ synchronized_session_data.left_eye_pupil_labs_data,+ synchronized_session_data.left_eye_socket_rotation_data,+ copy.deepcopy(synchronized_session_data.head_rotation_data),+ fixation_point_fr_xyz)++ # save that data+ self.save_gaze_data(synchronized_session_data)+ ####+ # Play laser skeleton animation (as both a cool thing and a debug tool)+ ####++ qt_gl_laser_skeleton = QtGlLaserSkeletonVisualizer(session_data=synchronized_session_data,+ move_data_to_origin=True, )+ # start_frame=self.vor_frame_start,+ # end_frame=self.vor_frame_end)+ qt_gl_laser_skeleton.start_animation()++ def save_gaze_data(self, synchronized_session_data):+ data_save_path = self.session_path / 'DataArrays'++ # right eye+ save_right_eye_data_path = data_save_path / 'right_eye_gaze_fr_xyz.npy'+ np.save(save_right_eye_data_path, synchronized_session_data.right_gaze_vector_endpoint_fr_xyz)++ # left eye+ save_left_eye_data_path = data_save_path / 'left_eye_gaze_fr_xyz.npy'+ np.save(save_left_eye_data_path, synchronized_session_data.left_gaze_vector_endpoint_fr_xyz)+++if __name__ == '__main__':+ # session_id = 'sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0'+ # vor_frame_start_in = 614+ # vor_frame_end_in = 1073++ session_id = 'sesh_2022-02-15_11_54_28_pupil_maybe'+ vor_frame_start_in = 1200+ vor_frame_end_in = 1500++ data_path = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data/')+ this_session_path = data_path / session_id+++ pupil_freemocap_calibration_pipeline_orchestrator = PupilFreemocapCalibrationPipelineOrchestrator(this_session_path,+ vor_frame_start=vor_frame_start_in,+ vor_frame_end=vor_frame_end_in)+ pupil_freemocap_calibration_pipeline_orchestrator.run()+ print('done :D ')diff --git a/jon_scratch/pupil_calibration_pipeline/pupil_freemocap_synchronizer.py b/jon_scratch/pupil_calibration_pipeline/pupil_freemocap_synchronizer.pynew file mode 100644index 00000000..42f19211--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/pupil_freemocap_synchronizer.py@@ -0,0 +1,288 @@+import logging++import numpy as np+from matplotlib import pyplot as plt+import matplotlib++from jon_scratch.pupil_calibration_pipeline.data_classes.freemocap_session_data_class import FreemocapSessionDataClass+from jon_scratch.pupil_calibration_pipeline.data_classes.pupil_dataclass_and_handler import PupilLabsDataClass++matplotlib.use('qt5agg')+logger = logging.getLogger(__name__)+++class PupilFreemocapSynchronizer:+ """+ synchronize pupil and freemocap timestamps, return synchronized session data (exactly the same number of synchronized frames in each data stream+ """++ def __init__(self, raw_session_data: FreemocapSessionDataClass):+ self.raw_session_data = raw_session_data+ self.synchronized_session_data: FreemocapSessionDataClass = None++ def synchronize(self, debug: bool = False, vor_frame_start: int = None, vor_frame_end: int = None):+ """+ align freemocap and pupil timestamps and clip the starts and ends of the various data traces so that everything covers the same timespacn+ """+ # find start and end frames shared by all datastreams+ freemocap_timestamps = self.raw_session_data.timestamps+ right_eye_timestamps = self.raw_session_data.right_eye_pupil_labs_data.timestamps+ left_eye_timestamps = self.raw_session_data.left_eye_pupil_labs_data.timestamps++ start_time_unix = np.max((freemocap_timestamps[0], right_eye_timestamps[0], left_eye_timestamps[0]))+ end_time_unix = np.min((freemocap_timestamps[-1], right_eye_timestamps[-1], left_eye_timestamps[-1]))++ # freemocap+ if any(freemocap_timestamps >= start_time_unix):+ freemocap_start_frame = np.where(freemocap_timestamps >= start_time_unix)[0][0]+ else:+ freemocap_start_frame = 0++ if any(freemocap_timestamps <= end_time_unix):+ freemocap_end_frame = np.where(freemocap_timestamps <= end_time_unix)[0][-1]+ else:+ freemocap_end_frame = len(freemocap_timestamps)++ # right eye+ if any(right_eye_timestamps >= start_time_unix):+ right_eye_start_frame = np.where(right_eye_timestamps >= start_time_unix)[0][0]+ else:+ right_eye_start_frame = 0++ if any(right_eye_timestamps <= end_time_unix):+ right_eye_end_frame = np.where(right_eye_timestamps <= end_time_unix)[0][-1]+ else:+ right_eye_end_frame = len(right_eye_timestamps)++ # left eye+ if any(left_eye_timestamps >= start_time_unix):+ left_eye_start_frame = np.where(left_eye_timestamps >= start_time_unix)[0][0]+ else:+ left_eye_start_frame = 0++ if any(left_eye_timestamps <= end_time_unix):+ left_eye_end_frame = np.where(left_eye_timestamps <= end_time_unix)[0][-1]+ else:+ left_eye_end_frame = len(left_eye_timestamps)++ self.right_eye_start_frame = right_eye_start_frame+ self.right_eye_end_frame = right_eye_end_frame+ self.left_eye_start_frame = left_eye_start_frame+ self.left_eye_end_frame = left_eye_end_frame++ # rebase time onto freemocap's framerate (b/c it's slower than pupil) <- sloppy, assumes mocap slower than eye tracker, which is untrue for, say, GoPros+ self.synchronized_timestamps = self.raw_session_data.timestamps[freemocap_start_frame:freemocap_end_frame ]++ logger.warning(+ "SLOPPY ASSUMPTION: assuming freemocap framerate is always slower than eye tracker (true for webcams, not true for GoPros)")++ self.clip_eye_data()+ self.resample_eye_data()+ # self.normalize_eye_data()+++ self.synchronized_timestamps = self.synchronized_timestamps - self.synchronized_timestamps[0]++ assert (self.synchronized_timestamps.shape[0] == self.right_eye_theta.shape[0])+ assert (self.synchronized_timestamps.shape[0] == self.left_eye_theta.shape[0])++ if debug:+ self.show_debug_plots(vor_frame_start, vor_frame_end)++ synchronized_right_eye_data = PupilLabsDataClass(timestamps=self.synchronized_timestamps,+ theta=self.right_eye_theta,+ phi=self.right_eye_phi,+ pupil_center_normal_x=self.right_eye_pupil_center_normal_x,+ pupil_center_normal_y=self.right_eye_pupil_center_normal_y,+ pupil_center_normal_z=self.right_eye_pupil_center_normal_z,+ eye_d=0)+ synchronized_left_eye_data = PupilLabsDataClass(timestamps=self.synchronized_timestamps,+ theta=self.left_eye_theta,+ phi=self.left_eye_phi,+ pupil_center_normal_x=self.left_eye_pupil_center_normal_x,+ pupil_center_normal_y=self.left_eye_pupil_center_normal_y,+ pupil_center_normal_z=self.left_eye_pupil_center_normal_z,+ eye_d=1)++ synchronized_session_data = FreemocapSessionDataClass(timestamps=self.synchronized_timestamps,+ mediapipe_skel_fr_mar_dim=self.raw_session_data.mediapipe_skel_fr_mar_dim[+ freemocap_start_frame:freemocap_end_frame,+ :, :],+ right_eye_pupil_labs_data=synchronized_right_eye_data,+ left_eye_pupil_labs_data=synchronized_left_eye_data)+ return synchronized_session_data++ def clip_eye_data(self):+ self.right_eye_timestamps_clipped = self.raw_session_data.right_eye_pupil_labs_data.timestamps[+ self.right_eye_start_frame:self.right_eye_end_frame]++ self.right_eye_pupil_center_normal_x_clipped = self.raw_session_data.right_eye_pupil_labs_data.pupil_center_normal_x[+ self.right_eye_start_frame:self.right_eye_end_frame]++ self.right_eye_pupil_center_normal_y_clipped = self.raw_session_data.right_eye_pupil_labs_data.pupil_center_normal_y[+ self.right_eye_start_frame:self.right_eye_end_frame]++ self.right_eye_pupil_center_normal_z_clipped = self.raw_session_data.right_eye_pupil_labs_data.pupil_center_normal_z[+ self.right_eye_start_frame:self.right_eye_end_frame]++ self.right_eye_theta_clipped = self.raw_session_data.right_eye_pupil_labs_data.theta[+ self.right_eye_start_frame:self.right_eye_end_frame]++ self.right_eye_phi_clipped = self.raw_session_data.right_eye_pupil_labs_data.phi[+ self.right_eye_start_frame:self.right_eye_end_frame]++ self.left_eye_timestamps_clipped = self.raw_session_data.left_eye_pupil_labs_data.timestamps[+ self.left_eye_start_frame:self.left_eye_end_frame]++ self.left_eye_pupil_center_normal_x_clipped = self.raw_session_data.left_eye_pupil_labs_data.pupil_center_normal_x[+ self.left_eye_start_frame:self.left_eye_end_frame]++ self.left_eye_pupil_center_normal_y_clipped = self.raw_session_data.left_eye_pupil_labs_data.pupil_center_normal_y[+ self.left_eye_start_frame:self.left_eye_end_frame]++ self.left_eye_pupil_center_normal_z_clipped = self.raw_session_data.left_eye_pupil_labs_data.pupil_center_normal_z[+ self.left_eye_start_frame:self.left_eye_end_frame]++ self.left_eye_theta_clipped = self.raw_session_data.left_eye_pupil_labs_data.theta[+ self.left_eye_start_frame:self.left_eye_end_frame]++ self.left_eye_phi_clipped = self.raw_session_data.left_eye_pupil_labs_data.phi[+ self.left_eye_start_frame:self.left_eye_end_frame]++ def resample_eye_data(self):+ freemocap_timestamps = self.synchronized_timestamps+ right_eye_timestamps = self.right_eye_timestamps_clipped+ left_eye_timestamps = self.left_eye_timestamps_clipped++ self.right_eye_pupil_center_normal_x = np.interp(freemocap_timestamps, right_eye_timestamps,+ self.right_eye_pupil_center_normal_x_clipped)+ self.right_eye_pupil_center_normal_y = np.interp(freemocap_timestamps, right_eye_timestamps,+ self.right_eye_pupil_center_normal_y_clipped)+ self.right_eye_pupil_center_normal_z = np.interp(freemocap_timestamps, right_eye_timestamps,+ self.right_eye_pupil_center_normal_z_clipped)+ self.right_eye_theta = np.interp(freemocap_timestamps, right_eye_timestamps,+ self.right_eye_theta_clipped)+ self.right_eye_phi = np.interp(freemocap_timestamps, right_eye_timestamps,+ self.right_eye_phi_clipped)++ self.left_eye_pupil_center_normal_x = np.interp(freemocap_timestamps, left_eye_timestamps,+ self.left_eye_pupil_center_normal_x_clipped)+ self.left_eye_pupil_center_normal_y = np.interp(freemocap_timestamps, left_eye_timestamps,+ self.left_eye_pupil_center_normal_y_clipped)+ self.left_eye_pupil_center_normal_z = np.interp(freemocap_timestamps, left_eye_timestamps,+ self.left_eye_pupil_center_normal_z_clipped)+ self.left_eye_theta = np.interp(freemocap_timestamps, left_eye_timestamps,+ self.left_eye_theta_clipped)+ self.left_eye_phi = np.interp(freemocap_timestamps, left_eye_timestamps,+ self.left_eye_phi_clipped)++ def normalize_eye_data(self):+ self.right_eye_pupil_center_normal_x = self.right_eye_pupil_center_normal_x / np.linalg.norm(+ self.right_eye_pupil_center_normal_x)+ self.right_eye_pupil_center_normal_y = self.right_eye_pupil_center_normal_y / np.linalg.norm(+ self.right_eye_pupil_center_normal_y)+ self.right_eye_pupil_center_normal_z = self.right_eye_pupil_center_normal_z / np.linalg.norm(+ self.right_eye_pupil_center_normal_z)++ self.left_eye_pupil_center_normal_x = self.left_eye_pupil_center_normal_x / np.linalg.norm(+ self.left_eye_pupil_center_normal_x)+ self.left_eye_pupil_center_normal_y = self.left_eye_pupil_center_normal_y / np.linalg.norm(+ self.left_eye_pupil_center_normal_y)+ self.left_eye_pupil_center_normal_z = self.left_eye_pupil_center_normal_z / np.linalg.norm(+ self.left_eye_pupil_center_normal_z)++ def show_debug_plots(self, vor_frame_start, vor_frame_end):++ ###########################+ # Plot Raw Data+ ###########################+ fig = plt.figure(num=653412, figsize=(10, 20))+ fig.suptitle('Raw data')+ ax1 = fig.add_subplot(411)+ ax1.plot(self.raw_session_data.right_eye_pupil_labs_data.timestamps,+ self.raw_session_data.right_eye_pupil_labs_data.pupil_center_normal_x, '.-',+ label='right_eye_pupil_center_normal_x')+ ax1.plot(self.raw_session_data.right_eye_pupil_labs_data.timestamps,+ self.raw_session_data.right_eye_pupil_labs_data.pupil_center_normal_y, '.-',+ label='right_eye_pupil_center_normal_y')+ ax1.plot(self.raw_session_data.right_eye_pupil_labs_data.timestamps,+ self.raw_session_data.right_eye_pupil_labs_data.pupil_center_normal_z, '.-',+ label='right_eye_pupil_center_normal_z')+ ax1.legend(loc='upper left')++ ax2 = fig.add_subplot(412)+ ax2.plot(self.raw_session_data.right_eye_pupil_labs_data.timestamps,+ self.raw_session_data.right_eye_pupil_labs_data.theta, '.-',+ label='right_eye_theta')+ ax2.plot(self.raw_session_data.right_eye_pupil_labs_data.timestamps,+ self.raw_session_data.right_eye_pupil_labs_data.phi, '.-',+ label='right_eye_phi')+ ax2.legend(loc='upper left')++ ax3 = fig.add_subplot(413)+ ax3.plot(self.synchronized_timestamps, self.left_eye_pupil_center_normal_x, '.-',+ label='left_eye_pupil_center_normal_x')+ ax3.plot(self.synchronized_timestamps, self.left_eye_pupil_center_normal_y, '.-',+ label='left_eye_pupil_center_normal_y')+ ax3.plot(self.synchronized_timestamps, self.left_eye_pupil_center_normal_z, '.-',+ label='left_eye_pupil_center_normal_z')+ ax3.legend(loc='upper left')++ ax4 = fig.add_subplot(414)+ ax4.plot(self.raw_session_data.left_eye_pupil_labs_data.timestamps,+ self.raw_session_data.left_eye_pupil_labs_data.theta, '.-',+ label='left_eye_theta')+ ax4.plot(self.raw_session_data.left_eye_pupil_labs_data.timestamps,+ self.raw_session_data.left_eye_pupil_labs_data.phi, '.-',+ label='left_eye_phi')+ ax4.legend(loc='upper left')++ ###########################+ # Plot synchronized data+ ###########################++ fig = plt.figure(num=65341, figsize=(10, 20))+ fig.suptitle('Synchronized data')+ ax1 = fig.add_subplot(411)+ ax1.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.right_eye_pupil_center_normal_x[vor_frame_start:vor_frame_end], '.-',+ label='right_eye_pupil_center_normal_x')+ ax1.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.right_eye_pupil_center_normal_y[vor_frame_start:vor_frame_end], '.-',+ label='right_eye_pupil_center_normal_y')+ ax1.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.right_eye_pupil_center_normal_z[vor_frame_start:vor_frame_end], '.-',+ label='right_eye_pupil_center_normal_z')+ ax1.legend(loc='upper left')++ ax2 = fig.add_subplot(412)+ ax2.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.right_eye_theta[vor_frame_start:vor_frame_end], '.-',+ label='right_eye_theta')+ ax2.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.right_eye_phi[vor_frame_start:vor_frame_end], '.-',+ label='right_eye_phi')+ ax2.legend(loc='upper left')++ ax3 = fig.add_subplot(413)+ ax3.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.left_eye_pupil_center_normal_x[vor_frame_start:vor_frame_end], '.-',+ label='left_eye_pupil_center_normal_x')+ ax3.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.left_eye_pupil_center_normal_y[vor_frame_start:vor_frame_end], '.-',+ label='left_eye_pupil_center_normal_y')+ ax3.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.left_eye_pupil_center_normal_z[vor_frame_start:vor_frame_end], '.-',+ label='left_eye_pupil_center_normal_z')+ ax3.legend(loc='upper left')++ ax4 = fig.add_subplot(414)+ ax4.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.left_eye_theta[vor_frame_start:vor_frame_end], '.-',+ label='left_eye_theta')+ ax4.plot(self.synchronized_timestamps[vor_frame_start:vor_frame_end],+ self.left_eye_phi[vor_frame_start:vor_frame_end], '.-',+ label='left_eye_phi')+ ax4.legend(loc='upper left')++ plt.show()diff --git a/jon_scratch/pupil_calibration_pipeline/qt_gl_laser_skeleton_visualizer.py b/jon_scratch/pupil_calibration_pipeline/qt_gl_laser_skeleton_visualizer.pynew file mode 100644index 00000000..7b35f0be--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/qt_gl_laser_skeleton_visualizer.py@@ -0,0 +1,440 @@+# adapted from - https://gist.github.com/markjay4k/da2f55e28514be7160a7c5fbf95bd243+import logging+from pathlib import Path+from typing import Union, List++from pyqtgraph.Qt import QtCore, QtGui+import pyqtgraph.opengl as gl+import pyqtgraph as pg+import numpy as np+import sys+from mediapipe.python.solutions import holistic as mp_holistic++from jon_scratch.pupil_calibration_pipeline.data_classes.freemocap_session_data_class import FreemocapSessionDataClass+from jon_scratch.pupil_calibration_pipeline.data_classes.rotation_data_class import RotationDataClass+from jon_scratch.pupil_calibration_pipeline.session_data_loader import SessionDataLoader++logger = logging.getLogger(__name__)+++class QtGlLaserSkeletonVisualizer():+ session_path: Path = None++ def __init__(self,+ session_data: FreemocapSessionDataClass = None,+ mediapipe_skel_fr_mar_xyz: np.ndarray = None,+ start_frame: int = None,+ end_frame: int = None,+ move_data_to_origin: bool = True):++ if session_data is not None:+ self.session_data = session_data+ self.mediapipe_fr_mar_xyz = session_data.mediapipe_skel_fr_mar_dim+ elif mediapipe_skel_fr_mar_xyz is not None:+ self.mediapipe_fr_mar_xyz = mediapipe_skel_fr_mar_xyz+ else:+ raise ValueError("Must provide either session data or mediapipe skeleton data")++ if move_data_to_origin:+ self.move_data_to_origin()++ if start_frame is None:+ self.start_frame_number = 0+ else:+ self.start_frame_number = start_frame+ if end_frame is None:+ self.end_frame_number = self.mediapipe_fr_mar_xyz.shape[0]+ else:+ self.end_frame_number = end_frame++ self.current_frame_number = self.start_frame_number++ def initialize_display_window(self):+ self.create_app_window()+ self.create_grid_planes()+ self.get_mediapipe_connections()+ self.initialize_skel_dottos()+ self.initialize_skel_lines()+ self.initialize_head_axes()+ self.initialize_eye_socket_axes()+ if self.session_data.right_gaze_vector_endpoint_fr_xyz is not None:+ self.initialize_gaze_lasers()+ self.initialize_gaze_laser_tails(tail_length=30)++ def create_app_window(self):+ self.app = pg.mkQApp("Laser Skeleton")+ self.gl_view_widget = gl.GLViewWidget()+ self.gl_view_widget.opts['distance'] = 2000+ self.gl_view_widget.setWindowTitle('FreeMoCap - Laser Skeleton')+ self.gl_view_widget.setGeometry(0, 110, 1920, 1080)+ self.gl_view_widget.show()++ def create_grid_planes(self):+ grid_scale = 2e3+ # create the background grids+ grid_plane_x = gl.GLGridItem()+ grid_plane_x.setSize(grid_scale, grid_scale, grid_scale)+ grid_plane_x.setSpacing(grid_scale / 10, grid_scale / 10, grid_scale / 10)+ grid_plane_x.rotate(90, 0, 1, 0)+ grid_plane_x.translate(-grid_scale, 0, 0)+ self.gl_view_widget.addItem(grid_plane_x)++ grid_plane_y = gl.GLGridItem()+ grid_plane_y.setSize(grid_scale, grid_scale, grid_scale)+ grid_plane_y.setSpacing(grid_scale / 10, grid_scale / 10, grid_scale / 10)+ grid_plane_y.rotate(90, 1, 0, 0)+ grid_plane_y.translate(0, -grid_scale, 0)+ self.gl_view_widget.addItem(grid_plane_y)++ grid_plane_z = gl.GLGridItem()+ grid_plane_z.setSize(grid_scale, grid_scale, grid_scale)+ grid_plane_z.setSpacing(grid_scale / 10, grid_scale / 10, grid_scale / 10)+ grid_plane_z.translate(0, 0, -grid_scale)+ self.gl_view_widget.addItem(grid_plane_z)++ def get_mediapipe_connections(self):+ self.mediapipe_body_connections = [this_connection for this_connection in mp_holistic.POSE_CONNECTIONS]+ self.mediapipe_hand_connections = [this_connection for this_connection in mp_holistic.HAND_CONNECTIONS]+ self.mediapipe_face_connections = [this_connection for this_connection in mp_holistic.FACEMESH_TESSELATION]++ def initialize_skel_dottos(self):+ self.skeleton_scatter_item = gl.GLScatterPlotItem(+ pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, :, :], color=(0, 1, 1, 1), size=10,+ pxMode=False+ )+ self.gl_view_widget.addItem(self.skeleton_scatter_item)++ def initialize_skel_lines(self):+ self.skeleton_connections_list = []+ for this_connection in self.mediapipe_body_connections:+ this_skel_line = gl.GLLinePlotItem(+ pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, this_connection, :])+ self.skeleton_connections_list.append(this_skel_line)+ self.gl_view_widget.addItem(this_skel_line)++ def initialize_gaze_lasers(self):+ # right eye+ this_frame_right_gaze_origin = self.session_data.right_eye_socket_rotation_data.local_origin_fr_xyz[0, :]+ this_frame_right_gaze_endpoint = self.session_data.right_gaze_vector_endpoint_fr_xyz[0, :]+ this_frame_right_gaze_laser = np.array([this_frame_right_gaze_origin, this_frame_right_gaze_endpoint])++ self.right_gaze_vector_line_item = gl.GLLinePlotItem(pos=this_frame_right_gaze_laser,+ color=(1, 0, 1, 1),+ width=3)++ self.gl_view_widget.addItem(self.right_gaze_vector_line_item)++ # left eye+ this_frame_left_gaze_origin = self.session_data.left_eye_socket_rotation_data.local_origin_fr_xyz[0, :]+ this_frame_left_gaze_endpoint = self.session_data.left_gaze_vector_endpoint_fr_xyz[0, :]+ this_frame_left_gaze_laser = np.array([this_frame_left_gaze_origin, this_frame_left_gaze_endpoint])++ self.left_gaze_vector_line_item = gl.GLLinePlotItem(pos=this_frame_left_gaze_laser,+ color=(0, 1, 1, 1),+ width=3)++ self.gl_view_widget.addItem(self.left_gaze_vector_line_item)++ def initialize_gaze_laser_tails(self, tail_length):+ self.tail_length = tail_length+ dummy_tail = np.empty((self.tail_length, 3))+ dummy_tail[:] = np.nan+ self.right_gaze_laser_tail_line_item = gl.GLLinePlotItem(pos=dummy_tail, color=(1, 0, 1, .5), width=2)+ self.left_gaze_laser_tail_line_item = gl.GLLinePlotItem(pos=dummy_tail, color=(0, 1, 1, .5), width=2)++ self.gl_view_widget.addItem(self.right_gaze_laser_tail_line_item)+ self.gl_view_widget.addItem(self.left_gaze_laser_tail_line_item)++ def initialize_eye_socket_axes(self):+ self.eye_axes_scale = 1e2 / 2++ # right eye+ self.right_eye_socket_axes_x_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.right_eye_socket_rotation_data,+ dimension='x',+ scale=self.eye_axes_scale,+ color=(1, 0, 0, 1),+ width=3)+ self.right_eye_socket_axes_y_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.right_eye_socket_rotation_data,+ dimension='y',+ scale=self.eye_axes_scale,+ color=(0, 1, 0, 1),+ width=3)+ self.right_eye_socket_axes_z_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.right_eye_socket_rotation_data,+ dimension='z',+ scale=self.eye_axes_scale,+ color=(0, 0, 1, 1),+ width=3)++ self.gl_view_widget.addItem(self.right_eye_socket_axes_x_vector_line_item)+ self.gl_view_widget.addItem(self.right_eye_socket_axes_y_vector_line_item)+ self.gl_view_widget.addItem(self.right_eye_socket_axes_z_vector_line_item)++ # left eye+ self.left_eye_socket_axes_x_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.left_eye_socket_rotation_data,+ dimension='x',+ scale=self.eye_axes_scale,+ color=(1, 0, 0, 1),+ width=3)+ self.left_eye_socket_axes_y_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.left_eye_socket_rotation_data,+ dimension='y',+ scale=self.eye_axes_scale,+ color=(0, 1, 0, 1),+ width=3)+ self.left_eye_socket_axes_z_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.left_eye_socket_rotation_data,+ dimension='z',+ scale=self.eye_axes_scale,+ color=(0, 0, 1, 1),+ width=3)++ self.gl_view_widget.addItem(self.left_eye_socket_axes_x_vector_line_item)+ self.gl_view_widget.addItem(self.left_eye_socket_axes_y_vector_line_item)+ self.gl_view_widget.addItem(self.left_eye_socket_axes_z_vector_line_item)++ def initialize_head_axes(self):+ self.head_axes_scale = 1e2++ first_frame_head_center_xyz = self.session_data.head_rotation_data.local_origin_fr_xyz[0, :]++ self.head_axes_x_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.head_rotation_data,+ dimension='x',+ scale=self.head_axes_scale,+ color=(1, 0, 0, 1),+ width=3)+ self.head_axes_y_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.head_rotation_data,+ dimension='y',+ scale=self.head_axes_scale,+ color=(0, 1, 0, 1),+ width=3)+ self.head_axes_z_vector_line_item = self.create_axis_vector_line_item(0,+ self.session_data.head_rotation_data,+ dimension='z',+ scale=self.head_axes_scale,+ color=(0, 0, 1, 1),+ width=3)++ self.gl_view_widget.addItem(self.head_axes_x_vector_line_item)+ self.gl_view_widget.addItem(self.head_axes_y_vector_line_item)+ self.gl_view_widget.addItem(self.head_axes_z_vector_line_item)++ def create_axis_vector_line_item(self,+ frame_number: int,+ rotation_data: RotationDataClass,+ dimension: Union[str, int] = None,+ scale: Union[int, float] = 1,+ color=(1, 1, 1, 1),+ width=3,+ ) -> np.ndarray:++ this_frame_axis_vector_xyz = self.unit_vector_from_rotation_matrix(frame_number,+ rotation_data,+ dimension=dimension,+ scale=scale)+ this_frame_axis_vector_line_item = gl.GLLinePlotItem(pos=this_frame_axis_vector_xyz,+ color=color,+ width=width)+ self.gl_view_widget.addItem(this_frame_axis_vector_line_item)+ return this_frame_axis_vector_line_item++ def unit_vector_from_rotation_matrix(self,+ frame_number: int,+ rotation_data: RotationDataClass,+ dimension: Union[str, int] = None,+ scale: Union[int, float] = 1,+ ) -> np.ndarray:++ try:+ if dimension == 'x':+ dimension = 0+ elif dimension == 'y':+ dimension = 1+ elif dimension == 'z':+ dimension = 2++ if dimension > rotation_data.rotation_matricies[0].shape[1]:+ raise ValueError(+ 'dimension must be `x`, `y`, `z`, or an integer less than the number of dimensions in the rotation matrix')+ except:+ raise ValueError('something weird about the `dimension` argument')++ this_rot_mat = rotation_data.rotation_matricies[frame_number]+ this_axis_vector_endpoint_xyz = rotation_data.local_origin_fr_xyz[frame_number, :] + (+ this_rot_mat[dimension, :] * scale)+ this_axis_vector_origin_xyz = rotation_data.local_origin_fr_xyz[frame_number, :]++ this_axis_vector_line_xyz = np.array([this_axis_vector_origin_xyz, this_axis_vector_endpoint_xyz])++ return this_axis_vector_line_xyz++ def update(self):+ self.update_frame_number()+ if self.current_frame_number % 10 == 0:+ print(f'frame number: {self.current_frame_number}')++ self.skeleton_scatter_item.setData(+ pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, :, :]+ )+ self.update_skeleton_lines()+ self.update_head_axis_lines()+ self.update_eye_axis_lines()+ if self.session_data.right_gaze_vector_endpoint_fr_xyz is not None:+ self.update_gaze_lasers()+ self.update_gaze_laser_tails()++ def update_skeleton_lines(self):+ for this_skeleton_line_number, this_connection in enumerate(self.mediapipe_body_connections):+ self.skeleton_connections_list[this_skeleton_line_number].setData(+ pos=self.mediapipe_fr_mar_xyz[self.current_frame_number, this_connection, :]+ )++ def update_head_axis_lines(self):+ # X+ this_frame_head_axis_x_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.head_rotation_data,+ dimension='x',+ scale=self.head_axes_scale)+ self.head_axes_x_vector_line_item.setData(pos=this_frame_head_axis_x_vector_xyz)++ # Y+ this_frame_head_axis_y_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.head_rotation_data,+ dimension='y',+ scale=self.head_axes_scale)+ self.head_axes_y_vector_line_item.setData(pos=this_frame_head_axis_y_vector_xyz)++ # Z+ this_frame_head_axis_z_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.head_rotation_data,+ dimension='z',+ scale=self.head_axes_scale)+ self.head_axes_z_vector_line_item.setData(pos=this_frame_head_axis_z_vector_xyz)++ def update_eye_axis_lines(self):+ # right right eye+ # X+ this_frame_right_eye_socket_axis_x_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.right_eye_socket_rotation_data,+ dimension='x',+ scale=self.eye_axes_scale)+ self.right_eye_socket_axes_x_vector_line_item.setData(pos=this_frame_right_eye_socket_axis_x_vector_xyz)++ # Y+ this_frame_right_eye_socket_axis_y_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.right_eye_socket_rotation_data,+ dimension='y',+ scale=self.eye_axes_scale)+ self.right_eye_socket_axes_y_vector_line_item.setData(pos=this_frame_right_eye_socket_axis_y_vector_xyz)++ # Z+ this_frame_right_eye_socket_axis_z_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.right_eye_socket_rotation_data,+ dimension='z',+ scale=self.eye_axes_scale)+ self.right_eye_socket_axes_z_vector_line_item.setData(pos=this_frame_right_eye_socket_axis_z_vector_xyz)++ # left eye+ # X+ this_frame_left_eye_socket_axis_x_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.left_eye_socket_rotation_data,+ dimension='x',+ scale=self.eye_axes_scale)+ self.left_eye_socket_axes_x_vector_line_item.setData(pos=this_frame_left_eye_socket_axis_x_vector_xyz)++ # Y+ this_frame_left_eye_socket_axis_y_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.left_eye_socket_rotation_data,+ dimension='y',+ scale=self.eye_axes_scale)+ self.left_eye_socket_axes_y_vector_line_item.setData(pos=this_frame_left_eye_socket_axis_y_vector_xyz)++ # Z+ this_frame_left_eye_socket_axis_z_vector_xyz = self.unit_vector_from_rotation_matrix(self.current_frame_number,+ self.session_data.left_eye_socket_rotation_data,+ dimension='z',+ scale=self.eye_axes_scale)+ self.left_eye_socket_axes_z_vector_line_item.setData(pos=this_frame_left_eye_socket_axis_z_vector_xyz)++ def update_gaze_lasers(self):+ # right eye+ this_right_gaze_laser_line = np.array(+ [self.session_data.right_eye_socket_rotation_data.local_origin_fr_xyz[self.current_frame_number, :],+ self.session_data.right_gaze_vector_endpoint_fr_xyz[self.current_frame_number, :]])+ self.right_gaze_vector_line_item.setData(pos=this_right_gaze_laser_line)++ # left eye+ this_left_gaze_laser_line = np.array(+ [self.session_data.left_eye_socket_rotation_data.local_origin_fr_xyz[self.current_frame_number, :],+ self.session_data.left_gaze_vector_endpoint_fr_xyz[self.current_frame_number, :]])++ self.left_gaze_vector_line_item.setData(pos=this_left_gaze_laser_line)++ def update_gaze_laser_tails(self):+ if self.current_frame_number < self.tail_length:+ return++ # right eye+ this_right_gaze_laser_tail = self.session_data.right_gaze_vector_endpoint_fr_xyz[+ self.current_frame_number - self.tail_length: self.current_frame_number, :]+ self.right_gaze_laser_tail_line_item.setData(pos=this_right_gaze_laser_tail)++ # left eye+ this_left_gaze_laser_tail = self.session_data.left_gaze_vector_endpoint_fr_xyz[+ self.current_frame_number - self.tail_length: self.current_frame_number, :]+ self.left_gaze_laser_tail_line_item.setData(pos=this_left_gaze_laser_tail)++ def update_frame_number(self):+ self.current_frame_number += 1+ if self.current_frame_number >= self.end_frame_number:+ self.current_frame_number = self.start_frame_number++ def start(self):+ print('starting animation')+ pg.exec()++ def start_animation(self):+ print('initializing display window')+ self.initialize_display_window()+ timer = QtCore.QTimer()+ timer.timeout.connect(self.update)+ timer.start(33)+ self.start()++ def move_data_to_origin(self):+ mean_position_xyz = np.nanmedian(np.nanmedian(self.mediapipe_fr_mar_xyz, axis=0), axis=0)++ self.mediapipe_fr_mar_xyz[:, :, 0] -= mean_position_xyz[0]+ self.mediapipe_fr_mar_xyz[:, :, 1] -= mean_position_xyz[1]+ self.mediapipe_fr_mar_xyz[:, :, 2] -= mean_position_xyz[2]++ self.session_data.head_rotation_data.local_origin_fr_xyz[:, 0] -= mean_position_xyz[0]+ self.session_data.head_rotation_data.local_origin_fr_xyz[:, 1] -= mean_position_xyz[1]+ self.session_data.head_rotation_data.local_origin_fr_xyz[:, 2] -= mean_position_xyz[2]++ if self.session_data.right_gaze_vector_endpoint_fr_xyz is not None:+ self.session_data.right_gaze_vector_endpoint_fr_xyz[:, 0] -= mean_position_xyz[0]+ self.session_data.right_gaze_vector_endpoint_fr_xyz[:, 1] -= mean_position_xyz[1]+ self.session_data.right_gaze_vector_endpoint_fr_xyz[:, 2] -= mean_position_xyz[2]++ self.session_data.left_gaze_vector_endpoint_fr_xyz[:, 0] -= mean_position_xyz[0]+ self.session_data.left_gaze_vector_endpoint_fr_xyz[:, 1] -= mean_position_xyz[1]+ self.session_data.left_gaze_vector_endpoint_fr_xyz[:, 2] -= mean_position_xyz[2]+++if __name__ == '__main__':+ # session_id = 'sesh_2022-02-15_11_54_28_pupil_maybe'+ session_id = 'sesh_2022-05-03_13_43_00_JSM_treadmill_day2_t0'+ data_path = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data/')+ session_path = data_path / session_id++ session_data_loader = SessionDataLoader(session_path)+ mediapipe_skel_fr_mar_xyz_in = session_data_loader.load_mediapipe_data(move_to_origin=True)+ print(f'mediapipe_skel_fr_mar_xyz.shape: {mediapipe_skel_fr_mar_xyz_in.shape}')+ qt_gl_laser_skeleton = QtGlLaserSkeletonVisualizer(mediapipe_skel_fr_mar_xyz=mediapipe_skel_fr_mar_xyz_in, )+ qt_gl_laser_skeleton.start_animation()diff --git a/jon_scratch/pupil_calibration_pipeline/rotation_matrix_calculator.py b/jon_scratch/pupil_calibration_pipeline/rotation_matrix_calculator.pynew file mode 100644index 00000000..162bd161--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/rotation_matrix_calculator.py@@ -0,0 +1,284 @@+import logging+from typing import List++import numpy as np++import matplotlib.pyplot as plt+import matplotlib+import keyboard++from jon_scratch.pupil_calibration_pipeline.data_classes.rotation_data_class import RotationDataClass++logger = logging.getLogger(__name__)+logger.setLevel(logging.INFO)++matplotlib.use('qt5agg')+++def pin_point_new_origin(point_xyz, new_origin_xyz):+ return point_xyz - new_origin_xyz+++def normalize_length(point_xyz):+ return point_xyz / get_norm_length(point_xyz, axis=1)[:, None]+++def get_norm_length(point_xyz, axis=1):+ return np.linalg.norm(point_xyz, axis=axis)+++class RotationMatrixCalculator:+ mediapipe_skeleton_fr_mar_xyz: np.ndarray++ def __init__(self, mediapipe_skeleton_fr_mar_xyz):+ self.mediapipe_skeleton_fr_mar_xyz = mediapipe_skeleton_fr_mar_xyz++ def calculate_head_rotation_matricies(self,+ normalize_length_by_x: bool = False,+ debug: bool = False):+ """+ # Calculate head rotation matricies++ calculate orthonormal basis vectors of the head on each frame, stack them on top of each other to make 3x3 rotation matricies, because apparently that's how that works?!++ 1 - calculate `head_center_xyz` as mean of left and right ear markers+ 2 - define x+ as vector between `head_center_xyz` and `nose_xyz`- normalize to get `x_hat`+ 3 - define y+ as vector between `head_center_xyz` and `left_ear_xyz` - normalize to get `y_hat_temp` (will be replaced later, to ensure orthogonal \)+ 4 - define z+ as cross product of `x_hat` and `y_hat`+ 5 - replace `y_hat_temp` with `z_hat` crossed wtih `x_hat` (Pretty sure that makes it right handed...)++ Stack x_hat, y_hat, and z_hat on top of each other, and that's your head rotation matrix on that frame++ """+ nose_id = 0+ left_ear_id = 7+ right_ear_id = 8++ nose_xyz = self.mediapipe_skeleton_fr_mar_xyz[:, nose_id, :].copy()+ right_ear_xyz = self.mediapipe_skeleton_fr_mar_xyz[:, right_ear_id, :].copy()+ left_ear_xyz = self.mediapipe_skeleton_fr_mar_xyz[:, left_ear_id, :].copy()+ head_center_xyz = (right_ear_xyz + left_ear_xyz) / 2++ head_rotation_data = self.define_basis_vectors(center_point_fr_xyz=head_center_xyz,+ x_direction_fr_xyz=nose_xyz,+ y_direction_fr_xyz=left_ear_xyz,+ normalize_length_by_x=normalize_length_by_x)+ if debug:+ self.show_head_rotation_debug_plot(head_rotation_data)++ return head_rotation_data++ def calculate_eye_rotation_matricies(self,+ eye: str,+ normalize_length_by_x: bool = False,+ debug: bool = False,+ ) -> List[np.ndarray]:+ nose_index = 0+ if eye == 'left':+ inner_eye_index = 1+ eye_mid_index = 2+ outer_eye_index = 3+ x_direction_fr_xyz = self.mediapipe_skeleton_fr_mar_xyz[:, inner_eye_index,+ :] # this will make Z+ point forward+ elif eye == 'right':+ inner_eye_index = 4+ eye_mid_index = 5+ outer_eye_index = 6+ x_direction_fr_xyz = self.mediapipe_skeleton_fr_mar_xyz[:, outer_eye_index,+ :] # this will make Z+ point forward+ else:+ raise ValueError('`eye` must be "left" or "right"')++ y_direction_fr_xyz = self.mediapipe_skeleton_fr_mar_xyz[:, nose_index, :]++ eye_mid_fr_xyz = self.mediapipe_skeleton_fr_mar_xyz[:, eye_mid_index, :]++ eye_socket_rotation_data = self.define_basis_vectors(center_point_fr_xyz=eye_mid_fr_xyz,+ x_direction_fr_xyz=x_direction_fr_xyz,+ y_direction_fr_xyz=y_direction_fr_xyz,+ normalize_length_by_x = normalize_length_by_x)++ return eye_socket_rotation_data++ def define_basis_vectors(self,+ center_point_fr_xyz: np.ndarray = None,+ x_direction_fr_xyz: np.ndarray = None,+ y_direction_fr_xyz: np.ndarray = None,+ z_direction_fr_xyz: np.ndarray = None,+ normalize_length_by_x: bool = True) -> RotationDataClass:+ """+ create orthonormal basis vectors based on a center point and at least TWO of the following = a point defining X direction, a point defining Y direction, and a point defining Z direction+ """++ x_known = True+ y_known = True+ z_known = True++ if x_direction_fr_xyz is None:+ x_known = False+ x_direction_fr_xyz = np.empty(center_point_fr_xyz.shape)+ x_direction_fr_xyz[:] = np.nan++ if y_direction_fr_xyz is None:+ y_known = False+ y_direction_fr_xyz = np.empty(center_point_fr_xyz.shape)+ y_direction_fr_xyz[:] = np.nan++ if z_direction_fr_xyz is None:+ z_known = False+ z_direction_fr_xyz = np.empty(center_point_fr_xyz.shape)+ z_direction_fr_xyz[:] = np.nan++ if sum([x_known, y_known, z_known]) < 2:+ raise ValueError(+ "Must specify at least two of the following: x_direction_fr_xyz, y_direction_fr_xyz, z_direction_fr_xyz")++ if center_point_fr_xyz is not None: # if center point is specified, then use it to zero out the x,y,z directions vectors+ zero_x_direction_xyz = pin_point_new_origin(x_direction_fr_xyz, center_point_fr_xyz)+ zero_y_direction_xyz = pin_point_new_origin(y_direction_fr_xyz, center_point_fr_xyz)+ zero_z_direction_xyz = pin_point_new_origin(z_direction_fr_xyz, center_point_fr_xyz)+ else:+ zero_x_direction_xyz = x_direction_fr_xyz+ zero_y_direction_xyz = y_direction_fr_xyz+ zero_z_direction_xyz = z_direction_fr_xyz++ x_hat_xyz = None+ y_hat_xyz = None+ z_hat_xyz = None++ if x_known:+ x_norm_length = np.nanmean(get_norm_length(zero_x_direction_xyz))+ x_hat_xyz = normalize_length(zero_x_direction_xyz)+ if y_known: # we know x and y directions and need to find Z by X cross Y+ z_hat_xyz = normalize_length(np.cross(x_hat_xyz, zero_y_direction_xyz))+ y_hat_xyz = normalize_length(np.cross(z_hat_xyz, x_hat_xyz))++ else: # we know x and z directions and need to find Y by Z cross X+ y_hat_xyz = normalize_length(np.cross(zero_z_direction_xyz, x_hat_xyz))+ z_hat_xyz = normalize_length(np.cross(x_hat_xyz, y_hat_xyz))++ elif y_known: # we know y and z directions and need to find X by Y cross Z+ y_hat_xyz = normalize_length(zero_y_direction_xyz)+ x_hat_xyz = normalize_length(np.cross(y_hat_xyz, zero_z_direction_xyz))+ z_hat_xyz = normalize_length(np.cross(x_hat_xyz, y_hat_xyz))++ if normalize_length_by_x:+ x_hat_xyz = x_hat_xyz * x_norm_length+ y_hat_xyz = y_hat_xyz * x_norm_length+ z_hat_xyz = z_hat_xyz * x_norm_length++ # #umm, this shouldn't be necessary.... but it makes it work, so....+ # x_hat_xyz = -x_hat_xyz+ # y_hat_xyz = -y_hat_xyz+ # z_hat_xyz = z_hat_xyz++ # create 3x3 cosine rotation matricies by stacking X_hat, Y_hat, and Z_hat+ rotation_matricies = []+ frame_number = -1+ for this_x_hat_xyz, this_y_hat_xyz, this_z_hat_xyz in zip(x_hat_xyz, y_hat_xyz, z_hat_xyz):+ frame_number += 1+ if frame_number > 1000:+ f = 9+ this_rotation_matrix = np.squeeze(np.dstack((this_x_hat_xyz, this_y_hat_xyz, this_z_hat_xyz)))++ assert this_rotation_matrix.shape == (3, 3)++ if not np.isnan(this_x_hat_xyz).any():+ assert (this_rotation_matrix[:, 0] == this_x_hat_xyz).all()++ rotation_matricies.append(this_rotation_matrix)++ rotation_matricies = [np.transpose(this_rotation_martix) for this_rotation_martix in rotation_matricies]++ return RotationDataClass(rotation_matricies=rotation_matricies,+ local_origin_fr_xyz=center_point_fr_xyz,+ x_hat_norm_fr_xyz=x_hat_xyz,+ y_hat_norm_fr_xyz=y_hat_xyz,+ z_hat_norm_fr_xyz=z_hat_xyz,+ )++ def show_head_rotation_debug_plot(self, rotation_data: RotationDataClass):+ rotation_matricies = rotation_data.rotation_matricies+ local_origin_fr_xyz = rotation_data.local_origin_fr_xyz++ x_hat_xyz = np.asarray([this_rot_mat[0, :] for this_rot_mat in rotation_matricies])+ y_hat_xyz = np.asarray([this_rot_mat[1, :] for this_rot_mat in rotation_matricies])+ z_hat_xyz = np.asarray([this_rot_mat[2, :] for this_rot_mat in rotation_matricies])++ test_point_xyz_og = np.array((1, 1, 1))+ test_point_xyz_rot = np.empty(z_hat_xyz.shape)+ logger.warning('We shouldnt have to transpose the rotation matix here, need to fix this upstream')++ for frame_number in range(len(rotation_matricies)):+ # see https://peps.python.org/pep-0465/ for explanation of the `@` operator (which is a "matrix multiplication" operator)+ test_point_xyz_rot[frame_number, :] = np.transpose(rotation_matricies[frame_number]) @ test_point_xyz_og++ z_mediapipe_skeleton_fr_mar_xyz = np.empty(self.mediapipe_skeleton_fr_mar_xyz.shape)+ for this_marker_number in range(self.mediapipe_skeleton_fr_mar_xyz.shape[1]):+ z_mediapipe_skeleton_fr_mar_xyz[:, this_marker_number, :] = pin_point_new_origin(+ self.mediapipe_skeleton_fr_mar_xyz[:, this_marker_number, :],+ local_origin_fr_xyz)+ plt.close('all')++ fig = plt.figure(num=124)+ fig.suptitle('Head Rotation Matrix - Debug Plot - Press ESC to Quit')+ ax = fig.add_subplot(111, projection='3d')++ ax_range = np.nanmean(get_norm_length(x_hat_xyz)) * 2++ for frame_number in range(1200, x_hat_xyz.shape[0]):+ ax.clear()++ # plot head orthonormal basis vectors+ ax.plot(x_hat_xyz[frame_number - 30:frame_number, 0],+ x_hat_xyz[frame_number - 30:frame_number, 1],+ x_hat_xyz[frame_number - 30:frame_number, 2], 'r-')++ ax.plot(y_hat_xyz[frame_number - 30:frame_number, 0],+ y_hat_xyz[frame_number - 30:frame_number, 1],+ y_hat_xyz[frame_number - 30:frame_number, 2], 'g-')++ ax.plot(z_hat_xyz[frame_number - 30:frame_number, 0],+ z_hat_xyz[frame_number - 30:frame_number, 1],+ z_hat_xyz[frame_number - 30:frame_number, 2], 'b-')++ ax.plot([0, x_hat_xyz[frame_number, 0]],+ [0, x_hat_xyz[frame_number, 1]],+ [0, x_hat_xyz[frame_number, 2]], 'r-o', label='x_hat')++ ax.plot([0, y_hat_xyz[frame_number, 0]],+ [0, y_hat_xyz[frame_number, 1]],+ [0, y_hat_xyz[frame_number, 2]], 'g-o', label='y_hat')++ ax.plot([0, z_hat_xyz[frame_number, 0]],+ [0, z_hat_xyz[frame_number, 1]],+ [0, z_hat_xyz[frame_number, 2]], 'b-o', label='z_hat')++ # plot test point rotated with head rotation matricies++ ax.plot(test_point_xyz_rot[frame_number - 30:frame_number, 0],+ test_point_xyz_rot[frame_number - 30:frame_number, 1],+ test_point_xyz_rot[frame_number - 30:frame_number, 2], 'k-')++ ax.plot([0, test_point_xyz_rot[frame_number, 0]],+ [0, test_point_xyz_rot[frame_number, 1]],+ [0, test_point_xyz_rot[frame_number, 2]], 'k-o', label='test point')++ # plot a friendly skeleton+ ax.scatter(z_mediapipe_skeleton_fr_mar_xyz[frame_number, :, 0],+ z_mediapipe_skeleton_fr_mar_xyz[frame_number, :, 1],+ z_mediapipe_skeleton_fr_mar_xyz[frame_number, :, 2],+ 'ko')++ ax.set_xlim([-ax_range, ax_range])+ ax.set_ylim([-ax_range, ax_range])+ ax.set_zlim([-ax_range, ax_range])++ ax.legend()+ plt.pause(.01)++ if keyboard.is_pressed('esc'):+ print('You Pressed Escape!')+ break # finishing the loop++ plt.close(fig)diff --git a/jon_scratch/pupil_calibration_pipeline/session_data_loader.py b/jon_scratch/pupil_calibration_pipeline/session_data_loader.pynew file mode 100644index 00000000..fb20ed00--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/session_data_loader.py@@ -0,0 +1,107 @@+import json+import logging+from pathlib import Path+from typing import Union+import numpy as np+import pandas as pd++from jon_scratch.pupil_calibration_pipeline.data_classes.pupil_dataclass_and_handler import PupilLabsDataClass, PupilDataHandler++logger = logging.getLogger(__name__)+++class SessionDataLoader:+ _session_path: Path = None++ def __init__(self, session_path):++ self._session_path = Path(session_path)++ @property+ def session_path(self):+ return self._session_path+++ def load_freemocap_unix_timestamps(self):+ """+ load/calculate the unix timestamps for the freemocap data (as mean of each camera's timestamp on each frame)++ """+ freemocap_unix_timestamp_path = self.session_path / 'unix_synced_timestamps.csv'+ freemocap_unix_timestamps_dataframe = pd.read_csv(freemocap_unix_timestamp_path)+ each_cam_timestamps_list = []+ camera_names = [column_name for column_name in list(freemocap_unix_timestamps_dataframe.columns) if+ 'Cam' in column_name]++ for this_camera_name in camera_names:+ each_cam_timestamps_list.append(np.array(freemocap_unix_timestamps_dataframe[this_camera_name]))++ cameras_timestamp_array = np.stack(each_cam_timestamps_list, axis=1)+ cameras_timestamp_array[cameras_timestamp_array == -1] = np.nan # replace -1 with nan++ freemocap_timestamps = np.nanmean(cameras_timestamp_array, axis=1)+ return freemocap_timestamps++ def load_mediapipe_data(self,+ session_path: Union[Path, str] = None,+ include_hands: bool = True,+ include_face: bool = False,+ move_to_origin: bool = False, ) -> np.ndarray:++ if session_path is not None:+ self.session_path(session_path)++ if self._session_path is None:+ raise ValueError('Session path is not specified - Can\'t load mediapipe data without a session path')++ mediapipe_data_path = self.session_path / 'DataArrays' / 'mediaPipeSkel_3d_smoothed.npy'+ logger.info(f'loading mediapipe data from {mediapipe_data_path}')+ mediapipe_fr_mar_xyz = np.load(str(mediapipe_data_path))++ if move_to_origin:+ mean_position_xyz = np.nanmedian(np.nanmedian(mediapipe_fr_mar_xyz, axis=0), axis=0)+ mediapipe_fr_mar_xyz[:, :, 0] -= mean_position_xyz[0]+ mediapipe_fr_mar_xyz[:, :, 1] -= mean_position_xyz[1]+ mediapipe_fr_mar_xyz[:, :, 2] -= mean_position_xyz[2]++ if include_face:+ return mediapipe_fr_mar_xyz # body + hands + face++ if include_hands:+ return mediapipe_fr_mar_xyz[:, :75, :] # body + hands, but not face++ return mediapipe_fr_mar_xyz[:, :33, :] # just the body marker++ def load_pupil_data(self)->PupilDataHandler:+ """+ # load in data recorded with a Pupil Labs eye tracker.+ - pupil data from `pupil_positions.csv` in the **pupil exports** folder+ - definitions in `pupil_gaze_positions_info.txt` of same folder+ - pupil start time from `info.player.json` for syncing+ """+ pupil_data_path = self.session_path / 'pupil_000'+ pupil_data_exports_path = pupil_data_path / 'exports' / '000'+ pupil_positions_path = pupil_data_exports_path / 'pupil_positions.csv'+ pupil_recording_info_path = pupil_data_path / 'info.player.json'+ freemocap_unix_timestamp_path = self.session_path / 'unix_synced_timestamps.csv'++ logger.info(f'loading pupil data from {pupil_positions_path}')+ pupil_dataframe = pd.read_csv(pupil_positions_path)+ pupil_recording_info_json = json.load(open(pupil_recording_info_path))+ freemocap_unix_timestamps_dataframe = pd.read_csv(freemocap_unix_timestamp_path)++ pupil_data_handler = PupilDataHandler(pupil_dataframe)+ pupil_data_handler.convert_to_unix_timestamps(pupil_recording_info_json)++ return pupil_data_handler+++if __name__ == '__main__':+ data_path = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data/')+ session_path = data_path / 'sesh_2022-02-15_11_54_28_pupil_maybe'++ session_data_loader = SessionDataLoader(session_path)+ pupil_data = session_data_loader.load_pupil_data()+ print('pupil data loaded, but I don\'t have an easy way to print out the shape of things yet, lol')+ mediapipe_skel_fr_mar_xyz = session_data_loader.load_mediapipe_data()+ print(f'mediapipe_skel_fr_mar_xyz.shape: {mediapipe_skel_fr_mar_xyz.shape}')diff --git a/jon_scratch/pupil_calibration_pipeline/spherical_coordiate_converters/__init__.py b/jon_scratch/pupil_calibration_pipeline/spherical_coordiate_converters/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/jon_scratch/pupil_calibration_pipeline/spherical_coordiate_converters/cart2sph_and_sph2cart.py b/jon_scratch/pupil_calibration_pipeline/spherical_coordiate_converters/cart2sph_and_sph2cart.pynew file mode 100644index 00000000..473a4354--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/spherical_coordiate_converters/cart2sph_and_sph2cart.py@@ -0,0 +1,15 @@+#copied from https://stackoverflow.com/questions/30084174/efficient-matlab-cart2sph-and-sph2cart-functions-in-python+import numpy as np+++def cart2sph(x,y,z):+ azimuth = np.arctan2(y,x)+ elevation = np.arctan2(z,np.sqrt(x**2 + y**2))+ r = np.sqrt(x**2 + y**2 + z**2)+ return azimuth, elevation, r++def sph2cart(azimuth,elevation,r):+ x = r * np.cos(elevation) * np.cos(azimuth)+ y = r * np.cos(elevation) * np.sin(azimuth)+ z = r * np.sin(elevation)+ return x, y, z\ No newline at end of filediff --git a/jon_scratch/pupil_calibration_pipeline/vor_calibrator.py b/jon_scratch/pupil_calibration_pipeline/vor_calibrator.pynew file mode 100644index 00000000..f9d4c71e--- /dev/null+++ b/jon_scratch/pupil_calibration_pipeline/vor_calibrator.py@@ -0,0 +1,282 @@+import logging+from typing import Union, List++import keyboard+import numpy as np+from scipy import optimize+from matplotlib import pyplot as plt+import matplotlib+from scipy.spatial.transform import Rotation++from jon_scratch.pupil_calibration_pipeline.data_classes.freemocap_session_data_class import FreemocapSessionDataClass+from jon_scratch.pupil_calibration_pipeline.data_classes.pupil_dataclass_and_handler import PupilLabsDataClass+from jon_scratch.pupil_calibration_pipeline.data_classes.rotation_data_class import RotationDataClass+from jon_scratch.pupil_calibration_pipeline.spherical_coordiate_converters.cart2sph_and_sph2cart import cart2sph, \+ sph2cart++matplotlib.use('qt5agg')+logger = logging.getLogger(__name__)+++class VorCalibrator:+ """+ Calibrate synchronized eye data from a Pupil Labs eye tracker with FreeMoCap skeleton+ see methods sections of (Matthis, Yates, and Hayhoe, Current Biology, 2018) and (Matthis, Muller, Bonnen, Hayehoe, PLoS Computation Biology 2022) for details+ """++ def __init__(self, mediapipe_skel_fr_mar_xyz: np.ndarray,+ vor_start_frame: int = None,+ vor_end_frame: int = None,+ debug: bool = False):+ self.debug = debug+ self.mediapipe_skel_fr_mar_xyz = mediapipe_skel_fr_mar_xyz+ if vor_start_frame is not None:+ self.vor_start_frame = vor_start_frame+ if vor_end_frame is None:+ raise ValueError("Must specify vor_end_frame if vor_start_frame is specified")+ self.vor_end_frame = vor_end_frame++ def calibrate(self,+ pupil_labs_eye_data: PupilLabsDataClass,+ eye_socket_rotation_data: RotationDataClass,+ head_rotation_data: RotationDataClass,+ fixation_point_fr_xyz: np.ndarray) -> np.ndarray:++ if not fixation_point_fr_xyz.shape[1] == 3:+ raise Exception("fixation_point_fr_xyz must be a numpy array with 3 columns")++ eye_socket_origin_fr_xyz = eye_socket_rotation_data.local_origin_fr_xyz+ eye_socket_rotation_matricies = eye_socket_rotation_data.rotation_matricies++ head_rotation_matricies = head_rotation_data.rotation_matricies++ # gaze_unit_vector_end_point_fr_xyz = eye_socket_rotation_data.z_hat_norm_fr_xyz+ self.fixation_distance = self.get_distance_between_two_points(fixation_point_fr_xyz,+ eye_socket_origin_fr_xyz[+ self.vor_start_frame:self.vor_end_frame, :])++ # not sure why I need to do this, but Karl Muller did it so it's likely necessary+ gaze_y = -pupil_labs_eye_data.pupil_center_normal_y+ gaze_x = pupil_labs_eye_data.pupil_center_normal_z+ gaze_z = pupil_labs_eye_data.pupil_center_normal_x++ gaze_x = gaze_x * np.nanmean(self.fixation_distance)+ gaze_y = gaze_y * np.nanmean(self.fixation_distance)+ gaze_z = gaze_z * np.nanmean(self.fixation_distance)++ use_dummy_gaze_data = False+ if use_dummy_gaze_data:+ gaze_x = np.zeros(pupil_labs_eye_data.pupil_center_normal_x.shape) * np.nanmedian(self.fixation_distance)+ gaze_y = np.zeros(pupil_labs_eye_data.pupil_center_normal_y.shape) * np.nanmedian(self.fixation_distance)+ gaze_z = np.ones(pupil_labs_eye_data.pupil_center_normal_z.shape) * np.nanmedian(self.fixation_distance)++ self.uncalibrated_gaze_vector_endpoint_fr_xyz = self.rotate_gaze_lasers([0, 0, 0],+ eye_socket_origin_fr_xyz,+ eye_socket_rotation_matricies,+ head_rotation_matricies,+ gaze_x,+ gaze_y,+ gaze_z)++ self.calibrated_rotational_offset = self.calculate_optimal_rotational_offset(eye_socket_origin_fr_xyz,+ eye_socket_rotation_matricies,+ head_rotation_matricies,+ fixation_point_fr_xyz,+ gaze_x,+ gaze_y,+ gaze_z)++ gaze_vector_endpoint_fr_xyz = self.rotate_gaze_lasers(self.calibrated_rotational_offset,+ eye_socket_origin_fr_xyz,+ eye_socket_rotation_matricies,+ head_rotation_matricies,+ gaze_x,+ gaze_y,+ gaze_z+ )++ return gaze_vector_endpoint_fr_xyz++ def calculate_optimal_rotational_offset(self,+ eye_socket_origin_fr_xyz: np.ndarray,+ eye_socket_rotation_matricies: List[np.ndarray],+ head_rotation_matricies,+ fixation_point_fr_xyz: np.ndarray,+ gaze_x: np.ndarray,+ gaze_y: np.ndarray,+ gaze_z: np.ndarray) -> List[float]:+ """+ Calculate the optimal rotational offset to apply to Pupil Labs eye tracker data in order to minimize the distance+ between teh end of the gaze vector and the fixation point on each frame+ (based on the VOR calibration methods described in Matthis, Yates, and Hayhoe, Current Biology, 2018 and Matthis, Muller, Bonnen, Hayhoe, PLoS Computation Biology 2022)++ """+ initial_rotational_offset_guess = [0, 0, 0]++ vor_frames = np.arange(self.vor_start_frame, self.vor_end_frame)++ optimization_results = optimize.least_squares(self.get_error_for_a_given_rotational_offset_guess,+ initial_rotational_offset_guess,+ args=(eye_socket_origin_fr_xyz[vor_frames, :],+ eye_socket_rotation_matricies[+ self.vor_start_frame:self.vor_end_frame],+ head_rotation_matricies[+ self.vor_start_frame:self.vor_end_frame],+ fixation_point_fr_xyz,+ gaze_x[vor_frames],+ gaze_y[vor_frames],+ gaze_z[vor_frames],),+ gtol=1e-10,+ verbose=2)+ calibrated_rotational_offset = optimization_results.x+ return calibrated_rotational_offset++ def get_error_for_a_given_rotational_offset_guess(self,+ rotational_offset_guess,+ eye_socket_origin_fr_xyz,+ eye_socket_rotation_matricies,+ head_rotation_matricies,+ fixation_point_fr_xyz,+ gaze_x,+ gaze_y,+ gaze_z):++ gaze_laser_endpoint_during_vor_fr_xyz = self.rotate_gaze_lasers(rotational_offset_guess,+ eye_socket_origin_fr_xyz,+ eye_socket_rotation_matricies,+ head_rotation_matricies,+ gaze_x,+ gaze_y,+ gaze_z)++ distance_between_gaze_endpoint_and_fixation_point_fr_xyz = self.get_distance_between_two_points(+ gaze_laser_endpoint_during_vor_fr_xyz,+ fixation_point_fr_xyz)+ distance_error = np.sqrt(np.nanmean(distance_between_gaze_endpoint_and_fixation_point_fr_xyz ** 2))++ gaze_tip_velocity_fr_xyz = np.diff(gaze_laser_endpoint_during_vor_fr_xyz, axis=0)+ mean_gaze_tip_velocity_per_frame = np.nanmean(gaze_tip_velocity_fr_xyz, axis=0)+ velocity_error = np.sqrt(np.nanmean(mean_gaze_tip_velocity_per_frame ** 2))++ error = distance_error + velocity_error++ if self.debug:+ self.plot_optimization_error(error, gaze_laser_endpoint_during_vor_fr_xyz, eye_socket_origin_fr_xyz,+ fixation_point_fr_xyz)+ return error++ def plot_optimization_error(self, error, gaze_laser_endpoint_during_vor_fr_xyz, eye_socket_origin_fr_xyz,+ fixation_point_fr_xyz):+ figure_number = 13451++ if not plt.fignum_exists(figure_number):+ fig = plt.figure(figure_number)+ ax = fig.add_subplot(111, projection='3d')+ else:+ fig = plt.gcf()+ ax = plt.gca()++ mean_eye_origin_x = np.nanmean(eye_socket_origin_fr_xyz[:, 0])+ mean_eye_origin_y = np.nanmean(eye_socket_origin_fr_xyz[:, 1])+ mean_eye_origin_z = np.nanmean(eye_socket_origin_fr_xyz[:, 2])++ fig.suptitle(f'error: {error}')+ ax_range = 1e3+ ax.clear()++ ax.plot(mean_eye_origin_x, mean_eye_origin_y, mean_eye_origin_z, 'mh', label='mean eye origin')++ ax.plot(self.uncalibrated_gaze_vector_endpoint_fr_xyz[self.vor_start_frame:self.vor_end_frame, 0],+ self.uncalibrated_gaze_vector_endpoint_fr_xyz[self.vor_start_frame:self.vor_end_frame, 1],+ self.uncalibrated_gaze_vector_endpoint_fr_xyz[self.vor_start_frame:self.vor_end_frame, 2], 'k-o',+ label='original gaze xyz')++ ax.plot(gaze_laser_endpoint_during_vor_fr_xyz[:, 0],+ gaze_laser_endpoint_during_vor_fr_xyz[:, 1],+ gaze_laser_endpoint_during_vor_fr_xyz[:, 2], 'r-o',+ label='gaze_rotated_by_guess_then_head_rotation_xyz')++ ax.plot(fixation_point_fr_xyz[:, 0],+ fixation_point_fr_xyz[:, 1],+ fixation_point_fr_xyz[:, 2], 'b-o', label='mean_fixation_point_xyz')++ ax.set_xlim([mean_eye_origin_x - ax_range, mean_eye_origin_x + ax_range])+ ax.set_ylim([mean_eye_origin_y - ax_range, mean_eye_origin_y + ax_range])+ ax.set_zlim([mean_eye_origin_z - ax_range, mean_eye_origin_z + ax_range])+ ax.legend()+ plt.pause(.01)++ if keyboard.is_pressed('esc'):+ print('You Pressed Escape!')+ self.debug = False+ plt.close(fig)++ def get_distance_between_two_points(self, local_origin_fr_xyz, fixation_point_fr_xyz):+ return np.linalg.norm(local_origin_fr_xyz - fixation_point_fr_xyz, axis=1)++ def create_unrotated_gaze_lasers_from_eye_rotation_matrixies(self,+ gaze_vector_start_point_fr_xyz: np.ndarray,+ gaze_unit_vector_end_point_fr_xyz: np.ndarray,+ calbration_distance,+ ) -> np.ndarray:+ """+ Creates an unrotated gaze laser for each frame in the skeleton/gaze data. Returns a vector the same length as the calibration_distance, shooting our of the skeletons eye_sockets (as if they eyes were paralyzed).+ """+ if not gaze_vector_start_point_fr_xyz.shape == gaze_unit_vector_end_point_fr_xyz.shape:+ raise Exception(+ "`gaze_vector_start_point_fr_xyz` and `gaze_unit_vector_end_point_fr_xyz` must have the same shape")++ # scale gaze vector so it's the same norm/length as the calibration distance+ gaze_unit_vector_end_point_fr_xyz = gaze_unit_vector_end_point_fr_xyz * calbration_distance++ # move it from an zero-centered reference frame to the eye socket reference frame+ gaze_vector_end_point_fr_xyz = gaze_unit_vector_end_point_fr_xyz + gaze_vector_start_point_fr_xyz++ return gaze_vector_end_point_fr_xyz++ def rotate_gaze_lasers(self,+ rotational_offset,+ gaze_vector_start_point_fr_xyz,+ eye_socket_rotation_matricies,+ head_rotation_matricies,+ gaze_x,+ gaze_y,+ gaze_z) -> np.ndarray:+ """"+ - pin gaze to origin+ - rotate gaze vector by `calibrated_rotational_offset`+ - rotate by measured eye movemens (from pupil labs)+ - translate gaze vector back to the eye socket (by adding eyeball center xyz)+ """++ # - rotate gaze vector by `calibrated_rotational_offset`+ offset_x_euler = rotational_offset[0]+ offset_y_euler = rotational_offset[1]+ offset_z_euler = rotational_offset[2]+ offset_rotation_matrix_guess = Rotation.from_euler('xyz',+ (offset_x_euler, offset_y_euler, offset_z_euler)).as_matrix()++ original_gaze_fr_xyz = np.array([gaze_x, gaze_y, gaze_z])++ # rotate gaze vector by rotational offset to align pupil labs gaze with the eye sockets gaze+ gaze_rotated_by_guess_fr_xyz = [+ offset_rotation_matrix_guess @ original_gaze_fr_xyz[:, this_frame_number] for+ this_frame_number in+ range(original_gaze_fr_xyz.shape[1])]+ gaze_rotated_by_guess_fr_xyz = np.array(gaze_rotated_by_guess_fr_xyz)++ # THEN rotate gaze vector by eye_socket_rotation_matrix to align it with the eye sockets gaze direction (i think)+ gaze_fr_xyz = [+ np.transpose(head_rotation_matricies[this_frame_number]) @ gaze_rotated_by_guess_fr_xyz[+ this_frame_number, :] for+ this_frame_number in+ range(gaze_rotated_by_guess_fr_xyz.shape[0])]+ gaze_fr_xyz = np.array(gaze_fr_xyz)++ # 6 - translate gaze vector back to the eye socket (by adding eyeball center xyz)+ gaze_laser_fr_xyz = np.empty(gaze_fr_xyz.shape)+ gaze_laser_fr_xyz[:, 0] = gaze_fr_xyz[:, 0] + gaze_vector_start_point_fr_xyz[:, 0]+ gaze_laser_fr_xyz[:, 1] = gaze_fr_xyz[:, 1] + gaze_vector_start_point_fr_xyz[:, 1]+ gaze_laser_fr_xyz[:, 2] = gaze_fr_xyz[:, 2] + gaze_vector_start_point_fr_xyz[:, 2]++ return gaze_laser_fr_xyzdiff --git a/jon_scratch/sleap/__init__.py b/jon_scratch/sleap/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/jon_scratch/sleap/load_sleap_tracks_to_blender.py b/jon_scratch/sleap/load_sleap_tracks_to_blender.pynew file mode 100644index 00000000..e48ed557--- /dev/null+++ b/jon_scratch/sleap/load_sleap_tracks_to_blender.py@@ -0,0 +1,37 @@+import bpy+import numpy as np++print("loading gaze data as empties")++path_to_sleap_xyz_npy_path = r"C:\Users\jonma\Dropbox\FreeMoCapProject\FreeMocap_Data\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0\DataArrays\sleap_3d_points.npy"+sleap_fr_mar_xyz = np.load(path_to_sleap_xyz_npy_path)+++print(f'shape of sleap_fr_mar_xyz: {sleap_fr_mar_xyz.shape}')++sleap_fr_mar_x = sleap_fr_mar_xyz[:,:,0]*.001+sleap_fr_mar_y = sleap_fr_mar_xyz[:,:,1]*.001+sleap_fr_mar_z = sleap_fr_mar_xyz[:,:,2]*.001++#%%+print(f'loading sleap_data...')+++for this_sleap_track_num in range(sleap_fr_mar_xyz.shape[1]):+ bpy.ops.object.empty_add(type='PLAIN_AXES')+ this_empty = bpy.context.active_object+ print(this_empty)+ for frame_num in range(sleap_fr_mar_xyz.shape[0]):+ if frame_num % 100 == 0:+ print(f'frame {frame_num}')++ this_empty.location = (sleap_fr_mar_x[frame_num, this_sleap_track_num],+ sleap_fr_mar_y[frame_num, this_sleap_track_num],+ sleap_fr_mar_z[frame_num, this_sleap_track_num])+ this_empty.scale = (0.1, 0.1, .1)++ # this_empty.name = 'right_eye_gaze'+ bpy.context.view_layer.update()+ this_empty.keyframe_insert(data_path='location',frame=frame_num)++print('done!')\ No newline at end of filediff --git a/jon_scratch/sleap/sleap_freemocap_processing copy.ipynb b/jon_scratch/sleap/sleap_freemocap_processing copy.ipynbnew file mode 100644index 00000000..ef40f088--- /dev/null+++ b/jon_scratch/sleap/sleap_freemocap_processing copy.ipynb@@ -0,0 +1,505 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "# notebook to post-process results from SLEAP into freemocap format\n",+ "\n",+ "expected processing steps:\n",+ "\n",+ "1 - load data from SLEAP h5's (one per video, produced by SLEAP's GUI process)\n",+ "2 - Reformat into standard freemocap 2d npy format (dimensions: camera, frame, marker, XY)\n",+ "2a,b - save and test by producing annotated videos\n",+ "3 - triangulate using anipose triangulation\n",+ "4 - save triangulation results to freemocap formated 3d npy (frame, marker, XYZ) (and another npy for reprojection error)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "ename": "ModuleNotFoundError",+ "evalue": "No module named 'sleap'",+ "output_type": "error",+ "traceback": [+ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",+ "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",+ "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_61232\\3622061885.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mpathlib\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mPath\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0maniposelib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcameras\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mCameraGroup\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0msleap\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",+ "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sleap'"+ ]+ }+ ],+ "source": [+ "import numpy as np\n",+ "import h5py\n",+ "from pathlib import Path\n",+ "from aniposelib.cameras import CameraGroup\n",+ "import sleap"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": []+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "session_id = 'sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0'\n",+ "\n",+ "freemocap_data_folder = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data')\n",+ "\n",+ "session_folder_path = freemocap_data_folder / session_id\n",+ "\n",+ "calibration_file_name = session_id + '_calibration.toml'\n",+ "camera_calibration_file_path = session_folder_path / calibration_file_name\n",+ "\n",+ "sleap_data_path = session_folder_path / 'sleap_data'\n",+ "sleap_h5_files_path = sleap_data_path / 'output_h5_files'\n",+ "\n",+ "sleap_predictions_file = sleap_data_path / 'labels.v000.slp'"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {},+ "outputs": [+ {+ "ename": "NameError",+ "evalue": "name 'sleap' is not defined",+ "output_type": "error",+ "traceback": [+ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",+ "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)",+ "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_61232\\726452517.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0msleap_labels_object\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msleap\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mload_file\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msleap_predictions_file\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[0msleap_labels_object\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",+ "\u001b[1;31mNameError\u001b[0m: name 'sleap' is not defined"+ ]+ }+ ],+ "source": [+ "sleap_labels_object = sleap.load_file(str(sleap_predictions_file))\n",+ "sleap_labels_object"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "anipose_camera_calibration_object = CameraGroup.load(camera_calibration_file_path)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "===filename===\n",+ "C:\\Users\\jonma\\Dropbox\\FreeMoCapProject\\FreeMocap_Data\\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0\\sleap_data\\output_h5_files\\labels_v000_analysis_video_0.h5\n",+ "\n",+ "===HDF5 datasets===\n",+ "['edge_inds', 'edge_names', 'instance_scores', 'node_names', 'point_scores', 'track_names', 'track_occupancy', 'tracking_scores', 'tracks']\n",+ "\n",+ "===locations data shape===\n",+ "(5090, 10, 2, 1)\n",+ "\n",+ "===nodes===\n",+ "0: red_ball\n",+ "1: green_ball\n",+ "2: orange_ball\n",+ "3: wobble_0\n",+ "4: wobble_1\n",+ "5: wobble_2\n",+ "6: wobble_3\n",+ "7: wobble_4\n",+ "8: wheel_center\n",+ "9: wheel_edge\n",+ "\n",+ "===filename===\n",+ "C:\\Users\\jonma\\Dropbox\\FreeMoCapProject\\FreeMocap_Data\\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0\\sleap_data\\output_h5_files\\labels_v000_analysis_video_1.h5\n",+ "\n",+ "===HDF5 datasets===\n",+ "['edge_inds', 'edge_names', 'instance_scores', 'node_names', 'point_scores', 'track_names', 'track_occupancy', 'tracking_scores', 'tracks']\n",+ "\n",+ "===locations data shape===\n",+ "(5090, 10, 2, 1)\n",+ "\n",+ "===nodes===\n",+ "0: red_ball\n",+ "1: green_ball\n",+ "2: orange_ball\n",+ "3: wobble_0\n",+ "4: wobble_1\n",+ "5: wobble_2\n",+ "6: wobble_3\n",+ "7: wobble_4\n",+ "8: wheel_center\n",+ "9: wheel_edge\n",+ "\n",+ "===filename===\n",+ "C:\\Users\\jonma\\Dropbox\\FreeMoCapProject\\FreeMocap_Data\\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0\\sleap_data\\output_h5_files\\labels_v000_analysis_video_2.h5\n",+ "\n",+ "===HDF5 datasets===\n",+ "['edge_inds', 'edge_names', 'instance_scores', 'node_names', 'point_scores', 'track_names', 'track_occupancy', 'tracking_scores', 'tracks']\n",+ "\n",+ "===locations data shape===\n",+ "(5090, 10, 2, 1)\n",+ "\n",+ "===nodes===\n",+ "0: red_ball\n",+ "1: green_ball\n",+ "2: orange_ball\n",+ "3: wobble_0\n",+ "4: wobble_1\n",+ "5: wobble_2\n",+ "6: wobble_3\n",+ "7: wobble_4\n",+ "8: wheel_center\n",+ "9: wheel_edge\n",+ "\n",+ "===filename===\n",+ "C:\\Users\\jonma\\Dropbox\\FreeMoCapProject\\FreeMocap_Data\\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0\\sleap_data\\output_h5_files\\labels_v000_analysis_video_3.h5\n",+ "\n",+ "===HDF5 datasets===\n",+ "['edge_inds', 'edge_names', 'instance_scores', 'node_names', 'point_scores', 'track_names', 'track_occupancy', 'tracking_scores', 'tracks']\n",+ "\n",+ "===locations data shape===\n",+ "(5090, 10, 2, 1)\n",+ "\n",+ "===nodes===\n",+ "0: red_ball\n",+ "1: green_ball\n",+ "2: orange_ball\n",+ "3: wobble_0\n",+ "4: wobble_1\n",+ "5: wobble_2\n",+ "6: wobble_3\n",+ "7: wobble_4\n",+ "8: wheel_center\n",+ "9: wheel_edge\n",+ "\n",+ "===filename===\n",+ "C:\\Users\\jonma\\Dropbox\\FreeMoCapProject\\FreeMocap_Data\\sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0\\sleap_data\\output_h5_files\\labels_v000_analysis_video_4.h5\n",+ "\n",+ "===HDF5 datasets===\n",+ "['edge_inds', 'edge_names', 'instance_scores', 'node_names', 'point_scores', 'track_names', 'track_occupancy', 'tracking_scores', 'tracks']\n",+ "\n",+ "===locations data shape===\n",+ "(5090, 10, 2, 1)\n",+ "\n",+ "===nodes===\n",+ "0: red_ball\n",+ "1: green_ball\n",+ "2: orange_ball\n",+ "3: wobble_0\n",+ "4: wobble_1\n",+ "5: wobble_2\n",+ "6: wobble_3\n",+ "7: wobble_4\n",+ "8: wheel_center\n",+ "9: wheel_edge\n",+ "\n"+ ]+ }+ ],+ "source": [+ "\n",+ "\n",+ "per_video_labels_fr_point_xy = []\n",+ "per_video_label_names = []\n",+ "for this_h5_file in sleap_h5_files_path.glob('*.h5'):\n",+ " with h5py.File(this_h5_file, \"r\") as f:\n",+ " dset_names = list(f.keys())\n",+ " locations = f[\"tracks\"][:].T\n",+ " node_names = [n.decode() for n in f[\"node_names\"][:]]\n",+ "\n",+ " per_video_labels_fr_point_xy.append(locations)\n",+ " per_video_label_names.append(node_names)\n",+ " print(\"===filename===\")\n",+ " print(this_h5_file)\n",+ " print()\n",+ "\n",+ " print(\"===HDF5 datasets===\")\n",+ " print(dset_names)\n",+ " print()\n",+ "\n",+ " print(\"===locations data shape===\")\n",+ " print(locations.shape)\n",+ " print()\n",+ "\n",+ " print(\"===nodes===\")\n",+ " for i, name in enumerate(node_names):\n",+ " print(f\"{i}: {name}\")\n",+ " print()\n",+ "\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "number_of_videos = len(per_video_labels_fr_point_xy)\n",+ "number_of_frames = len(per_video_labels_fr_point_xy[0])\n",+ "number_of_markers = len(per_video_labels_fr_point_xy[0][0])\n",+ "sleap2d_nCams_nFrames_nImgPts_XY = np.empty((number_of_videos, number_of_frames, number_of_markers, 2))"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "put data into freemocap 2d_npy format"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "data": {+ "text/plain": [+ "(5, 5090, 10, 2)"+ ]+ },+ "execution_count": 6,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "for this_video_number in range(len(per_video_labels_fr_point_xy)):\n",+ " sleap2d_nCams_nFrames_nImgPts_XY[this_video_number,:,:,:] = np.squeeze(per_video_labels_fr_point_xy[this_video_number])\n",+ "\n",+ "sleap2d_nCams_nFrames_nImgPts_XY.shape"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "load anipose calibration cgroup whosit"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Reconstructing 3d points...\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|βββββββββββββββββββββββββ| 50900/50900 [00:08<00:00, 6245.40it/s]\n"+ ]+ }+ ],+ "source": [+ "\n",+ "sleap_flattened_nCams_nTotalPoints_XY = sleap2d_nCams_nFrames_nImgPts_XY.reshape(number_of_videos, -1, 2) # reshape data to collapse across 'frames' so it becomes [numCams, numFrames*numPoints, XY]\n",+ "\n",+ "print('Reconstructing 3d points...')\n",+ "sleap_data3d_flat = anipose_camera_calibration_object.triangulate(sleap_flattened_nCams_nTotalPoints_XY, progress=True)\n",+ "\n",+ "sleap_data_reprojerr_flat = anipose_camera_calibration_object.reprojection_error( sleap_data3d_flat, sleap_flattened_nCams_nTotalPoints_XY, mean=True)\n",+ "\n",+ "##return:\n",+ "sleap_fr_mar_xyz = sleap_data3d_flat.reshape(number_of_frames, number_of_markers, 3)\n",+ "sleap_fr_mar_reprojectionError = sleap_data_reprojerr_flat.reshape(number_of_frames, number_of_markers)\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "sleap_fr_mar_xyz.shape: (5090, 10, 3)\n",+ "sleap_fr_mar_reprojectionError.shape: (5090, 10)\n"+ ]+ }+ ],+ "source": [+ "\n",+ "print(f'sleap_fr_mar_xyz.shape: {sleap_fr_mar_xyz.shape}')\n",+ "print(f'sleap_fr_mar_reprojectionError.shape: {sleap_fr_mar_reprojectionError.shape}')"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Using matplotlib backend: TkAgg\n"+ ]+ }+ ],+ "source": [+ "import matplotlib.pyplot as plt\n",+ "%matplotlib \n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 10,+ "metadata": {},+ "outputs": [],+ "source": [+ "plt.close('all')\n",+ "fig = plt.figure()\n",+ "\n",+ "for this_marker_number in range(sleap_fr_mar_xyz.shape[1]):\n",+ " ax = fig.add_subplot(5,2,this_marker_number+1)\n",+ " ax.plot(sleap_fr_mar_xyz[:,this_marker_number,:])"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "save sleap data to `DataArrays` folder "+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 11,+ "metadata": {},+ "outputs": [],+ "source": [+ "session_data_arrays_path = session_folder_path / 'DataArrays'\n",+ "\n",+ "sleap_3d_save_path = session_data_arrays_path / 'sleap_3d_points.npy'\n",+ "sleap_reproj_err_save_path = session_data_arrays_path / 'sleap_reprojection_error.npy'\n",+ "label_name_csv_path = session_data_arrays_path / 'sleap_label_names.csv'\n",+ "\n",+ "np.save(sleap_3d_save_path, sleap_fr_mar_xyz)\n",+ "np.save(sleap_reproj_err_save_path, sleap_fr_mar_reprojectionError)\n",+ "\n",+ "import csv\n",+ "with open(label_name_csv_path,'w') as label_name_file:\n",+ " wr = csv.writer(label_name_file)\n",+ " wr.writerow(node_names)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 17,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "(5090,)"+ ]+ },+ "execution_count": 17,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "sleap_fr_mar_xyz[:,5,0].shape"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": []+ }+ ],+ "metadata": {+ "interpreter": {+ "hash": "122c9931a57dd91c579842728ee2ee1c7ffefd770186c077f442b22e7e57f48e"+ },+ "kernelspec": {+ "display_name": "Python 3.7.13 ('freemocap-env')",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.7.13"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 0+}diff --git a/jon_scratch/sleap/sleap_freemocap_processing.ipynb b/jon_scratch/sleap/sleap_freemocap_processing.ipynbnew file mode 100644index 00000000..07072623--- /dev/null+++ b/jon_scratch/sleap/sleap_freemocap_processing.ipynb@@ -0,0 +1,529 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "# notebook to post-process results from SLEAP into freemocap format\n",+ "\n",+ "expected processing steps:\n",+ "\n",+ "1 - load data from SLEAP h5's (one per video, produced by SLEAP's GUI process)\n",+ "2 - Reformat into standard freemocap 2d npy format (dimensions: camera, frame, marker, XY)\n",+ "2a,b - save and test by producing annotated videos\n",+ "3 - triangulate using anipose triangulation\n",+ "4 - save triangulation results to freemocap formated 3d npy (frame, marker, XYZ) (and another npy for reprojection error)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "import numpy as np\n",+ "import h5py\n",+ "from pathlib import Path\n",+ "from aniposelib.cameras import CameraGroup\n"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "session_id = 'sesh_2022-05-07_17_15_05_pupil_wobble_juggle_0'\n",+ "\n",+ "freemocap_data_folder = Path('C:/Users/jonma/Dropbox/FreeMoCapProject/FreeMocap_Data')\n",+ "\n",+ "session_folder_path = freemocap_data_folder / session_id\n",+ "\n",+ "calibration_file_name = session_id + '_calibration.toml'\n",+ "camera_calibration_file_path = session_folder_path / calibration_file_name\n",+ "\n",+ "sleap_data_path = session_folder_path / 'sleap_data'\n",+ "sleap_h5_files_path = sleap_data_path / 'output_h5_files'\n",+ "\n",+ "sleap_predictions_file = sleap_data_path / 'labels.v000.slp'"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "anipose_camera_calibration_object = CameraGroup.load(camera_calibration_file_path)"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "\n",+ "\n",+ "per_video_labels_fr_point_xy = []\n",+ "per_video_label_names = []\n",+ "for this_h5_file in sleap_h5_files_path.glob('*.h5'):\n",+ " with h5py.File(this_h5_file, \"r\") as f:\n",+ " dset_names = list(f.keys())\n",+ " locations = f[\"tracks\"][:].T\n",+ " node_names = [n.decode() for n in f[\"node_names\"][:]]\n",+ "\n",+ " per_video_labels_fr_point_xy.append(locations)\n",+ " per_video_label_names.append(node_names)\n",+ " print(\"===filename===\")\n",+ " print(this_h5_file)\n",+ " print()\n",+ "\n",+ " print(\"===HDF5 datasets===\")\n",+ " print(dset_names)\n",+ " print()\n",+ "\n",+ " print(\"===locations data shape===\")\n",+ " print(locations.shape)\n",+ " print()\n",+ "\n",+ " print(\"===nodes===\")\n",+ " for i, name in enumerate(node_names):\n",+ " print(f\"{i}: {name}\")\n",+ " print()\n",+ "\n",+ "\n"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "number_of_videos = len(per_video_labels_fr_point_xy)\n",+ "number_of_frames = len(per_video_labels_fr_point_xy[0])\n",+ "number_of_markers = len(per_video_labels_fr_point_xy[0][0])\n",+ "sleap2d_nCams_nFrames_nImgPts_XY = np.empty((number_of_videos, number_of_frames, number_of_markers, 2))"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "markdown",+ "source": [+ "put data into freemocap 2d_npy format"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%% md\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "for this_video_number in range(len(per_video_labels_fr_point_xy)):\n",+ " sleap2d_nCams_nFrames_nImgPts_XY[this_video_number,:,:,:] = np.squeeze(per_video_labels_fr_point_xy[this_video_number])\n",+ "\n",+ "sleap2d_nCams_nFrames_nImgPts_XY.shape"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "markdown",+ "source": [+ "load anipose calibration cgroup whosit"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%% md\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "\n",+ "sleap_flattened_nCams_nTotalPoints_XY = sleap2d_nCams_nFrames_nImgPts_XY.reshape(number_of_videos, -1, 2) # reshape data to collapse across 'frames' so it becomes [numCams, numFrames*numPoints, XY]\n",+ "\n",+ "print('Reconstructing 3d points...')\n",+ "sleap_data3d_flat = anipose_camera_calibration_object.triangulate(sleap_flattened_nCams_nTotalPoints_XY, progress=True)\n",+ "\n",+ "sleap_data_reprojerr_flat = anipose_camera_calibration_object.reprojection_error( sleap_data3d_flat, sleap_flattened_nCams_nTotalPoints_XY, mean=True)\n",+ "\n",+ "##return:\n",+ "sleap_fr_mar_xyz = sleap_data3d_flat.reshape(number_of_frames, number_of_markers, 3)\n",+ "sleap_fr_mar_reprojectionError = sleap_data_reprojerr_flat.reshape(number_of_frames, number_of_markers)\n"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "\n",+ "print(f'sleap_fr_mar_xyz.shape: {sleap_fr_mar_xyz.shape}')\n",+ "print(f'sleap_fr_mar_reprojectionError.shape: {sleap_fr_mar_reprojectionError.shape}')"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "import matplotlib.pyplot as plt\n",+ "%matplotlib \n"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "plt.close('all')\n",+ "fig = plt.figure()\n",+ "\n",+ "for this_marker_number in range(sleap_fr_mar_xyz.shape[1]):\n",+ " ax = fig.add_subplot(5,2,this_marker_number+1)\n",+ " ax.plot(sleap_fr_mar_xyz[:,this_marker_number,:])"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "markdown",+ "source": [+ "save sleap data to `DataArrays` folder "+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%% md\n"+ }+ }+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "outputs": [],+ "source": [+ "session_data_arrays_path = session_folder_path / 'DataArrays'\n",+ "\n",+ "sleap_3d_save_path = session_data_arrays_path / 'sleap_3d_points.npy'\n",+ "sleap_reproj_err_save_path = session_data_arrays_path / 'sleap_reprojection_error.npy'\n",+ "label_name_csv_path = session_data_arrays_path / 'sleap_label_names.csv'\n",+ "\n",+ "np.save(sleap_3d_save_path, sleap_fr_mar_xyz)\n",+ "np.save(sleap_reproj_err_save_path, sleap_fr_mar_reprojectionError)\n",+ "\n",+ "import csv\n",+ "with open(label_name_csv_path,'w') as label_name_file:\n",+ " wr = csv.writer(label_name_file)\n",+ " wr.writerow(node_names)"+ ],+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ }+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "put data into freemocap 2d_npy format"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 12,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "data": {+ "text/plain": [+ "(5, 5090, 10, 2)"+ ]+ },+ "execution_count": 12,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "for this_video_number in range(len(per_video_labels_fr_point_xy)):\n",+ " sleap2d_nCams_nFrames_nImgPts_XY[this_video_number,:,:,:] = np.squeeze(per_video_labels_fr_point_xy[this_video_number])\n",+ "\n",+ "sleap2d_nCams_nFrames_nImgPts_XY.shape"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "load anipose calibration cgroup whosit"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 13,+ "metadata": {+ "collapsed": false,+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Reconstructing 3d points...\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|βββββββββββββββββββββββββ| 50900/50900 [00:07<00:00, 6795.51it/s]\n"+ ]+ }+ ],+ "source": [+ "\n",+ "sleap_flattened_nCams_nTotalPoints_XY = sleap2d_nCams_nFrames_nImgPts_XY.reshape(number_of_videos, -1, 2) # reshape data to collapse across 'frames' so it becomes [numCams, numFrames*numPoints, XY]\n",+ "\n",+ "print('Reconstructing 3d points...')\n",+ "sleap_data3d_flat = anipose_camera_calibration_object.triangulate(sleap_flattened_nCams_nTotalPoints_XY, progress=True)\n",+ "\n",+ "sleap_data_reprojerr_flat = anipose_camera_calibration_object.reprojection_error( sleap_data3d_flat, sleap_flattened_nCams_nTotalPoints_XY, mean=True)\n",+ "\n",+ "##return:\n",+ "sleap_fr_mar_xyz = sleap_data3d_flat.reshape(number_of_frames, number_of_markers, 3)\n",+ "sleap_fr_mar_reprojectionError = sleap_data_reprojerr_flat.reshape(number_of_frames, number_of_markers)\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 14,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "sleap_fr_mar_xyz.shape: (5090, 10, 3)\n",+ "sleap_fr_mar_reprojectionError.shape: (5090, 10)\n"+ ]+ }+ ],+ "source": [+ "\n",+ "print(f'sleap_fr_mar_xyz.shape: {sleap_fr_mar_xyz.shape}')\n",+ "print(f'sleap_fr_mar_reprojectionError.shape: {sleap_fr_mar_reprojectionError.shape}')"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 15,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Using matplotlib backend: TkAgg\n"+ ]+ }+ ],+ "source": [+ "import matplotlib.pyplot as plt\n",+ "%matplotlib \n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 22,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "plt.close('all')\n",+ "fig = plt.figure()\n",+ "\n",+ "for this_marker_number in range(sleap_fr_mar_xyz.shape[1]):\n",+ " ax = fig.add_subplot(5,2,this_marker_number+1)\n",+ " ax.plot(sleap_fr_mar_xyz[:,this_marker_number,:])"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "pycharm": {+ "name": "#%% md\n"+ }+ },+ "source": [+ "save sleap data to `DataArrays` folder "+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": [+ "session_data_arrays_path = session_folder_path / 'DataArrays'\n",+ "\n",+ "sleap_3d_save_path = session_data_arrays_path / 'sleap_3d_points.npy'\n",+ "sleap_reproj_err_save_path = session_data_arrays_path / 'sleap_reprojection_error.npy'\n",+ "label_name_csv_path = session_data_arrays_path / 'sleap_label_names.csv'\n",+ "\n",+ "np.save(sleap_3d_save_path, sleap_fr_mar_xyz)\n",+ "np.save(sleap_reproj_err_save_path, sleap_fr_mar_reprojectionError)\n",+ "\n",+ "import csv\n",+ "with open(label_name_csv_path,'w') as label_name_file:\n",+ " wr = csv.writer(label_name_file)\n",+ " wr.writerow(node_names)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": []+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {+ "pycharm": {+ "name": "#%%\n"+ }+ },+ "outputs": [],+ "source": []+ }+ ],+ "metadata": {+ "interpreter": {+ "hash": "122c9931a57dd91c579842728ee2ee1c7ffefd770186c077f442b22e7e57f48e"+ },+ "kernelspec": {+ "display_name": "Python 3.7.13 ('freemocap-env')",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.7.13"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 0+}\ No newline at end of filediff --git a/jon_scratch/tensorflow_gpu_tester.py b/jon_scratch/tensorflow_gpu_tester.pynew file mode 100644index 00000000..583ba6e8--- /dev/null+++ b/jon_scratch/tensorflow_gpu_tester.py@@ -0,0 +1,2 @@+import tensorflow as tf+print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))