From 3671f4e2c92f28319e9c8e44071604e98d8d783f Mon Sep 17 00:00:00 2001 From: Blaise Thompson Date: Mon, 16 Apr 2018 08:45:36 -0500 Subject: 2018-04-16 08:45 --- processing/chapter.tex | 204 ++++++++++++++++++++++++++++--------------------- 1 file changed, 119 insertions(+), 85 deletions(-) (limited to 'processing/chapter.tex') diff --git a/processing/chapter.tex b/processing/chapter.tex index 0d89055..1d11788 100644 --- a/processing/chapter.tex +++ b/processing/chapter.tex @@ -507,8 +507,8 @@ WrightTools. % \texttt{wt.data.from\_shimamdzu} & Shimadzu brand spectrometers. \\ \hline \texttt{wt.data.from\_Tensor27} & Tensor 27 FT-IR. \\ \hline \end{tabular} - \caption[CAPTION TODO]{ - CAPTION TODO + \caption{ + WrightTools data import functions. } \label{pro:tab:from_functions} \end{table} @@ -586,10 +586,11 @@ After importing and manipulating data, one typically wants to create a plot. % The artists sub-package contains everything users need to plot their data objects. % This includes both ``quick'' artists, which generate simple plots as quickly as possible, and a full figure layout toolkit that allows users to generate full publication quality figures. % -It also includes ``specialty'' artists which are made to perform certain popular plotting -operations, as I will describe below. % +It also includes ``specialty'' artists which are made to perform certain common plotting +operations. % -Currently the artists sub-package is built on-top of the wonderful matplotlib library. % +Currently the artists sub-package is built on-top of the wonderful matplotlib library +\cite{HunterJohnD2007a}. % In the future, other libraries (e.g. Mayavi \cite{Mayavi}), may be incorporated. % \subsection{Strategies for 2D visualization} % --------------------------------------------------- @@ -598,19 +599,36 @@ Representing two-dimensional data is an important capability for WrightTools, so discussion about how such representations work is warranted. % WrightTools data is typically very structured, with values recorded at a grid of positions. % To represent two-dimensional data, then, WrightTools needs to map the values onto a color axis. % -There are better and worse choices of colormap... % TODO: elaborate +When doing this mapping, choosing an appropriate colormap and interpolation type is crucial. % \subsubsection{Colormap} +For the purposes of this discussion, there are two types of colormaps: 1. qualitative 2. +perceptual. % +Qualitative colormaps have random orderings of color. % +They are best used to represent unordered things, and they typically have high dynamic range. % +Perceptual colormaps are monotonic in lightness, and are best at representing ordered information +(like signal levels in MR-CMDS). % + +Historically the Wright Group has used a qualitative colormap for all plotting. % \autoref{pro:fig:cmaps} shows the red, green, and blue components of four different colormaps. % -The black line is the net brightness of each color (larger value means lighter color). % +The black line is the net lightness of each color (larger value means lighter color). % Below each figure is a gray-scale representation of the corresponding colormap. % The r, g, and b components are scaled according to human perception. % TODO: values, from where -The traditional Wright Group colormap (derived from jet) is shown first. % +The traditional Wright Group colormap (derived from jet) is shown first. i% Following are two perceptual colormaps, cubehelix from \texcite{GreenDA2009a}, and viridis, the new matplotlib default. WrightTools uses the algorithm from \textcite{GreenDA2009a} to define a custom cubehelix colormap with good perceptual properties and familiar Wright Group coloration. % +The lightness is not perfectly straight as in viridis and cubehelix, but it is monotonic at +least. % + +\autoref{pro:fig:cmap_comparison} compares the new (top row) and old (bottom row) colormaps in full +color and greyscale. % +While the old colormap shows a greater dynamic range, the new colormap is more perceptually +uniform. % +In my opinion, users should use logarithmic scaling of an appropriate base if more dynamic range is +required, rather than using a non-perceptual colormap. % \subsubsection{Interpolation type} @@ -635,22 +653,27 @@ plots. % \includegraphics[scale=0.5]{"processing/cubehelix_cmap"} \includegraphics[scale=0.5]{"processing/viridis_cmap"} \includegraphics[scale=0.5]{"processing/default_cmap"} - \caption[CAPTION TODO]{ - CAPTION TODO} + \caption[Colormap lightness.]{ + Comparison of four different colormaps. + R, G, and B components are each shown. + Black line is human-perceived lightness. + } \label{pro:fig:cmaps} \end{figure} \begin{figure} \includegraphics[width=\textwidth]{"processing/cmap_comparison"} - \caption[CAPTION TODO]{ - CAPTION TODO} + \caption{ + Colormap comparison. + } \label{pro:fig:cmap_comparison} \end{figure} \begin{figure} \includegraphics[width=\textwidth]{"processing/fill_types"} - \caption[CAPTION TODO]{ - CAPTION TODO} + \caption{ + Interpolation types in 2D plotting. + } \label{pro:fig:fill_types} \end{figure} @@ -684,39 +707,40 @@ These functionalities are derived from \python{wt.Data.chop}, discussed further \begin{figure} \includegraphics[width=0.5\textwidth]{"processing/quick1D 000"} \includepython{"processing/quick1D.py"} - \caption[CAPTION TODO]{ - CAPTION TODO} + \caption[quick1D example.]{ + Minimal example of the \python{quick1D} function. + } \label{pro:fig:quick1D} \end{figure} \begin{figure} \includegraphics[width=0.5\textwidth]{"processing/quick2D 000"} \includepython{"processing/quick2D.py"} - \caption[CAPTION TODO]{ - CAPTION TODO} + \caption[quick2D example.]{ + Minimal example of the \python{quick2D} function. + } \label{pro:fig:quick2D} \end{figure} % TODO: signed data (with and without dynamic_range=True) -\subsection{API} % ------------------------------------------------------------------------------- +% \subsection{API} % ------------------------------------------------------------------------------- -The artists sub-package offers a thin wrapper on the default matplotlib object-oriented figure -creation API. % -The wrapper allows WrightTools to add the following capabilities on top of matplotlib: -\begin{ditemize} - \item More consistent multi-axes figure layout. - \item Ability to plot data objects directly. -\end{ditemize} -Each of these is meant to lower the barrier to plotting data. % -Without going into every detail of matplotlib figure generation capabilities, this section -introduces the unique strategy that the WrightTools wrapper takes. % +% The artists sub-package offers a thin wrapper on the default matplotlib object-oriented figure +% creation API. % +% The wrapper allows WrightTools to add the following capabilities on top of matplotlib: +% \begin{ditemize} +% \item More consistent multi-axes figure layout. +% \item Ability to plot data objects directly. +% \end{ditemize} +% Each of these is meant to lower the barrier to plotting data. % +% Without going into every detail of matplotlib figure generation capabilities, this section +% introduces the unique strategy that the WrightTools wrapper takes. % -% \subsection{Gotchas} % ------------------------------------------------------------------------- +% % \subsection{Gotchas} % ------------------------------------------------------------------------- % TODO: mention gotcha of apparently narrowing linewidths with wigners (how to READ colormaps) -\clearpage \section{Variables and channels} % =============================================================== Data objects are made up of many component channels and variables, each array having the same @@ -728,7 +752,7 @@ may not have even moved across some (or any) dimensions. % To avoid unnecessarily large arrays, WrightTools allows Channels and Variables to have different sizes than the parent data. % As an example, consider the following object. -\begin{codefragment}{bash} % TODO: need to use bash here because of box charachters :-( +\begin{codefragment}{bash} >>> import WrightTools as wt; from WrightTools import datasets >>> data = wt.data.from_COLORS(datasets.COLORS.v2p1_MoS2_TrEE_movie) >>> data.print_tree() @@ -763,18 +787,34 @@ propagated. % \section{Axes} \label{pro:sec:axes} % ============================================================ -The axes have the joint shape of their component variables. % -Although not shown in this example, channels also may have axes with length 1. +Axes are the primary interface to coordinate positions in WrightTools. % +Axes are \emph{not} arrays, although they do behave like arrays. % +They are merely \emph{interfaces} into the information stored in one or more variables. % + +Each axis has an expression, like \python{'w1'}, \python{'d1=d2'}, \python{2*w3} or +\python{'w1+w2-wm'}. % +These expressions describe an unambiguous mathematical operation involving one or more +variables. % +The axis computes these expressions on the fly when needed, giving users real flexability over how +they would like to represent and process their results. % Axes, variables, and channels are array-likes, so they support slicing operations. % +The axes have the joint shape of their component variables. % In addition, all three classes have \python{points} and \python{full} attributes that return the squeezed and broadcasted array, respectively. % +One of the coolest features enabled by this approach to axes is \emph{transformation}. % +\autoref{pro:fig:transform} shows how easy it is to transform data from one coordinate system into +another. % +Transforming is essentially free, since axes compute coordinates on the fly. % + \begin{figure} - \includegraphics[width=\textwidth]{"processing/fringes_transform"} + \includegraphics[width=0.8\textwidth]{"processing/fringes_transform"} \includepython{"processing/fringes_transform.py"} - \caption[CAPTION TODO]{ - CAPTION TODO} + \caption{ + Example using transform. + } + \label{pro:fig:transform} \end{figure} \section{Math} % ================================================================================= @@ -783,8 +823,6 @@ Now that we know the basics of how the WrightTools \python{Data} class stores da some data manipulation. % Let's start with some elementary algebra. % -% TODO: mention chunkwise strategy - \subsection{In-place operators} \label{pro:sec:in_place} % --------------------------------------- In Python, operators are symbols that carry out some computation. % @@ -847,10 +885,11 @@ Variables also support in-place operators. % \subsection{Clip} % ------------------------------------------------------------------------------ Clip allows users to exclude values outside of a certain range. % +Excluded values can be replaced with a user-specified value or with not-a-number. % This can be particularly useful in cases like fitting. % -See section ... for an example. % TODO: link to section +See \autoref{pro:sec:fit} for an example. % -It's also useful for when noise in a certain region of a spectrum obscures useful data... +It's also useful for when noise in a certain region of a spectrum obscures useful data. % Particularly true for normalized and signed data. % \subsection{Symmetric root} % -------------------------------------------------------------------- @@ -885,7 +924,7 @@ keyword arguments. % These may be slightly faster than \python{channel.log(base=10)} and \python{channel.log(base=2)}. % -\subsection{Level} % ----------------------------------------------------------------------------- +% \subsection{Level} % ----------------------------------------------------------------------------- % TODO: figure from wright.tools @@ -894,6 +933,7 @@ These may be slightly faster than \python{channel.log(base=10)} and Trim uses statistical treatment to find and remove outliers from a dataset. % It is useful in cases where the naive strategy employed by \python{wt.Channel.clip} is not sufficient, and when preparing for fitting. % +See \autoref{pro:fig:outlier} for an example of outlier detection. % Currently \python{trim} only supports one statistical treatment: the z-test. % Z-testing compares each pixel to its multidimensional neighborhood of pixels. % @@ -902,8 +942,6 @@ neighborhood standard deviation) it is either masked, replaced with \python{np.n with the neighborhood mean. % All outliers are found before any outliers are modified, so the algorithm is not directional. % -% TODO: z-test citation - \python{wt.Channel.trim} can easily be enhanced with other statistical methods as needed. % \subsection{Smooth} % ---------------------------------------------------------------------------- @@ -922,13 +960,20 @@ can confidently smooth collected data in post to achieve clean results. % This strategy is similar to that accomplished in time domain CMDS where a low-pass filter is applied on the very high resolution raw data. % +\begin{figure} + \includegraphics[width=0.8\textwidth]{"processing/outlier"} + \caption{ + Outlier detection using a z-test. + } + \label{pro:fig:outlier} +\end{figure} + % TODO: figure: example of smoothed data -\clearpage \section{Dimensionality manipulation} % ========================================================== WrightTools offers several strategies for reducing the dimensionality of a data object. % -Also consider using the fit sub-package. % TODO: more info, link to section +Also consider using the fit sub-package described in \autoref{pro:sec:fit}. % \subsection{Chop} \label{pro:sec:chop} % --------------------------------------------------------- @@ -941,7 +986,7 @@ This dataset can be chopped to it's component 2D \python{('wm'', 'w1')} spectra. >>> import WrightTools as wt; from WrightTools import datasets >>> data = wt.data.from_PyCMDS(datasets.PyCMDS.wm_w2_w1_000) data created at /tmp/lzyjg4au.wt5::/ - axes ('wm', 'w2', 'w1') + axes ('wm'', 'w2', 'w1') shape (35, 11, 11) >>> chopped = data.chop('wm', 'w1') chopped data into 11 piece(s) in ('wm', 'w1'') @@ -977,8 +1022,8 @@ chopped data into 1 piece(s) in ('wm', 'w1') >>> chopped.w2.points array([1580.0]) \end{codefragment} -Note the [0]... % TODO -This same syntax used in artists... % TODO +% Note the [0]... % TODO +% This same syntax used in artists... % TODO \subsection{Collapse} % -------------------------------------------------------------------------- @@ -1016,8 +1061,9 @@ To do this, the variable and channel names must agree. % \begin{figure} \includegraphics[width=\textwidth]{"processing/join_example"} - \caption[CAPTION TODO]{ - CAPTION TODO} + \caption{ + Join example. + } \label{pro:fig:join_example} \end{figure} @@ -1032,7 +1078,7 @@ Rather than attempt to visualize \python{w1, w2} at some specific value of \pyth powerful to instead consider the parameters (amplitude, offset, and time constant) of an exponential fit at each \python{w1, w2} coordinate. % On a more practical note, this kind of slice-by-slice dimensionality reduction via fitting can -greatly simplify automated instrumental calibration (see ...) % TODO: link to opa chapter +greatly simplify automated instrumental calibration (see \autoref{cha:opa}). % WrightTools employs some simple tricks to enable these kind of fit operations, described here. % % TODO: consider inserting figures that demonstrate this story (need to use wt2?) @@ -1043,22 +1089,16 @@ One challenge of slice-by-slice fitting is making a good intial guess to optimiz It is not tractable to ask the user to provide a guess for each slice, so some kind of reasonable automated guessing must be used. % WrightTools ``function'' objects are self contained describers of a particular function. % -As an example, consider the \python{wt.fit.Expontial} class... -It has parameters... -Fit... -Evaluate... -Guess... - -Can be used directly... - -[USERS CAN WRITE THEIR OWN FUNCTION OBJECTS] +As an example, consider the \python{wt.fit.Expontial} class. +It can be used directly, as shown in \autoref{pro:fig:fit_function} \begin{figure} \includegraphics[width=0.5\textwidth]{"processing/fit_function"} \includepython{"processing/fit_function.py"} - \caption[CAPTION TODO]{ - CAPTION TODO + \caption{ + Fitting a Gaussian. } + \label{pro:fig:fit_function} \end{figure} \subsection{Fitter} % ---------------------------------------------------------------------------- @@ -1117,13 +1157,11 @@ the fit parameters, in this case amplitude, tau, and offset. Accordingly, outs i than the original data object. model contains the fit evaluated at each coordinate of the original dataset—it’s really useful for inspecting the quality of your fit procedure. -[ALSO GOOD FOR WORKUP OF TUNING DATA: SEE SECTION ...] - \begin{figure} \includegraphics[width=0.4\textwidth]{"processing/fit_amplitude"} \includegraphics[width=0.4\textwidth]{"processing/fit_tau"} - \caption[CAPTION TODO]{ - CAPTION TODO + \caption{ + Fitting as dimensionality reduction. } \label{pro:fig:fitted_movie} \end{figure} @@ -1197,12 +1235,12 @@ returned to again and again. % Developers can collaborate together to edit the software by making incremental changes that are easy to review. % -WrightTools uses git \cite{git} for source control, and the package is hosted on GitHub -\cite{GitHub, WrightToolsGitHub}. % +WrightTools uses git for source control, and the package is hosted on GitHub +\cite{GitHub}. % As of 2018-04-08, WrightTools has 1,018 commits from seven developers, as shown in \autoref{pro:tab:commits}. % -In addition to simply hosting the git repo, GitHub gives us issue tracking and automated -integration with zenodo. % +In addition to simply hosting the git repository, GitHub gives us issue tracking, continuous +integration, and Zenodo support. % The WrightTools package has a developer controlled version as well, following the semantic versioning convention \cite{SemanticVersioning}. % @@ -1223,9 +1261,8 @@ These attributes can be accessed through python: \python{wt.__version__} and Daniel Kohler & 3 & 113 & 29 \\ \hline Rachel Swedin & 1 & 5,197 & 0 \\ \hline \end{tabular} - \caption[CAPTION TODO]{ + \caption[Commits to WrightTools.]{ Commits to WrightTools by individual, ordered by number of commits. - Wright Stuff is... Note that datasets... } \label{pro:tab:commits} @@ -1260,7 +1297,8 @@ of the tests and receive a report on what failed and why. % If a future programmer unintentionally breaks \python{convert}, the above test will fail and alert her to the unexpected side effect of her modification. % -[PARAGRAPH ABOUT CONTINUOUS INTEGRATION] +WrightTools uses continuous integration services to run unit tests every time a pull request is +made on GitHub. % \subsection{Distribution} \label{pro:sec:distribution} % ------------------------------------------ @@ -1272,8 +1310,7 @@ Luckily for us, distribution is fairly simple within the Python ecosystem. % The same tools that are used to distribute hugely popular packages like numpy are also available for anyone else, including WrightTools. % -The Python Package Index (PyPI), affectionately known as the cheese shop, is the official third -party software repository for Python. % +The Python Package Index (PyPI) is the official third party software repository for Python. % It is community maintained, and supported by the Python Software Foundation and The Python Packaging Group. % As of 2018-04-08 PyPI hosts 134,758 Python packages, all for free. % @@ -1291,10 +1328,9 @@ Since the scientific Python ecosystem has so many non-Python binary dependencies popular choice---especially on Windows where the necessary compilers are not typically pre-installed. % Unlike pip, conda is not tied to a single repository. % -There is the official repository, maintained by Anaconda, the company that develops conda. -[CITE] % +There is the official repository, maintained by Anaconda, the company that develops conda. % There is also the popular conda-forge repository, which is maintained by the community via -GitHub. [CITE] % +GitHub. % WrightTools is on conda-forge: % \begin{codefragment}{bash} conda config --add channels conda-forge @@ -1308,13 +1344,11 @@ using such a thing immediately. % To this end, it is important to have easy to use, searchable documentation with end-users in mind. % -wright.tools - -Built and hosted by readthedocs - -Both master and development versions... +I have built a website for WrightTools documentation at \href{http://wright.tools/}. % +The documentation is built and hosted by Read the Docs. \cite{ReadTheDocs}. % +Both master and development versions are built, so users of both can get documentation that is up +to date for them. % -% \clearpage % \section{Future directions} % ==================================================================== % Single variable decomposition. % -- cgit v1.2.3