# Controlling Variables ## Orthogonal Array Testing A multivariate experimental method popularized by Genichi Taguchi using orthogonal arrays. The test "array" is a list of experiments where multiple variables are tested together in a statistically useful structure which allows all variables to be tested in combination with all others. > an orthogonal array (more specifically, a fixed-level orthogonal array) is a "table" (array) whose entries come from a fixed finite set of symbols (for example, $\set{1,2,...,v}$), arranged in such a way that there is an integer $t$ so that for every selection of $t$ columns of the table, all ordered $t$-tuples of the symbols, formed by taking the entries in each row restricted to these columns, appear the same number of times. \- Orthogonal Array article, Wikipedia An example *fixed-level* orthogonal array of symbols from the set $\set{0,1}$ forms this [[Vector Geometric Algebra|matrix]]: $ \begin{bmatrix}1&1&1 \\ 0&0&1 \\ 1&0&0 \\ 0&1&0\end{bmatrix} $ With a set cardinality of only 2, this appears to be a simple permutation matrix, however at larger cardinalities it becomes clear that there is more going on. In the next example, each column represents a variable to be tested and each entry represents a setting for that variable from the set $\set{H,N,L}$ representing *h*igh, *n*ormal (baseline), and *l*ow values for each of the 3 variables. $ \begin{bmatrix} N&N&N \\ N&L&L \\ N&H&H\ \\ L&N&L\ \\ L&L&H\ \\ L&H&N\ \\ H&N&H\ \\ H&L&N\ \\ H&H&L\ \end{bmatrix} $ ### Tracking Effectiveness It is ideal to have objective measurable criteria broken up among multiple dimensions to determine the success of any set of experiments. This methodology is no different, but doing so may produce additional insights, so keeping track of it in a table is important. Assigning some kind of score to each result, and weighting them as any fitness function against your target, will help produce the best results. ### Calculating Variable Influence Each variable can be tested with multiple settings in this way and their results compared for statistical differences using simple addition of the tests with that setting. To see which setting is most effective, you simply add the scores for the rows with that setting and compare it against the total from each other setting. For example, using the HNL table above: To compare variable 1's influence at setting H you would add the scores together for the first 3 runs. Then the N runs for the second three. And finally for the L runs of the final 3. If you have a half-decent scoring system, then it should show you very quickly if that variable had any notable effect. Additional steps can be used to detect if variables are compounding or conflicting in effectiveness in the same way. ### Rules 1. All variable settings must be tested an equal number of times 2. All settings must be tested against all others settings equally ### Efficiency This allows on average $\sqrt{n}$ where $n$ is the traditional combinatorial matrix which rigorously tests each possible combination in isolation^{[citation needed]}. # References ## Taguchi Methods & Orthogonal Arrays ```cardlink url: https://youtu.be/5oULEuOoRd0 title: "Test Multiple Variables at Once to Optimize Anything" description: "Check out my sponsor Brilliant, free for 30 days (and get 20% off an annual premium subscription!) by using this link: https://brilliant.org/nighthawkIn this..." host: youtu.be favicon: https://www.youtube.com/s/desktop/0fccf9b4/img/favicon_32x32.png image: https://i.ytimg.com/vi/5oULEuOoRd0/maxresdefault.jpg ``` - **Quick reference tables:** https://www.me.psu.edu/cimbala/me345/Lectures/Taguchi_orthogonal_arrays.pdf - https://support.minitab.com/en-us/minitab/21/help-and-how-to/statistical-modeling/doe/supporting-topics/taguchi-designs/taguchi-designs/ - Niel Sloane's Orthogonal Array Sites - [A Library of Orthogonal Arrays](http://neilsloane.com/oadir/) ([old version archived](https://web.archive.org/web/20081210112132/http://www.research.att.com/~njas/oadir/)) - [Sloane](https://scholar.google.com/citations?hl=en&user=daC6bnkAAAAJ)'s work is referenced a lot, as is [Art Owen](https://scholar.google.com/citations?user=owV8q1cAAAAJ)'s - Software libraries for orthogonal arrays - [The Orthogonal Array Package](http://pietereendebak.nl/oapackage/) ([source](https://github.com/eendebakpt/oapackage) in [[C++]]) - [Sage Math's Orthogonal Arrays](http://match.stanford.edu/reference/combinat/sage/combinat/designs/orthogonal_arrays.html) ([source](https://github.com/sagemath/sage/) in [[Python]]) - [OARS](https://github.com/afnanenayet/oars) in [[Rust]] - https://people.sc.fsu.edu/~jburkardt/datasets/oa/oa.html - [Construct Orthogonal Arrays in a Spreadsheet (YouTube)](https://youtu.be/847pIrKgZec) - [Sudoku-like arrays, codes and orthogonality](https://link.springer.com/article/10.1007/s10623-016-0190-y) (paywalled, [archived](https://annas-archive.org/md5/75a88109ab13863570073cb7553df507)) - [Constructing Ordered Orthogonal Arrays via Sudoku](https://annas-archive.org/md5/3e1498380e88a47db4544d6c3d49c9db) ([open access](https://arxiv.org/abs/1309.1680v2), archived) - [Taguch's Orthogonal Arrays Are Classical Designs of Experiments (NIST PDF)](https://nvlpubs.nist.gov/nistpubs/jres/096/jresv96n5p577_A1b.pdf) - [Constructing Orthogonal Arrays (MIT Course Slides)](https://ocw.mit.edu/courses/16-881-robust-system-design-summer-1998/dc71137bc61a68bd65d8416928727c07_l8_orth_arrays.pdf) - https://engineering.purdue.edu/LSM/courses/MEEM5990/ExpDesign_Lecture15.pdf - https://stackoverflow.com/questions/37851038/how-to-create-orthogonal-array - https://en.wikipedia.org/wiki/Taguchi_methods - https://en.wikipedia.org/wiki/Orthogonal_array ## Multivariate Testing (in general) - https://www.oxfordbibliographies.com/display/document/obo-9780199756810/obo-9780199756810-0145.xml - https://s4be.cochrane.org/blog/2021/09/09/multivariate-analysis-an-overview/