Research Design in Modelling & Simulation

Instructor Contact:
Instructor: PK Douglas, PhD
Office: Partnership II, Room 322
Email: pdouglas@ist.ucf.edu
Office Hours/Web Hours: After class, or by appointment

Course Description:
This course provides an overview to modelling and simulation research techniques. This is a core course
for graduate students in the Modelling and Simulation department, though it is open to students from
other disciplines. It is designed to support development of skills related to machine learning and pattern
classification of 4 dimensional data (x,y,z, time), as well as dynamic systems modeling approaches to
study the brain.

Learning Objectives: Develop the ability to 1) identify important and novel lines of inquiry in the field of
Modeling and Simulation research, 2) appreciate how to frame a research problem with the appropriate
methodology and experimental design, 3) understand how to implement both hypothesis-driven and
model free analytic approaches to study data, 4) recognize the limitations of different research
approaches, and determine appropriate conclusions within these constraints. A key theme of the
course will focus on identifying alternate model hypotheses, and methods for model selection.

Prerequisites: Although there are no formal prerequisites, a background knowledge of linear algebra
will be useful, and the ability to code (e.g., Matlab, R, Python) would be helpful for the second half of
the course. I am aware that students in the M&S department come from a variety of backgrounds. I will
therefore provide tutorials, links, and sufficient time to get up to speed on maths and relevant coding
prior to homework assignments.
Required Text: There is no required text for this course. Instead, readings will be assigned from the
literature and selections from several texts, which will be made available online. Students may find the
introductory ML book “Introduction to Machine Learning,” 3rd. Ed” by Ethem Alpaydin to be a useful
reference for that portion of the course.

Course Outline:
The course is divided into four main parts. In part 1, you will learn the fundamentals of research
including topic formulation, experimental design, and classic data types. In part 2 (Model --> Data), you
will learn methods for going from a model (hypothesis/hypotheses) to testing how well that model fits
the data, compared to alternative models. In part 3, (Data --> Model), you will learn methods for
allowing the data themselves to nominate the model using a number of unsupervised and clustering
techniques. In part 4, we will discuss methods for critically evaluating these models.

Part I: Experimental Design
Module 1: Introduction to Research Design
1 HW1: Please take the Pretest—This will fulfill the student engagement requirement for the university.
Note – completion of this quiz will receive full credit (5pts). The purpose of this quiz is for me to
better understand the students’ backgrounds in the course.
Module 2: Formulating and testing hypotheses
1 Steps in constructing testable hypotheses
2 Correlation and Causation (What inferences can I make?)
3 HW2: Topic formation and Introduction. In a few sentences in a discussion thread, please describe 3
topics that interest you in modeling and simulation. Please take this time to introduce yourself
to the class and describe your educational/work background. (5pts)

Module 3: Data sources & Data Types
1 Data types (descriptive, correlational, etc)
2 Potential threats to validity, generalizability, and replicability
3 Pilot Testing & Identifying population implications a priori and their influence on statistical choices
4 HW3: (Group Assignment) Forming Research Hypotheses—Discussion post of 2 research questions,
each with 3 testable hypotheses. Response with 2 recommendations, requests for clarification,
and/ or additional potential hypotheses. *You must submit an email of who your group
members are by week 4 (groups of up to 3 people) (5 pts)

Module 4: Data collection & Design
1 Quantitative (e.g. data types, computational techniques, physiological)
2 Sampling and Statistics
3 HW 4: (Group Assignment) —Discussion post. Post a diagram of your research design. Include DVs and
IVs, interventions/manipulations. Use your diagram to show how information collected will help
answer your research question by testing your hypotheses. Do you plan to use an existing data
set or a simulated data set for your group project? (10 pts)

Part II: Model Hypothesis --> Data

Module 5: Classical Inference
1 Classical Hypothesis Testing
2 General Linear Model
3 HW 5: GLM exercise (5pts)

Module 6: Correction for Multiple Comparisons
1 False discovery rate (e.g., Benjamini-Hockberg)
2 Other methods for correction for multiple comparison
3 HW 6: Watch the video on Gaussian Random Field theory and complete the quiz. (5pts)

Module 7: Inductive Bias & Correction for Multiple Comparison
1 Inductive Bias In Data Modeling & Simulation
2 Feature Selection, Cross-validation, permutation tests
3 HW 7: Machine learning exercise (5pts)

Module 8: Machine Learning - Part I
1 Feature Selection - Is this step important?
2 Dimension Reduction (Curse of Dimensionality, Kernel Trick)
3 HW 8: Download Weka, and complete the feature selection exercises (5pts)

Module 9: Machine Learning - Part II
1 Interpreting Classifiers
2 How noise and redundancy can effect interpretation
3 HW 9: Complete Matlab LDA assignment (10 pts)

Module 10: Generative Models
1 Dynamic Systems Modeling
2 Model Inversion (Parameter Estimation Techniques)
3 HW 10: Complete Matlab ODE assignment (10 pts)

Module 11: Frequency Domain Analysis
1 Laplace Transform
2 Stability constraints when modeling in the Frequency Domain
3 HW 11: Complete Frequency Domain Assignment (5pts)

Part III: Data --> Model
Module 12: Unsupervised Methods
1 Why run an unsupervised analysis?
2 Review of Linear Algebra Techniques
3 HW 12: Complete Linear Algebra Review (5pts)

Module 13: Eigenvalues and More
1 Singular Value Decomposition (SVD, PCA, ICA, CCA)
2 HW 13: Complete Matlab Assignment (10 pts)

Module 14: Model Selection
4 Sparse vs. Complex Models
5 AIC, BIC, and Free Energy
6 HW 11: Complete Model Selection Exercise (5pts)

Part IV: Evaluating Model
Module 15: Critiquing Models
1 Understanding limitations of models
2 HW 15: Read posted manuscript and post discussion of limitations of the method in the discussion
section. (10 pts)

Module 16: Group Week
1 Work in your groups on your project

Course Requirements:
Attendance, participation, completion of homework and final exam are required for course completion.
According to the university, this class is "mixed mode". Here is the official description: M – Mixed
Mode/Reduced Seat Time courses include both required classroom attendance and online instruction.
Classes have substantial activity conducted over the web, which substitutes for some classroom
meetings.

Evaluation & Grading:
Homework and participation in the discussion topics will account for 100 pts of your grade. The final
project will also be worth 100 pts (oral presenation, 50 pts, and final paper 50 pts). The class will follow
a traditional grading scale shown below:
Letter Grade Points
A 90-100
B 80-89
C 70-79
D 60-69
F 59 or below

Make up Exams:
Homeworks, group projects, and exams will be posted online, and students may take them any time up
until their final due date. All group members are expected to contribute to the best of their ability, and
are expected to participate in the final group presentation and contribute to the final paper. Groups
should submit final papers in research manuscript format, and indicate the contribution of each group
member. If for some reason, you cannot make it in-person to the final group presentation, please
contact me to discuss alternatives. Failure to do so will result in a lower grade for the course.

Attendance Policy:
According to the university, this class is "mixed mode". Here is the official description: M – Mixed
Mode/Reduced Seat Time courses include both required classroom attendance and online instruction.
Classes have substantial activity conducted over the web, which substitutes for some classroom
meetings. While I do not directly penalize for missed classes, attendance it is extremely important for
success and class participation, and should be considered mandatory.

Academic Engagement:
All instructors/faculty are required to document students’ academic activity at the beginning of each
course. In order to document that you began this course, please complete the pretest as soon as
possible. Failure to do so may result in a delay in the disbursement of your financial aid. To document
your academic engagement, you will complete a quiz by the end of the first week of class.

Final Paper: The final paper will be due at the time assigned by UCF. This time will be posted on
Webcourses within the first week.

Changes/Announcements: It is possible that adjustments to the schedule may be made

during the course. In the event that anything in this syllabus changes (e.g. classroom moves, changes in
due dates, contact information), I will use a broad announcement so that all students will be informed
immediately. It is critically important that you set your webcourse announcements so that you receive
all notifications and be sure to check the class website regularly.

Academic Integrity:
Students are encouraged to discuss problems with colleagues, but the final assignment handed in should
be the student’s own work. Plagiarism and cheating of any kind on an examination, quiz, or assignment
will result at least in an "F" for that assignment (and may, depending on the severity of the case, lead to
an "F" for the entire course) and may be subject to appropriate referral to the Office of Student Conduct
for further action. See the UCF Golden Rule for further information. I will also adhere to the highest
standards of academic integrity, so please do not ask me to change your grade.

Accessibility Statement:
The University is committed to providing reasonable accommodations for all persons with disabilities.
This syllabus is available in alternate formats upon request. Students with disabilities who need
accommodations in this course must contact the professor at the beginning of the semester to discuss
needed accommodations. No accommodations will be provided until the student has met with the
professor to request accommodations. Students who need accommodations must be registered with
Student Accessibility Services, Ferrell Commons, 7F, Room 185, phone (407) 823-2371, TTY/TDD only
phone (407) 823-2116, before requesting accommodations from the professor.

Copyright:
This course may contain copyright protected materials such as audio or video clips, images, text
materials, etc. These items are being used with regard to the Fair Use doctrine in order to enhance the
learning environment. Please do not copy, duplicate, download or distribute these items. The use of
these materials is strictly reserved for this online classroom environment and your use only. All
copyright materials are credited to the copyright holder.