- #references - Title: AI Research, Replicability and Incentives - Meta: - Tags: #ref/Other - Authored by:: Denny Britz - Year: [[2020]] - Publication: no_info - URL: https://dennybritz.com/blog/ai-replication-incentives/ - Citekey: britzAIResearchReplicability2020 - Content - Placeholder - Abstract - A look at replicability issues in Artificial Intelligence research, and how academic incentive systems are driving the community towards certain types of research. - #quotes - One would think that implementing the same model in two different frameworks would lead to identical results. But that's not the case. Subtle differences in ^^framework implementations, insufficient documentation, hidden hyperparameters, and bugs^^ can cascade and lead to different outcomes. If you go through the Github issues and forums of popular Deep Learning frameworks, you can find many examples of researchers obtaining unexpected results (like [here](https://l7.curtisnorthcutt.com/towards-reproducibility-benchmarking-keras-pytorch) or [here](https://github.com/keras-team/keras/pull/9965) or [here](https://github.com/keras-team/keras/issues/4444) or [here](https://github.com/keras-team/keras/issues/8672) or [here](https://github.com/kuangliu/pytorch-cifar/issues/45) or [here](https://github.com/Microsoft/MMdnn/issues/595)). From what I have seen, high-level frameworks like Keras that hide low-level implementation details and come with implicit hyperparameter choices already made for you, are the most common source of confusion. - this is also an #example-of [[context]] - [[📝 lit-notes]] - [[SOTA]] -  -  ###### Discourse Context - **Informs::** [[CLM - Prevailing incentives in academia are bad for science]]