Not Every Model Has a Separate "Loss Function"

"Loss function" is one of the most basic concepts today in deep learning. Despite that, it is actually not necessarily a good programming abstraction when designing general-purpose systems. A system should not assume that a model always comes together with a "loss function".

Read more

How to Maintain Clean Core APIs for Research

Building a library for research and experiments is quite different from building other types of software. A key challenge is that, in research, abstractions and APIs are rarely set in stone: users may want to propose a slight variant or modification to literally ANYWHERE in the whole program, just because they have a new idea.

Read more

Automatically Flatten & Unflatten Nested Containers

This post is about a small functionality that is found useful in TensorFlow / JAX / PyTorch.

Low-level components of these systems often use a plain list of values/tensors as inputs & outputs. However, end-users that develop models often want to work with more complicated data structures: Dict[str, Any], List[Any], custom classes, and their nested combinations. Therefore, we need bidirectional conversion between nested structures and a plain list of tensors. I found that different libraries invent similar approaches to solve this problem, and it's interesting to list them here.

Read more

TorchScript: Tracing vs. Scripting

PyTorch provides two methods to turn an nn.Module into a graph represented in TorchScript format: tracing and scripting. This article will:

  1. Compare their pros and cons, with a focus on useful tips for tracing.
  2. Try to convince you that torch.jit.trace should be preferred over torch.jit.script for deployment of non-trivial models.
Read more

谈谈Github上如何交流(3): 如何管理issue

我听过不少人凭借爱好开源了自己的项目后, 却对 issue 太乱感到困扰, 甚至想干脆直接禁用 issue. 事实上, 任何项目达到一定规模后, 如果不对 issue 进行适当管理, 都会使 issue 信噪比过低, 失去原本的功能.

这篇文章主要从 maintainer 的角度说说, 在具备规模的项目中管理 issue 的一些方法和原则.

Read more

谈谈Github上如何交流(2): 如何科学的报bug

报告错误 / 报 bug 是用户与开发者间最常见的一类交流, 也是常见的 github issue. 但是很多用户并不会科学的报 bug, maintainer 对此也缺乏引导. 因此这篇文章讨论如何科学的报 bug.

Read more

谈谈Github上如何交流(1)

相比传统的邮件列表 / bugzilla/sourceforge 等开源平台, github 把开源社区交流的成本 / 门槛降的很低, 因此交流的质量也常常随之下降.

我计划写几篇文章, 从 用户 (User)维护者 (Maintainer) 两者的角度写写开源社区中如何使用 issue/PR 进行沟通, 希望能够:

  • 让普通用户学会提问, 学会以 maintainer 更容易接受的方式使用 issue/PR 做贡献.
  • 给 maintainer 提供一些管理 issue/PR 的经验, 不要再为 issue 头疼.
Read more

Effective Use of Python 'logging' Module

In large systems, logs can be terrifying: they are huge in volume, and hard to understand. This note lists some suggestions and common misuse of Python's logging module, with the aim of:

  • Reduce redundant logs & spams from libraries.
  • Allow more control of logging behaviors.
  • Make logs more informative to users.
Read more

Where Are Pixels? -- a Deep Learning Perspective

Technically, an image is a function that maps a continuous domain, e.g. a box , to intensities such as (R, G, B). To store it on computer memory, an image is discretized to an array array[H][W], where each element array[i][j] is a pixel.

How does discretization work? How does a discrete pixel relate to the abstract notion of the underlying continuous image? These basic questions play an important role in computer graphics & computer vision algorithms.

This article discusses these low-level details, and how they affect our CNN models and deep learning libraries. If you ever wonder which resize function to use or whether you should add/subtract 0.5 or 1 to some pixel coordinates, you may find answers here. Interestingly, these details have contributed to many accuracy improvements in Detectron and Detectron2.

Read more