Yuxin's Bloghttps://www.gravatar.com/avatar/0c9f23b0750710ed47c897702096e1f52025-11-20T08:00:00.000Zhttps://ppwwyyxx.com/blog/Yuxin Wuppwwyyxxc@gmail.comHexoCommon Python Reference Cycle Patternshttps://ppwwyyxx.com/blog/2025/Ref-Cycle-Patterns/2025-11-20T08:00:00.000Z2025-11-20T08:00:00.000ZIn Python, when a set of objects constructs a reference cycle, none of them would reach a zero refcount.In this case, even if these objects all go out-of-scope and are no longer accessible, they will not be immediately released.
The Python ecosystem typically accepts reference cycles as an inevitable issue, and relies on garbage collection (GC) to avoid leaks.A GC is triggered by the Python interpreter from time to time;it will detect all non-reachable objects, and release them regardless of their refcount.
However, in high performance deep learning systems, GC is not always a good choice.
The GPU memory is so precious that we care a lot about its allocations and deallocations.We want a GPU tensor to be freed as soon as possible,and cannot risk leaving them non-reachable and wait for GC to kick in at a later time.
GC causes a random, significant pause of the interpreter.This is not acceptable for large-scale synchronous jobs, because once any worker is running GC, all otherswill have to wait for it in order to finish a collective communication.
Instead, high performance deep learning systems often disable the automatic GC, and run them manually at specific sync point,rather infrequently.See the relevant code intorchtitanand Megatron-LMfor example.
Therefore a deep learning system engineer has to be aware of refcycles and avoid them.This post shares a few patterns of refcycles that are worth knowing.We'll also use objgraph to visualize the refcycle.
In this post, we'll assume GC is disabled, so that any object involved in a refcycle is considered "leaked".
import gc import sys import objgraph from types import SimpleNamespace
obj = object() a, b = SimpleNamespace(obj=obj), SimpleNamespace() print(sys.getrefcount(a)) # 2 a.b = b; b.a = a print(sys.getrefcount(a)) # 3 del a, b
In code reviews, I often frown upon any closure that captures self: when a closurerefers to self, we have to make sure self does not directly or indirectly refer to the closure.Below is a simple example of a refcycle that may arise:
import sys classA: def__init__(self, obj): self.obj = obj self.print_obj = lambda: print(self.obj) obj = object() for k inrange(10): a = A(obj) print(sys.getrefcount(obj)) # Refcount keep increasing!
Any bound method also has a reference to self. So the following creates a refcycle as well:
import sys classA: def__init__(self, obj): self.obj = obj self.method_alias = self.method defmethod(self): pass obj = object() for k inrange(10): a = A(obj) print(sys.getrefcount(obj)) # Refcount keep increasing!
The lesson learned from both examples is to prefer defining methods using the normal def syntax, instead of assignments.
Similarly, code below is an alarming pattern, because the active thread keeps a reference to self,despite self owning the thread.
In addition to capturing self, a closure that's a recursive function is also a refcycle by itself,because it captures the closure definition.
deffunc(obj): deffoo(n): if n == 0: return0 return foo(n-1) + obj result = foo(3)
obj = 12313123123 for k inrange(10): func(obj) print(sys.getrefcount(obj)) # Refcount keep increasing!
In the example above, the function foo will leak because it refers to itself.This is sometimes OK because a function definition is cheap.But leaking foo causes obj to leak as well because foo refers to obj.This could be a severe problem when obj is a resource-heavy object.
A simple fix to this example is to remove the function object from the local scope after using it:
- result = foo(3) + try: + result = foo(3) + finally: + foo = None
PyTorch has seen a few similar bugs in the past where tensors get leaked due to recursive local functions.For example,[1],[2]and a more complicated one involving functions calling each other:[3].@dzhulgakov recently posted another example on X.
Keep in mind that ANY Python class definition is self-referential -- i.e. any class definition will be leaked.This has been discussed in the CPython tracker a long time ago and hasn't been fixed.It is typically fine, as most class definitions are cheap, and they are createdat import time and expected to survive the entire lifetime of the program anyway.
However, similar to recursive functions, it is a giant footgun to define a class that is a closure on local objects --the local objects will be leaked. The following example demonstrates this:
An exception stores its stack trace with all stack frames.Storing an exception object to a variable will create a refcycle"frame → variables of the frame → exception object → stack → frame".This leaks the exception object, together with the entire stack!
import sys deff(obj): try: assertFalse except Exception as e: err = e obj = object() for k inrange(10): f(obj) print(sys.getrefcount(obj)) # Refcount keep increasing!
This is a big problem, because any library we depend on may have a seemingly innocent err = e statement,and it will cause the entire stack that calls into this library to leak in a very hard-to-debug way.This PR is an example where a 3rd-party library storingthe exception costs 100+GB of memory leak.
To properly store an exception, PyTorch provides a nice ExceptionWrapper that avoids refcycles.See its commentsfor details.
In autograd, any tensor in the graph has a grad_fn that refers to a node in the autograd graph,which may, in cases of custom autograd.Function or custom saved_tensor_hook, refer backto the tensor itself.[1][2][3][4]are some bugs of this family.What's worse is that the cycles involve C++ objects, therefore gc.collect() would not help.
The fix is to always apply a detach() whenever we need to manually save a tensor for backward(even if the tensor does not require grad).
A unittest.mock.MagicMock has some "magic" that allows mocked methods and sub-mocks (mocks created from a mock) toaccess their parents. This introduces refcycles:
x = object() for k inrange(10): f(x) print(sys.getrefcount(x)) # Refcount keep increasing!
I attempted to fix it in CPython but it wasn't accepted.
Due to this issue, we should be careful when using mock in resource-heavy unittests (e.g. GPU tests) --some intermediate results may get leaked.
omegaconf is a config library. Its config object is self-referential:
from omegaconf import DictConfig import sys deff(obj): x = DictConfig({'a': obj}, flags={'allow_objects': True}) obj = object() for i inrange(5): f(obj) print(sys.getrefcount(obj)) # Refcount keep increasing!
This is because it allows a sub-config node to access its parent.We should keep this in mind when storing any resource-expensive objects in the config.
More generally,when any object has access to its conceptual parent/owner, i.e. if A owns B but B has access to A,care has to be taken to avoid refcycle.Ideally, we would structure software such that an owner has access to its children objects, but not the opposite.But it is often not the reality.Bidirectional access is helpful when flexibility is preferred, as we have seen in MagicMock and DictConfig.
As another example, detectron2's trainer owns several hooks -- which are functions to execute during training.These hooks themselves may need access to the trainer -- in order to access information about the training.A weakref is usedthere to avoid refcycle.
Eliminating refcycles is important in deep learning systems.This post lists a few patterns of refcycle that I have seen frequently when debugging memory leaks.When needed, objgraph can be helpful to identify refcycles.
]]><p>In Python, when a set of objects constructs a reference cycle, none of them would reach a zero refcount.
In this case, even if these objects all go out-of-scope and are no longer accessible, they will not be immediately released.</p>
<p>The Python ecosystem typically accepts reference cycles as an inevitable issue, and relies on garbage collection (GC) to avoid leaks.
A GC is triggered by the Python interpreter from time to time;
it will detect all non-reachable objects, and release them regardless of their refcount.</p>
<p>However, in high performance deep learning systems, GC is not always a good choice.</p>写在 wechat-dump 项目的第十年https://ppwwyyxx.com/blog/2025/wechat-dump-10-years/2025-01-31T08:00:00.000Z2025-01-31T08:00:00.000Z在过年的这几天, 为了从焦虑的工作中换一个心情, 我给我的 wechat-dump 项目添加了几个当年没做出来的功能, 解决了一些遗留问题. 意外的发现这个项目始于 2014 年末, 到今天已经超过十年了. 有多少人会有给自己十年前的代码补充新 feature 的经历呢? 突然有了一些感触想要写下来.
amend 会导致 rebase branchX on branchA 的时候产生 conflict. (下一节解释)
如果在 branchA 里加入一个不干净的 commitA-fix (即并不需要最终保留在 main branch 里的 commit), 在 rebase branchA on main 时会带来很多不必要的痛苦. 例如: 假设 commitA 和 commitA-fix 都对函数 func 进行了修改. 而与此同时 main branch 的 func 也有了变化. 这时候 rebase branchA on main 时需要 resolve conflict 两次.
😞无论怎样, 都需要再手动切换到 branchX, rebase on branchA, 再 force push branchX, 才能维护好它们的依赖关系.
Example 3: 接着上一个 example, 在经过一些 review 后, 我们发现需要额外对函数 S 进行修改才能更好的实现 feature A. 也即依赖关系为S <- A <- X. 在 Meta 我会这么做:
$ hg log --graph @ changeset: 30:abcdef123456 | user: John Doe | summary: Fix a small bug. | | o changeset: 29:789012345678 | | user: John Doe | | summary: Implemented feature X | | | | o changeset: 27:abcdef012345 | |/ user: John Doe | | summary: Updated documentation for module B | | | o changeset: 26:456789abcdef | | user: John Doe | | summary: Fix module B | | | o changeset: 25:abcdef012345 |/ user: John Doe | summary: Change module A |
]]><p>Meta 与 Google 内部的代码管理工具都支持一种被称作 "stacked diffs / stacked PRs"
的 workflow. 然而, 基于 git 的主流平台 (github, gitlab) 都不支持这种 workflow.
许多离开 Meta 后不得不使用 github 的朋友表示, stacked diffs 对于工程师是一个 "ultimate
productivity tool", 我也深有同感. 这篇文章介绍一下什么是 stacked diffs workflow,
以及为什么它能够极大的提升团队开发效率.</p>Registration Does Not Scale Wellhttps://ppwwyyxx.com/blog/2023/Registration-Does-Not-Scale-Well/2023-04-23T07:00:00.000Z2023-04-23T07:00:00.000ZPeople have many different opinions about config systems. Having worked withvarious styles of configs, I also want to write about what a greatconfig subsystem in a large-scale (in terms of system complexity, number ofusers, etc.) system should look like.
The design space is complex, so in this article I'll start with a smaller topic:registration in config systems. I'll show why this common pattern, though works fine forsmall-scale projects, does not scale well in the long term. I'll also discussan alternative.
Configs are often constrained to include only primitive types (numbers andstrings), and there are a lot of good reasons to keep this property.But users often want to refer to more complex objects.
A global registry in a config system is typically a Mapping[str, Any].Its purpose is to allow users to refer to complex objects through simple strings in a config,overcoming the constraint of primitive types.
Objects can be added to a registry like this:
# Maps string to a class/type. @ModelRegistry.register classMyModel(): ...
# Maps string to an object. DatasetRegistry.register("my_dataset", MyDataset(...))
This allows users to select which model / data to use by settingcfg.model.name = "MyModel" or cfg.dataset.train = "my_dataset".
Registration has a clear benefit, but at a larger scale, some of its downside are serious.
Users should only pay (compute cost and mental cost) for what they use is a general design philosophy I found pretty important in almost all aspects of software design.
The registration pattern breaks this philosophy by running unnecessaryregistration code: users will only provide one (or very few) string in theirconfig, but they have to pay the overhead of registering many candidate stringsthat users might need.
To make matter worse, the overhead has to happen very early in a program, typically at import time. Import speed is crucial for developer ergonomics: unlike other code that may run async with development, importblocks developers, slowing down their day-to-day work.
The registration overhead includes:
Cost to import extra Python modules that contain registration code, and most importantly, all their dependencies.
For registries that map strings to non-trivial objects (not just types/functions), the cost to create these objects.
A better practice is to avoid such registries: don't store objectsin the registry, but store functions that create these objects if possible. Howeverthis does not always solve the problem: the function may have to be aclosure that close on non-trivial objects, in which case the objects stillhave to be created at registration time.
These costs are negligible for small-scale projects, but they can become quitebad when there are hundreds or more objects to register.If a design permits bad patterns, bad patterns are guaranteed to appear at larger scale:There will be users doingnon-trivial registration (e.g. register objects in a for loop) that's slow oreven has unintended side effects. I had to work with many projects that take >10s to import and the most common reason of slow import is registration.
The import overhead is also greatly magnified by Python's multiprocessing module: all subprocesses will have to spend the time and RAM to rerun the imports.
It's not uncommon that different users register different objects under thesame name -- at a large scale that's guaranteed to happen.
Such conflicts can live in two users' own code for a long time, unnoticed,until one day someone needs to depend on both.The only viable solution is usually to rename one, hence break all its users.
To complicate the issue even more, people sometimes decide to resolve nameconflicts by overwriting what's already registered. For example, an "overwrite"option is provided inthe registry of iopath,mobile_cv,and paxml.Using this option may introduce hard-to-debug problems, because now aninnocent import statement may silently change the behavior of user code.
Despite of this, note that overwriting is actually necessary when working in notebooks, where it's common to "reload" code (therefore re-register objects) on the fly. Here is some code I use to always enable overwrite during reload.
When running a function using a multiprocessing.Process created with a safe start_method like "spawn", the child process receives a pickled closure from its parent, so it knows what to run. However, this pickle does not include any global states. This implies that if a function access global states, it may behave differently depending on if it runs in the subprocess or the parent process. Python's documentation has a clear warning about this:
if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that Process.start was called.
Since the registration is globally accessible, it's not easy to find where inthe code an object is registered (or modified, if overwrite is allowed) just byreading the code. When a user sees cfg.dataset.name = 'dataset_X' and iscurious what is "dataset_X", a global string search is almost the only way tofind it out without running the code. And the search does not always work: ifthe name is programmatically generated, then the string cannot be founddirectly in source code, e.g.:
for dataset in ["ds1", "ds2"]: for r in [0.1, 0.5]: DatasetRegistry.register(f'my_{dataset}_ratio{r}', MyDataset(dataset, r))
In this case, users will have to be more creative about what strings to search.
In C++, registries cause even more trouble becauseconstruction and destruction of global objects are verytricky. In Safe Static Initialization, No DestructionI talked about a few PyTorch's C++ bugs related to this. Luckily,in Python, there are better alternatives.
If the only goal of registration is to provide a name → object mapping, then asimple alternative in Python is to use obj.__module__ + '.' + {variable name} as the name,which may look like some_library.some_module.MyClass."Variable name" can be obj.__qualname__ for classes & functions.
Given this string, one can then call a simple function such as the builtinpydoc.locate to obtain the object it refers to: modules will be importedon-demand by importlib.
# main.py --name=MyModel: from my_lib import ModelRegistry import my_lib.my_module # import to register model = ModelRegistry.get(args.name)
# my_lib/my_module.py: classMyModel(...): ...
# main.py --name=my_lib.my_module.MyModel model = pydoc.locate(args.name)
This pattern has some obvious advantages over registration:
No need to import any unused modules. Modules are imported on-demand.
No global states.
There are some common concerns of this pattern, but they are not hard to address.
It's slightly harder to dynamically create candidates: there is no "registry" to add objects to,and the only equivalence is to edit the globals() dictionary directly.
for dataset in ["ds1", "ds2"]: for r in [0.1, 0.5]: globals()[f"my_{dataset}_{r}"] = create_dataset(dataset, r)
I don't consider this an issue, because it's actually discouraging bad practice:in a proper config system (e.g. one that's based on recursive instantiation)there should be no need to dynamically generate candidates like above.I hope to get to this in a future article.
The names in config have to match the names of classes/functions in code.
This has the benefit of clarity on one hand. But on the other hand, code owners have more responsibilityto maintain backward compatibility, especially after renaming their classes and files.The standard good practice suffices to address this: distinguish private vs. public symbols;keep an alias from the deprecated name to the new name; etc.
Overall, I think the benefit of matching config and code outweighs the concern that config may be broken by code.
The names are too long.
This is a real problem. Here are some ways to address it:
A $PATH-like mechanism can be used to specify which modules to search for names.The search path can include common prefixes like "my_project" so that users only have to provide "submodule.MyModel".
There can be a registry-like Mapping[str, str] that maps from "MyModel" to"my_lib.my_module.MyModel" so that users don't have to write long strings.This mapping doesn't have to be global and doesn't introduce import overhead.This can help with problem (2) as well.
This is just a UI-level issue.Having a better config frontend, e.g. using Python code as the config language,can make this issue disappear! Let me save this for a future article.
]]><p>People have many different opinions about config systems. Having worked with
various styles of configs, I also want to write about what a great
config subsystem in a large-scale (in terms of system complexity, number of
users, etc.) system should look like.</p>
<p>The design space is complex, so in this article I'll start with a smaller topic:
<strong>registration</strong> in config systems. I'll show why this common pattern, though works fine for
small-scale projects, does not scale well in the long term. I'll also discuss
an alternative.</p>Safe Static Initialization, No Destructionhttps://ppwwyyxx.com/blog/2023/Safe-Static-Initialization-No-Destruction/2023-04-06T07:00:00.000Z2023-04-06T07:00:00.000ZSince I joined Google Brain, I brought PyTorch to Google's internal infra andowned its maintenance. Being a "tech island", it's well known that almosteverything in Google works differently from the outside world, and thatcreates many challenges when building a massive library like PyTorch.
Among those challenges, there are a few tricky bugs related to staticinitialization order fiasco(SIOF) and their destructions. This time I was forced to learn a lot more detailsthan I'd like to know about these topics, so it's good to write them down before I forget.
"Static initialization" is an ambiguous term because "static" is very overloaded in C++.In our context, it is supposed to mean "initialization of objects that have static storage duration",i.e. objects that live through the lifetime of a program.The word "static" actually talks about the object lifetime, not about initialization.
Meanwhile, initialization of such objects can have two steps:
Zero/Constant initialization (it's also confusinglycalled "static initialization" 😱):allocate memory and fill inzero or constantbytes. For objects with static storage duration, this step is done during compile time.
Dynamic initialization: call constructor (if non-trivial, and if not constexpr) of the object.
Objects with static storage duration can be categorized into following two types, based on when their "dynamic initialization" happen:
Global: these objects are dynamic-initialized at program launch (before main()).
SIOF typically refers to the problem that the dynamic initialization order of objects from different translation units is undefined, e.g.:
// a.cpp: Object a; // b.cpp: AnotherObject b;
If a and b have non-trivial constructors, and the constructor of b somehow needs to access a, the program may crash or behave unexpectedly because a may be initialized after b.
PyTorch heavily uses registrations, which all have static storage duration. A few SIOF bugs were found when Itried to build PyTorch in Google. As an example, when an ATen operator has many overloads, initialization order affects which overload is called, because an overload that's initialized earlier will be preferred over those initialized later.
Standard ways to avoid SIOF problems are:
Avoid dynamic initialization: change object type to something that can be zero/const-initialized. totw/140 shows a few examples on how to replace std::string with non-dynamic counterparts.
Use well-defined initialization order: there is a guarantee that objects within the same translation unit are dynamically initialized according to the well-defined program order. So we can sometimes just move code into the same translation unit. In another PyTorch bug where one global depends on another,I simply merged two files so that their constructors are properly sequenced.
Construct on first use: it's often not practical to merge files. A better solution is the "construct on first use" idiom:
By doing this, anyone that needs to access a will have to call get_a(). Because function-local static is guaranteed to initialize on first use, we can rest assured that a will not be used before initialization.
The "construct on first use" idiom may look differently, because sometimes we don't need to use a directly but do need to observe the side effects of its constructor. In such cases we just manually call get_a to make sure a is constructed. I used this to fix another PyTorch bug .
There are more ways things can go wrong in the destruction of objects with static storage duration.
In general, we have to carefully avoid use-after-free, i.e. access a global/function-local variable after it's destructed. This is typically protected by this rule:
Non-local objects with static storage duration are destroyed in the reverse order of the completion of their constructor.
Given this rule, we can deduce that:The above result sounds nice and is often enough protection, but people tend to overlook a few ways things can still go wrong:
The program can, in theory, pass around the address of b. This should be discouraged, but it means that technically ANY object could access b in their destructor. If any of these objects are destructed after b, we're doomed.
When one thread crashes and destructs objects, there is a possibility that at this point another thread is still running and have access to the object. We're doomed again.
The above "reverse-ordering" rule, though written in cppreference.com, is not always true under certain build options!
Objects with static storage duration are forbidden unless they are trivially destructible.
This "no destruction" rule implies that the following code is illegal
Object& get_a(){ static Object a; return a;}
if Object is not trivially destructible. C++ FAQ advises the same.
Writing static Object* a = new Object; return *a; is safe as long as we never call delete, but this introduces a heap-allocation overhead.The last trick is to use a NoDestructorwrapper class to bypass RAII(the trick is placement new operator):
Safe, but has heap allocation overhead
Safe and low overhead
Object& get_a(){ static Object* a = newObject(); return *a; }
Finally, as an alternative to "no destruction",another way to safely run destructors is toref-counting all such objects,but it's perhaps not worth the complexity. "No destruction" is usually a good enough solution.
In conclusion, to safely construct and destruct objects with static storage duration + dynamic initialization, follow these rules of thumb:
Safe Initialization: use "construct on first use" idiom
No Destruction: don't run any non-trivial destructors
]]><p>Since I joined Google Brain, I brought PyTorch to Google's internal infra and
owned its maintenance. Being a "tech island", it's well known that almost
everything in Google works differently from the outside world, and that
creates many challenges when building a massive library like PyTorch.</p>
<p>Among those challenges, there are a few tricky bugs related to <a href="https://en.cppreference.com/w/cpp/language/siof" aria-label="Static Initialization Order Fiasco - cppreference.com" class="hint--top hint--rounded hint--no-animate hint--no-arrow">static
initialization order fiasco</a>
(SIOF) and their destructions. This time I was forced to learn a lot more details
than I'd like to know about these topics, so it's good to write them down before I forget.</p>Some Useful Terminal Escape Sequenceshttps://ppwwyyxx.com/blog/2023/Terminal-Escape-Sequences/2023-01-15T08:00:00.000Z2023-01-15T08:00:00.000Z最近学习到了一些 Terminal Escape Sequences, 其中尤其对 OSC52 相见恨晚. 这里稍微记录一下各种 Sequences.
]]><p>最近学习到了一些 Terminal Escape Sequences, 其中尤其对 OSC52 相见恨晚.
这里稍微记录一下各种 Sequences.</p>
<p>Terminal Escape Sequences 是终端应用向 stdout 打出的一些具有特殊含义的字符串.
终端看到这些串之后不会显示它们, 而是执行这些串所对应的终端高级功能.</p>Demystify RAM Usage in Multi-Process Data Loadershttps://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multiprocess-DataLoader/2022-12-24T08:00:00.000Z2022-12-24T08:00:00.000ZA typical PyTorch training program on 8 GPUs with 4 dataloaderworkers per GPU would create at least processes.A naive use of PyTorch dataset and dataloader can easilyreplicate your dataset's RAM usage by 40 times. This issue has probably affected everyone who has done anything nontrivial with PyTorch.In this post, we will explain why it happens, and how to avoid the 40x RAM usage.
All code examples and experiment results are available on github at ppwwyyxx/RAM-multiprocess-dataloader.The content is not specific to PyTorch: it applies to any user of Python's multiprocessing library on Linux.
Datasets for machine learning are usually not stored in RAM. But it's common to store their "metadata" in RAM, and this may still cause nontrivial RAM usage. The metadata could be:
For ImageNet dataset: A million file names and their labels.
For COCO dataset: 100k file names and their bounding boxes, segmentations, etc.
As a concrete case, loading the metadata of COCO training set into Python takes ~2.4G of RAM:
# Download from https://huggingface.co/datasets/merve/coco/resolve/main/annotations/instances_train2017.json defcreate_coco() -> list[Any]: withopen("instances_train2017.json") as f: obj = json.load(f) return obj["annotations"]
We obviously don't want to replicate this 2.4G of RAM across all processes.
We acknowledge that there are ways to offload these metadata to disk. For example, people sometimes do:
Store all the metadata together with raw data on disk, so metadata are not stored in RAM.
Read sequentially from a combined single-file dataset, so that file names or indices are not stored in RAM.
By doing these, the RAM usage of a dataset becomes negligible. However, these methods will sacrifice flexibility and capabilities, such as random-access, perfect shuffle, merging datasets arbitrarily, custom subsampling support, etc.Notably, PyTorch's commonly used map-style datasets supportrandom access & sampling.All of these capabilities require certain metadata in RAM.
This article ignores any of these offloading methods. Instead, we'll discuss how to reduce the RAM usage without moving these data out of RAM. The idea is simple: we'll try to let all processes share a single copy of the dataset.
First let's build tools to measure RAM usage - which is not as easy as it sounds.
Common tools like top -p PID or psutil.Process(PID).memory_info() obtains memory statistics from /proc/{PID}/statm or /proc/{PID}/status, but they are insufficient for our analysis. Instead, we'll use the information provided in
/proc/{PID}/smaps: per-memory-mapping RAM usage information, documented inthis man page
/proc/{PID}/smaps_rollup: aggregation of data from smaps
We'll derive the following important measurements from it:
USS (Unique Set Size): RAM that's unique/private to this process, i.e. not shared with any other process. This is obtained by the sum of "private_*" entries in smaps.
Shared: RAM in this process that's also shared with other processes. This is obtained by the sum of "shared_*" entries in smaps.
Note that this is not equal to the "shared" column in /proc/{PID}/statm or top, which uses different accounting.
Shared_File: RAM that's shared with other processes through files. It is no larger than "Shared".
PSS (Proportional Set Size): Like RSS, but it avoids overcounting shared memory multiple times across all processes that are sharing it. It's basically "USS + Shared / (number of processes sharing it)". By definition, we should use total PSS to count the total RAM usage of N processes.
To obtain these measurements, we use psutil.Process(PID).memory_maps() which parses smaps under the hood:
defget_mem_info(pid: int) -> dict[str, int]: res = defaultdict(int) for mmap in psutil.Process(pid).memory_maps(): res['rss'] += mmap.rss res['pss'] += mmap.pss res['uss'] += mmap.private_clean + mmap.private_dirty res['shared'] += mmap.shared_clean + mmap.shared_dirty if mmap.path.startswith('/'): # looks like a file path res['shared_file'] += mmap.shared_clean + mmap.shared_dirty return res
Then we create a MemoryMonitor utility to measure and print the results for a list of PIDs.The code is straightforward and can be found here.
Then we launch subprocesses to read from this dataset with the list of COCO data. To make a cleaner demo, we don't use PyTorch's dataloader, but just launch 4 subprocesses by ourselves:
defworker(_, dataset: torch.utils.data.Dataset): whileTrue: for sample in dataset: # read the data, with a fake latency time.sleep(0.000001) result = pickle.dumps(sample)
We then added our MemoryMonitor to it. The full code and its output logs are available on github. Each segment in the log contains memory measurements for the main process + 4 workers:
The code above looks completely innocent, don't you think? However, if we plot the memoryusage of any dataloader worker over time, we seem to find a memory leak!This is the notorious "dataloader leaks memory" issue that is discussed at multiple places, e.g. this PyTorch issue and Edward's podcast.
In fact, the growth of RAM usage does stop in the end, so this issue is not a memory leak. But in reality, users often do not see the end before the system OOMs, and they may wrongly conclude this as a "memory leak".
The root cause of this issue is "copy-on-read" of forked CPython objects.
Linux has a copy-on-write mechanism: when a process forks, the child process will share its entire memory space with the parent, and only copy the relevant pages when necessary, i.e. when the child process needs to write to the page. This mechanism allows read-only pages to be shared to reduce total memory usage.
The copy-on-write behavior can be clearly observed in the above figure:at time=0, the worker has 2.6G of shared RAM, 0 USS (memory unique to the worker), and of PSS because the RAM is shared among 5 processes (4 workers + 1 main).
However, this mechanism did not help us when we read our dataset. The problemis that our dataset is a large nested data structure that contains many smallPython objects. Even though the dataset is "read-only" in theory, accessingany Python object will increment its refcount - causing a lot of memorywrites. With these writes, memory can no longer be shared among parent andchild processes. In other words, objects are not only copy-on-write, but also copy-on-read.Therefore, in the figure we see that the "Shared" RAM decreases and "USS" increases,since many pages are copied from shared memory into each process.
The end game is that each child process has to replicate all the pages that contain object refcounts in the dataset. For a dataset with many objects, this is almost the size of the dataset itself. In the output log, we see that this program uses10G total PSS in the end,where each child process replicates 1.8G of USS.
The copy-on-read issue is due to CPython's reference counting.There are ways to change CPython's behavior, e.g. by gc.freeze, but it has far-reaching consequences and I failed to make it work for the example here. However, there is a simple and transparent way to solve the issue: store the dataset with very few number of Python objects, so there are very few refcounts!Below is a minimal implementation that stores a listusing 2 numpy arrays:
classNumpySerializedList: def__init__(self, lst: list[Any]): lst = [np.frombuffer(pickle.dumps(x), dtype=np.uint8) for x in lst] self._addr = np.cumsum([len(x) for x in lst]) self._lst = np.concatenate(lst)
Detectron2 enables this type of serialization by default (since this commit by Yanghan). To compare different serialization mechanisms,we borrow its code into a serialization util, and use it here:
The total PSS usage is only 1.6G -- a 6x RAM reduction.
All processes have almost 0 USS, which means everything is shared! In fact, from the logs we can see that 1.6G is exactly the memory usage of the main process before starting subprocesses. Subprocesses add no extra memory usage.
The 6x reduction factor is better than #processes because pickle.dumps notonly serializes but also compresses the data. We benefit from both sharingand compression by applying this optimization, at the cost of a tinypickle.loads overhead in each access.
Actually, after compression, the dataset only takes ~500M (printed at the beginning of log). So a question arises: why does the main process use 1.6G RAM before starting subprocesses?
This is in fact just an artifact of modern memory allocators: it does not always release memory back to the OS. In fact, if we run this simple serialization/compression code:
Using a better allocator, e.g. by export LD_PRELOAD=libjemalloc.so, can make this issue largely disappear.
This artifact is typically not a big concern, since allocators will find opportunities to reuse these free buffers.(Well, they may be concerning in start_method="fork" because reusing these free buffers may trigger copy-on-write!But I'm not going to talk more about that.)
In our code above, we launched subprocesses using a start_method="fork" argument."fork, spawn, forkserver" are the 3 "start methods" of Python's multiprocessing library. This article is a good reference that explains their differences.
Since start_method="fork" is unsafe (in practice, it causes various crashes & deadlocks) and might no longer be the default in the future, we want to rerun our code above with start_method="spawn" or "forkserver". Sadly, the serialized array is no longer shared among workers. Each worker has a large USS:
The reason why our trick no longer works is that "spawn" and "forkserver" don't benefit from the copy-on-write mechanism. They will start a "fresh" subprocess with fresh memory space, instead of sharing with the parent. Everything the child process needs to access is pickled in the parent process and sent to the child. This ensures safe behavior, but is bad for start-up speed and memory usage.
In our case, the entire dataset will be pickled and sent to child processes. This is why each child process consumes a large USS.
It turns out there is a simple fix to this problem: just store the serialized dataset in a torch.Tensor instead of a numpy array. The reason why it works, is that multiprocessing uses a customizable pickle implementation called ForkingPickler, and PyTorch customizes how torch.Tensor should be pickled by it: the tensor data will not be serialized to bytes. Instead, during pickling the tensor will be moved to shared memory files (typically under /dev/shm) to be accessed by other processes directly.
To test tensor-based serialization, we run ./main-torchserialize.py spawn using the code here, and observes the following memory usage in workers (raw log is here):
"Shared_File" grows because workers will load from the shared torch.Tensor as needed. This is different from start_method="fork" where the entire memory space is shared at the beginning.
"Shared_File" stops growing when the worker has accessed the entire shared tensor.
The size of "Shared_File" in the end is roughly 500M(size of serialized dataset) + 170M, where 170M is thesize of all the binary files that import torch needs to load such as libtorch.so.This can be easily verified by printing the measurements after import torch.
After applying tensor-based serialization,the total PSS usage in the end is 2.2G-- still worse than our earlier number using start_method="fork".Next section will optimize it further.
The last culprit in the above experiment is the 160MBper-worker USS in the above figure: this is just the memory footprint of import torch,mainly for PyTorch's global variables, etc. Since every child process launched by "spawn / forkserver" is a "fresh" one, they all need to import torch independently, hence each has 160MB of USS.
Luckily, "forkserver" provides a way to share the import torch RAM usage through copy-on-write. By calling the undocumented Python API multiprocessing.set_forkserver_preload(["torch"]) before launching processes, each child process will be "less fresh": the torch library is preloaded (and shared), and don't need to be imported by each process independently.
Below are the experiment results. Code and full logs are on github:
The total PSS is only 1.7G, which is roughly the same as our best number using start_method="fork".
The USS of each worker is negligible, which means we've successfully shared everything. There is no per-worker memory overhead anymore.
(Note that this optimization may be unsafe if import torch creates any threads.My observation is that threads are indeed created due to import numpy inside torch, but they can be disabled with environment variables.)
So far we've only looked at a single dataloader (with 4 workers). In reality, the only scalable way to use PyTorch on multiple GPUs is to use one process per GPU, each will have its own dataloader and dataloader workers. This gives a total of #GPUs x (#DL workers + 1) processes organized like below:
We modified the previous experiment slightly into this code to run on 2 GPUs. The memory usage looks like this:
Our previous optimization on dataloader workers is still effective - dataloader workers have a tiny USS. However, RAM usage is now replicated by #GPUs times because we let each GPU worker read the dataset independently.
An inconvenient solution to this problem is to load and serialize the dataset before launching GPU workers. By doing this, all GPU workers share the dataset just like what dataloader workers do. However, this limits flexibility and often requires significant refactoring, due to reasons such as:
Dataset would have to be made ready much earlier than usual
Per-GPU data loading logic (e.g. sharding) may need to be modified
Another simple solution to this problem is again to use torch.Tensor and ForkingPickler to share the dataset among GPU workers, except that now we need tomanage the sharing explicitly like this:
if comm.get_local_rank() == 0: # GPU0 reads data and moves it to shared memory. # Move data to shared memory, obtain a handle to send to each local worker. handles = [None] + [ bytes(mp.reduction.ForkingPickler.dumps(tensor_dataset)) for _ inrange(comm.get_local_size() - 1)] else: handles = None # Each GPU receives its handle from GPU0. handle = local_scatter(handles)
if comm.get_local_rank() > 0: # Materialize a tensor from shared memory. tensor_dataset = ForkingPickler.loads(handle)
This logic is implemented as another serialization utilhere.When using it as a drop-in replacement (full code here),the dataset is no longer replicated by GPU workers:
GPU worker 1 still has a small amount of extra USS, and that's just the footprint of import torch that we saw earlier, and can be avoided using set_forkserver_preload.
Note that the multiprocessing library itself also provides shared memory support.This PR contains an implementation of our serialization util without using PyTorch.
We've successfully reduced the total RAM usage by (approximately) a factor of
The essence of the solution is to let all processes share memory through a single torch.Tensor object, which needs to be moved to Linux shared memory by PyTorch's custom pickling routine. The TLDR on how to achieve sharing is:
Don't let dataloader workers access many Python objects in their parent. Serialize all objects into a single torch.Tensor (but not numpy array) for workers to access.
Don't let all GPU workers load data independently. Load in one GPU worker, and share with others through a torch.Tensor.
For list-like data, all of these can be implemented transparently using the serialization routines developed in this article.
Multi-processing is often the only way to achieve trueparallelism in Python(until PEP703),but it comes with many tricky problems.This article hopefully provides an in-depth view of the problem of RAM usage.
"Loss function" may mean different things in different systems.The version I'm going to criticize is the most common one that looks like below:
Bad
Worse
deftrainer_bad(data, model, loss_func): whileTrue: inputs = next(data) predictions = model(inputs) loss = loss_func(predictions, inputs) # compute gradients and update model
deftrainer_worse(data, model, loss_func): whileTrue: inputs, labels = next(data) predictions = model(inputs) loss = loss_func(predictions, labels) # compute gradients and update model
The key property of a bad "loss function" abstraction is:Users are asked to provide a "loss function" that's executed separately after the "model / forward logic".Such abstraction appears in a few open source systems: Keras model.compile(loss=),fast.ai Learner(loss_func=), Lingvo BaseModel.ComputeLoss.
The main problem is not with the function itself, but that the users' algorithm logic is forced to separate into two parts: model and loss_func.
As an alternative, trainer_good below no longer separates "loss_func" from the model, and has equal functionalitieswith trainer_bad.
deftrainer_good(data, model): whileTrue: inputs: Any = next(data) loss: Scalar = model(inputs) # or losses: dict[str, Scalar] # compute gradients and update model
In this article, I want to argue that this is a better design because:
Separating out "loss function" causes many trouble.
With trainer_good, users can still separate their model into two parts if they like, but they don't have to.
Although trainer_good is no longer aware of the separation, it does not matter.
(Apparently, trainer_good == partial(trainer_bad, loss_func=lambda x, y: x).So trainer_bad can still be used - we just set loss_func to a no-op if we don't like it.But trainer_good is cleaner.)
The separation is not convenient for a model with many optional losses.Take a multi-task model for example:
Separation
No Separation
classModule(nn.Module): defforward(self, inputs): out = {} if has_task1: out["out1"] = # get outputs for task 1 if has_task2: out["out2"] = # get outputs for task 2 # ... return out
defloss_func(predictions, inputs): losses = {} if has_task1: # Or: if "out1" in predictions: losses["loss1"] = # get loss for task 1 if has_task2: losses["loss2"] = # get loss for task 2 # ... return losses
classModule(nn.Module): defforward(self, inputs): losses = {} if has_task1: out = # get outputs for task 1 losses["task1"] = # get loss for task 1 if has_task2: out = # get outputs for task 2 losses["task2"] = # get loss for task 2 # ... return losses
The right one is simpler in that it does not duplicate thebranches that enable different tasks/losses.In reality, these conditions are often more complex than a simple if,and branching is generally less straightforward to maintain.So it's beneficial to not have to repeat the logic.
Note: If you think a wrapper likemulti_loss_func({"task1": loss_func1, "task2": loss_func2})will help (like what Keras supports), it is not going to work wellbecause it doesn't know how to route the inputs/outputs to loss functions.
One may argue that separating "loss" from "model" is nice becausethen we can easily switch different loss functions independent of "model".That is indeed useful in many cases.However, in many algorithms, loss computation is simply not independent ofthe model and should not be switched arbitrarily.This could be due to:
Loss computation depends on internal states computed during model.forward, e.g.:
Loss needs to know which part of training data is sampled during forward.
Some predicted auxiliary attributes control whether a sample in a batch should participate in losses.
Losses such as activation regularization should naturally happen during forward.
In these cases, forcing a separation of "loss" and "model" will require "model" to return its internal states, causing an abstraction leak.
Different loss functions expect different representations of model's predictions. For example, these representations could be:
Discrete vs. one-hot encoding of class labels.
Boxes represented as absolute coordinates, or as reference anchors plus offsets.
Segmentation masks represented as polygons, binary bitmasks, or many other formats.
Since conversion between representations may be expensive or lossy, we'd like the model toproduce the exact representation needed by loss computation.Therefore, a separation would not make model independent of losses.On the contrary, it's even worse because loss-relatedlogic will be unnaturally split like this:
Separation
No Separation
classModel(nn.Module): def__call__(self, inputs): hidden_representation = layers(inputs) if use_loss1: # Return proper representation for loss1 return predict1(hidden_representation) if use_loss2: return predict2(hidden_representation)
defloss_func(pred, inputs): # pred could have different representations # depending on the loss function used if use_loss1: return loss1(pred, inputs, some_options) if use_loss2: return loss2(pred, inputs, other_options)
We can see in the above snippet that the model is in factnot independent of losses.It also makes loss_func a bad abstraction because the semanticsof its prediction argument is complex: it should bein different formats depending on which of loss{1,2} is used.In the version with no separation, it's very clearthat the losses are computed using the right representation.
One may argue that the separation is helpful because it's nice to let the "model"return the same data in training and inference.This makes sense for simple models where training and inference share most of the logic.For example, in a standard classification model shown below,we can let the "model" object return logits, which will be useful in bothtraining and inference.
But many models don't have a clean separation like this.In theory, training and inference only have to share (some) trained weights,but don't necessarily have to share any logic.Many object detection models, for example, do not compute "predictions" in trainingand do not compute losses in inference.A simplified diagram of Region-Proposal Network (RPN)of a two-stage detector looks like this during training:
Any attempt to split a complicated algorithm like this into "model" and "loss function" will:
Force "model" to return its internal algorithmic details that are not useful anywhere,except in a corresponding "loss function". This is an abstraction leak.
Make code harder to read, because closely related logic is separated in an unnatural way.
Lead to higher memory usage (if executing eagerly), because some internal stateshave to be kept alive until "loss function" is called after the "model".
Therefore, it's unrealistic to expect that there is a nice separation, or that "model" can producea consistent format in both training and inference.A better design is to include loss computation in the model's training-mode forward, i.e., let model outputlosses in training, but predictions in inference.
Trainer Does Not Need to Know about the Separation¶
In the "no separation" design, users provide a "model" that returns losses.This model internally can still use separation of "loss function" and "forward logic"as long as it makes sense for this specific model.However, trainer is no longer aware of the separation,and the trainer can no longer obtain the "outputs".
Will this become a limitation of the "no separation" design? What if we'd like to do something with"outputs"? My answer is:
For 99% of the use cases where the "outputs" don't directly affect the training loop structure,trainer doesn't need to know about "outputs".
For use cases where the trainer does something non-critical (not affecting loop structure) with "outputs",a proper design would move such responsibility away from trainer.
For example, writing "outputs" to tensorboard shouldn't be a responsibility of trainer. A commonapproach is to use a context-based systemthat allows users to simply call write_summary(outputs) inside their model.
For other obscure use cases, they should have custom trainers anyway.
Design is always a trade-off.Adding assumptions to a system might result in some benefits, but at the same time can cause trouble when the assumption isn't true.Finding a balance in between is difficult and often subjective.
The assumption that models have to come together with a separate "loss function", in my opinion, brings more trouble than it's worth.
]]><p>"Loss function" is one of the most basic concepts today in deep learning.
Despite that,
it is actually not necessarily a good programming abstraction when
designing general-purpose systems. A system should not assume that
a model always comes together with a separate "loss function".</p>How to Maintain Clean Core APIs for Researchhttps://ppwwyyxx.com/blog/2022/Maintain-Clean-Core-APIs-for-Research/2022-09-19T07:00:00.000Z2022-09-19T07:00:00.000ZBuilding a library for research and experiments is quite different from building other types of software.A key challenge is that, in research, abstractions and APIs are rarely set in stone:users may want to propose a slight variant or modification to literally ANYWHERE in the whole program,just because they have a new idea.
In deep learning libraries, these variants can be a different implementation of a layer,a change in optimization algorithm, or a small modification to the training logic, etc.
Designing and maintaining these "research APIs" is difficultdue to how frequently users want to change their behaviors.Such changes are often implemented by simply adding featuresto the target API they want to modify, e.g. by adding a new flag to the API,or by adding a new abstraction that generalizes the target API towards the users’ use case.
However, when maintaining a generic, core library meant to be adopted by diverse use cases for a long term,the above approach does not scale and poses many problems (discussed morebelow).
This note lists a few principles when working with "research APIs" that should help answer:
How to maintain a clean set of core APIs in research libraries.
How library maintainers & users collaborate to achieveusers’ diverse needs without complicating the core APIs.
Researchers' job is about doing things in new ways.Hence their needs are so diverse that a core library should not aim to include or implementfeatures for all possible use cases. The library should aim to only include the most popularand standardized features (more on the criteria later).
Core should allow most features to be implemented out-of-core¶
For features not included in the core, there should be a way for users to implementthem in a non-intrusive way, often out-of-core as extensions,without too much overhead / repetition.
This requires a continuous design evolution to make the core more modular and composable,so that core code can be reused in users’ new implementation.
A good sanity check for library maintainers is to ask the following question:
For any feature currently in the core library, suppose we remove it today, how much effort would it takefor users to reimplement it out-of-core?
A well-designed library should be decoupled such that most of its features are just extensions of itself, and they can beimplemented out-of-core the same way as it is in the core.
There are 3 criteria for feature inclusion in core, ordered by their importance.
Popularity: Whether the feature is used by many users
Standardization: Whether the feature’s API is standardized/agreed among its users
Simplicity: Whether the feature is simple to implement
To understand the criteria more, let’s ask: what if the feature is —
Popular but not standardized: sometimes a feature is popular, but its users don’t yet align on the properparameterization, its API, or the subtle implementation details. Including such features is risky, as it may create unclearsemantics orimpede its standardization in the future. It’s still OK to include it if it’s very popular (popularity is the #1 most important criteria),but try to do it in a composable way and with warning signs.
As a negative example, "Transformer" is a popular but not standarized feature.It's included in Pytorch, but received many complaints,and many projects (e.g. fairseq, detr)eventually have to fork and reimplement their own Transformer.
Simple but not popular/standardized: Simplicity alone is not sufficient for inclusion, no matter how simple it is.Because if everyone adds a simple feature they need, together it becomes complex.
Popular, standardized but not simple: Simplicity is the #3 important factors.If something is complex but very popular & standardized (e.g. BatchNorm being a headachefor DL library developers), it should be included. In fact this is where a library couldprovide a lot of value to users.
When a user wants to change the behavior of a "research API" def func() defined in core,adding new arguments is often the quickest way to get things done. But it may introduce a number of maintenance problems.
deffunc(flag=False): ... if flag: Variant here ...
deffunc(multiplier_of_x=1.0): ... x = x * multiplier_of_x ..
Adding a simple argument to control the behavior like above is OK,if we think that the new option is very clear and popular.But as a "research API", many users will want to add their own customizations.This could lead to the following problems:
Poor Code health: The library may gradually accumulates too many features that are:
Hard to read due to branching (as there are too many flags).Ideally, readers should not pay too much extra mental overhead for logic they don’t care about
Hard to maintain because the contextual knowledge about them is distributed among different developers
Confusing behaviors: More and more features added over time may not interact with each other in a clear way,causing confusing or silent wrong behaviors
E.g. featureA becomes a no-op when featureB is enabled
E.g. featureA and featureB are conflicting / overlapping in semantics
E.g. featureA’s semantics becomes undefined when featureB is enabled
Good characterization of the abovementioned problems
"More general" may become "less general":A common argument for adding options like this, is thatit doesn't change existing behavior and"makes the function more general".
However, keep in mind that when a function becomes more general in one aspect,it's often less general in other aspects.Generalizing towards one direction may not be a net win, becauseresearch code has too many possible directions to generalize towards, andpicking one direction may affect its eligibility to pick others in the future.We will show what this means shortly.
To add a new behavior to func, one can also add a callback argument:
Inject custom behaviors through callbacks:
Use object.method as callbacks:
deffunc(callback): ... callback(x) ...
deffunc(obj): ... obj.do(x) ...
This appears useful, since the custom logic is not implemented in core,but in a user-provided callback.For example, given the original code below (left), a researcher who wants to compute y differently may proposea compute_y_fn argument like below (right).
Original:
With callbacks:
deffunc(): a = something ... x = something ... y = x.norm() z = ... ...
deffunc(compute_y_fn=lambda x: x.norm()): a = something ... x = something ... y = compute_y_fn(x) z = ... ...
However, this design often turns out to be more problematic:
Premature abstractions: Assumptions/constraints are implicitly created about where the callback is triggered,what arguments it needs and what it returns. These assumptions may be bad.
In the example above, a 2nd researcher may want to computey using both x and a; a 3rd researcher may want to compute y, zin one function compute_y_z_fn because it's more efficient. These variants conflict with the 1st researcher's design.
In the future, after seeing enough use cases, we might realize that a xyz = compute_xyz(a) is a truly good abstraction.However, at that time the premature abstraction of compute_y_fn will get in our way implementing compute_xyz.In other words, although the current design makes the computation of y "more general", the abstraction limits our abilityto generalize the function in other ways. That's why we said earlier that "more general means less general".
Obscure logic: readers can't easily figure out what this function does by reading its code: they needto look at the caller of this function to see which callback is supplied, and thenlook at the implementation of the callback function. The aforementioned issue of "confusing behaviors" also applies here.
Sometimes callbacks are good and useful abstractions. But because it is too powerful, I saw it frequently abused to altera behavior into something that's strongly overfitted to a small number of use cases.This happens especially often in research code.In code reviews, I usually frown upon APIs that require callbacks/user-defined functions.
To customize a "research API" def func() defined in core, we have the following options:
Out-of-core, e.g. a def func_v2() in user code.(Or a class ClassV2 for classes).
In-core, but keep existing APIs unaffected, e.g. a def func_v2() in core.
In-core, and change existing APIs, e.g. a new option in def func(option).
The best choice is heavily subjective and should be evaluated case-by-case.Due to the concern of new arguments,in general we recommend methods (1) and (2), i.e. prefer forking func() over changing func().
If a fork will create significant code duplication, choose (2) and try to reduce duplication with private abstractions (see next section).
Adding flags / simple args is acceptable for simple, clean, popular additions.
Adding callbacks / new abstraction requires scrutiny, and should come with more than a handful of use cases in mind.
This also echoesFlax design philosophy thatsays "prefer duplication over adding options / bad abstractions".
Users/developers may find that the core design is not good enough yet, and recreating a variantof func() without touching it may lead to too much code duplication.For example, ... is duplicated between the two functions below.
Existing API in core
New variant
deffunc(): ... ...
deffunc_v2(): ... New custom logic in between. ...
Such duplication is acceptable for a short term.We do NOT mean to encourage users to heavily fork core code.Instead, users and core developers should engage and aim to evolve the core design to reduce duplication— but design change takes time to happen, and duplication is preferred before a good design is found.
The most risk-free way to reduce duplications is by moving them into shared reusable code:
Existing API in core
New variant
deffunc(): ... # fewer duplications than before _reusable_parts() ...
deffunc_v2(): ... # fewer duplications than before _reusable_parts() Variant logic inserted here ...
This should be the preferred way to reduce duplications. The benefits are:
No change to the API of func(), hence little risk.
Create reusable sub-routines that may be useful to new use cases.
However, there are also challenges:
This adds a new API (_reusable_parts()) to maintain.
Sometimes it's difficult to identify a clean & reusable subset that can be easily split from the duplicated code.It may require small refactoring to expose a clean subset.Also, remember that the approach that reduces the most duplications might not be the one with the best abstraction.
The above challenges are less significant if _reusable_parts() is private. Therefore:
If func_v2() is in core, make _reusable_parts() private.
If func_v2() must be out-of-core, consider _reusable_parts() as "internal/experimental APIs".
Inheritance, e.g. class ModuleV2(ModuleCore) may also reduce duplication between two variants.However, this is generally less preferable than composition like above. The reason is similar towhy callbacks are not preferred: overriding methods is like passing callbacks - they are both user-definedfunctions and suffer from the same limitations: users are constrained by the assumption ofwhen/where/how the methods/callbacks are triggered.
It is discussed that we prefer "adding a new implementation" over"adding new conditional branches to the existing implementation",but sometimes branching has to happen somewhere anyway -to choose between the two implementations.
If branching has to happen, we prefer it at earlier, shallower code path:
Branch earlier
Branch later
classModule(): def__call__(self, flag): ... if flag: func() else: func_v2() ...
By branching earlier, we keep a clean func() unaffected by the new variant.This recommendation is consistent with the preference to fork func_v2(), not to add flag to func().
]]><p>Building a library for research and experiments is quite different from building other types of software.
A key challenge is that, in research, abstractions and APIs are rarely set in stone:
users may want to propose a slight variant or modification to literally <strong>ANYWHERE</strong> in the whole program,
just because they have a new idea.</p>Automatically Flatten & Unflatten Nested Containershttps://ppwwyyxx.com/blog/2022/Automatically-Flatten-Unflatten-Nested-Containers/2022-06-16T07:00:00.000Z2022-06-16T07:00:00.000ZThis post is about a small functionality that is found useful in TensorFlow / JAX / PyTorch.
Low-level components of these systems often use a plain list of values/tensorsas inputs & outputs.However, end-users that develop models often want to work with morecomplicated data structures:Dict[str, Any], List[Any], custom classes, and their nested combinations.Therefore, we need bidirectional conversion between nested structures and a plain list of tensors.I found that different libraries invent similar approaches to solve this problem, and it's interesting to list them here.
Though many simple deep learning models just needs a few inputs/outputs tensors,nested containers are useful abstractions in advanced models.This is because many concepts are naturally represented by >1 tensors, e.g.:
A sparse tensor consists of values + indices
A masked tensor (common in transformers) is represented by a tensor + its binary mask
A segmentation mask can be represented in different ways:
single whole-image bitmask tensor
shape + bounding box + mask within the box (aka "RoIMask")
shape + list of polygons
shape + run-length encoding
Detected objects in an image are represented by boxes + scores + labels + many possible attributes
A list of variable-length vectors may be represented by a concatenated vector + a length vector, i.e.:
When a frequently-used concept has natural complexity like above, representing itin a flat structure consisting of only regular tensors (e.g. Dict[str, Tensor]) may result in ugly code.A multi-level nested structure is a better alternative.Take sparse tensor as a simple example:
Use nested containers
Use a flat Dict[str, Tensor]
Representation
{"a": SparseTensor, "b": Tensor}
SparseTensor can be a namedtuple/dataclass, or a new class.
Despite the benefits, lower-level stacks often ignore these abstractionsand choose to use a "flat" interface: their inputs & outputs are a flat list of values / Tensors.This is because:(i) the abstraction may no longer be useful in lower level;(ii) a simple structure simplifies their implementation;(iii) a flat list is a data structure available even in lower-level languages & systems.
Therefore, conversion from a nested structure to a plain list of values is important.This is often referred to as "flatten".It is pretty straightforward to flatten a container recursively -- like the following flatten function:
defflatten(container): ifisinstance(container, Sequence): returnlist(itertools.chain.from_iterable(flatten(x) for x in container)) elifisinstance(container, Mapping): return flatten(list(container.values())) elifisinstance(container, Tensor): return [container] # ... handle other containers
x, y, z = torch.rand(3) obj = [{"a": x, "b": [y, {"c": z}]}] flatten(obj) # ==> [x, y, z]
The inverse of flatten is also important: given new values [x2, y2, z2],we want the unflatten function below to construct obj2 that has the samestructure as obj.
How do we implement unflatten?Apparently, we need to give it a representation of structure (noted as a placeholder ??? in the above code).There are two high-level approaches to solve this problem:
Schema-based: when flattening a container, explicitly record its structure/schema to be used for unflatten.Its API may look like this:
>>>from jax.tree_util import tree_flatten, tree_unflatten >>>obj = [3, ([5, 6], {"name": [7, 9], "name2": 3})] >>>res, schema = tree_flatten(obj) >>>res # Flattened results: [3, 5, 6, 7, 9, 3] >>>schema # An explicit representation of the container's structure PyTreeDef([*, ([*, *], {'name': [*, *], 'name2': *})]) >>># Construct a nested container using the given values and the structure/schema: >>>tree_unflatten(schema, [1, 2, 3, 4, 5, 6]) [1, ([2, 3], {'name': [4, 5], 'name2': 6})]
JAX's low level components accept/return flat tensors, so functions can be transformed and optimized more easily.Since end-users need nested containers, JAX transformations supports pytree containers,which by default includes flattening & unflattening for common Python containers.It further allows users to register custom classes byregister_pytree_node.
Pytree uses a schema-based implementation that we already show-cased above.
When we need to independently process each leaf of the container, JAX provides another handyfunction tree_map:
deftree_map(f, obj): values, schema = flatten(obj) new_values = [f(x) for x in values] return unflatten(schema, new_values)
PyTorch also adds a similar implementation of pytree at herethat is used in its FX tracing.
torch.jit.trace(model, inputs) executes the model with given inputs, and returns a graph representationof the model's execution.This is one of the most common methods (and the best IMO) how PyTorch models are exported today.However, it limits model's input & output format.
In order to trace models with more complicated inputs & outputs,I created the TracingAdaptertool in detectron2, that flattens/unflattens a model's inputs and outputs into simple Tuple[Tensor] to make it traceable.A minimal implementation of it may look like this:
defforward(self, *flattened_inputs): inputs = self.inputs_schema(flattened_inputs) # This does unflatten outputs = self.model(inputs) flattened_outputs, self.outputs_schema = flatten(outputs) return flattened_outputs # torch.jit.trace(model, inputs) # Fails because inputs is a nested container flattened_inputs, _ = flatten(inputs) torch.jit.trace(TracingAdapter(model, inputs), flattened_inputs) # Succeeds
where flatten uses a schema-based implementation that can be found in this file.Coincidentally, its interface looks like JAX's pytree:
>>>from detectron2.export.flatten import flatten_to_tuple as flatten >>>obj = [3, ([5, 6], {"name": [7, 9], "name2": 3})] >>>res, schema = flatten(obj) >>>res # Flattened results: (3, 5, 6, 7, 9, 3) >>>schema # An explicit representation of the container's structure ListSchema(schemas=[IdentitySchema(), TupleSchema(schemas=[ListSchema(schemas=[IdentitySchema(), IdentitySchema()], sizes=[1, 1]), DictSchema(schemas=[ListSchema(schemas=[IdentitySchema(), IdentitySchema()], sizes=[1, 1]), IdentitySchema()], sizes=[2, 1], keys=['name', 'name2'])], sizes=[2, 3])], sizes=[1, 5]) >>># Construct a nested container using the given values and the structure/schema: >>>schema((1, 2, 3, 4, 5, 6)) [1, ([2, 3], {'name': [4, 5], 'name2': 6})]
Perception models in Meta accept a wide range of inputs/outputs formats:they may take any number of images plus auxiliary data as inputs, andpredict boxes, masks, keypoints or any other interesting attributes as outputs.But deployment prefers a flat interface for optimizability and interoperability.TracingAdapter's automatic flattening and unflattening mechanism has freed engineers fromwriting format conversion glue code when deploying these models.
In addition to deployment, TracingAdapter is also useful in a few other places to smooththe experience of torch.jit.trace:
Tensorboard graph visualization: PyTorch's tensorboard writer has a add_graphmethod that visualizes the graph structure in tensorboard.The method requires flattened inputs,therefore TracingAdapter can be used like this.
PyTorch's ONNX export is also based on tracing. So TracingAdapter is useful as well, e.g. here.
The unflatten function requires a container, and it will flatten this container on-the-fly whilesimultaneously "pack" flat values into the structure of this container. Here is an official example (note that dict values are ordered by keys):
tf.nest.{flatten,pack_sequence_as} are widely used in TensorFlow because many low-level components have a flat interface, especially forinterop with C APIs.
~/tensorflow/tensorflow/python$ ag pack_sequence_as | wc -l 243
TFv1's session.run(fetches) supports fetching nested containers.This is demonstrated in an example from theofficial documentation:
a = tf.constant([10, 20]) b = tf.constant([1.0, 2.0]) MyData = collections.namedtuple('MyData', ['a', 'b'])
v = session.run({'k1': MyData(a, b), 'k2': [b, a]}) # v is a dict with: # v['k1'] is a MyData namedtuple with 'a' (the numpy array [10, 20]) and # 'b' (the numpy array [1.0, 2.0]) # v['k2'] is a list with the numpy array [1.0, 2.0] and the numpy array # [10, 20].
This powerful interface exists in TF's Python client only.The client interacts with the C API's TF_SessionRunwhich only accepts a plain array of inputs/outputs.Therefore, the client needs to:
Flatten the container to a plain array of tensors
Send this array to the C API to obtain an array of results
Unflatten / reconstruct the container using the results
The flatten/unflatten logic uses a schema-based implementation in the client's FetchMapper.This implementation is a bit more complicated due toan extra guarantee thatthe flattened tensors are unique. (This is to ensure the client won't fetch the same tensor twice in one call;this cannot be done by using tf.nest.)
In addition to builtin Python containers, FetchMapper supports a few other TF containers(such as SparseTensor) and can be extended to new containers by registering conversion functions.
]]><p>This post is about a small functionality that is found useful in TensorFlow / JAX / PyTorch.</p>
<p>Low-level components of these systems often use a plain list of values/tensors
as inputs & outputs.
However, end-users that develop models often want to work with more
complicated data structures:
<code>Dict[str, Any]</code>, <code>List[Any]</code>, custom classes, and their nested combinations.
Therefore, we need bidirectional conversion between nested structures and a plain list of tensors.
I found that different libraries invent similar approaches to solve this problem, and it's interesting to list them here.</p>TorchScript: Tracing vs. Scriptinghttps://ppwwyyxx.com/blog/2022/TorchScript-Tracing-vs-Scripting/2022-05-23T06:59:00.000Z2022-05-23T06:59:00.000ZPyTorch provides two methods to turn an nn.Module into agraph represented in TorchScript format: tracing and scripting.This article will:
Compare their pros and cons, with a focus on useful tips for tracing.
Try to convince you that torch.jit.trace should be preferred over torch.jit.scriptfor deployment of non-trivial models.
The second point might be an uncommon opinion:If I Google "tracing vs scripting", the first articlerecommends scripting as default.But tracing has many advantages.In fact, by the time I left, "tracing as default, scripting only when necessary" is thestrategy all detection & segmentation models in Facebook/Meta products are deployed.
Why tracing is better? TL;DR: (i) it will not damage the code quality; (ii) its main limitations can beaddressed by mixing with scripting.
We start by disambiguate some common terminologies:
Export: refers to the process that turns a model written in eager-mode Pythoncode into a graph that describes the computation.
Tracing: An export method. It runs a model with certain inputs, and "traces / records" all the operationsthat are executed into a graph.
torch.jit.trace is an export API that uses tracing, used like torch.jit.trace(model, input).See its tutorialand API.
Scripting: Another export method. It parses the Python source code of the model, and compiles the code into agraph.
torch.jit.script is an export API that uses scripting, used like torch.jit.script(model).See its tutorialand API.
TorchScript: This is an overloaded term
It often refers to the representation / format of the exported graph.
But sometimes it refers to the scripting export method.
To avoid confusion, I'll never use "TorchScript" alone in this article.I'll use "TS-format" to refer to the format, and "scripting" to refer to the export method.
Because this term is used with ambiguity, it may have caused the impression that "scripting" is the"official / preferred" way to create a TS-format model. But that's not necessarily true.
(Torch)Scriptable: A model is "scriptable" if torch.jit.script(model) succeeds, i.e. it canbe exported by scripting.
Traceable: A model is "traceable" if torch.jit.trace(model, input) succeeds for atypical input.
Generalize: A traced model (returned object of trace()) "generalizes" to other inputs(different from the inputs given during tracing), if it can inference correctly when given other inputs.Scripted models always generalize.
Dynamic control flow or data-dependent control flow: control flow where the operatorsto be executed depend on the input data, e.g. for a Tensor x:
if x[0] == 4: x += 1 is a dynamic control flow.
model: nn.Sequential = ... for m in model: x = m(x)
is NOT a dynamic control flow.
classA(nn.Module): backbone: nn.Module head: Optiona[nn.Module] defforward(self, x): x = self.backbone(x) ifself.head isnotNone: x = self.head(x) return x
If anyone says "we'll make Python better by writing a compiler for it", you should immediatelybe alarmed and know that this is extremely difficult.Python is too big and too dynamic. A compiler can only support a subset of its syntax features and builtins, at best --the scripting compiler in PyTorch is no exception.
What subset of Python does this compiler support?A rough answer is: the compiler hasgood support for the most basic syntax, but medium to no support for anything more complicated (classes, builtins like range and zip, dynamic types, etc.).But there is no clear answer: even the developers of the compiler usually need to run the code to see if it can be compiled or not.
The incomplete Python compiler limits how users can write code.Though there isn't a clear list of constraints,I can tell from my experience what impact they have had on large projects:code quality is the cost of scriptability.
To make their code scriptable / compilable by the scripting compiler,most projects choose to stay on the "safe side" to only use basic syntax of Python:no/few custom structures, no builtins, no inheritance, no Union, no **kwargs, no lambda, no dynamic types, etc.
This is because these "advanced" compiler features are either not supported at all, or with "partial support"which is not robust enough: they may work in some cases but fail in others.And because there is no clear spec of what is supported,users are unable to reason about or workaround the failures.Therefore, eventually users move to and stay on the safe side.
The terrible consequence is that:developers stop making abstractions / exploring useful language featuresdue to concerns in scriptability.
A related hack that many projects do is to rewrite part of the code for scripting:create a separate, inference-only forward codepath that makes the compiler happy.This also makes the project harder to maintain.
Detectron2 supports scripting, but the story was a bit different: it did not go downhill in code quality which we value a lot in research.Instead, with some creativity and direct support from PyTorch team (and some volunteered help from Alibaba engineers), we managed to make most modelsscriptable without removing any abstractions.
However, it is not an easy task:we had to add dozens of syntax fixes to the compiler, find creative workarounds,and develop some hacky patches in detectron2 that are inthis file(which honestly could affect maintainability in the long term).I would not recommend other large projects to aim for "scriptability without losing abstractions" unlessthey are also closely supported by PyTorch team.
If you think "scripting seems to work for my project"so let's embrace it, I might advise against it for the following reasons,based on my past experiences with a few projects that support scripting:
What "works" might be more brittle than you think (unless you limit yourself to the basic syntax):Your code might happen to compile now, but one day you'll add a few innocent changes to your modeland find that the compiler refuses it.
Basic syntax is not enough:Even if more complex abstractions don't appear necessary to your project at the moment,if the project is expected to grow, it will require more language features in the future.
Take a multi-task detector for example:
There could be 10s of inputs, so it's preferable to use some structures/classes.
The same data can have different representations (e.g. different ways to represent a segmentation mask),which demands Union or more dynamic types.
There are many architectural choices of a detector, which makes inheritance useful.
Large, growing projects definitely need evolving abstractions to stay healthy.
Code quality could severely deteriorate:Ugly code starts to accumulate, because clean code sometimes just doesn't compile.Also, due to syntax limitations of the compiler,abstractions cannot be easily made to clean up the ugliness.The health of the project gradually goes downhill.
Below is a complaint in PyTorch issues.The issue itself is just one small papercut of scripting,but similar complaints were heard many times.The status-quo is: scripting forces you to write ugly code, so only use it when necessary.
What it takes to make a model traceable is very clear, and has a much smaller impact on code health.
First, neither scripting nor tracing works if the model is not even a proper single-device, connected graph representable in TS-format.For example, if the model has DataParallel submodules, or if the modelconverts tensors to numpy arrays and calls OpenCV functions, etc, you'll have to refactor it.
Apart from this obvious constraint, there are only two extra requirements for traceability.
Input/output format
Model's inputs/outputs have to be Union[Tensor, Tuple[Tensor], Dict[str, Tensor]]or their nested combinations. Note that values in a dict have to belong to the same type.
Similar constraints exist for scripting as well.However, in tracing the constraint does not apply to submodules:submodules can use any input/output format: dicts of Any, classes, kwargs, anything that Python supports.Only the top-level model is required to use the constraint format.
This makes the constraint very easy to satisfy.If the model uses richer formats, just create a simple wrapper around it that converts to/fromTuple[Tensor].Detectron2 even automates this for all its models by a universal wrapperlike this:
outputs = model(inputs) # inputs/outputs are rich structure, e.g. dicts or classes # torch.jit.trace(model, inputs) # FAIL! unsupported format adapter = TracingAdapter(model, inputs) traced = torch.jit.trace(adapter, adapter.flattened_inputs) # Can now trace the model
# Traced model can only produce flattened outputs (tuple of tensors): flattened_outputs = traced(*adapter.flattened_inputs) # Adapter knows how to convert it back to the rich structure (new_outputs == outputs): new_outputs = adapter.outputs_schema(flattened_outputs)
Expressions like tensor.size(0), tensor.size()[1], tensor.shape[2]are integers in eager mode, but Tensors in tracing mode.Such difference is necessary so that during tracing, shape computation can becaptured as symbolic operations in the graph.An example is given in the next section about generalization.
Due to different return types,a model may be untraceable if parts of it assume shapes are integers.This usually can be fixed quite easily by handling both types in the code.A helpful function is torch.jit.is_tracingwhich checks if the code is executed in tracing mode.
That's all it takes for traceability - most importantly, any Python syntax is allowed in model implementation, because tracing does not careabout syntax at all.
Just being "traceable" is not sufficient.The biggest problem with tracing, is that it may not generalize to other inputs.This problem happens in the following cases:
In this example, due to dynamic control flow,the trace only keeps one branch of the condition, and will not generalize to certain (negative) inputs.
Capture variables as constants:
>>>a, b = torch.rand(1), torch.rand(2) >>>deff1(x): return torch.arange(x.shape[0]) >>>deff2(x): return torch.arange(len(x)) >>># See if the two traces generalize from a to b: >>>torch.jit.trace(f1, a)(b) tensor([0, 1]) >>>torch.jit.trace(f2, a)(b) tensor([0]) # WRONG! >>># Why f2 does not generalize? Let's compare their code: >>>print(torch.jit.trace(f1, a).code, torch.jit.trace(f2, a).code) def f1(x: Tensor) -> Tensor: _0 = ops.prim.NumToTensor(torch.size(x, 0)) _1 = torch.arange(annotate(number, _0), dtype=None, layout=0, device=torch.device("cpu"), pin_memory=False) return _1 def f2(x: Tensor) -> Tensor: _0 = torch.arange(1, dtype=None, layout=0, device=torch.device("cpu"), pin_memory=False) return _0
Intermediate computation results of a non-Tensor type (in this case, an int type) may be captured as constants, using thevalue observed during tracing. This causes the trace to not generalize.
In addition to len(), this issue can also appear in:
.item() which converts tensors to int/float.
Any other code that converts torch types to numpy/python primitives.
Similarly, operators that accept a device argument will remember the device used during tracing (this canbe seen in m.code).So the trace may not generalize to inputs on a different device.Such generalization is almost never needed, because deployment usually has a target device.
The above problems are annoying and often silent (warnings, but no errors),but they can be successfully addressed by good practice and tools:
Pay attention to TracerWarning: In the first two examples above, torch.jit.trace actually emits warnings.The first example prints:
a.py:3: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if x.sum() > 0:
Paying attention to these warnings (or even better, catch them)will expose most generalization problems of tracing.
Note that the "capture device" case does not print warnings because tracing was not designed to support such generalization at all.
Unittests for parity: Unittests should be done after export and before deployment, to verify thatthe exported model produces the same outputs as the original eager-mode model, i.e.
If generalization across shapes is needed (not always needed), input2 should have differentshapes from input1.
Detectron2 has many generalization tests, e.g. thisand this.Once a gap is found, inspecting the code of the exported TS-format model can uncover the place whereit fails to generalize.
Avoid unnecessary "special case" conditions:Avoid conditions like
that handles special cases such as empty inputs.Instead, improve self.layers or its underlying kernel so it supports empty inputs.This would result in cleaner code and also improve tracing.This is why I'm involved in many PyTorch issues that improve support for emptyinputs, such as#12013,#36530,#56998.Most PyTorch operations work perfectly with empty inputs,so such branching is hardly needed.
Use symbolic shapes: As mentioned earlier, tensor.size() returns Tensor during tracing, sothat shape computations are captured in the graph.Users should avoid accidentally turning tensor shapes into constants:
Use tensor.size(0) instead of len(tensor) because the latter is an int.For custom classes, implement a .size method or use .__len__() instead of len(), e.g. like here.
Do not convert sizes by int() or torch.as_tensor because they will capture constants.This helper functionis useful to convert sizes into a tensor, in a way that works in both tracing and eager mode.
Mix tracing and scripting: they can be mixed together, so you can use scriptingon the small portion of code that tracing does not work correctly.This can fix almost all problems of tracing. More on this below.
Tracing and scripting both have their own problems, and thebest solution is usually to mix them together.This gives us the best of both worlds.
To minimize the negative impact on code quality,we should use tracing for the majority of logic, and use scripting only when necessary.
Use @script_if_tracing: Inside torch.jit.trace, the @script_if_tracingdecorator can compile functions by scripting.Typically, this only requires a small refactor of the forward logic to separate the parts that need tobe compiled (the parts with control flow):
defforward(self, ...): # ... some forward logic @torch.jit.script_if_tracing def_inner_impl(x, y, z, flag: bool): # use control flow, etc. return ... output = _inner_impl(x, y, z, flag) # ... other forward logic
By scripting only the parts that need it,the code quality damage is strictly smaller than making the entire model scriptable,and it does not affect the module's forward interface at all.
The function decorated by @script_if_tracing has to be a pure function that does not contain modules.Therefore, sometimes a bit more refactoring is needed:
Before Refactoring
After Refactoring
# This branch cannot be compiled by # @script_if_tracing, because it # refers to `self.layers` if x.numel() > 0: x = preprocess(x) output = self.layers(x) else: # Create empty outputs output = torch.zeros(...)
# This branch can be compiled by @script_if_tracing if x.numel() > 0: x = preprocess(x) else: # Create empty inputs x = torch.zeros(...) # Needs to make sure self.layers accept empty # inputs. If necessary, add such condition branch # into self.layers as well. output = self.layers(x)
In fact, for most vision models, dynamic control flow is needed only in a few submodules whereit's easy to be scriptable.To show how rare it is needed, the entire detectron2 only has two functions decorated with @script_if_tracing due to control flows:paste_masksand heatmaps_to_keypoints,both for post-processing only.A few other functions are also decorated to generalize across devices (a very rare requirement).
In this example, suppose submodule cannot be traced correctly, we can script it before tracing.However I do not recommend it.If possible, I will suggest using @script_if_tracinginside submodule.forward instead,so that scripting is limited to the internals of the submodule,without affecting the module's interface.
this uses a traced submodule during scripting.This looks nice, but is not so useful in practice: it will affect the interfaceof submodule, requiring it to only accept/return Tuple[Tensor] -- this is abig constraint that might hurt code quality even more than scripting.
A rare scenario where "tracing a submodule" is useful, is this:
classA(nn.Module): defforward(self, x): # Dispatch to different submodules based on a dynamic, data-dependent condition: returnself.submodule1(x) if x.sum() > 0elseself.submodule2(x)
@script_if_tracing cannot compile such control flow because it only supports pure functions.If submodule{1,2} are complex and cannot be scripted,using traced submodules in a scripted parent A is the best option.
Merge multiple traces:
Scripted models support two more features that traced models don't:
Control flow conditioned on attributes: a scripted module can have mutable attributes (e.g. a boolean flag)that affect control flows. Traced modules do not have control flows.
Multiple methods: a traced module only supports forward(), but a scripted module can havemultiple methods.
Actually, both features above are doing the same thing: they allow an exported model to be used indifferent ways, i.e. execute different sequences of operators as requested by the caller.
Below is an example scenario where such feature is useful: if Detector is scripted, the caller can mutate itsdo_keypoint attribute to control its behavior, or call predict_keypoint methoddirectly if needed.
If a model is both traceable and scriptable,tracing always generates same or simpler graph (therefore likely faster).
Why?Because scripting tries to faithfully representyour Python code, even some of it are unnecessary. For example:it is not always smart enough to realize that someloops or data structures in the Python code are actually static and can be removed:
This example is very simple, so it actually has workarounds for scripting (use tuple instead of list),or the loop might get optimized in a later optimization pass.But the point is: the graph compiler is not always smart enough. For complicated models, scripting mightgenerate a graph with unnecessary complexity that's hard to optimize.
Tracing has clear limitations:I spent most of this article talking about the limitations of tracing and how to fix them.I actually think this is the advantage of tracing: it has clear limitations (and solutions),so you can reason about whether it works.
On the contrary, scripting is more like a black box:no one knows if it works before trying.I didn't mention a single trick about how to fix scripting:there are many of them, but it's not worth your time to probe and fix a black box.
Tracing has small blast radius:Both tracing and scripting affect how code can be written, but tracing has a much smaller blastradius, and causes much less damage:
It limits the input/output format, but on the outer-most module only. (And this issue can be automaticallysolved as discussed above.)
It needs some code changes to generalize (e.g. to mix scripting in tracing), but these changes only go into theinternal implementation of the affected modules, not their interfaces.
On the other hand, scripting has an impact on:
The interface of every module & submodule involved.
IMO, this is the biggest damage:Advanced syntax features are needed in interfaces, and I'm not willing to compromise on interface design.
This may end up affecting training as well because interface is often shared between training and inference.
Pretty much every line of code in the inference forward path.
Having a large blast radius is why scripting can do great harm to code quality.
Control flow vs. other Python syntax:PyTorch is loved by its users because they can "just write Python", and most importantly writePython control flows. But other syntax of Python are important as well.If being able to write Python control flow (scripting) means losing other great syntax,I'd rather give up on the ability to write Python control flow.
In fact, if PyTorch is less obsessed with Python control flow, and offers mesymbolic control flows such as torch.cond like this (similar to the API of tf.cond):
Then f could be traced correctly and I would be happy to use this, no longer having to worryabout scripting.TensorFlow AutoGraphis a great example that automates this idea.
]]><p>PyTorch provides two methods to turn an <code>nn.Module</code> into a
graph represented in TorchScript format: tracing and scripting.
This article will:</p>
<ol>
<li>Compare their pros and cons, with a focus on useful tips for tracing.</li>
<li>Try to convince you that <strong><code>torch.jit.trace</code> should be preferred over <code>torch.jit.script</code></strong>
for deployment of non-trivial models.</li>
</ol>谈谈Github上如何交流(4): {Feature,Pull} Requesthttps://ppwwyyxx.com/blog/2022/Github-Communications-4/2022-05-18T09:00:00.000Z2022-05-18T09:00:00.000Z
"We don't support X" 更多是关于服务范围的声明, 而不是关于项目功能的声明: "We don't support X" 不代表 "X doesn't work". "We don't support X" 的意思是 "We won't help you about X", 也即 "我们不管 X 能不能用".
例如 detectron2 一直以来都在 windows 上可用甚至还有 CI 测试, 但是从不 "support windows". 对于与 windows 有关的 issue 我们也就不提供帮助.
开源社区中的 "support" 一词大多数时候都是这个含义.
例如 Ubuntu 的 LTS (Long-term support) 中 "support" 的意思, 官方是这么解释的:"commitment to update, patch and maintain the software".
报 bug 时, 如果尝试了方案 X 没用, 不要说 "It still doesn't work after X". 因为这是模糊的人类语言, 没有区分下面两种情况:
如果程序的输出和之前相同, 那就说 "输出和之前完全相同"
如果程序的行为有任何变化, 那么应复制粘贴新的 log
代词常常自带歧义, 应注意或减少代词的使用.
这篇文章中有个例子:"I started FooApp. It put up a warning window. I tried to close it and it crashed." 这句话里每个 "it" 到底指什么?
类似的例子还有很多. 尽量使用更准确的语言来交流技术问题是个重要的好习惯.
]]>
<p>相比传统的邮件列表 / bugzilla/sourceforge 等开源平台,
github 把开源社区交流的成本 / 门槛降的很低, 因此交流的质量也常常随之下降.</p>
<p>我计划写几篇文章, 从 <strong>用户 (User)</strong> 和 <strong>维护者 (Maintainer)</strong> 两者的角度写写开源社区中如何使用 issue/PR 进行沟通, 希望能够:</p>
<ul>
<li>让普通用户学会提问, 学会以 maintainer 更容易接受的方式使用 issue/PR 做贡献.</li>
<li>给 maintainer 提供一些管理 issue/PR 的经验, 不要再为 issue 头疼.</li>
</ul>Effective Use of Python 'logging' Modulehttps://ppwwyyxx.com/blog/2022/Effective-Python-Logging/2022-04-23T07:00:00.000Z2022-04-23T07:00:00.000ZIn large systems, logs can be terrifying: they are huge in volume, and hard to understand.This note lists some suggestions and common misuse of Python's logging module,with the aim of:
Loggers are globally identified by a dot-separated name given to logging.getLogger(name), such as library.module.submodule.The logger named by an empty string is the "root logger".
Libraries must not call logging.basicConfigor configure the root logger in any way, unless requested by users.
Configuration of the root logger affects all logs, which is beyond the responsibility of anysingle library. Only application developers, i.e. those who create the program that interacts withusers, should determine how to configure the root logger.
Never call functions like logging.{info,error} from within a library, because they write to theroot logger. I've added this advice into CPython's official documentation.
When a library writes to the root logger, applications that use the library lose control over thelibrary's logging behavior: they cannot turn on/off the logs from the library, apply custom filter/formatter, or redirect thelogs from the library.
Instead, A library should write to a logger with easily and uniquely identifiable name, using
__name__, i.e. the current module name, is often a good logger name.Occasionally, __name__ is not good enough:
In some deep, internal submodules, a suffix of __name__ may beuninformative and can be removed, e.g. my_lib.submodule._internal._impl.Note that there is a trade-off between the name simplicity and the granularity of control.
In a monorepo, many modules may share a common prefix in __name__, e.g. company.company3.organization.Removing such a common prefix can simplify the logger names while still keeping them unique.
The "current function/class name" is often a bad logger name because:
The name is likely not unique.
The name could be the library's internal detail and therefore not identifiable to users.
The name may change more frequently (compared to module names), breaking users' logger configuration.
I wrote a simple scriptthat processes Python source files to automatically replace all logging.xxx by logging.getLogger(__name__).xxx.This script has created PRs in a few projects that misuse the root logger, such aspytorch/72649,tensorboardX/662.
I hope someone could create a linter that performs this check.
Libraries should not add a handler that's visible to users¶
Handlercan be attached to loggers to decide where/how to log a record.
Unless requested by users,a library should not add a handler anywhere (even not to its own logger),if the handler has a visible effect on users.This is because the application developer should make the final call onhow each library's logs are processed.Pre-existing handlers may cause issues such as duplicated logs.This suggestion is present in CPython's documentation here.
Examples of invisible handlers that libraries may add to their loggers are:
Logs are not only strings. logging.LogRecordis a rich structure with useful attributes, and users can even tag logs with custom attributes through the extra= argument.
Large, distributed systems should not rely on printing as the sole method of logging.Whenever logs are printed (to terminal or files), they have to be converted to strings.A lot of useful attributes, such as stack trace and line number, are often lost. The lack of structure also makes it difficult toparse and analyze the logs.
In addition to printing, we can also use an additional Handler to send structured logs to a logging service,such as Google Cloud Logging or Humio.The advantages include:
Can log a much larger volume than what a terminal can hold.
Can keep the rich structure of logs, enabling many features like querying by filters,rendering as different views, etc.Here is a list of logging featuresGoogle Cloud provides.
Can persist and aggregate logs across different instances/services/users, allowing developers to find patterns from big data.
In an MPI-like distributed job (e.g. many data-parallel deep learning training), workers often print almost identical logs.We should avoid printing them all to the terminal.
A good strategy could be:
Print all logs from the main process / first rank. Print some logs (e.g. only errors) from other workers.
Save all logs from all workers to separate files.
Save all logs from all workers to a structured logging service, tagged by worker rank.
Be aware that it's insufficient to only rely on logging module.A Python program may produce useful logs bypassing the logging module, e.g.:
print statements: they should be avoided, but may still exist.
Logs from C libraries linked to the process.
Logs from the interpreter itself, e.g. stack traces.
Logs printed by the shell, e.g. "Segmentation fault".
To not miss important logs, a comprehensive logging solution needs to integrateboth the structured logs from Python and the less common unstructured logs from the above sources.
]]><p>In large systems, logs can be terrifying: they are huge in volume, and hard to understand.
This note lists some suggestions and common misuse of Python's <a href="https://docs.python.org/3/library/logging.html" aria-label="logging — Logging facility for Python — Python 3.11.1 documentation" class="hint--top hint--rounded hint--no-animate hint--no-arrow"><code>logging</code></a> module,
with the aim of:</p>
<ul>
<li>Reduce redundant logs & spams from libraries.</li>
<li>Allow more control of logging behaviors.</li>
<li>Make logs more informative to users.</li>
</ul>How To Do Ablation Experimentshttps://ppwwyyxx.com/blog/2022/How-to-do-Ablation-Experiments/2022-04-08T07:00:00.000Z2022-04-08T07:00:00.000Z
Novel computer vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies.
...
We hope our work encourages further scrutiny in maintaining consistent methodology for both proposed innovations and baselines alike.
相关性使得不同变量带来的效果常常不可叠加: A 变量和 B 变量可能各自能够将结果提高 1%, 但是合在一起也只能提高 1%.
举个直观的例子: 假如某 paper 发明了一个新的 loss function 能够提高结果, 但也许这个 loss function 的主要原理是改变了 gradient magnitude. 这时候, 把旧的 loss function 的系数调一调也能得到一样的效果. 在这里, "loss function" 和它的系数就是两个相关的变量, 他们带来的效果是可以互相替代的. 如果研究者不注意, 写了这样一篇 loss function 的 paper, 被人发现跟调系数没区别, 那 paper 的价值就消失了.
正例: ResNet paper 多次 report 了 training error (Fig. 4, 6), 这也许会显得奇怪, 毕竟 training error 不是一个大家常用的 metric. 这是因为文章的大 claim 是关于 residual connection 对训练 / 优化有帮助, 而 deep plain network 难以优化.Training error 才是跟这个 claim 直接相关的 metric, validation error 变好只是 training error 变好的一个副产品.
正 / 反例: Optimizer 的根本目标是降低 training loss, 所以比较不同的 optimizer (例如 SGD/Adam) 的时候不能不看 training loss. 这篇 paper Sec. 5 就吐了这个槽: 有的 optimizer 跑出来的 validation error 更低就声称自己更好, 但是实际上发现它的 training loss 更高.
如何找一个好钉子, 设计实验来巧妙的突出 claim, 是一项技术活, 也是增加一篇 paper 的说服力的重要手段.
]]>
<p>延续 <a href="/blog/2021/DL-Experiments-and-Claims/" aria-label="Deep Learning Experiments and Claims" class="hint--top hint--rounded hint--no-animate hint--no-arrow">上一篇文章</a>,
再说一说怎么科学的在 paper 里做 ablations.</p>Where Are Pixels? -- a Deep Learning Perspectivehttps://ppwwyyxx.com/blog/2021/Where-are-Pixels/2021-06-11T07:00:00.000Z2021-06-11T07:00:00.000ZTechnically, an image is a function that maps a continuous domain, e.g.a box , to intensities such as (R, G, B).To store it on computer memory, an image is discretized to an array array[H][W], where each elementarray[i][j] is a pixel.
How does discretization work? How does a discrete pixel relate to the abstract notion of the underlying continuous image?These basic questions play an important role in computer graphics & computer vision algorithms.
This article discusses these low-level details, and how they affect our CNN models and deep learning libraries.If you ever wonder which resize function to use or whether you should add/subtract 0.5 or 1 to some pixel coordinates,you may find answers here.Interestingly, these details have contributed to many accuracy improvements in Detectronand Detectron2.
Sampling theory tells us howa continuous 2D signal is turned into a discrete array by sampling and filtering.
We choose a rectangular grid of points, from which we will draw samples.In order to make the best use of the produced samples, we have to know the exact location where every sample onthis grid is chosen.
Values on these sampled points are not directly retrieved from the original signal, but come from afiltering step that removes high-frequency components.A bad choice of filters can lead to aliasing effects.
Sampling and filtering are both important in basic image processing operations, such as resize.Resize operation takes a discrete image, resamples it, and creates a new image.The choice of sampling grid and sampling filter will then affect how such a basic operation is implemented.
For example, the paper On Buggy Resizing Libraries and Surprising Subtleties in FID Calculationstudies the filtering issues, and shows that the resize operations in many libraries(OpenCV, PyTorch, TensorFlow) don't take into account the low-pass filtering. This then leadsto incorrect deep learning evaluation.
In this article, we ignore the issue of sampling filter, and only study the coordinates of sampling grid.We'll see that this choice is also inconsistent among libraries, and can affect the design and performance of CNN models.
Pixels are located on a sampling grid we choose.Naturally, we would like to only consider rectangular grids where pixels are spaced evenly.But there are many other factors to be concerned with:
Offset: where is the first pixel located relative to the beginning of the signal?
Stride: what's the distance between two neighboring pixels?
Resolution: how many pixels are there?
(These terminologies may have a different meaning elsewhere, but this is how I define them in this article.)
For simplicity, we look at the one-dimensional case instead.We want to answer this question: for a 1D signal defined on , what is the sampling grid with stride=1?There are a few different choices:
In this figure, the green bars represent the 1D signal of length , and blue dotsrepresent the locations where point samples are taken.On top of each sample we labeled their coordinates,while on the bottom are their zero-based pixel indices.More formally, given a stride (which equals to 1 here), the offset and resolution of the grid are defined by the following table (assume is an integer):
They (or at least the first two) are all valid interpretations when we are given an array of pixels.The interpretation we choose affects how we implement operations and models,because they each have some unique weird properties.To understand them more, let's check how a 2x resize operation should be implemented under each interpretation.
We'll now see that a simple "2x resize" operation has many possible implementations.
A unique undesired property of ① is that, stride is not the inverse of resolution.So a 2x resize is ambiguous: we have to be clear about whether we want half of stride, or twice more pixels.The new grids after resize look like these:
Resize for grid ② & ③ aren't ambiguous:
You can easily verify that the 4 different resized grids still match thecorresponding definition in our table above.
For 2D case, the 2x resize in ①(twice more pixels) and ② look liks this (image credit: here),from which you can see why ①(twice more pixels) is also called align_corners:
These 4 different versions of 2x resize have some issues:
Extrapolation: ② and ③ both need extrapolation outside the border of the original grid to perform resize, but ① only needs interpolation.Extrapolation is sometimes undesirable.
Asymmetry: ③ is asymmetric, and it's probably a good reason to never use it. One consequence is that resize(flip(x)) != flip(resize(x)). All others are symmetric.
Information Loss: in ①(half of stride) and ③ , about half of the points on the new grid exist in the old grid.By not having to interpolate their values, we minimize the loss of information.However, in ①(twice more pixels) and ②, most or all of the new pixels need to be recomputed.
For resize with other arbitrary scale factors, all versions have information loss. But 2x/0.5x resize aremost common in deep learning.
The DeepLab series of segmentation models are famous for using grid ①(half of stride) for all the 2x resize.See here for words from its author.This matches the inconvenient image shapes they use, such as 321x513.I've heard opinions that the benefits of "no information loss" and "no extrapolation" may let itoutperform ② in segmentation, but I have yet to see more evidence.
What do libraries use? Situation is a bit messy. I'll list what I know and look forward to your help to add more.No guarantee they are all correct, since I didn't check the source code for all of them.
Library & Operation
Pixel Grid Convention
OpenCV cv2.resize
interpolation=LINEAR/CUBIC: ② interpolation=NEAREST: buggy, none of the above. issue interpolation=NEAREST_EXACT: ②
Pillow Image.resize
②
scikit-image transform.resize
②
PyTorch F.interpolate
mode=linear/cubic, align_corners=False: ② mode=linear/cubic, align_corners=True: ① mode=nearest: buggy like OpenCV. issue mode=nearest_exact: ②
PyTorch F.grid_sample
align_corners=False which I requested: ② align_corners=True: ①
TensorFlow tf.image.resize
TFv1 method=BILINEAR/NEAREST, align_corners=False: ③ TFv1 method=BILINEAR/NEAREST, align_corners=True: ① TFv2 method=BILINEAR/NEAREST: ② (In TFv2, align_corners option was removed)
It seems the mess is unique in the deep learning world. How come?From what I can tell, the history looks like this:
TensorFlow is the first place that introduces ③, in its initial open source.This was later considered as a bug and fixedin v1.14 using a new optionnamed half_pixel_centers=True that follows grid ②.
align_corners=True(①) appeared in TensorFlow 0.7 in 2016.I guess this was probably intended for DeepLab development and not for general use.
In TensorFlow v2, grid ② becomes the only version of resize, but it was too late.During all these years, the uncommon version (①) and the wrong version (③) have propagated to people'smodels and other libraries.
PyTorch's interpolate comes originally from upsample operation. Nearest upsample was buggy when it's firstadded in LuaTorch in 2014.Bilinear upsample was first added in LuaTorch in 2016 andused grid ①. Grid ② was added in 2018 to PyTorch under an align_corners=False option,and became the default since then.
We pick ② as the convention for grid locations, but this is not the end of the story!We now know the grid locations relative to the beginning of the signal are 0.5, 1.5, , but what are their absolute coordinates?In other words, where is the origin (0, 0) ?
This is just a choice of convention and has no substantial effect on any algorithms.Two of the graphics literature I listed above put the origin on the first pixel.This has the benefit that all pixel locations have integer coordinates, but then it's weird that the signal lies oninterval .This convention is sometimes referred to as "integer centers".
Another convention, "integer corners", or "half-integer centers", puts the origin at the beginning of the signal, so the first pixel is centered at (0.5, 0.5).The two conventions are demonstrated in this figure:
We choose "integer corners", and then willhave the following relationship between continuous coordinates and discrete pixel indices:
The choice doesn't matter for resize because absolute coordinates are not part of its API.However, for functions that accept or return absolute coordinates, we should be aware of their convention. For example:
cv2.findContours returns integer polygons represented by indices. So we always add 0.5 pixel to itsresults to obtain coordinates that match our convention.
cv2.warpAffine uses "integer centers" and this is complained about in this issue.In fact most OpenCV functions use the "integer centers" convention.
pycocotools.mask.frPyObjects renders polygons as masks. It accepts polygons in "integer corners"convention.Same for PIL.ImageDraw.polygon, but its results are 0.5 pixel "fatter" due to howits implemented. This has affected cityscapes annotations.
RoIAlign in torchvision takes a box in absolute coordinates that match our "integer corners" convention.
scipy.ndimage.map_coordinates takes coordinates in "integer centers" convention.
If a dataset is annotated with coordinates, we also need to know its choice of coordinate system. Thisinformation is often not provided by dataset owner, so we make guesses. For example, in COCO itappears that polygon annotations match our convention, butkeypoint annotations do not and should be incremented by 0.5.
Now that we have a convention for the coordinate system, it's a good practice in computer vision systems toalways use coordinates rather than indices to represent geometries, such as boxes and polygons. This is because indices are integers, andcan easily lose precision during geometric operations.Using indices for bounding boxes has caused some issues in Detectron.
Models in Detectron / Detectron2all involve localization of objects in images, so the convention of pixels and coordinates matters a lot.Various improvements and bugfixes in the two libraries are related to pixels.
In detection models, bounding box regression typically predicts "deltas" between the ground truth (GT) box and a reference box (e.g. anchor).In training, GT box is encoded to deltas as training target. In inference, the predicted deltas are decoded to become output boxes.
Boxes in Detectron often use integer indices, instead of coordinates. So the width of abox is given by instead of . Its box transform code looked like this for a long time (showing only one dimension for brevity):
defencode(x0, x1): # given reference box and gt box w = x1 - x0 + 1 center = x0 + 0.5 * w dx = (center - ref_center) / ref_w # delta between centers dw = log(w / ref_w) # delta between widths return dx, dw
defdecode(dx, dw): # given reference box and deltas center = dx * ref_w + ref_center # undo the encoding w = exp(dw) * ref_w x0, x1 = center - 0.5 * w, center + 0.5 * w return x0, x1
As innocent as the code seems, the two functions are not inverse of each other: decode(encode(x0, x1)) != (x0, x1).x1 is incorrectly decoded: it should be center + 0.5 * w - 1 instead.
This bug appeared in the py-faster-rcnn project around 2015, and is still theretoday.It was carried into Detectron and negatively affected results in the Mask R-CNN paper.Then it's fixed in late 2017after I found it, and contributed to an improvement of 0.4~0.7 box AP.Detectron went open source in 2018 with this fix.In Detectron2, we adopt the rule to always use floating-point coordinates for boxes, so the issue nolonger exists.
How to horizontally flip a geometry? Although pixel indices should be flipped by ,we should follow the rule to always use coordinates, and coordinates should be flipped by under "integer corner" system.
Detectron isn't so rigorous on this and it uses for coordinates. IIRC, fixing the issue leads to ~0.5 mask AP improvement.
The augmentation library "imgaug" also made this fix.
COCO's instance segmentation data is annotated with polygons that have sub-pixel precision.Converting polygons to binary masks loses the precision due to quantization,and the lost might become more severe during augmentations.Therefore it's preferrable to keep the polygon representation and delay the conversion as much as possible.
In both Detectron and Detectron2, polygon representation are kept during flipping, scaling, and RoI cropping.Masks are not created until the second stage's box predictions are made, where the boxes are used to crop the groundtruth polygonsand generate the mask training target.
On the contrary, in TensorFlow's detection code hereand herepolygons are turned to binary masks immediately at dataset creation time.
The code to generate anchors in Detectron is quite long,because it tries to generate integer-valued anchor boxes.By adopting coordinates for all boxes in Detectron2, integer boxes are not needed.This simplifies all the logic to just a few lines of code.
This does not affect accuracy, because the exact values of anchors are not that important as long asthe same is used in training & testing.
The RoIAlign operation crops a region from an image and resize it to certain shape.It's easy to make mistakes becausetwo images and two coordinate systems are involved.Let's derive how to perform RoIAlign.
Given an image and a region (the green box), we want to resample a K K output imagethat corresponds to the region.W.l.o.g. we assume the input image has stride=1.Since we know the resolution and the absolute length of output,the output stride derived from the definition of grid ② is .Because grid offset is 0.5stride, the location of output pixel [i,j] isLet's call it .To compute resampled values at location , an easy way is to do bilinear interpolation with its 4nearest pixels (this corresponds to RoIAlign with sampling_ratio=1).We show the 4 neighboring input pixels of output[0,0] in the figure.The indices of 4 nearest pixels of are obtained after subtracting 0.5 to align their coordinate system:
The original implementation of RoIAlign in Detectron doesn't subtract 0.5 in the end, so it's actually not very aligned.It turns out this detail does not affect accuracy of R-CNNs, because RoIAlign is applied on CNN features, and CNNis believed to be able to fit slightly misaligned features.
However, we have new use cases of RoIAlign in other places, e.g. to crop mask head training targets from the ground truth mask, soI fixed it in the detectron2 / torchvision RoIAlign with an aligned=True option.Its unittestdemonstrates how the old version is misaligned.
Btw, once we figured out the coordinate transform formula, it's easy to implement RoIAlign using grid_sample.This shows that RoIAlign is nothing more than a fused bilinear sampling + averaging.Using grid_sample is about 10%-50% slower than the RoIAlign CUDA kernel.
Mask R-CNN is trained to predict masks of fixed resolution (e.g. 28x28) restrained inside given boxes(we call it "RoIMask").But in the end, we often want to obtain full-image masks.A "paste mask" operation is needed to paste the small RoIMask into the given region in the image.
This operation is an inverse of RoIAlign, so it should be implemented similar to our derivation above.In Detectron, this was implementedwith some magic rounding & resize that are not exactly the inverse of RoIAlign.Fixing it in detectron2 increases the mask AP by 0.1~0.4.
Obviously, the paste mask operation can introduce aliasing in the results due to the low resolution RoIMask.This is the motivation behind our work ofPointRend.
PointRend is a segmentation method that focuses on point-wise features, where a "point" isnot necessarily a pixel, but any real-valued coordinates.Pointly-Supervised Instance Segmentation, also from our team, uses point-wise annotations to train segmentation models.Both projects involve heavy use of point sampling and coordinate transforms.Having a clear and consistent convention of pixels and coordinates was important to their success.
Due to some sloppy code in the early days of deep learning libraries,today we're facing multiple versions of resize functions.Together with the two different coordinate system conventions, they easily cause hidden bugs in computer vision code.
This article revisits these historical technical debts and shows how these fun details matter in modeling and training.I hope they will help you make proper choices.
"Harking" 这个词最早出现在心理学研究里. 在 deep learning 中也有对此的批评: HARK Side of Deep Learning. 它之所以是一种不太好的 research practice, 是因为它背离了科学研究的目标. 科学与迷信的一大区别, 是科学应当不仅能够解释已知, 还能够预测未知."天上打雷是雷公电母生气了" 就是对天气现象的一种解释, 但它不是科学. 如果一个工作仅仅追求找到一个解释, 来与现有的少量实验结果兼容的话, 这个解释未必在其他实验中适用, 因此可能不是一个科学的结论.
