![]() Mmap Error handling is done via global handlers that might interfere with third party libraries. SIGBUS (bus error) is a signal that happens when you try to access memory that has not been physically mapped. Mmap code produces SIGBUS signals that are difficult to centralise and complicate error handling code. Good Exception handling is key for any critical application.This may or may not be possible depending on your application. As mentioned in their post, they handle it via releasing memory mapped locks on idle. The difference is that Windows keeps a lock on the file, not allowing it to be deleted. There is one major difference in implementations of memory mapped files as mentioned by Sublime HQ in their post on memory maps. It has its own construct called MapViewOfFile.The higher level APIs in languages like Python hide this behind an abstraction. But, many times applications have better knowledge about their workloads that they can use to make intelligent decisions that Kernel can’t do for you. This is a good start and if your application can work with it effectively then you should be fine using mmap. The kernel decides on the page eviction when memory runs low. When you use mmap you let the kernel take the responsibility of maintaining caches for both reads and writes.When your application process makes a page fault on an mmap region then you see major IO stalls. This is easy for programmers but you lose the flexibility to make things asynchronous and parallel. The programming model offered by mmap is synchronous and blocking.I collected below mentioned reasons why to avoid using mmap. Now that we understand mmap, let’s talk about its issues. Mmap is powerful and makes it easy for database designers to leave memory management to the operating system. We will start by pulling the latest Python docker image. We will use a Python docker image to create our experimentation playground. Let’s do some quick experimentation with mmap to better understand it. Virtual memory gives programs an illusion that they have more memory available than they do. Mmap allows you to read files that are much bigger than the physical memory available to the system. Mmap takes advantage of file system caches by asking the operating system to map the needed files in virtual memory in order to access that memory directly. If you miss the cache then it causes a page fault, prompting the kernel to go to fetch the corresponding data to the disk. If the data happens to be in the cache then the kernel is bypassed and reads & writes are done from the memory. This allows an application to read data from the application address space just like an array. It makes a section of the application address space to directly refer to the page caches that contain the file data. It is an OS level feature that memory-maps the file into application address space. But, before we do that let’s first understand mmap. I will list all of the reasons I could find in my research and from Andy’s video in this post. Given that so many databases use mmap I wanted to understand why Andy recommended us to not use mmap. Some of the databases that use mmap are RavenDB, ElasticSearch, LevelDB, InfluxDB, LMDB, BoltDB, moss (key-value store from Couchbase), etc. MongoDB is not the only database that uses mmap. It allowed them to achieve faster time to market but later they had to replace it with a new storage engine wiredtiger because of the issues they faced with mmap. I was aware that MongoDB used to use an mmap based storage engine. I have not used mmap before so I was intrigued to understand it in more detail. ![]() He went on to say that if there is only one thing you should get from his database course is to never use mmap when building and designing database management systems. A couple of months back I watched a video by Andy Pavlo, Associate Professor of Databases Carnegie Mellon, where he made a point that databases should not use mmap.
0 Comments
Leave a Reply. |