mirror of
https://github.com/linuxkit/linuxkit.git
synced 2026-01-05 07:18:08 +00:00
Add Memorizer to projects
This commits an initial version of the Memorizer tracing tool. It collects and outputs detailed data on the objects (traced from kmalloc/kmem_cache_alloc) and accesses, tracking the context of each event with respect to thread ID, program counter, and for allocations name of process. Signed-off-by: Nathan Dautenhahn <ndd@cis.upenn.edu>
This commit is contained in:
Nathan Dautenhahn
92
projects/memorizer/docs/memorizer.txt
Normal file
92
projects/memorizer/docs/memorizer.txt
Normal file
@@ -0,0 +1,92 @@
|
||||
|
||||
+--------------------------------------------------+
|
||||
| Memorizer: Kernel Memory Access Patterns (KMAPs) |
|
||||
+--------------------------------------------------+
|
||||
|
||||
Introduction
|
||||
============
|
||||
|
||||
Memorizer is a tool to record information about access to kernel objects:
|
||||
specifically, it counts memory accesses from distinct IP addresses in the
|
||||
kernel source and also the PID that accessed, thereby providing spatial and
|
||||
temporal dimensions.
|
||||
|
||||
Interface via debugfs
|
||||
=====================
|
||||
|
||||
The tool has a very simple interface at the moment. It can:
|
||||
|
||||
- Print out some statistics about memory allocations and memory accesses
|
||||
- Control enable/disable of memory object allocation tracking and memory access
|
||||
tracing
|
||||
- Print the KMAP using the debugfs file system
|
||||
|
||||
Enable object allocation tracking:
|
||||
```bash
|
||||
echo 1 > /sys/kernel/debug/memorizer/memorizer_enabled
|
||||
```
|
||||
|
||||
Enable object access tracking:
|
||||
```bash
|
||||
echo 1 > /sys/kernel/debug/memorizer/memorizer_log_access
|
||||
```
|
||||
|
||||
Show allocation statistics:
|
||||
```bash
|
||||
cat /sys/kernel/debug/memorizer/show_stats
|
||||
```
|
||||
|
||||
Clear free'd objects:
|
||||
```bash
|
||||
echo 1 > /sys/kernel/debug/memorizer/clear_object_list
|
||||
```
|
||||
|
||||
Using Memorizer to Collect KMAPs
|
||||
================================
|
||||
|
||||
Memorizer lacks push style logging and clearing of the object lists, therefore
|
||||
it has the propensity of overflowing memory. The only way to manage the log and
|
||||
current set of objects is to manually clear and print the KMAPs.
|
||||
|
||||
Therefore, a typical run using memorizer to create KMAPs includes:
|
||||
|
||||
```bash
|
||||
# mount the debugfs filesystem if it isn't already
|
||||
mount -t debugfs nodev /sys/kernel/debug
|
||||
# clear free objects: the current system traces from boot with a lot of
|
||||
# uninteresting data
|
||||
echo 1 > /sys/kernel/debug/clear_object_list
|
||||
# enable memorizer object access tracking, which by default is off
|
||||
echo 1 > /sys/kernel/debug/memorizer_log_access
|
||||
# Now run whatever test
|
||||
tar zcf something.tar.gz /somedir &
|
||||
ssh u@h:/somefile
|
||||
...
|
||||
# Disable access logging
|
||||
echo 0 > /sys/kernel/debug/memorizer/memorizer_log_access
|
||||
# Disable memorizer object tracking: isn't necessary but will reduce noise
|
||||
echo 0 > /sys/kernel/debug/memorizer/memorizer_enabled
|
||||
# Cat the results: make sure to pipe to something
|
||||
cat /sys/kernel/debug/memorizer/kmap > test.kmap
|
||||
```
|
||||
|
||||
Output Format
|
||||
=============
|
||||
|
||||
Memorizer outputs data as text, which may change if space is a problem. The
|
||||
format of the kmap file is as follows:
|
||||
|
||||
alloc_ip,pid,obj_va_ptr,size,alloc_jiffies,free_jiffies,free_ip,executable
|
||||
access_ip,access_pid,write_count,read_count
|
||||
access_ip,access_pid,write_count,read_count
|
||||
access_ip,access_pid,write_count,read_count
|
||||
...
|
||||
...
|
||||
|
||||
There are a few special error codes:
|
||||
|
||||
- Not all free_ip's could be obtained correctly and therefore some of these
|
||||
will be 0.
|
||||
- There is a bug where we insert into the live object map over another
|
||||
allocation, this implies that we are missing a free. So for now we mark
|
||||
the free_ip as 0xDEADBEEF.
|
||||
36
projects/memorizer/docs/scripts/README.txt
Normal file
36
projects/memorizer/docs/scripts/README.txt
Normal file
@@ -0,0 +1,36 @@
|
||||
Files: memorizer.py, test_memorizer.py
|
||||
|
||||
Dependencies:
|
||||
In order to run the test_memorizer w/ linux test suite, you must
|
||||
wget the latest version from the ltp github repo and set it up.
|
||||
Ex:
|
||||
wget https://github.com/linux-test-project/ltp/releases/download/20170116/ltp-full-20170116.tar.bz2
|
||||
tar xvfj ltp-full-20170116.tar.bz2
|
||||
# cd into the untarred dir
|
||||
./configure
|
||||
make
|
||||
sudo make install
|
||||
|
||||
Good documentation / examples: http://ltp.sourceforge.net/documentation/how-to/ltp.php
|
||||
|
||||
memorizer.py: accepts processes to run in quotes.
|
||||
Ex: python memorizer.py "ls" "mkdir dir"
|
||||
In order to run the script, you must have your user be in the
|
||||
memorizer group, which you should setup if not.
|
||||
How-to: sudo groupadd memorizer; sudo usermod -aG memorizer <user>
|
||||
You will be queried to enter your pw in order to set group
|
||||
permissions on the /sys/kernel/debug dirs which include ftrace
|
||||
and memorizer.
|
||||
|
||||
test_memorizer.py: accepts either -e, -m, or -h flags.
|
||||
Ex: python test_memorizer.py -e
|
||||
*All modes will run the setup/cleanup checks to ensure all virtual nodes
|
||||
are being set correctly.
|
||||
-e: Runs ls, wget, and tar sequentially.
|
||||
-m: Runs the linux test suite and saves a human-readable log to
|
||||
/opt/ltp/results/ltp.log
|
||||
-h: runs both -e and -m
|
||||
As with the memorizer.py, you will need your user to be in the
|
||||
memorizer group. Additionally, you will be queried to enter your
|
||||
pw in order to set group permissions on the /opt/ltp dirs.
|
||||
|
||||
152
projects/memorizer/docs/scripts/memorizer.py
Executable file
152
projects/memorizer/docs/scripts/memorizer.py
Executable file
@@ -0,0 +1,152 @@
|
||||
import sys,threading,os,subprocess,operator,time
|
||||
|
||||
mem_path = "/sys/kernel/debug/memorizer/"
|
||||
directory = ""
|
||||
completed = False
|
||||
|
||||
def worker(cmd):
|
||||
ret = os.system(cmd)
|
||||
if(ret != 0):
|
||||
print "Failed attempt on: " + cmd
|
||||
exit(1)
|
||||
|
||||
def basic_cleanup():
|
||||
print "Basic tests completed. Now cleaning up."
|
||||
ret = os.system("rm UPennlogo2.jpg")
|
||||
|
||||
def memManager():
|
||||
while(not completed):
|
||||
stats = subprocess.check_output(["free"])
|
||||
stats_list = stats.split()
|
||||
total_mem = float(stats_list[7])
|
||||
used_mem = float(stats_list[8])
|
||||
memory_usage = used_mem / total_mem
|
||||
if(memory_usage > 0.8):
|
||||
ret = os.system("cat " + mem_path + "kmap >> " + directory + "test.kmap")
|
||||
if ret != 0:
|
||||
print "Failed to append kmap to temp file"
|
||||
exit(1)
|
||||
ret = os.system("echo 1 > " + mem_path + "clear_printed_list")
|
||||
if ret != 0:
|
||||
print "Failed to clear printed list"
|
||||
exit(1)
|
||||
time.sleep(2)
|
||||
|
||||
def startup():
|
||||
ret = os.system("sudo chgrp -R memorizer /opt/")
|
||||
if ret != 0:
|
||||
print "Failed to change group permissions of /opt/"
|
||||
exit(1)
|
||||
os.system("sudo chmod -R g+wrx /opt/")
|
||||
if ret != 0:
|
||||
print "Failed to grant wrx permissions to /opt/"
|
||||
exit(1)
|
||||
# Setup group permissions to ftrace & memorizer directories
|
||||
ret = os.system("sudo chgrp -R memorizer /sys/kernel/debug/")
|
||||
if ret != 0:
|
||||
print "Failed to change memorizer group permissions to /sys/kernel/debug/"
|
||||
exit(1)
|
||||
ret = os.system("sudo chmod -R g+wrx /sys/kernel/debug/")
|
||||
if ret != 0:
|
||||
print "Failed to grant wrx persmissions to /sys/kernel/debug/"
|
||||
exit(1)
|
||||
# Memorizer Startup
|
||||
ret = os.system("echo 1 > " + mem_path + "clear_object_list")
|
||||
if ret != 0:
|
||||
print "Failed to clear object list"
|
||||
exit(1)
|
||||
ret = os.system("echo 0 > " + mem_path + "print_live_obj")
|
||||
if ret != 0:
|
||||
print "Failed to disable live object dumping"
|
||||
exit(1)
|
||||
ret = os.system("echo 1 > " + mem_path + "memorizer_enabled")
|
||||
if ret != 0:
|
||||
print "Failed to enable memorizer object allocation tracking"
|
||||
exit(1)
|
||||
ret = os.system("echo 1 > " + mem_path + "memorizer_log_access")
|
||||
if ret != 0:
|
||||
print "Failed to enable memorizer object access tracking"
|
||||
exit(1)
|
||||
|
||||
def cleanup():
|
||||
# Memorizer cleanup
|
||||
ret = os.system("echo 0 > " + mem_path + "memorizer_log_access")
|
||||
if ret != 0:
|
||||
print "Failed to disable memorizer object access tracking"
|
||||
exit(1)
|
||||
ret = os.system("echo 0 > " + mem_path + "memorizer_enabled")
|
||||
if ret != 0:
|
||||
print "Failed to disable memorizer object allocation tracking"
|
||||
exit(1)
|
||||
# Print stats
|
||||
ret = os.system("cat " + mem_path + "show_stats")
|
||||
if ret != 0:
|
||||
print "Failed to display memorizer stats"
|
||||
exit(1)
|
||||
ret = os.system("echo 1 > " + mem_path + "print_live_obj")
|
||||
if ret != 0:
|
||||
print "Failed to enable live object dumping"
|
||||
exit(1)
|
||||
# Make local copies of outputs
|
||||
ret = os.system("cat " + mem_path + "kmap >> " +directory+ "test.kmap")
|
||||
if ret != 0:
|
||||
print "Failed to copy live and freed objs to kmap"
|
||||
exit(1)
|
||||
ret = os.system("echo 1 > " + mem_path + "clear_object_list")
|
||||
if ret != 0:
|
||||
print "Failed to clear all freed objects in obj list"
|
||||
exit(1)
|
||||
|
||||
def main(argv):
|
||||
global completed
|
||||
global directory
|
||||
if len(sys.argv) == 1:
|
||||
print "Invalid/missing arg. Please enter -e for basic tests, -m for ltp tests, and/or specify a full process to run in quotes. Specify path using the -p <path> otherwise default to ."
|
||||
return
|
||||
startup()
|
||||
processes = []
|
||||
easy_processes = False
|
||||
next_arg = False
|
||||
for arg in argv:
|
||||
if next_arg:
|
||||
next_arg = False
|
||||
directory = str(arg) + "/"
|
||||
elif arg == '-p':
|
||||
next_arg = True
|
||||
#User wants to run ltp
|
||||
elif arg == '-m':
|
||||
print "Performing ltp tests"
|
||||
processes.append("/opt/ltp/runltp -p -l ltp.log")
|
||||
print "See /opt/ltp/results/ltp.log for ltp results"
|
||||
#User wants to run wget,ls,etc.
|
||||
elif arg == '-e':
|
||||
easy_processes = True
|
||||
print "Performing basic ls test"
|
||||
processes.append("ls")
|
||||
print "Performing wget test"
|
||||
processes.append("wget http://www.sas.upenn.edu/~egme/UPennlogo2.jpg")
|
||||
print "Attempting to remove any existing kmaps in the specified path"
|
||||
os.system("rm " + directory + "test.kmap")
|
||||
print "Startup completed. Generating threads."
|
||||
manager = threading.Thread(target=memManager, args=())
|
||||
manager.start()
|
||||
threads = []
|
||||
for process in processes:
|
||||
try:
|
||||
t = threading.Thread(target=worker, args=(process,))
|
||||
threads.append(t)
|
||||
t.start()
|
||||
except:
|
||||
print "Error: unable to start thread"
|
||||
for thr in threads:
|
||||
thr.join()
|
||||
completed = True
|
||||
manager.join()
|
||||
print "Threads ran to completion. Cleaning up."
|
||||
basic_cleanup()
|
||||
cleanup()
|
||||
print "Cleanup successful."
|
||||
return 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv)
|
||||
Reference in New Issue
Block a user