Linux smaps_rollup 及 status 查看 Java 进程内存占用

需求

排查Java进程频繁被Kubernetes的OOMKiller，需查看Java进程占用的总内存。

Java除了堆内存外，还有非堆内存和堆外内存。

排查工具

可以使用Java的NMT(Native Memory Tracking)以及Linux的进程信息文件smaps_rollup。

NMT vs smaps_rollup

不能仅通过NMT的数据就判断JVM占用的内存。JVM通过mmap分配的大量内存都是先reserve再commit之后实际往里面写入数据的时候，才会真正分配物理内存。

同时，JVM还会动态释放一些内存，这些内存可能不会立刻被操作系统回收。

NMT是JVM认为自己向操作系统申请的内存，与实际操作系统分配的内存是有所差距的，所以不能只查看NMT去判断，还需要查看能体现真正内存占用指标。

进程监控文件smaps_rollup看出具体的内存占用，一般不看Rss，因为如果涉及多个虚拟地址映射同一个物理地址的话会有不准确，所以主要关注Pss即可，但是Pss更新不是实时的，但也差不多，这就可以理解为进程占用的实际物理内存。

进程监控文件 smaps_rollup

cat /proc/[pid]/smaps_rollup

一般Kubernetes中Pod的服务进程ID都为1。

主要查看Pss这一项

cat /proc/1/smaps_rollup

显示Pss为867463kB(847M)

bash-4.4$ cat /proc/1/smaps_rollup 
e0c00000-7ffd3e94d000 ---p 00000000 00:00 0                              [rollup]
Rss:              875588 kB
Pss:              867463 kB
Pss_Anon:         857780 kB
Pss_File:           9683 kB
Pss_Shmem:             0 kB
Shared_Clean:      15996 kB
Shared_Dirty:          0 kB
Private_Clean:      1812 kB
Private_Dirty:    857780 kB
Referenced:       764932 kB
Anonymous:        857780 kB
LazyFree:              0 kB
AnonHugePages:    323584 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB

进程状态文件 status

主要查看VmRSS这一项

cat /proc/1/status

VmRSS为856072 kB(836M)

bash-4.4$ cat /proc/1/status
Name:   java
Umask:  0022
State:  S (sleeping)
Tgid:   1
Ngid:   0
Pid:    1
PPid:   0
TracerPid:      0
Uid:    1000    1000    1000    1000
Gid:    0       0       0       0
FDSize: 64
Groups: 0 
NStgid: 1
NSpid:  1
NSpgid: 1
NSsid:  1
VmPeak:  4820760 kB
VmSize:  4755224 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:    887180 kB
VmRSS:    856072 kB
RssAnon:          856072 kB
RssFile:               0 kB
RssShmem:              0 kB
VmData:   950668 kB
VmStk:       132 kB
VmExe:         4 kB
VmLib:     25304 kB
VmPTE:      2212 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
Threads:        53
SigQ:   0/126776
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 2000000181005ccf
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 00000000a80425fb
CapAmb: 0000000000000000
NoNewPrivs:     0
Seccomp:        0
Seccomp_filters:        0
Speculation_Store_Bypass:       vulnerable
Cpus_allowed:   ff
Cpus_allowed_list:      0-7
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        23
nonvoluntary_ctxt_switches:     12

NMT

Total统计的committed为725458KB(708M)。与smaps_rollup的Pss的867463KB(847M)相差了139M。

bash-4.4$ jcmd 1 VM.native_memory
1:

Native Memory Tracking:

(Omitting categories weighting less than 1KB)

Total: reserved=1047322KB, committed=725458KB
       malloc: 45610KB #427315
       mmap:   reserved=1001712KB, committed=679848KB

-                 Java Heap (reserved=512000KB, committed=512000KB)
                            (mmap: reserved=512000KB, committed=512000KB) 

-                     Class (reserved=83309KB, committed=9389KB)
                            (classes #14121)
                            (  instance classes #13194, array classes #927)
                            (malloc=1389KB #28990) 
                            (mmap: reserved=81920KB, committed=8000KB) 
                            (  Metadata:   )
                            (    reserved=65536KB, committed=57152KB)
                            (    used=56802KB)
                            (    waste=350KB =0.61%)
                            (  Class space:)
                            (    reserved=81920KB, committed=8000KB)
                            (    used=7669KB)
                            (    waste=331KB =4.14%)

-                    Thread (reserved=30474KB, committed=3958KB)
                            (thread #53)
                            (stack: reserved=30336KB, committed=3820KB)
                            (malloc=79KB #316) 
                            (arena=59KB #102)

-                      Code (reserved=250026KB, committed=37302KB)
                            (malloc=2338KB #11379) 
                            (mmap: reserved=247688KB, committed=34964KB) 

-                        GC (reserved=69549KB, committed=69549KB)
                            (malloc=17649KB #13372) 
                            (mmap: reserved=51900KB, committed=51900KB) 

-                 GCCardSet (reserved=123KB, committed=123KB)
                            (malloc=123KB #1527) 

-                  Compiler (reserved=465KB, committed=465KB)
                            (malloc=300KB #1081) 
                            (arena=165KB #5)

-                  Internal (reserved=567KB, committed=567KB)
                            (malloc=563KB #15855) 
                            (mmap: reserved=4KB, committed=4KB) 

-                     Other (reserved=112KB, committed=112KB)
                            (malloc=112KB #21) 

-                    Symbol (reserved=13324KB, committed=13324KB)
                            (malloc=11909KB #331813) 
                            (arena=1415KB #1)

-    Native Memory Tracking (reserved=7148KB, committed=7148KB)
                            (malloc=471KB #6715) 
                            (tracking overhead=6677KB)

-        Shared class space (reserved=12288KB, committed=11968KB)
                            (mmap: reserved=12288KB, committed=11968KB) 

-               Arena Chunk (reserved=197KB, committed=197KB)
                            (malloc=197KB) 

-                   Tracing (reserved=32KB, committed=32KB)
                            (arena=32KB #1)

-                    Module (reserved=106KB, committed=106KB)
                            (malloc=106KB #1439) 

-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB) 

-           Synchronization (reserved=136KB, committed=136KB)
                            (malloc=136KB #1470) 

-            Serviceability (reserved=18KB, committed=18KB)
                            (malloc=18KB #17) 

-                 Metaspace (reserved=65817KB, committed=57433KB)
                            (malloc=281KB #157) 
                            (mmap: reserved=65536KB, committed=57152KB) 

-      String Deduplication (reserved=1592KB, committed=1592KB)
                            (malloc=1592KB #12987) 

-                   Unknown (reserved=32KB, committed=32KB)
                            (mmap: reserved=32KB, committed=32KB)

参考

https://juejin.cn/post/7225871227743043644