支持从 k8s 临时 debug 容器 attach 目标 JVM（跨 mount namespace）#3247
Open
killingwolf wants to merge 1 commit into
Open
Conversation
…ontainer When attaching from a k8s ephemeral debug container, the debug container shares the pid namespace but NOT the mount namespace with the target app container, which breaks the JDK attach mechanism. This adds cross mount namespace support (auto-gated by comparing /proc/<pid>/ns/mnt), so normal same-container / non-Linux usage is unaffected: - expose the target attach socket via a /proc/<pid>/root symlink - copy the full arthas home into the target container (incl. arthas-spy.jar, async-profiler, arthas.properties), chowned to the target uid/gid - clean up the temp arthas home on stop/destroy (gated by a marker file) - post-attach telnet port health check with a clear error pointing to arthas.log - as.sh: skip the user/owner consistency check when running as root Co-Authored-By: Claude <noreply@anthropic.com>
|
|
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
该 PR 为 kubectl debug 临时容器（与目标共享 pid namespace，但 mount namespace 不同）场景下的 Arthas attach 增强：通过暴露 attach socket、复制完整 arthas home 到目标容器可见路径，并在 stop/destroy 时清理临时目录，从而避免“attach 假成功但 server 未真正启动”的问题。
Changes:
- 在
Arthas#attachAgent中自动检测跨 mount namespace，并实现 socket 软链暴露、arthas home 递归复制 + chown、attach 后端口健康校验 - 在
ArthasBootstrap#destroy中基于标记文件清理跨 ns 场景复制的临时 arthas home as.sh放宽 root 用户的进程属主一致性校验
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| core/src/main/java/com/taobao/arthas/core/Arthas.java | 跨 mount ns attach 的核心逻辑：socket 暴露、home 复制/属主对齐、启动健康校验 |
| core/src/main/java/com/taobao/arthas/core/server/ArthasBootstrap.java | stop/destroy 时基于 marker 清理跨 ns 临时目录 |
| common/src/main/java/com/taobao/arthas/common/ArthasConstants.java | 新增跨 ns 临时目录 marker 常量 |
| bin/as.sh | root 用户跳过“当前用户=目标进程属主”的一致性检查 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+897
to
+921
| private void cleanupCrossNsTempHome() { | ||
| try { | ||
| File home = new File(arthasHome()); | ||
| File marker = new File(home, ArthasConstants.CROSS_NS_TEMP_HOME_MARKER); | ||
| if (!marker.exists()) { | ||
| return; // 正常安装(非跨 ns 临时拷贝)，不清理 | ||
| } | ||
| deleteRecursively(home); | ||
| logger().info("Cleaned up cross-ns temp arthas home: {}", home); | ||
| } catch (Throwable e) { | ||
| logger().warn("Failed to cleanup cross-ns temp arthas home: {}", e.toString()); | ||
| } | ||
| } | ||
|
|
||
| private static void deleteRecursively(File file) { | ||
| if (file.isDirectory()) { | ||
| File[] children = file.listFiles(); | ||
| if (children != null) { | ||
| for (File child : children) { | ||
| deleteRecursively(child); | ||
| } | ||
| } | ||
| } | ||
| file.delete(); | ||
| } |
Comment on lines
+303
to
+316
| try (Stream<Path> stream = Files.walk(arthasHome)) { | ||
| for (Path src : (Iterable<Path>) stream::iterator) { | ||
| Path dest = targetHome.resolve(arthasHome.relativize(src).toString()); | ||
| if (Files.isDirectory(src)) { | ||
| Files.createDirectories(dest); | ||
| setPosixPermissions(dest, "rwxr-xr-x"); | ||
| } else { | ||
| Files.createDirectories(dest.getParent()); | ||
| Files.copy(src, dest, StandardCopyOption.REPLACE_EXISTING); | ||
| // 放开 other 读权限，确保目标容器里的(普通用户)进程可读；源文件可执行则保留执行位 | ||
| setPosixPermissions(dest, Files.isExecutable(src) ? "rwxr-xr-x" : "rw-r--r--"); | ||
| } | ||
| } | ||
| } |
Comment on lines
+401
to
+430
| private static String detectTargetTmpDir(long targetPid) { | ||
| boolean cmdlineReadable = false; | ||
| try { | ||
| String cmdline = new String(Files.readAllBytes(Paths.get("/proc/" + targetPid + "/cmdline"))); | ||
| cmdlineReadable = true; | ||
| for (String arg : cmdline.split("\0")) { | ||
| if (arg.startsWith("-Djava.io.tmpdir=")) { | ||
| return arg.substring("-Djava.io.tmpdir=".length()); | ||
| } | ||
| } | ||
| } catch (Throwable e) { | ||
| // ignore | ||
| } | ||
| try { | ||
| String environ = new String(Files.readAllBytes(Paths.get("/proc/" + targetPid + "/environ"))); | ||
| for (String env : environ.split("\0")) { | ||
| if (env.startsWith("TMPDIR=")) { | ||
| return env.substring("TMPDIR=".length()); | ||
| } | ||
| } | ||
| } catch (Throwable e) { | ||
| // ignore | ||
| } | ||
| if (!cmdlineReadable) { | ||
| // 连 cmdline 都读不到(通常是权限不足)，无法确认目标是否用了自定义 tmpdir——明确告警而非静默 | ||
| AnsiLog.warn("Cannot read /proc/{}/cmdline to detect target java.io.tmpdir (permission denied?), " | ||
| + "assuming /tmp. If the target uses a custom tmpdir, attach may fail.", targetPid); | ||
| } | ||
| return "/tmp"; | ||
| } |
Comment on lines
+278
to
+286
| private static String[] resolveAgentPathsForTarget(long targetPid, String agentJar, String coreJar) { | ||
| try { | ||
| return copyArthasHomeIntoTarget(targetPid, agentJar, coreJar); | ||
| } catch (Throwable e) { | ||
| AnsiLog.warn("Copy arthas home into target failed ({}), fallback to /proc/self/root path", e.toString()); | ||
| String selfRoot = "/proc/" + PidUtils.currentPid() + "/root"; | ||
| return new String[] { selfRoot + agentJar, selfRoot + coreJar }; | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景 / 问题
使用
kubectl debug的临时容器（ephemeral container）调试应用容器里的 Java 进程时，attach 会失败，典型报错依次是：com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /tmp/.java_pid<pid>com.sun.tools.attach.AgentLoadException: Agent JAR not found or no Agent-Class attributeAttach process N success），但arthas-client connect 127.0.0.1:3658报Connection refused原因分析
临时容器与应用容器共享 pid namespace，但 mount namespace 相互独立，而 JDK 的 attach 机制（
sun.tools.attach.LinuxVirtualMachine）默认双方在同一文件系统：${java.io.tmpdir}/.java_pid<pid>，发起方在自己容器的 tmpdir 里找不到。loadAgent让目标 JVM 去打开 agent/core jar 路径，但那是发起方容器的路径，目标侧不存在；当双方 UID 不同（root 的 debug 容器 + 普通用户应用）时，目标进程也无权读取发起方的/proc/<pid>/root。ArthasBootstrap启动时还要从 arthas home 读取arthas-spy.jar（注入 bootstrap classloader，缺失即抛异常）、async-profiler、arthas.properties等资源；资源不全会导致 bind 失败，而AgentBootstrap的 binding 线程会吞掉该异常，于是loadAgent“成功”但 telnet 端口根本没监听。方案
在
com.taobao.arthas.core.Arthas#attachAgent中增加跨 mount namespace 支持，全部逻辑由inDifferentMountNamespace(targetPid)（比较/proc/self/ns/mnt与/proc/<pid>/ns/mnt）自动 gating——普通同容器 / 非 Linux 场景完全不受影响：/proc/<pid>/root<targetTmp>/.java_pid<pid>。arthas-spy.jar/async-profiler/arthas.properties等）递归复制进目标容器 tmpdir，返回目标侧本地路径；同 UID 时退回/proc/<selfpid>/root前缀方案。chown），保证目标侧普通用户在stop时能删除该临时目录。arthas-<targetPid>命名（幂等、止漏）并写标记文件；ArthasBootstrap#destroy见到标记才递归清理临时目录（正常安装无标记，不受影响）。loadAgent后主动探测 telnet 端口，失败则明确报错并指路目标的arthas.log，消除“假成功”。改动清单
core/src/main/java/com/taobao/arthas/core/Arthas.java：跨 mount namespace 的 socket 软链、arthas home 复制、属主对齐、tmpdir 探测告警、attach 后健康校验。core/src/main/java/com/taobao/arthas/core/server/ArthasBootstrap.java：destroy()见标记清理临时 arthas home。common/src/main/java/com/taobao/arthas/common/ArthasConstants.java：新增临时目录标记常量CROSS_NS_TEMP_HOME_MARKER。bin/as.sh：root 用户跳过属主一致性检查。兼容性
as.sh仅对 root 放宽，非 root 的原有校验保持不变。测试
在 k8s 中手动验证：
kubectl debug临时容器（root）attach 普通用户应用容器（共享 pid namespace）：stop后目标/tmp/arthas-<pid>被清理；arthas.log，而非“假成功”。（跨 mount namespace 场景较难做单元测试，主要为手动集成验证。）
局限性与说明
.java_pid命名、tmpdir 约定、.attach_pid+SIGQUIT 触发），JDK 版本变更可能需要适配；冷启动路径依赖经/proc/<pid>/cwd写.attach_pid，受权限影响。kubectl exec进应用容器、两容器挂共享emptyDir、或走 tunnel server。本 PR 面向“无法改部署、又需要直接 attach”的场景。关联 issue
#3246