SSH-MITM Version
SSH-MITM 4.0.0
Platform detail
Linux gw 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux
Arguments used to start SSH-MITM
venv/bin/python3 -m sshmitm --paramiko-log-level debug server --session-log-dir session-logs --store-ssh-session --remote-host $REMOTE_HOST --remote-port 22 --host-key host_key_file --banner-name "$BANNER" --listen-port 22
SSH client used
All of them. MitM proxy was reachable by global traffic.
SSH server used
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2
What steps will reproduce the bug?
Let the server run publicly and be reachable by global ingress. The below error in "additional details" was produced after running the MitM proxy for around 4 hours to proxy incoming traffic on port 22 to another machine.
I observed lots of traffic of approximately one connection every 10 seconds. They were, in fact, all short-living connections and terminated in a few seconds.
However, they allocate a socket and a thread of a server. My ss listed hundreds of open listening sockets spawned by the MitM proxy. At some point, it could not spawn more threads and this error crashed the server when trying to accept a new incoming connection.
The KeyboardInterrupt was made by me after I found out that the server was effectively down.
Note:
I was using my own fork which had slight modifications.
Last commit from you was eabae16. The software was running 57c1001 from my fork.
I do not expect that these modifications are the cause of the error, but the summary of these modifications are: adding a few command line arguments, added a tracking mechanism that made HTTP requests with login credentials in one thread (it did not spawn new threads!) and a provisioning mechanism that created the requested user on the target, in combination with a retry mechanism. This way, it accepts all usernames and all passwords and provides a valid shell on the target machine (in my configuration, since the "provisioning command" is a useradd on the remote machine).
This is not the only such error, I will post a second issue later, maybe. TL;DR: Same behavior, but the error was OSError: Too many open files due to the number of open sockets (see above) filled all available file descriptors (default: 1024), before I increased the shell's ulimit.
What should have happened?
The server should not go down. When the proxied TCP connection ends, the socket and thread should be freed to avoid this problem (if the problem is what I suspect).
Additional information
If you need more details, I can provide more logs; this is an extract.
ERROR Unknown exception: can't start new thread
ERROR Traceback (most recent call last):
ERROR File
"/home/mitm-nl/mitm/sshmitm/workarounds/transport.p
y", line 160, in transport_run
ERROR self.packetizer.start_handshake(self.handshake_
timeout)
ERROR File
"/home/mitm-nl/mitm/venv/lib/python3.11/site-packag
es/paramiko/packet.py", line 252, in
start_handshake
ERROR self.__timer.start()
ERROR File "/usr/lib/python3.11/threading.py", line
957, in start
ERROR _start_new_thread(self._bootstrap, ())
ERROR RuntimeError: can't start new thread
ERROR
ERROR internal error, abort authentication!
╭─────── Traceback (most recent call last) ───────╮
│ /home/mitm-nl/mitm/sshmitm/authentication.py:36 │
│ 6 in authenticate │
│ │
│ 363 │ │ │ │ │ self.session.remote_a │
│ 364 │ │ │ │ ) │
│ 365 │ │ │ if self.session.password: │
│ ❱ 366 │ │ │ │ return self.auth_password │
│ 367 │ │ │ │ │ self.session.username │
│ 368 │ │ │ │ │ self.session.remote_a │
│ 369 │ │ │ │ │ self.session.remote_a │
│ │
│ /home/mitm-nl/mitm/sshmitm/authentication.py:54 │
│ 2 in auth_password │
│ │
│ 539 │ │ return self.connect(username, hos │
│ 540 │ │
│ 541 │ def auth_password(self, username: str │
│ ❱ 542 │ │ return self.connect(username, hos │
│ password=password) │
│ 543 │ │
│ 544 │ def auth_publickey(self, username: st │
│ 545 │ │ """ │
│ │
│ /home/mitm-nl/mitm/sshmitm/authentication.py:48 │
│ 7 in connect │
│ │
│ 484 │ │ ) │
│ 485 │ │ self.pre_auth_action() │
│ 486 │ │ try: │
│ ❱ 487 │ │ │ first_success = client.connec │
│ 488 │ │ │ if first_success: │
│ 489 │ │ │ │ self.session.ssh_client = │
│ 490 │ │ │ │ auth_status = paramiko.co │
│ │
│ /home/mitm-nl/mitm/sshmitm/clients/ssh.py:121 │
│ in connect │
│ │
│ 118 │ │ │
│ 119 │ │ try: │
│ 120 │ │ │ if self.method is Authenticat │
│ ❱ 121 │ │ │ │ self.transport.connect(us │
│ 122 │ │ │ elif self.method is Authentic │
│ 123 │ │ │ │ self.transport.connect(us │
│ pkey=self.key) │
│ 124 │ │ │ elif self.method is Authentic │
│ │
│ /home/mitm-nl/mitm/venv/lib/python3.11/site-pac │
│ kages/paramiko/transport.py:1351 in connect │
│ │
│ 1348 │ │ │ gssapi_requested=gss_kex or │
│ 1349 │ │ ) │
│ 1350 │ │ │
│ ❱ 1351 │ │ self.start_client() │
│ 1352 │ │ │
│ 1353 │ │ # check host key if we were give │
│ 1354 │ │ # If GSS-API Key Exchange was pe │
│ │
│ /home/mitm-nl/mitm/venv/lib/python3.11/site-pac │
│ kages/paramiko/transport.py:704 in start_client │
│ │
│ 701 │ │ │ if not self.active: │
│ 702 │ │ │ │ e = self.get_exception() │
│ 703 │ │ │ │ if e is not None: │
│ ❱ 704 │ │ │ │ │ raise e │
│ 705 │ │ │ │ raise SSHException("Nego │
│ 706 │ │ │ if event.is_set() or ( │
│ 707 │ │ │ │ timeout is not None and │
│ │
│ /home/mitm-nl/mitm/sshmitm/workarounds/transpor │
│ t.py:160 in transport_run │
│ │
│ 157 │ │ │ # shell. │
│ 158 │ │ │ # Make sure we can specify a │
│ 159 │ │ │ # Re-use the banner timeout f │
│ ❱ 160 │ │ │ self.packetizer.start_handsha │
│ 161 │ │ │ self._send_kex_init() │
│ 162 │ │ │ self._expect_packet(MSG_KEXIN │
│ 163 │
│ │
│ /home/mitm-nl/mitm/venv/lib/python3.11/site-pac │
│ kages/paramiko/packet.py:252 in start_handshake │
│ │
│ 249 │ │ """ │
│ 250 │ │ if not self.__timer: │
│ 251 │ │ │ self.__timer = threading.Time │
│ ❱ 252 │ │ │ self.__timer.start() │
│ 253 │ │
│ 254 │ def handshake_timed_out(self): │
│ 255 │ │ """ │
│ │
│ /usr/lib/python3.11/threading.py:957 in start │
│ │
│ 954 │ │ with _active_limbo_lock: │
│ 955 │ │ │ _limbo[self] = self │
│ 956 │ │ try: │
│ ❱ 957 │ │ │ _start_new_thread(self._boot │
│ 958 │ │ except Exception: │
│ 959 │ │ │ with _active_limbo_lock: │
│ 960 │ │ │ │ del _limbo[self] │
╰─────────────────────────────────────────────────╯
RuntimeError: can't start new thread
INFO ❗ Shutting down server ...
^CTraceback (most recent call last):
File "/home/mitm-nl/mitm/sshmitm/server/__init__.py", line 288, in start
thread.start()
File "/usr/lib/python3.11/threading.py", line 957, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/mitm-nl/mitm/sshmitm/__main__.py", line 3, in <module>
main()
File "/home/mitm-nl/mitm/sshmitm/cli.py", line 194, in main
available_subcommands[args.subparser_name].run_func(args)
File "/home/mitm-nl/mitm/sshmitm/server/cli.py", line 200, in run_server
proxy.start()
File "/home/mitm-nl/mitm/sshmitm/server/__init__.py", line 303, in start
thread.join()
File "/usr/lib/python3.11/threading.py", line 1112, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
^CException ignored in: <module 'threading' from '/usr/lib/python3.11/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.11/threading.py", line 1583, in _shutdown
lock.acquire()
KeyboardInterrupt:
Thanks for your patience ;)
SSH-MITM Version
SSH-MITM 4.0.0
Platform detail
Linux gw 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux
Arguments used to start SSH-MITM
venv/bin/python3 -m sshmitm --paramiko-log-level debug server --session-log-dir session-logs --store-ssh-session --remote-host $REMOTE_HOST --remote-port 22 --host-key host_key_file --banner-name "$BANNER" --listen-port 22SSH client used
All of them. MitM proxy was reachable by global traffic.
SSH server used
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2
What steps will reproduce the bug?
Let the server run publicly and be reachable by global ingress. The below error in "additional details" was produced after running the MitM proxy for around 4 hours to proxy incoming traffic on port 22 to another machine.
I observed lots of traffic of approximately one connection every 10 seconds. They were, in fact, all short-living connections and terminated in a few seconds.
However, they allocate a socket and a thread of a server. My
sslisted hundreds of open listening sockets spawned by the MitM proxy. At some point, it could not spawn more threads and this error crashed the server when trying to accept a new incoming connection.The KeyboardInterrupt was made by me after I found out that the server was effectively down.
Note:
I was using my own fork which had slight modifications.
Last commit from you was eabae16. The software was running 57c1001 from my fork.
I do not expect that these modifications are the cause of the error, but the summary of these modifications are: adding a few command line arguments, added a tracking mechanism that made HTTP requests with login credentials in one thread (it did not spawn new threads!) and a provisioning mechanism that created the requested user on the target, in combination with a retry mechanism. This way, it accepts all usernames and all passwords and provides a valid shell on the target machine (in my configuration, since the "provisioning command" is a
useraddon the remote machine).This is not the only such error, I will post a second issue later, maybe. TL;DR: Same behavior, but the error was
OSError: Too many open filesdue to the number of open sockets (see above) filled all available file descriptors (default: 1024), before I increased the shell'sulimit.What should have happened?
The server should not go down. When the proxied TCP connection ends, the socket and thread should be freed to avoid this problem (if the problem is what I suspect).
Additional information
If you need more details, I can provide more logs; this is an extract.
Thanks for your patience ;)