Creating network "nvidia-llm" with the default driver Creating milvus-etcd ... done Creating llm-inference-server ... done Creating milvus-minio ... done Creating milvus-standalone ... done Creating notebook-server ... done Creating evaluation ... done Creating chain-server ... done Creating llm-playground ... done Attaching to milvus-etcd, milvus-minio, llm-inference-server, milvus-standalone, notebook-server, evaluation, chain-server, llm-playground evaluation | [W 2023-12-27 11:02:52.710 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server. llm-inference-server | [1] Dec 27 11:02:48 [ INFO] - model_server - Reading the model directory. evaluation | [W 2023-12-27 11:02:52.715 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server. llm-inference-server | [1] Dec 27 11:02:48 [ INFO] - model_server - Model file format: HUGGINGFACE llm-inference-server | [1] Dec 27 11:02:48 [ INFO] - model_server - World Size: 8 llm-inference-server | [1] Dec 27 11:02:48 [ INFO] - model_server - Compute Capability: 8.0 llm-inference-server | [1] Dec 27 11:02:48 [ INFO] - model_server - Quantization: None evaluation | [I 2023-12-27 11:02:52.715 ServerApp] jupyter_lsp | extension was successfully linked. llm-inference-server | [1] Dec 27 11:02:48 [ INFO] - model_server.model - Calculating model hash. milvus-minio | WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated. milvus-minio | Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD evaluation | [I 2023-12-27 11:02:52.719 ServerApp] jupyter_server_terminals | extension was successfully linked. milvus-minio | WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.149Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.151Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"1000"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.151Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"4294967296"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.151Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_SNAPSHOT_COUNT","variable-value":"50000"} milvus-standalone | [2023/12/27 11:02:51.039 +00:00] [INFO] [roles/roles.go:294] ["starting running Milvus components"] milvus-standalone | [2023/12/27 11:02:51.039 +00:00] [INFO] [roles/roles.go:164] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so] milvus-standalone | 2023/12/27 11:02:51 maxprocs: Leaving GOMAXPROCS=255: CPU quota undefined milvus-standalone | milvus-standalone | __ _________ _ ____ ______ milvus-standalone | / |/ / _/ /| | / / / / / __/ milvus-standalone | / /|_/ // // /_| |/ / /_/ /\ \ milvus-standalone | /_/ /_/___/____/___/\____/___/ milvus-standalone | milvus-standalone | Welcome to use Milvus! milvus-standalone | Version: v2.3.1-gpu milvus-standalone | Built: Fri Sep 22 12:42:14 UTC 2023 milvus-standalone | GitCommit: 70cf65b05 milvus-standalone | GoVersion: go version go1.20.4 linux/amd64 milvus-standalone | milvus-standalone | open pid file: /run/milvus/standalone.pid milvus-standalone | lock pid file: /run/milvus/standalone.pid milvus-standalone | [2023/12/27 11:02:51.046 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource] milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.152Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","-advertise-client-urls=http://127.0.0.1:2379","-listen-client-urls","http://0.0.0.0:2379","--data-dir","/etcd"]} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.153Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/etcd","dir-type":"member"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.153Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]} evaluation | [W 2023-12-27 11:02:52.720 LabApp] 'token' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. milvus-standalone | [2023/12/27 11:02:51.047 +00:00] [DEBUG] [config/etcd_source.go:49] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"UseSSL\":false,\"Endpoints\":[\"etcd:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"] milvus-standalone | [2023/12/27 11:02:51.047 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3] notebook-server | [W 2023-12-27 11:02:52.667 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server. milvus-minio | milvus-minio | API: SYSTEM() milvus-minio | Time: 11:02:47 UTC 12/27/2023 milvus-minio | DeploymentID: 6df92764-c39a-4bd4-8988-5433678b8481 milvus-minio | Error: Invalid api configuration: Invalid value for transition workers (*fmt.wrapError) milvus-minio | 10: internal/logger/logger.go:258:logger.LogIf() milvus-minio | 9: cmd/config-current.go:546:cmd.applyDynamicConfigForSubSys() milvus-minio | 8: cmd/config-current.go:671:cmd.applyDynamicConfig() milvus-minio | 7: cmd/config-current.go:531:cmd.lookupConfigs() milvus-minio | 6: cmd/config-current.go:794:cmd.loadConfig() milvus-minio | 5: cmd/config.go:227:cmd.initConfig() milvus-minio | 4: cmd/config.go:199:cmd.(*ConfigSys).Init() milvus-minio | 3: cmd/server-main.go:465:cmd.initConfigSubsystem() milvus-minio | 2: cmd/server-main.go:432:cmd.initServer() milvus-minio | 1: cmd/server-main.go:637:cmd.serverMain() milvus-standalone | [2023/12/27 11:02:51.048 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"] notebook-server | [W 2023-12-27 11:02:52.672 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server. milvus-minio | MinIO Object Storage Server milvus-minio | Copyright: 2015-2023 MinIO, Inc. milvus-minio | License: GNU AGPLv3 milvus-minio | Version: RELEASE.2023-03-20T20-16-18Z (go1.19.7 linux/amd64) milvus-minio | evaluation | [W 2023-12-27 11:02:52.723 ServerApp] ServerApp.token config is deprecated in 2.0. Use IdentityProvider.token. evaluation | [I 2023-12-27 11:02:52.723 ServerApp] jupyterlab | extension was successfully linked. milvus-minio | Status: 1 Online, 0 Offline. milvus-minio | API: http://172.22.0.3:9000 http://127.0.0.1:9000 milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.158Z","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]} notebook-server | [I 2023-12-27 11:02:52.672 ServerApp] jupyter_lsp | extension was successfully linked. milvus-standalone | [2023/12/27 11:02:51.057 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=EtcdSource] milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.170Z","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"19002cfc6","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":255,"max-cpu-available":255,"member-initialized":true,"name":"default","data-dir":"/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":50000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://127.0.0.1:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":4294967296,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"1µs","auto-compaction-interval":"1µs","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} milvus-etcd | {"level":"warn","ts":1703674967.1718278,"caller":"fileutil/fileutil.go:57","msg":"check file permission","error":"directory \"/etcd\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"} notebook-server | [I 2023-12-27 11:02:52.676 ServerApp] jupyter_server_terminals | extension was successfully linked. milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.183Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/etcd/member/snap/db","took":"2.030868ms"} milvus-standalone | [2023/12/27 11:02:51.063 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource] milvus-standalone | [2023/12/27 11:02:51.063 +00:00] [INFO] [rootcoord/root_coord.go:159] ["update rootcoord state"] [state=Abnormal] milvus-standalone | [2023/12/27 11:02:51.063 +00:00] [INFO] [http/server.go:77] ["management listen"] [addr=:9091] evaluation | [I 2023-12-27 11:02:52.724 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret milvus-minio | Console: http://172.22.0.3:9001 http://127.0.0.1:9001 milvus-minio | milvus-minio | Documentation: https://min.io/docs/minio/linux/index.html milvus-minio | Warning: The standard parity is set to 0. This can lead to data loss. milvus-minio | milvus-minio | You are running an older version of MinIO released 9 months ago milvus-minio | Update: Run `mc admin update` milvus-minio | milvus-minio | milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.194Z","caller":"etcdserver/server.go:530","msg":"No snapshot found. Recovering WAL from scratch!"} notebook-server | [W 2023-12-27 11:02:52.677 LabApp] 'token' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.200Z","caller":"etcdserver/raft.go:529","msg":"restarting local member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","commit-index":281} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.200Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=()"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.200Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 4"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.200Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8e9e05c52164694d [peers: [], term: 4, commit: 281, applied: 0, lastindex: 281, lastterm: 4]"} milvus-standalone | [2023/12/27 11:02:51.063 +00:00] [DEBUG] [rootcoord/service.go:158] ["init params done.."] milvus-standalone | [2023/12/27 11:02:51.063 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3] milvus-standalone | [2023/12/27 11:02:51.063 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"] milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [DEBUG] [rootcoord/service.go:175] ["etcd connect done ..."] milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [DEBUG] [rootcoord/service.go:240] ["start grpc "] [port=53100] milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [DEBUG] [proxy/multi_rate_limiter.go:187] ["RateLimiter register for rateType"] [rateType=DDLPartition] [rateLimit=+inf] [burst=1.7976931348623157e+308] milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [DEBUG] [proxy/multi_rate_limiter.go:187] ["RateLimiter register for rateType"] [rateType=DDLIndex] [rateLimit=+inf] [burst=1.7976931348623157e+308] notebook-server | [W 2023-12-27 11:02:52.680 ServerApp] ServerApp.token config is deprecated in 2.0. Use IdentityProvider.token. milvus-etcd | {"level":"warn","ts":"2023-12-27T11:02:47.218Z","caller":"auth/store.go:1233","msg":"simple token is not cryptographically signed"} evaluation | [I 2023-12-27 11:02:52.984 ServerApp] notebook_shim | extension was successfully linked. notebook-server | [I 2023-12-27 11:02:52.680 ServerApp] jupyterlab | extension was successfully linked. milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["Start check query node health loop"] milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [DEBUG] [proxy/service.go:113] ["create a new Proxy instance"] [state=Abnormal] notebook-server | [I 2023-12-27 11:02:52.681 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.230Z","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":115} milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [DEBUG] [proxy/service.go:354] ["init Proxy server"] milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [DEBUG] [proxy/service.go:385] ["Proxy init service's parameter table done"] milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [DEBUG] [proxy/service.go:387] ["Proxy init http server's parameter table done"] milvus-standalone | [2023/12/27 11:02:51.064 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3] milvus-standalone | [2023/12/27 11:02:51.065 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3] milvus-standalone | [2023/12/27 11:02:51.065 +00:00] [DEBUG] [querycoord/service.go:213] [network] [port=19531] milvus-standalone | [2023/12/27 11:02:51.065 +00:00] [DEBUG] [datacoord/service.go:129] ["network port"] [port=13333] milvus-standalone | [2023/12/27 11:02:51.065 +00:00] [DEBUG] [proxy/service.go:394] ["init Proxy's parameter table done"] ["internal address"=172.22.0.5:19529] ["external address"=172.22.0.5:19530] milvus-standalone | [2023/12/27 11:02:51.065 +00:00] [DEBUG] [proxy/service.go:397] ["init Proxy's tracer done"] ["service name"="Proxy ip: 172.22.0.5, port: 19530"] milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.240Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":4294967296,"quota-size":"4.3 GB"} milvus-standalone | [2023/12/27 11:02:51.065 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3] notebook-server | [I 2023-12-27 11:02:52.948 ServerApp] notebook_shim | extension was successfully linked. notebook-server | [W 2023-12-27 11:02:52.964 ServerApp] All authentication is disabled. Anyone who can connect to this server will be able to run code. milvus-standalone | [2023/12/27 11:02:51.067 +00:00] [DEBUG] [querynode/service.go:98] [QueryNode] [port=21123] notebook-server | [I 2023-12-27 11:02:52.965 ServerApp] notebook_shim | extension was successfully loaded. milvus-standalone | [2023/12/27 11:02:51.068 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3] notebook-server | [I 2023-12-27 11:02:52.967 ServerApp] jupyter_lsp | extension was successfully loaded. milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [DEBUG] [proxy/service.go:301] ["Proxy internal server listen on tcp"] [port=19529] milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [DEBUG] [proxy/service.go:308] ["Proxy internal server already listen on tcp"] [port=19529] milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3] milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [INFO] [components/index_coord.go:39] ["IndexCoord running ..."] milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [DEBUG] [querynode/service.go:115] ["QueryNode connect to etcd successfully"] notebook-server | [I 2023-12-27 11:02:52.967 ServerApp] jupyter_server_terminals | extension was successfully loaded. milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [DEBUG] [proxy/service.go:340] ["create Proxy internal grpc server"] ["enforcement policy"="{\"MinTime\":5000000000,\"PermitWithoutStream\":true}"] ["server parameters"="{\"MaxConnectionIdle\":0,\"MaxConnectionAge\":0,\"MaxConnectionAgeGrace\":0,\"Time\":60000000000,\"Timeout\":10000000000}"] milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [DEBUG] [indexnode/indexnode.go:108] ["New IndexNode ..."] notebook-server | [I 2023-12-27 11:02:52.968 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.10/site-packages/jupyterlab milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [INFO] [proxy/service.go:424] ["Proxy server listen on tcp"] [port=19530] milvus-standalone | [2023/12/27 11:02:51.069 +00:00] [INFO] [proxy/service.go:428] ["Proxy server already listen on tcp"] [port=19530] milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.244Z","caller":"etcdserver/server.go:854","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.5","cluster-version":"to_be_decided"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.244Z","caller":"etcdserver/server.go:754","msg":"starting initial election tick advance","election-ticks":10} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.244Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"} milvus-standalone | [2023/12/27 11:02:51.070 +00:00] [DEBUG] [indexnode/service.go:85] [IndexNode] ["network address"=172.22.0.5:21121] ["network port: "=21121] milvus-standalone | [2023/12/27 11:02:51.070 +00:00] [INFO] [datanode/service.go:262] ["DataNode address"] [address=172.22.0.5:21124] milvus-standalone | [2023/12/27 11:02:51.070 +00:00] [DEBUG] [proxy/service.go:212] ["Get proxy rate limiter done"] [port=19530] milvus-standalone | [2023/12/27 11:02:51.070 +00:00] [DEBUG] [proxy/service.go:276] ["create Proxy grpc server"] ["enforcement policy"="{\"MinTime\":5000000000,\"PermitWithoutStream\":true}"] ["server parameters"="{\"MaxConnectionIdle\":0,\"MaxConnectionAge\":0,\"MaxConnectionAgeGrace\":0,\"Time\":60000000000,\"Timeout\":10000000000}"] milvus-standalone | [2023/12/27 11:02:51.070 +00:00] [INFO] [proxy/service.go:507] ["register Proxy http server"] milvus-standalone | [2023/12/27 11:02:51.070 +00:00] [DEBUG] [proxy/service.go:514] ["create RootCoord client for Proxy"] milvus-standalone | [2023/12/27 11:02:51.070 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] milvus-standalone | [2023/12/27 11:02:51.072 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.245Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]} milvus-standalone | [2023/12/27 11:02:51.072 +00:00] [DEBUG] [proxy/service.go:520] ["create RootCoord client for Proxy done"] milvus-standalone | [2023/12/27 11:02:51.072 +00:00] [DEBUG] [proxy/service.go:523] ["init RootCoord client for Proxy"] milvus-standalone | [2023/12/27 11:02:51.072 +00:00] [DEBUG] [proxy/service.go:528] ["init RootCoord client for Proxy done"] milvus-standalone | [2023/12/27 11:02:51.072 +00:00] [DEBUG] [proxy/service.go:530] ["Proxy wait for RootCoord to be healthy"] milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.245Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.245Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"} evaluation | [W 2023-12-27 11:02:53.000 ServerApp] All authentication is disabled. Anyone who can connect to this server will be able to run code. milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.247Z","caller":"embed/etcd.go:275","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://127.0.0.1:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.247Z","caller":"embed/etcd.go:584","msg":"serving peer traffic","address":"127.0.0.1:2380"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:47.247Z","caller":"embed/etcd.go:556","msg":"cmux::serve","address":"127.0.0.1:2380"} notebook-server | [I 2023-12-27 11:02:52.968 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab milvus-standalone | [2023/12/27 11:02:51.073 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:51.073 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.302Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d is starting a new election at term 4"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.302Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became pre-candidate at term 4"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.302Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgPreVoteResp from 8e9e05c52164694d at term 4"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.302Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became candidate at term 5"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.302Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 5"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.302Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became leader at term 5"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.302Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 5"} notebook-server | [I 2023-12-27 11:02:52.969 LabApp] Extension Manager is 'pypi'. notebook-server | [I 2023-12-27 11:02:52.971 ServerApp] jupyterlab | extension was successfully loaded. milvus-standalone | [2023/12/27 11:02:51.166 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] milvus-standalone | [2023/12/27 11:02:51.166 +00:00] [DEBUG] [rootcoord/service.go:181] ["grpc init done ..."] milvus-standalone | [2023/12/27 11:02:51.166 +00:00] [INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=rocksmq] notebook-server | [I 2023-12-27 11:02:52.972 ServerApp] Serving notebooks from local directory: /app notebook-server | [I 2023-12-27 11:02:52.972 ServerApp] Jupyter Server 2.12.1 is running at: milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.304Z","caller":"etcdserver/server.go:2054","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://127.0.0.1:2379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.304Z","caller":"embed/serve.go:100","msg":"ready to serve client requests"} evaluation | [I 2023-12-27 11:02:53.000 ServerApp] notebook_shim | extension was successfully loaded. notebook-server | [I 2023-12-27 11:02:52.972 ServerApp] http://e500f74bc338:8888/lab evaluation | [I 2023-12-27 11:02:53.002 ServerApp] jupyter_lsp | extension was successfully loaded. notebook-server | [I 2023-12-27 11:02:52.972 ServerApp] http://127.0.0.1:8888/lab evaluation | [I 2023-12-27 11:02:53.003 ServerApp] jupyter_server_terminals | extension was successfully loaded. notebook-server | [I 2023-12-27 11:02:52.972 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). evaluation | [I 2023-12-27 11:02:53.004 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.10/site-packages/jupyterlab milvus-standalone | [2023/12/27 11:02:51.166 +00:00] [DEBUG] [server/global_rmq.go:39] ["initializing global rmq"] [path=/var/lib/milvus/rdb_data] milvus-standalone | [2023/12/27 11:02:51.166 +00:00] [DEBUG] [rootcoord/service.go:184] ["RootCoord start to create DataCoord client"] notebook-server | [W 2023-12-27 11:02:52.979 ServerApp] No web browser found: Error('could not locate runnable browser'). milvus-standalone | [2023/12/27 11:02:51.166 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] notebook-server | [I 2023-12-27 11:02:53.014 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server evaluation | [I 2023-12-27 11:02:53.004 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab milvus-standalone | [2023/12/27 11:02:51.167 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] evaluation | [I 2023-12-27 11:02:53.005 LabApp] Extension Manager is 'pypi'. milvus-standalone | [2023/12/27 11:02:51.167 +00:00] [DEBUG] [querycoord/service.go:155] ["QueryCoord try to wait for RootCoord ready"] milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.304Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"} milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.304Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"} milvus-standalone | [2023/12/27 11:02:51.173 +00:00] [INFO] [datanode/service.go:271] ["initializing RootCoord client for DataNode"] milvus-standalone | [2023/12/27 11:02:51.173 +00:00] [DEBUG] [querynode/service.go:125] [QueryNode] [State=Initializing] milvus-standalone | [2023/12/27 11:02:51.175 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] milvus-standalone | [2023/12/27 11:02:51.175 +00:00] [INFO] [querynodev2/server.go:239] ["QueryNode session info"] [metaPath=by-dev/meta] milvus-standalone | [2023/12/27 11:02:51.178 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] evaluation | [I 2023-12-27 11:02:53.007 ServerApp] jupyterlab | extension was successfully loaded. milvus-etcd | {"level":"info","ts":"2023-12-27T11:02:48.305Z","caller":"embed/serve.go:146","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"[::]:2379"} milvus-standalone | [2023/12/27 11:02:51.178 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:51.178 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] evaluation | [I 2023-12-27 11:02:53.008 ServerApp] Serving notebooks from local directory: /app evaluation | [I 2023-12-27 11:02:53.008 ServerApp] Jupyter Server 2.12.1 is running at: milvus-standalone | [2023/12/27 11:02:51.173 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3] milvus-standalone | [2023/12/27 11:02:51.184 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.184 +00:00] [DEBUG] [rootcoord/service.go:201] ["RootCoord start to create QueryCoord client"] milvus-standalone | [2023/12/27 11:02:51.184 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.184 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] milvus-standalone | [2023/12/27 11:02:51.184 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.184 +00:00] [INFO] [indexnode/indexnode.go:197] ["IndexNode init"] [state=Initializing] milvus-standalone | [2023/12/27 11:02:51.184 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] evaluation | [I 2023-12-27 11:02:53.008 ServerApp] http://5b6a61e6957b:8889/lab evaluation | [I 2023-12-27 11:02:53.008 ServerApp] http://127.0.0.1:8889/lab milvus-standalone | [2023/12/27 11:02:51.185 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.185 +00:00] [INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=rocksmq] milvus-standalone | [2023/12/27 11:02:51.185 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] evaluation | [I 2023-12-27 11:02:53.008 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). milvus-standalone | [2023/12/27 11:02:51.185 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] evaluation | [W 2023-12-27 11:02:53.015 ServerApp] No web browser found: Error('could not locate runnable browser'). milvus-standalone | [2023/12/27 11:02:51.186 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true] milvus-standalone | [2023/12/27 11:02:51.187 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.188 +00:00] [DEBUG] [sessionutil/session_util.go:343] ["Session get serverID success"] [key=id] [ServerId=5] milvus-standalone | [2023/12/27 11:02:51.188 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=querynode] [address=172.22.0.5:21123] [id=5] milvus-standalone | [2023/12/27 11:02:51.188 +00:00] [INFO] [querynodev2/server.go:154] ["QueryNode init session"] [nodeID=5] ["node address"=172.22.0.5:21123] milvus-standalone | [2023/12/27 11:02:51.188 +00:00] [INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=rocksmq] milvus-standalone | [2023/12/27 11:02:51.188 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true] milvus-standalone | [2023/12/27 11:02:51.188 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=indexnode] [address=172.22.0.5:21121] [id=5] milvus-standalone | [2023/12/27 11:02:51.188 +00:00] [INFO] [indexnode/indexnode.go:204] ["IndexNode init session successful"] [serverID=5] milvus-standalone | WARNING: Logging before InitGoogleLogging() is written to STDERR milvus-standalone | I20231227 11:02:51.189056 65 knowhere_config.cc:93] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384 milvus-standalone | I20231227 11:02:51.189780 65 knowhere_config.cc:104] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0 milvus-standalone | I20231227 11:02:51.189882 65 knowhere_config.cc:52] [KNOWHERE][ShowVersion][milvus] Knowhere Version: v2.2.1-gpu milvus-standalone | [2023/12/27 11:02:51.190 +00:00] [INFO] [indexnode/indexnode.go:209] ["Init IndexNode finished"] [] milvus-standalone | [2023/12/27 11:02:51.190 +00:00] [DEBUG] [indexnode/service.go:72] ["IndexNode init done ..."] milvus-standalone | [2023/12/27 11:02:51.190 +00:00] [INFO] [indexnode/indexnode.go:221] [IndexNode] [State=Healthy] milvus-standalone | [2023/12/27 11:02:51.190 +00:00] [INFO] [indexnode/indexnode.go:224] ["IndexNode start finished"] [] milvus-standalone | [2023/12/27 11:02:51.190 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=indexnode] [ServerID=5] milvus-standalone | [2023/12/27 11:02:51.190 +00:00] [DEBUG] [indexnode/task_scheduler.go:242] ["IndexNode TaskScheduler start build loop ..."] milvus-standalone | [2023/12/27 11:02:51.193 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/indexnode-5] [value="{\"ServerID\":5,\"ServerName\":\"indexnode\",\"Address\":\"172.22.0.5:21121\",\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587875613004046610}"] milvus-standalone | [2023/12/27 11:02:51.193 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=indexnode] [serverID=5] milvus-standalone | [2023/12/27 11:02:51.195 +00:00] [DEBUG] [indexnode/service.go:204] ["IndexNode Register etcd success"] milvus-standalone | [2023/12/27 11:02:51.195 +00:00] [DEBUG] [indexnode/service.go:76] ["IndexNode start done ..."] milvus-standalone | [2023/12/27 11:02:51.195 +00:00] [DEBUG] [components/index_node.go:55] ["IndexNode successfully started"] milvus-standalone | [2023/12/27 11:02:51.210 +00:00] [DEBUG] [server/rocksmq_impl.go:179] ["Start rocksmq "] ["max proc"=255] [parallism=4] ["lru cache"=4294967296] evaluation | [I 2023-12-27 11:02:53.048 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server milvus-standalone | [2023/12/27 11:02:51.260 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=indexnode] [key=by-dev/meta/session/indexnode-5] milvus-standalone | [2023/12/27 11:02:51.274 +00:00] [WARN] [grpcclient/client.go:483] ["fail to get grpc client"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.274 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.280 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:51.280 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:51.358 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2023/12/27 09:24:29.997 +00:00] [save=2023/12/27 11:02:54.358 +00:00] [next=2023/12/27 11:02:51.358 +00:00] milvus-standalone | [2023/12/27 11:02:51.358 +00:00] [INFO] [msgstream/mq_factory.go:17] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data] milvus-standalone | [2023/12/27 11:02:51.359 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] milvus-standalone | [2023/12/27 11:02:51.359 +00:00] [DEBUG] [server/rocksmq_retention.go:80] ["Rocksmq retention goroutine start!"] milvus-standalone | [2023/12/27 11:02:51.359 +00:00] [INFO] [msgstream/mq_factory.go:17] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data] milvus-standalone | [2023/12/27 11:02:51.359 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] milvus-standalone | [2023/12/27 11:02:51.359 +00:00] [INFO] [msgstream/mq_factory.go:17] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data] milvus-standalone | [2023/12/27 11:02:51.361 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.362 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.364 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true] milvus-standalone | [2023/12/27 11:02:51.364 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=indexcoord] [address=172.22.0.5:13333] [id=5] milvus-standalone | [2023/12/27 11:02:51.364 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] milvus-standalone | [2023/12/27 11:02:51.367 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.367 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true] milvus-standalone | [2023/12/27 11:02:51.367 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=rootcoord] [address=172.22.0.5:53100] [id=5] milvus-standalone | [2023/12/27 11:02:51.367 +00:00] [INFO] [rootcoord/root_coord.go:159] ["update rootcoord state"] [state=Initializing] milvus-standalone | [2023/12/27 11:02:51.367 +00:00] [INFO] [storage/minio_chunk_manager.go:154] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files] milvus-standalone | [2023/12/27 11:02:51.368 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true] milvus-standalone | [2023/12/27 11:02:51.369 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=datacoord] [address=172.22.0.5:13333] [id=5] milvus-standalone | [2023/12/27 11:02:51.369 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"] milvus-standalone | [2023/12/27 11:02:51.369 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"] milvus-standalone | [2023/12/27 11:02:51.378 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2023/12/27 09:24:30.007 +00:00] [save=2023/12/27 11:02:54.369 +00:00] [next=2023/12/27 11:02:51.369 +00:00] milvus-standalone | [2023/12/27 11:02:51.378 +00:00] [INFO] [rootcoord/root_coord.go:366] ["id allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp] milvus-standalone | [2023/12/27 11:02:51.378 +00:00] [WARN] [grpcclient/client.go:483] ["fail to get grpc client"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.378 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.379 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:51.379 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:51.380 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2023/12/27 09:24:30.013 +00:00] [save=2023/12/27 11:02:54.379 +00:00] [next=2023/12/27 11:02:51.379 +00:00] milvus-standalone | [2023/12/27 11:02:51.380 +00:00] [INFO] [storage/minio_chunk_manager.go:154] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files] milvus-standalone | [2023/12/27 11:02:51.380 +00:00] [INFO] [rootcoord/root_coord.go:382] ["tso allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp] milvus-standalone | [2023/12/27 11:02:51.380 +00:00] [DEBUG] [rootcoord/suffix_snapshot.go:557] ["suffix snapshot GC goroutine start!"] milvus-standalone | [2023/12/27 11:02:51.380 +00:00] [INFO] [storage/minio_chunk_manager.go:154] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files] milvus-standalone | [2023/12/27 11:02:51.380 +00:00] [INFO] [querynodev2/server.go:293] ["queryNode try to connect etcd success"] [MetaRootPath=by-dev/meta] milvus-standalone | [2023/12/27 11:02:51.380 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:25] ["query node use concurrent safe scheduler"] [max_concurrency=255] milvus-standalone | [2023/12/27 11:02:51.381 +00:00] [INFO] [querynodev2/server.go:299] ["queryNode init scheduler"] [policy=fifo] milvus-standalone | [2023/12/27 11:02:51.381 +00:00] [INFO] [segments/segment_loader.go:112] ["SegmentLoader created"] [ioPoolSize=256] milvus-standalone | [2023/12/27 11:02:51.382 +00:00] [INFO] [rootcoord/meta_table.go:150] ["recover databases"] ["num of dbs"=1] milvus-standalone | [2023/12/27 11:02:51.383 +00:00] [INFO] [datacoord/meta.go:170] ["DataCoord meta reloadFromKV done"] [duration=2.744847ms] milvus-standalone | [2023/12/27 11:02:51.383 +00:00] [INFO] [datacoord/channel_store.go:171] ["channel store reload done"] [duration=267.726µs] milvus-standalone | [2023/12/27 11:02:51.384 +00:00] [DEBUG] [sessionutil/session_util.go:589] ["SessionUtil GetSessions "] [prefix=datanode] [key=datanode-2] [address=172.19.0.7:21124] milvus-standalone | [2023/12/27 11:02:51.384 +00:00] [INFO] [datacoord/server.go:479] ["DataCoord success to get DataNode sessions"] [sessions="{\"datanode-2\":{\"ServerID\":2,\"ServerName\":\"datanode\",\"Address\":\"172.19.0.7:21124\",\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587875610686164996}}"] milvus-standalone | [2023/12/27 11:02:51.384 +00:00] [INFO] [datacoord/channel_manager.go:348] ["register node with no reassignment"] ["registered node"=2] milvus-standalone | [2023/12/27 11:02:51.384 +00:00] [INFO] [datacoord/channel_manager.go:171] ["starting etcd states checker"] milvus-standalone | [2023/12/27 11:02:51.384 +00:00] [INFO] [datacoord/channel_manager.go:176] ["starting background balance checker"] milvus-standalone | [2023/12/27 11:02:51.384 +00:00] [INFO] [datacoord/channel_manager.go:179] ["cluster start up"] [nodes="[2]"] [oNodes="[]"] ["old onlines"="[]"] ["new onlines"="[2]"] [offLines="[]"] milvus-standalone | [2023/12/27 11:02:51.384 +00:00] [INFO] [rootcoord/meta_table.go:188] ["recover collections from db"] [collection_num=0] [partition_num=0] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-3] [address=172.20.0.5:21121] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-4] [address=172.21.0.5:21121] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-5] [address=172.22.0.5:21121] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=3] ["node address"=172.20.0.5:21121] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=3] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=3] ["IndexNode num"=1] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=4] ["node address"=172.21.0.5:21121] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=4] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=4] ["IndexNode num"=2] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=5] ["node address"=172.22.0.5:21121] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [WARN] [grpcclient/client.go:483] ["fail to get grpc client"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=5] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=5] ["IndexNode num"=3] milvus-standalone | [2023/12/27 11:02:51.385 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [INFO] [rootcoord/meta_table.go:204] ["RootCoord meta table reload done"] [duration=5.779702ms] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [INFO] [datacoord/garbage_collector.go:71] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=1h0m0s] [dropTolerance=3h0m0s] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [DEBUG] [datacoord/service.go:232] ["DataCoord init done ..."] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=5] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_0] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_1] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_2] milvus-standalone | [2023/12/27 11:02:51.386 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_3] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_4] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_5] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_6] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_7] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_8] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_9] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_10] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_11] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_12] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_13] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_14] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [WARN] [server/rocksmq_impl.go:400] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_15] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [INFO] [rootcoord/dml_channels.go:216] ["init dml channels"] [prefix=by-dev-rootcoord-dml] [num=16] milvus-standalone | [2023/12/27 11:02:51.387 +00:00] [DEBUG] [rootcoord/root_coord.go:451] ["RootCoord init QuotaCenter done"] milvus-standalone | [2023/12/27 11:02:51.390 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: indexcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:243\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] milvus-standalone | [2023/12/27 11:02:51.391 +00:00] [DEBUG] [rootcoord/kv_catalog.go:726] ["the key has existed"] [key=root-coord/credential/roles/admin] milvus-standalone | [2023/12/27 11:02:51.392 +00:00] [DEBUG] [rootcoord/kv_catalog.go:726] ["the key has existed"] [key=root-coord/credential/roles/public] milvus-standalone | [2023/12/27 11:02:51.398 +00:00] [DEBUG] [rootcoord/service.go:146] ["RootCoord init done ..."] milvus-standalone | [2023/12/27 11:02:51.398 +00:00] [INFO] [rootcoord/service.go:288] ["RootCoord Core start ..."] milvus-standalone | [2023/12/27 11:02:51.398 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=rootcoord] [ServerID=5] milvus-standalone | [2023/12/27 11:02:51.402 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] milvus-standalone | [2023/12/27 11:02:51.481 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.481 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=0] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:531\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:355\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] milvus-standalone | [2023/12/27 11:02:51.580 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.580 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=0] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:156\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:95\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] milvus-standalone | [2023/12/27 11:02:51.587 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.587 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=0] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:285\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:189\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] milvus-standalone | [2023/12/27 11:02:51.681 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.682 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:51.682 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:51.781 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.782 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:51.782 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:51.787 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.788 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:51.788 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:51.883 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.983 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:51.996 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:52.283 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:52.284 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:52.284 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:52.384 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:52.385 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:52.385 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:52.400 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:52.401 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:52.401 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:52.484 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:52.586 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:52.601 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:53.288 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:53.290 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:53.290 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:53.387 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:53.390 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:53.390 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:53.402 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:53.403 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.5:53100] milvus-standalone | [2023/12/27 11:02:53.403 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.5:53100] [serverID=3] milvus-standalone | [2023/12/27 11:02:53.492 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded: connection error: desc = \"transport: error while dialing: dial tcp 172.20.0.5:53100: i/o timeout\""] milvus-standalone | [2023/12/27 11:02:53.590 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:53.606 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="failed to connect 172.20.0.5:53100, reason: context deadline exceeded"] milvus-standalone | [2023/12/27 11:02:53.690 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=973771661721] milvus-standalone | [2023/12/27 11:02:53.690 +00:00] [INFO] [querynodev2/server.go:353] ["query node init successfully"] [queryNodeID=5] [Address=172.22.0.5:21123] milvus-standalone | [2023/12/27 11:02:53.690 +00:00] [DEBUG] [querynode/service.go:226] ["QueryNode init done ..."] milvus-standalone | [2023/12/27 11:02:53.690 +00:00] [INFO] [querynodev2/server.go:372] ["query node start successfully"] [queryNodeID=5] [Address=172.22.0.5:21123] [mmapEnabled=false] milvus-standalone | [2023/12/27 11:02:53.690 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=querynode] [ServerID=5] milvus-standalone | [2023/12/27 11:02:53.690 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:169] ["start execute loop"] milvus-standalone | [2023/12/27 11:02:53.694 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/querynode-5] [value="{\"ServerID\":5,\"ServerName\":\"querynode\",\"Address\":\"172.22.0.5:21123\",\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587875613004046692}"] milvus-standalone | [2023/12/27 11:02:53.694 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=querynode] [serverID=5] milvus-standalone | [2023/12/27 11:02:53.695 +00:00] [DEBUG] [querynode/service.go:231] ["QueryNode start done ..."] milvus-standalone | [2023/12/27 11:02:53.695 +00:00] [DEBUG] [components/query_node.go:58] ["QueryNode successfully started"] milvus-standalone | [2023/12/27 11:02:53.792 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=querynode] [key=by-dev/meta/session/querynode-5] milvus-standalone | [2023/12/27 11:02:54.410 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: indexcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:243\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:234\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] milvus-standalone | [2023/12/27 11:02:54.424 +00:00] [ERROR] [sessionutil/session_util.go:448] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:448\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:269\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:274\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:289\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:148\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"] chain-server | Traceback (most recent call last): chain-server | File "/usr/local/bin/uvicorn", line 8, in chain-server | sys.exit(main()) chain-server | File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__ chain-server | return self.main(*args, **kwargs) chain-server | File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main chain-server | rv = self.invoke(ctx) chain-server | File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke chain-server | return ctx.invoke(self.callback, **ctx.params) chain-server | File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke chain-server | return __callback(*args, **kwargs) chain-server | File "/usr/local/lib/python3.10/dist-packages/uvicorn/main.py", line 416, in main chain-server | run( chain-server | File "/usr/local/lib/python3.10/dist-packages/uvicorn/main.py", line 587, in run chain-server | server.run() chain-server | File "/usr/local/lib/python3.10/dist-packages/uvicorn/server.py", line 61, in run chain-server | return asyncio.run(self.serve(sockets=sockets)) chain-server | File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run chain-server | return loop.run_until_complete(main) chain-server | File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete chain-server | File "/usr/local/lib/python3.10/dist-packages/uvicorn/server.py", line 68, in serve chain-server | config.load() chain-server | File "/usr/local/lib/python3.10/dist-packages/uvicorn/config.py", line 467, in load chain-server | self.loaded_app = import_from_string(self.app) chain-server | File "/usr/local/lib/python3.10/dist-packages/uvicorn/importer.py", line 21, in import_from_string chain-server | module = importlib.import_module(module_str) chain-server | File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module chain-server | return _bootstrap._gcd_import(name[level:], package, level) chain-server | File "", line 1050, in _gcd_import chain-server | File "", line 1027, in _find_and_load chain-server | File "", line 1006, in _find_and_load_unlocked chain-server | File "", line 688, in _load_unlocked chain-server | File "", line 883, in exec_module chain-server | File "", line 241, in _call_with_frames_removed chain-server | File "/opt/RetrievalAugmentedGeneration/common/server.py", line 27, in chain-server | from pymilvus.exceptions import MilvusException, MilvusUnavailableException chain-server | File "/usr/local/lib/python3.10/dist-packages/pymilvus/__init__.py", line 26, in chain-server | from .bulk_writer.remote_bulk_writer import ( chain-server | File "/usr/local/lib/python3.10/dist-packages/pymilvus/bulk_writer/remote_bulk_writer.py", line 18, in chain-server | from minio import Minio chain-server | File "/usr/local/lib/python3.10/dist-packages/minio/__init__.py", line 41, in chain-server | from .api import Minio chain-server | File "/usr/local/lib/python3.10/dist-packages/minio/api.py", line 44, in chain-server | from urllib3.response import BaseHTTPResponse chain-server | ImportError: cannot import name 'BaseHTTPResponse' from 'urllib3.response' (/usr/local/lib/python3.10/dist-packages/urllib3/response.py) chain-server exited with code 1