Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit history

Commits on Nov 29, 2022

• fix airspider task queue max size bug

  Boris-code

  committed
  7ff9c79

  Copy full SHA for 7ff9c79

Commits on Nov 25, 2022

• 修复任务爬虫依赖其他爬虫的bug

  Boris-code

  committed
  f74f5d9

  Copy full SHA for f74f5d9

Commits on Nov 22, 2022

• 修复有依赖爬虫时，依赖爬虫不结束，新批次开启不了的bug

  Boris-code

  committed
  910a06d

  Copy full SHA for 910a06d

Commits on Nov 20, 2022

• Task spider 支持检查依赖爬虫的状态，依赖的爬虫做完才可结束自己

  Boris-code

  committed
  b7a983c

  Copy full SHA for b7a983c

Commits on Nov 18, 2022

• 批次爬虫支持设置不自动启动下一批次

  Boris-code

  committed
  0992fde

  Copy full SHA for 0992fde

Commits on Oct 28, 2022

• 优化批次爬虫采集速度计算逻辑

  Boris-code

  committed
  ef4164f

  Copy full SHA for ef4164f

Commits on Oct 27, 2022

• 1. 批次超时报警后，若后续批次完成，则发个批次完成的报警，提醒已恢复正常 2. 精简代码

  Boris-code

  committed
  771862d

  Copy full SHA for 771862d

Commits on Oct 26, 2022

• 修复爬虫心跳bug

  Boris-code

  committed
  f4e9e6d

  Copy full SHA for f4e9e6d

Commits on Oct 25, 2022

• air spider 支持去重

  Boris-code

  committed
  051c2ae

  Copy full SHA for 051c2ae

Commits on Oct 24, 2022

• 修复批次报警误报的bug

  Boris-code

  committed
  6ed6da5

  Copy full SHA for 6ed6da5

Commits on Oct 20, 2022

• 批次表的时间类型统一为datetime

  Boris-code

  committed
  e6d34a6

  Copy full SHA for e6d34a6

• spider的debug模式支持保存数据到数据库

  Boris-code

  committed
  1ce1758

  Copy full SHA for 1ce1758

Commits on Sep 29, 2022

• 下载器使用自定义配置生效

  Boris-code

  committed
  49d2eb7

  Copy full SHA for 49d2eb7

Commits on Aug 9, 2022

• 修复delete_keys的bug

  Boris-code

  committed
  47dcdac

  Copy full SHA for 47dcdac

Commits on Jul 31, 2022

• 1. request支持是否拼接绝对连接参数 2. 下载方法单独抽离出来，方便扩展

  Boris-code

  committed
  245bc93

  Copy full SHA for 245bc93

Commits on Jul 6, 2022

• 支持batch_interval

  Boris-code

  committed
  d5e81af

  Copy full SHA for d5e81af

• 修改注释

  Boris-code

  committed
  159b312

  Copy full SHA for 159b312

Commits on Jun 12, 2022

• 修改注释

  Boris-code

  committed
  a224573

  Copy full SHA for a224573

Commits on Jun 10, 2022

• TaskSpider 支持redis作为任务种子

  Boris-code

  committed
  49033b6

  Copy full SHA for 49033b6

• add TaskSpider

  Boris-code

  committed
  7e73bc5

  Copy full SHA for 7e73bc5

Commits on May 18, 2022

• 支持飞书报警

  Boris-code

  committed
  da85aeb

  Copy full SHA for da85aeb

Commits on May 17, 2022

• 优化collector

  Boris-code

  committed
  f2afac2

  Copy full SHA for f2afac2

Commits on Feb 12, 2022

• 优化好提示

  Boris-code

  committed
  dba0d0b

  Copy full SHA for dba0d0b

Commits on Dec 30, 2021

• 爬虫集成支持传参

  Boris-code

  committed
  3a1afb2

  Copy full SHA for 3a1afb2

Commits on Nov 18, 2021

• 适配python3.10

  Boris-code

  committed
  aa3df59

  Copy full SHA for aa3df59

Commits on Sep 22, 2021

• pipeline 支持close方法

  Boris-code

  committed
  493ff7b

  Copy full SHA for 493ff7b

Commits on Aug 27, 2021

• 框架主线程捕获异常，防止爬虫某个线程死掉 1.6.7-beta2

  Boris-code

  committed
  744608a

  Copy full SHA for 744608a

Commits on Aug 25, 2021

• 优化 redis锁，修复转字符串cookie为json时的bug，添加cookie池，response的源码支持重新赋值，以便浏览器渲染时使用

  Boris-code

  committed
  d1e0c78

  Copy full SHA for d1e0c78

Commits on Aug 17, 2021

• 数据入库失败 自动重试

  Boris-code

  committed
  1b00eb3

  Copy full SHA for 1b00eb3

Commits on Aug 15, 2021

• auto_stop_when_spider_done参数改为keep_alive

  Boris-code

  committed
  21d1737

  Copy full SHA for 21d1737

Commits on Aug 4, 2021

• 解决jion报错问题

  Boris-code

  committed
  32721c6

  Copy full SHA for 32721c6

• 集成打点监控

  Boris-code

  committed
  f5f17f9

  Copy full SHA for f5f17f9

Commits on Jul 15, 2021

• 优化批次爬虫下发新批次任务时需等待1分钟问题

  Boris-code

  committed
  239ba71

  Copy full SHA for 239ba71

• 优化锁的问题

  Boris-code

  committed
  15ed732

  Copy full SHA for 15ed732

Commits on Jun 18, 2021

• 日志支持带颜色输出

  Boris-code

  committed
  3f7491f

  Copy full SHA for 3f7491f