Apache DolphinScheduler 3.2.1 版本发布：增强功能与安全性的全面升级

2023 大语言模型技术报告.pdf

近期，Apache DolphinScheduler 社区激动地宣布 3.2.1 版本的发布。此次更新不仅着力解决了前一版本（3.2.0）中遗留的问题，而且引入了一系列的功能增强和优化措施。

原先的问题主要源于部分重要代码在发布过程中未能成功合并（cherry-pick），加之这部分代码的合并过程较为复杂，因此，3.2.1 版本基于 2024年2月的 dev 分支代码，剔除了一些不兼容的特性后发布。

全部 Changelog：https://github.com/apache/dolphinscheduler/releases/tag/3.2.1

下载地址：https://dolphinscheduler.apache.org/zh-cn/download/3.2.1

主要修复和功能增强

新特性和优化

SQL 任务现支持使用 druid 进行 SQL 分割，支持设置 maxRows。
支持自定义 HTTP body 渲染。
Kubernetes (k8s) 现支持自定义标签 (label)。
新增支持阿里云语音告警源。
Helm chart 现支持 JDBC 注册中心；支持任务类型过滤。

关键问题修复

修复从 3.1.x 升级到 3.2.x 的失败问题。
解决任务组件在使用资源中心时只能使用绝对全路径的限制。
修复启动参数优先级设置错误。
解决数据质量任务无法执行的问题。
修复任务组队列失效问题。
解决任务定义列表修改时任务消失的问题。
修复特殊情况下删除工作流实例导致的空指针异常（NPE）。
解决 Master 和 Worker 之间的通信问题。
修复 Kyuubi 数据源在 UI 中不显示的问题。

安全性改进

此版本也对几个关键的 CVE 问题进行了修复，包括：

CVE-2023-49250
CVE-2023-51770
CVE-2023-50270
CVE-2023-49068
CVE-2023-49109

BugFix

fix: Resource relate path invalid when tenant change (#15581)
[fix][worker]:Fix the issue of missing disk usage reporting information in the worker (#15565)
[Fix] Fix WorkflowInstance batch start failed will throw incorrect exception. (#15577)
Fix create parent directory will cause FileAlreadyExistsException (#15576)
Fix Recover WorkflowInstance will casue workflow Instance state is success but task insatnce is killed/paused (#15574)
fix: data quality may fail in docker mode (#15563)
fix: start param for wf not work (#15544)
fix: ddl without drop exists (#14128)
fix switch js (#15487)
fix: data quality can not use (#15551)
Fix createFile with permission will not work (#15556)
[Bug][force-success] force success add end time (#15144)
[Bug][Task Api] fix 'MACPATTERN' in ProcessUtils and cover all cases on MacOS in ProcessUtilsTest (#15480)
Fix TaskGroupQueue will never be wakeup due to wakeup failed at one time (#15528) Exit JVM when OOM (#15538)
Fix exception occur in RpcServer side, it will not be sent to RpcClient (#15536)
front: When you edit a task in the task definition list, the front task list is displayed (#12819)
[Fix] [Bug] Change default version of Workflow/TaskDefinition to 1 (#15498)
[Bug] Fix a bug, When the worker service offline, workerNodeInfo cache in master cannot delete the offline worker (#15459)
fix workflow will have same updatetime when import (#14810)
[BUG] #15013 Fix retryInterval in RetryPolicy will never be used in RetryUtils (#15014)
Throw IllegalArgumentException if parse time placeholder error (#15514)
Fix PostgresqlDatabaseContainerProvider get Image is incorrect (#15434)
[Bug][Api] Fix NPE when deleting a workflow instance (#15485)
Directly Throw exception when taskInstancy log path is empty which log need to be queried (#15511)
Fix notify failover WorkflowInstance will cause NPE (#15499)
[HotFix] Fix createTaskInstanceWorkingDirectory failed if the old path exist (#15377)
[bug] Exception when using host in ipv6 format (#14040)
[Bug][Master]serial_wait strategy workflow unable to wake up (#15270)
[BUG][Task] fix java task classpath (#15470)
[Bug] [Audit log] Fix Audit log UI query error (#15427)
[Bug][Registry] Optimizing waiting strategy (#15223)
Set TaskGroupQueue updateTime when force start (#15510)
TaskGroupPriority only compare When TaskGroup is same (#15486)
Remove taskQueue and looper in worker (#15292)
Display the resource file doesn't exist message in task create page (#15350)
Recreate new TaskInstance Working Directory when exist in worker (#15358)
[Bug] Close SSH session after remote shell finish (#15348)
Fix check value rather than key in AbstractDataSourceProcessor#checkOther (#15351)
Fix resource file usage(Delete Resource/ResourceUser which is deprecated)
[Bug][Master] send ACK event timeout (#15346)
Fix k8sTaskExecutionContext setting configYaml (#15116)
[Fix #15129] [Dependent] The date rules of the dependent node are ambiguous. (#15289)
Fix failover Master might not release taskGroup (#15287)
[HotFix] Fix TaskOutputParameterParser might OOM if meed a bad output param expression (#15264)
[Bug-15215][Api] non-admin should not modify tenantId and queue (#15254)
Set the tenant as the owner in final stage (#15256)
Use chown to set the file owner (#15240)
[Fix] Change HTTP plugin timeout param to number type (#15234)
fix switch condition (#15228)
Fix docs style is incorrect by CI pass (#15167)
Expire session when update user password (#15219)
Fix home page workflow instance miss status (#15193)
fix security issue (#15192)
fix can't stop bug (#15191)
Remove API Result in Service (#15181)
Exclude DataSourceAutoConfiguration in worker server (#15169)
[Bug] Fix TriggerRelationMapper cannot work due to miss DatabaseIdProvider (#15153)
Fix spotless (#15164)
Fix incorrect button display text (#15160)
[Fix][Data Quality] Change t_ds_dq_rule_input_entry field name fix PostgreSQL not support value issue (#14992)
fix missing 'KYUUBI' in droplist of datasource (#15140)
[Bug] Fix endless loop (#15092)
fix: execute sql error: datasource plugin 'doris' is not found。 (#15123)
Fix confusing constant string for unit convertor (#15126)
[fix-#11726] fix error when set connection proerty both in the URL and an argument (#15093)
[Fix-15072][Resource Center] Non-admin user can not query resource recursively (#15097)
[E2E][Bug] Fix k8s-e2e (#15098)
Fix SqlTask cannot split the given sql when browser in windows (#15062)
[Fix-15036] [API] Fix task definition edit doesn't work (#14801)
remove sub workflow finish notify (#15057)
Fix missing Kyuubi type in UI (#15051)
[Fix-14885][pom] fix spotless format file path (#14889)
Fix When the task instance status is 'STOP' (#14967)
Revert "[Bug] [Resource] fix resource delete bug (#15003)
[Bug] [Resource] fix resource delete bug (#15003) Delete File generated by UT (#15022)

Improvement

[Improvement][UT] Improve Worker registry coverage (#15380)
refactor comments & function name for confuse (#15546)
[Improvement][HTTP] support custom rendering of http body (#15531)
[improvement][api] Fix typo for controllers (#15438)
[Feature-15475][DinkyTask] DinkyTask supports Dinky-1.0.0 and common sql (#15479)
[Improvement][K8S]Optimize MDC for K8S tasks (#15390)
Enable set ServerLoadProtection fot Master/Worker (#15439)
[Feature] timed scheduler Improvement (#15449)
[Improvement][E2E] add e2e javatask case (#15469)
[Enhancement][API]Enhance mysql connection properties (#15433)
[Improvement][E2E]e2e improve add workflow httpTask e2e case (#15420)
Add config for defaultTenantEnabled (#15391)
Use DefaultUncaughtExceptionHandler to log the uncached exception (#15496)
adjust the sequence of alarm group and add validate (#15382)
Use Druid to split sql (#15367)
optimize add select filter (#15378)
[Improvement][Helm] using helm-docs to generate docs automatically (#15299)
[Improvement][K8S] Custom label of a K8S task can be passed to the pod (#15369)
Optimize server startup log (#15362)
[Improvement][E2E] support e2e compose v2 fix code style (#15325)
[Improvement] Ensure that HttpUtils can only get result from certification URL (#15288)
delete debugger (#15316)
Set maxRows in SqlTask (#15342)
[Feature-15146][dolphinscheduler-task-sqoop] add sqoop source/target type (#15146)
[Feature-15248][dolphinscheduler-alert-plugins] add alert plugin aliyun-voice (#15248)
[Improvement-15260][dolphinscheduler-datasource-hana] add hana related dependencies (#15260)
fail-fast for dependent check (#15197)
[Improvement] Move delay calculation to Master (#15278)
Add dolphinscheduler-extract-common module (#15266)
Support parse task output params under multiple log (#15244)
[Improvement-15009][Parameter] Change project parameter value to text (#15010)
Remove spring cache for dao (#15184)
[Improvement] Clean up Scheduler logic (#15198)
[Improvement][Alert] Add a test send feature when creating an alert instance (#15163)
[Improvement][Helm] support task type filter (#15179)
[Improvement][Resource Center] Display brief file name in file-details page (#15137)
[Improvement][Alert] Add timeout params for HTTP plugin (#15174)
[feature#14654] alert-spi support prometheus alertmanager (#15079)
[Improvement][K8S] Remove ResourceQuota (#14991)
[Improvement] Refactoring K8S task plugin with connections managed in connection center (#14977)
[DSIP-19] Support sagemaker connections in the connection center, as well as external connections to the connection center in sagemaker tasks (#14976)
[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks (#14434)
[Feature-14832][Listener]Implementation of Listener Mechanism (#14981)
Remove mapper usage in tools (#15073)
[Feature-14678][Master][UI]Dependent task parameter passing (#14702)
Add IT for mysql5/postgresql16 initialize/upgrade (#15063)
Add IT for dolphinscheduler-tools module (#15043)
Set kubectl version to v1.28.3 (#15053)
Add dolphinscheduler-dao-plugin module (#15019)
[improvement][Resources] Improve details page return to the previous list page (#14951)
[Improvement][Alert] Alert plugin enhance fail message (#15024)
[Improvement][Registry][Jdbc] Add jdbc registry config in helm charts (#14431)
[Improvement][Master] Calculate the remainTime then we set the delay execution. (#15012)

Document

[Doc][Docker] fix typo on start with docker (#15534)
[Doc] remove skywalking, update note (#15028)
Change download url in backend.yml (#15526)
[Doc][K8S] Add DS K8S Operator into k8s deployment character (#15516)
Add guideline link into DolphinScheduler mail list (#15447)
Remove unused cache-evict.png (#15220)
[Doc-15500][Task] Update cli opts of spark and flink (#15501)
doc write wrong,should be MinIO it's not MinION (#15395)
[Doc]remove temporary markdown comments (#15385)
doc: Classify docs to avoid misleading (#15282)
Add deploy on Terraform on README (#15189)
Modify the documentation that python task will not work properly when '\n' indicates the presence of a variable and needs to use 'repr(value)' (#15145)
[Docs] fix typo (#15032)

Chore

merge schema 330 into 321 and change docs (#15582)
Set the workflow instance ready state to running in failover (#15572)
cp: Reduce the size of tarball to continue ASF release (#15004)
chore: Docs change for 3.2.1 release (#15539)
[DS-15489][style]rename the vo object suffix (#15504)

致谢

感谢所有贡献者的辛勤付出，特别是以下成员（排名不分先后）：

致谢名单
AliceXiaoLu	Gallardot	clovelll
BaiJv	John Huang	davidzollo
CommandCV	JohnZp	dbac
Dyqer	Orange-Summer	destroydestiny
EricGao888	Radeity	devosend
SbloodyS	SusurHe	fuchanghai
aiwenmo	arlendp	imizao
c3Vu	caishunfeng	imp2002
izualzhy	kevinkelin	lenboo
lgcareer	liunaijie	liyou
lizhenglei	ly109974	pegasas
qingwli	reele	rickchengx
ruanwenjun	sdhzwc	sleo
tangjiaolong	xdu-chenrj	xinxingi
xjlgod	xujiaqiang	zhanqian
zhaohehuhu	zhihuasu	zhongjiajie
zhuangchong	zhutong6688